Image decoding apparatus and encoding apparatus, processing system and method, and program product

By setting the search range according to the motion vector in the image decoding and encoding device and segmenting the prediction block, the problem of large processing load in the prior art is solved, and efficient motion vector correction is achieved without reducing the accuracy.

CN118921458APending Publication Date: 2024-11-08KDDI CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411205419.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2018-09-21
Filing Date
2019-08-29
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In the existing image processing technology, the processing load for determining the correction reference position from the search range is large, which leads to heavy load caused by motion vector correction, but it is difficult to maintain correction accuracy.

Method used

In the image decoding and encoding device, the search range is set according to the motion vector and the prediction block is divided into sub-blocks for fine processing, and the correction reference position is calculated from the outside of the search range to reduce the processing load without reducing the correction Accuracy.

Benefits of technology

It effectively reduces the processing load caused by motion vector correction, while maintaining correction accuracy, and improving the efficiency of image processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118921458A_ABST
    Figure CN118921458A_ABST
Patent Text Reader

Abstract

The invention provides an image decoding device, an image encoding device, a processing system, a processing method, and a program product. This image decoding device is provided with a prediction unit that generates a prediction signal included in a prediction block on the basis of a motion vector. The prediction unit performs a refinement process in which a search range is set on the basis of a reference position specified by the motion vector, a corrected reference position at which a prescribed cost is minimum is specified from the search range, and the motion vector is corrected on the basis of the corrected reference position. When a block size of the prediction block is larger than a predetermined block size, the prediction unit divides the prediction block into sub-block groups, and performs the refinement processing on each of the sub-blocks. According to the invention, the processing load caused by the correction of the motion vector can be reduced without reducing the correction accuracy of the motion vector.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of an application with a filing date of August 29, 2019, an application number of 201980018417.1, and an invention name of “Image decoding device, image encoding device, image processing system and program”. Technical Field

[0002] The present invention relates to an image decoding device and encoding device, and a processing system and method, and a program product. Background Art

[0003] In the past, a technology has been proposed for generating a prediction residual signal which is the difference between a prediction signal generated by intra-frame prediction or inter-frame prediction and an input image signal, and performing transformation and quantization processing on the prediction residual signal (e.g., HEVC: High Efficiency Video Coding) (e.g., non-patent document 1).

[0004] In such image processing technology, a technology (DMVR: Decoder-side Motion Vector Refinement) is also proposed in which an image decoding device refines a motion vector received from an image decoding device. Specifically, the image decoding device determines a reference position based on the motion vector, and determines a modified reference position with the minimum specified cost from a search range set based on the determined reference position. The image decoding device modifies the motion vector based on the modified reference position (for example, non-patent document 2).

[0005] Prior art literature

[0006] Non-patent literature

[0007] Non-patent document 1: ITU-T H.265 High Efficiency Video Coding

[0008] Non-patent document 2: Algorithm Description of Joint Exploration Test Model 7 (JEM 7), JVET-G1001 Summary of the invention

[0009] Problems to be solved by the invention

[0010] In the above-mentioned image processing technology, the processing load of determining the correction reference position from the search range is large. Therefore, it is desirable to reduce the processing load caused by the correction of the motion vector without reducing the correction accuracy of the motion vector.

[0011] Therefore, the present invention is completed to solve the above-mentioned problems, and its purpose is to provide an image decoding device and encoding device, as well as a processing system and method, and a program product that can reduce the processing load caused by the correction of motion vectors without reducing the correction accuracy of motion vectors.

[0012] Solutions for solving problems

[0013] The image decoding device according to the first technical solution includes a prediction unit that generates a prediction signal included in a prediction block based on a motion vector. The prediction unit performs a refinement process, that is, sets a search range based on a reference position determined by the motion vector, determines a modified reference position with a minimum predetermined cost from the search range, and modifies the motion vector based on the modified reference position. When the block size of the prediction block is larger than a predetermined block size, the prediction unit divides the prediction block into sub-block groups and performs the refinement process on each of the sub-blocks.

[0014] The image encoding device according to the second technical solution includes a prediction unit that generates a prediction signal included in a prediction block based on a motion vector. The prediction unit performs a refinement process, that is, sets a search range based on a reference position determined by the motion vector, determines a modified reference position with a minimum predetermined cost from the search range, and modifies the motion vector based on the modified reference position. When the block size of the prediction block is larger than a predetermined block size, the prediction unit divides the prediction block into sub-block groups and performs the refinement process on each of the sub-blocks.

[0015] The image processing system involved in the third technical solution includes an image encoding device and an image decoding device. The image encoding device and the image decoding device include a prediction unit that generates a prediction signal included in a prediction block based on a motion vector. The prediction unit performs a refinement process, that is, sets a search range based on a reference position determined by the motion vector, determines a modified reference position with the minimum specified cost from the search range, and modifies the motion vector based on the modified reference position. When the block size of the prediction block is larger than a predetermined block size, the prediction unit divides the prediction block into sub-block groups and performs the refinement process on each of the sub-blocks.

[0016] The program product involved in the fourth technical solution includes a program that causes a computer to execute a prediction step of generating a prediction signal based on a motion vector. The prediction step includes the following steps: performing a refinement process, that is, setting a search range based on a reference position determined by the motion vector, determining a modified reference position with the minimum specified cost from the search range, and modifying the motion vector based on the modified reference position. In the step, when the block size of the prediction block is larger than a predetermined block size, the prediction block is divided into sub-block groups, and the refinement process is performed on each of the sub-blocks.

[0017] Effects of the Invention

[0018] According to one aspect, an image decoding device and encoding device, a processing system and method, and a program product that can reduce the processing load caused by correction of motion vectors without reducing the correction accuracy of motion vectors can be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 1 is a diagram showing an image processing system 10 according to an embodiment.

[0020] Figure 2 FIG. 1 is a diagram showing an image encoding device 100 according to an embodiment.

[0021] Figure 3 It is a diagram showing the inter-frame prediction unit 111 according to the embodiment.

[0022] Figure 4 It is a diagram for explaining the first process involved in the embodiment.

[0023] Figure 5 It is a diagram for explaining the second process involved in the embodiment.

[0024] Figure 6 It is a diagram for explaining the third process involved in the embodiment.

[0025] Figure 7 It is a diagram for explaining the third process involved in the embodiment.

[0026] Figure 8 FIG. 2 is a diagram showing an image decoding device 200 according to an embodiment.

[0027] Fig. 9 It is a diagram showing the inter-frame prediction unit 241 according to the embodiment.

[0028] Fig.10 This is a diagram for explaining the third process according to Modification Example 1.

[0029] Fig.11 This is a diagram for explaining the third process involved in Modification Example 2.

[0030] Fig.12 This is a diagram for explaining a method of determining a target search range according to Modification 3. DETAILED DESCRIPTION

[0031] Hereinafter, the embodiments will be described with reference to the drawings. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

[0032] However, it should be noted that the drawings are schematic, and the ratio of each dimension is sometimes different from the actual. Therefore, the specific dimensions should be judged with reference to the following description. In addition, of course, sometimes there are parts where the dimensional relationship or ratio is different from each other between the drawings.

[0033] Solution of the present invention

[0034] The image decoding device according to the scheme of the present invention comprises a prediction unit, which generates a prediction signal included in a prediction block based on a motion vector. The prediction unit performs a refinement process, that is, sets a search range based on a reference position determined by the motion vector, determines a modified reference position with the minimum prescribed cost from the search range, and modifies the motion vector based on the modified reference position. The prediction unit performs a first process and a second process in the refinement process, wherein the first process is to calculate the prescribed cost for four first candidate positions and the reference position for defining an outer frame of a target search range constituting at least a part of the search range, and the second process is to determine two second candidate positions for defining an outer frame of a divided search range obtained by dividing the target search range based on the reference first candidate position with the minimum prescribed cost among the four first candidate positions and the reference position, and calculate the prescribed cost for the two second candidate positions.

[0035] In the image decoding device involved in the scheme of the present invention, a first process and a second process are performed, wherein the first process is to calculate a prescribed cost for four first candidate positions and a reference position of an outer frame for defining a target search range, and the second process is to determine two second candidate positions of an outer frame of a segmented search range obtained by segmenting the target search range, and calculate a prescribed cost for the two second candidate positions. That is, by searching for a correction reference position from the outside of the target search range, the correction reference position can be appropriately searched while suppressing the number of pixels for which the prescribed cost is to be calculated. In other words, the processing load caused by the correction of the motion vector can be reduced without reducing the correction accuracy of the motion vector.

[0036] The image encoding device according to the scheme of the present invention comprises a prediction unit, which generates a prediction signal included in a prediction block based on a motion vector. The prediction unit performs a refinement process, that is, sets a search range based on a reference position determined by the motion vector, determines a modified reference position with the minimum prescribed cost from the search range, and modifies the motion vector based on the modified reference position. The prediction unit performs a first process and a second process in the refinement process, wherein the first process is to calculate the prescribed cost for four first candidate positions and the reference position of an outer frame for defining a target search range constituting at least a part of the search range, and the second process is to determine two second candidate positions of an outer frame for defining a divided search range obtained by dividing the target search range based on the reference first candidate position with the minimum prescribed cost among the four first candidate positions and the reference position, and calculate the prescribed cost for the two second candidate positions.

[0037] In the image encoding device involved in the scheme of the present invention, a first process and a second process are performed, wherein the first process is to calculate a prescribed cost for four first candidate positions and a reference position of an outer frame for defining a target search range, and the second process is to determine two second candidate positions of an outer frame for defining a segmented search range obtained by segmenting the target search range, and calculate a prescribed cost for the two second candidate positions. That is, by searching for a correction reference position from the outside of the target search range, the correction reference position can be appropriately searched while suppressing the number of pixels for which the prescribed cost is to be calculated. In other words, the processing load caused by the correction of the motion vector can be reduced without reducing the correction accuracy of the motion vector.

[0038] As an embodiment of the present invention, an image decoding method related to the operation of the above-mentioned image decoding device may be provided, and an image encoding method related to the operation of the above-mentioned image encoding device may be provided. As an embodiment of the present invention, an image processing system having the above-mentioned image decoding device and image encoding device may be provided. As an embodiment of the present invention, a program related to the operation of the above-mentioned image decoding device may be provided, and a program related to the operation of the above-mentioned image encoding device may be provided.

[0039] Implementation

[0040] (Image Processing System)

[0041] Hereinafter, an image processing system according to an embodiment will be described. Figure 1 1 is a diagram showing an image processing system 10 according to an embodiment.

[0042] like Figure 1As shown, the image processing system 10 has an image encoding device 100 and an image decoding device 200. The image encoding device 100 generates encoded data by encoding an input image signal. The image decoding device 200 generates an output image signal by decoding the encoded data. The encoded data can be sent from the image encoding device 100 to the image decoding device 200 via a transmission path. The encoded data can also be stored in a storage medium and provided from the image encoding device 100 to the image decoding device 200.

[0043] (Image Coding Device)

[0044] Hereinafter, the image encoding device according to the embodiment will be described. Figure 2 FIG. 1 is a diagram showing an image encoding device 100 according to an embodiment.

[0045] like Figure 2 As shown, the image encoding device 100 has an inter-frame prediction unit 111, an intra-frame prediction unit 112, a subtractor 121, an adder 122, a transform and quantization unit 131, an inverse transform and inverse quantization unit 132, an encoding unit 140, a loop filter processing unit 150 and a frame buffer 160.

[0046] The inter-frame prediction unit 111 generates a prediction signal through inter-frame prediction. Specifically, the inter-frame prediction unit 111 determines a reference block included in the reference frame by comparing the encoding target frame (hereinafter referred to as the target frame) with the reference frame stored in the frame buffer 160, and determines a motion vector for the determined reference block. The inter-frame prediction unit 111 generates a prediction signal included in the prediction block for each prediction block based on the reference block and the motion vector. The inter-frame prediction unit 111 outputs the prediction signal to the subtractor 121 and the adder 122. The reference frame is a frame different from the target frame.

[0047] The intra-frame prediction unit 112 generates a prediction signal by intra-frame prediction. Specifically, the intra-frame prediction unit 112 determines a reference block included in the target frame, and generates a prediction signal for each prediction block based on the determined reference block. The intra-frame prediction unit 112 outputs the prediction signal to the subtractor 121 and the adder 122. The reference block is a block that is referenced for predicting a target block (hereinafter referred to as a target block). For example, the reference block is a block adjacent to the target block.

[0048] The subtractor 121 subtracts the prediction signal from the input image signal and outputs a prediction residual signal to the transform and quantization section 131. Here, the subtractor 121 generates a prediction residual signal which is a difference between a prediction signal generated by intra prediction or inter prediction and the input image signal.

[0049] The adder 122 adds the prediction signal to the prediction residual signal output from the inverse transform and inverse quantization section 132, and outputs the pre-filtering decoded signal to the intra prediction section 112 and the loop filter processing section 150. The pre-filtering decoded signal constitutes a reference block used in the intra prediction section 112.

[0050] The transform and quantization unit 131 performs transform processing on the prediction residual signal and obtains the coefficient level value. In addition, the transform and quantization unit 131 can also quantize the coefficient level value. The transform processing is a process of transforming the prediction residual signal into a frequency component signal. In the transform processing, a basic pattern (transformation matrix) corresponding to a discrete cosine transform (DCT: Discrete Cosine Transform) can be used, and a basic pattern (transformation matrix) corresponding to a discrete sine transform (DST: Discrete Sine Transform) can also be used.

[0051] The inverse transform and inverse quantization unit 132 performs an inverse transform process on the coefficient level value output from the transform and quantization unit 131. Here, the inverse transform and inverse quantization unit 132 may also perform an inverse quantization on the coefficient level value before the inverse transform process. The inverse transform process and the inverse quantization are performed in the reverse order of the transform process and the quantization performed by the transform and quantization unit 131.

[0052] The encoding section 140 encodes the coefficient level value output from the transform and quantization section 131 and outputs the encoded data. For example, the encoding is entropy encoding in which codes of different lengths are assigned according to the occurrence probability of the coefficient level value.

[0053] In addition to encoding the coefficient level values, the encoding unit 140 also encodes control data used in the decoding process. The control data may include size data such as the encoding block size, the prediction block size, and the transform block size.

[0054] The loop filter processing unit 150 performs filtering processing on the decoded signal before filtering output from the adder 122, and outputs the decoded signal after filtering to the frame buffer 160. For example, the filtering processing is a deblocking filtering processing that reduces distortion generated at the boundary portion of a block (coding block, prediction block or transform block).

[0055] The frame buffer 160 accumulates reference frames used in the inter-frame prediction unit 111. The decoded signal after the filtering process constitutes a reference frame used in the inter-frame prediction unit 111.

[0056] (Inter-frame prediction unit)

[0057] Hereinafter, the inter-frame prediction unit according to the embodiment will be described. Figure 3 It is a diagram showing the inter-frame prediction unit 111 according to the embodiment.

[0058] like Figure 3 As shown, the inter prediction unit 111 includes a motion vector search unit 111A, a refinement unit 111B, and a prediction signal generation unit 111C. The inter prediction unit 111 is an example of a prediction unit that generates a prediction signal included in a prediction block based on a motion vector.

[0059] The motion vector search unit 111A determines a reference block included in the reference frame by comparing the target frame with the reference frame, and searches for a motion vector for the determined reference block. A known method can be used for searching the motion vector, and thus a detailed description thereof is omitted.

[0060] The refinement unit 111B performs refinement processing, that is, sets a search range based on the reference position determined by the motion vector, determines a correction reference position with the minimum prescribed cost from the search range, and corrects the motion vector based on the correction reference position.

[0061] The refinement unit 111B may perform refinement processing when the prescribed conditions are met. The prescribed conditions may include the following conditions: the prediction block is a block for bidirectional prediction, one reference frame is a frame that is in the past in time relative to the target frame, and the other reference frame is a frame that is in the future in time relative to the target frame. The prescribed conditions may also include a condition that the block size of the prediction block is smaller than a predetermined size. The block size may be the number of pixels in the horizontal direction of the prediction block or the number of pixels in the vertical direction of the prediction block. The block size may also be the number of pixels in the horizontal direction and the number of pixels in the vertical direction, whichever is smaller. The block size may also be the total number of pixels in the block (i.e., the value obtained by multiplying the number of pixels in the horizontal direction by the number of pixels in the vertical direction).

[0062] The prescribed conditions may also include a condition that the motion vector is encoded in a merge mode. The merge mode is a mode that transmits only the index of the motion vector of the encoded block adjacent to the prediction block. The prescribed conditions may also include a condition that the motion vector does not apply motion compensation prediction using affine transformation.

[0063] In the embodiment, the refinement unit 111B performs refinement processing according to the following steps. Here, an example will be given of a case where the displacement of the motion vector in the past direction and the future direction are linked to each other in the refinement processing. In other words, the sign of the displacement in the past direction is opposite to the sign of the displacement in the future direction. For example, when the displacement in the past direction is (-2, 2), the displacement in the future direction is (2, -2).

[0064] Under such a premise, the prescribed cost is the sum of the absolute values ​​of the differences between the values ​​of the pixels included in the reference block in the past direction (displaced) and the values ​​of the pixels included in the reference block in the future direction (displaced). The difference is the difference between the values ​​of the pixels at corresponding positions. In addition, this is just an example, and any other indicator (value) can be used as the prescribed cost as long as it is an indicator (value) that can be calculated based on the values ​​of the pixels included in the reference block in the past direction and the values ​​of the pixels included in the reference block in the future direction, such as the sum of the squares of the differences.

[0065] Hereinafter, a case will be exemplified in which the target search range is the same as the search range and the target search range is a range of integer pixels of ±2 in the horizontal direction and the vertical direction. The target search range has a square shape.

[0066] First, the refinement section 111B performs a first process of calculating prescribed costs for four first candidate positions and a reference position defining an outer frame (eg, four corners) of a target search range constituting at least a part of the search range.

[0067] Specifically, if Figure 4 As shown, the refinement unit 111B determines the reference position (in Figure 4 The refinement unit 111B sets the target search range based on the reference position. The refinement unit 111B selects four first candidate positions (in Figure 4 P1 to P4) and reference position (in Figure 4 The specified cost is calculated by P0).

[0068] Second, the refinement unit 111B performs a second process, namely, determining two second candidate positions of an outer frame (for example, four corners) of a segmentation search range obtained by defining the segmentation target search range based on a baseline first candidate position with the smallest specified cost among the four first candidate positions and a reference position, and calculating a specified cost for the two second candidate positions.

[0069] Specifically, an example is given where the prescribed cost of P2 is the smallest. In other words, P2 is the first candidate position of the reference. Figure 5 As shown, the refinement unit 111B generates a first candidate position according to the reference position (in Figure 5 P2 in the figure) and the reference position (in Figure 5 The segmentation target region has a square shape with a straight line from P0 to P2 as a diagonal line. The refinement unit 111B determines two second candidate positions (in Figure 5 P5 and P6 in the figure), calculate the specified cost for the two second candidate positions.

[0070] Here, the refinement unit 111B may determine the position with the smallest predetermined cost among the calculated candidate positions (P0, P2, P5, P6) as the correction reference position, and correct (displace) the motion vector based on the correction reference position. However, in the embodiment, the refinement unit 111B continues the following process.

[0071] Third, the refinement unit 111B performs a third process, that is, determining a third candidate position from the reference second candidate position with the smaller prescribed cost among the four calculated candidate positions of the outer frame (e.g., four corners) defining the segmentation search range, and calculating the prescribed cost for the third candidate position. Here, a case where two reference second candidate positions are selected as reference second candidate positions and pixels arranged on a straight line between the two reference second candidate positions are determined as third candidate positions is described.

[0072] For example, Figure 6 As shown, the four calculated candidate positions (in Figure 6 In the example, P0, P2, P5, P6) two second candidate positions are selected in the order of the specified cost from small to large (in Figure 6 In this case, the refinement unit 111B arranges pixels on a straight line between the two reference second candidate positions (in Figure 6 The refinement unit 111B determines the third candidate position (in Figure 6 Here, P7 and P9 are 1 / 2 pixels and have values ​​interpolated from integer pixels. That is, the third candidate position includes 1 / 2 pixels.

[0073] Or, if Figure 7 As shown, the four calculated candidate positions (in Figure 7 In the example, P0, P2, P5, P6) two second candidate positions are selected in the order of the specified cost from small to large (in Figure 7 In this case, the refinement unit 111B arranges pixels on a straight line between the two reference second candidate positions (in Figure 7 The refinement unit 111B determines the third candidate position (in Figure 7 Here, P7 and P9 are 1 / 2 pixels and have values ​​interpolated from integer pixels. That is, the third candidate position includes 1 / 2 pixels.

[0074] The refinement unit 111B determines the position with the smallest predetermined cost among the calculated candidate positions (P0, P2, P5 to P9) as the correction reference position, and corrects (shifts) the motion vector based on the correction reference position.

[0075] The prediction signal generation unit 111C generates a prediction signal based on the motion vector. Specifically, when the motion vector is not corrected, the prediction signal generation unit 111C generates a prediction signal based on the motion vector input from the motion vector search unit 111A. On the other hand, when the motion vector is corrected, the prediction signal generation unit 111C generates a prediction signal based on the corrected motion vector input from the refinement unit 111B.

[0076] (Image decoding device)

[0077] Hereinafter, an image decoding device according to an embodiment will be described. Figure 8 FIG. 2 is a diagram showing an image decoding device 200 according to an embodiment.

[0078] like Figure 8 As shown, the image decoding apparatus 200 includes a decoding unit 210 , an inverse transform and inverse quantization unit 220 , an adder 230 , an inter prediction unit 241 , an intra prediction unit 242 , a loop filter processing unit 250 , and a frame buffer 260 .

[0079] The decoding unit 210 decodes the coded data generated by the image coding device 100 to decode the coefficient level values. For example, the decoding is entropy decoding that is the reverse of the entropy coding procedure performed by the coding unit 140 .

[0080] The decoding unit 210 can obtain the control data by decoding the coded data. As described above, the control data may include size data such as coding block size, prediction block size, and transform block size. The control data may also include an information element indicating an input source for generating a prediction sample of the second component.

[0081] The inverse transform and inverse quantization unit 220 performs an inverse transform process on the coefficient level value output from the decoding unit 210. Here, the inverse transform and inverse quantization unit 220 may perform an inverse quantization on the coefficient level value before the inverse transform process. The inverse transform process and the inverse quantization are performed in the reverse order of the transform process and the quantization performed by the transform and quantization unit 131.

[0082] The adder 230 adds the prediction signal to the prediction residual signal output from the inverse transform and inverse quantization section 220, and outputs the pre-filtering decoded signal to the intra prediction section 262 and the loop filter processing section 250. The pre-filtering decoded signal constitutes a reference block used in the intra prediction section 262.

[0083] The inter-frame prediction unit 241 generates a prediction signal by inter-frame prediction similarly to the inter-frame prediction unit 111. Specifically, the inter-frame prediction unit 241 generates a prediction signal for each prediction block based on a motion vector decoded from the coded data and a reference signal included in a reference frame. The inter-frame prediction unit 241 outputs the prediction signal to the adder 230.

[0084] The intra-frame prediction unit 262 generates a prediction signal by intra-frame prediction similarly to the intra-frame prediction unit 112. Specifically, the intra-frame prediction unit 262 determines a reference block included in the target frame, and generates a prediction signal for each prediction block based on the determined reference block. The intra-frame prediction unit 262 outputs the prediction signal to the adder 230.

[0085] The loop filter processing unit 250 performs filtering processing on the pre-filtering decoded signal output from the adder 230, similarly to the loop filter processing unit 150, and outputs the filtered decoded signal to the frame buffer 260. For example, the filtering processing is a deblocking filtering processing that reduces distortion generated at the boundary portion of a block (coding block, prediction block, or transform block).

[0086] The frame buffer 260 accumulates reference frames used in the inter-frame prediction unit 241, similarly to the frame buffer 160. The decoded signal after the filtering process constitutes a reference frame used in the inter-frame prediction unit 241.

[0087] (Inter-frame prediction unit)

[0088] Hereinafter, the inter-frame prediction unit according to the embodiment will be described. Fig. 9 It is a diagram showing the inter-frame prediction unit 241 according to the embodiment.

[0089] like Fig. 9 As shown, the inter prediction unit 241 includes a motion vector decoding unit 241A, a refinement unit 241B, and a prediction signal generation unit 241C. The inter prediction unit 241 is an example of a prediction unit that generates a prediction signal included in a prediction block based on a motion vector.

[0090] The motion vector decoding unit 241A obtains a motion vector by decoding the control data received from the image encoding device 100 .

[0091] The refinement unit 241B performs refinement processing similarly to the refinement unit 111B, that is, sets a search range based on a reference position determined by a motion vector, determines a correction reference position with a minimum predetermined cost from the search range, and corrects the motion vector based on the correction reference position.

[0092] The prediction signal generating unit 241C generates a prediction signal based on the motion vector, similarly to the prediction signal generating unit 111C.

[0093] (Function and Effect)

[0094] In the image encoding device 100 and the image decoding device 200 involved in the scheme of the present invention, a first process and a second process are performed, wherein the first process is to calculate a prescribed cost for four first candidate positions and a reference position of an outer frame for defining a target search range, and the second process is to determine two second candidate positions of an outer frame for defining a segmented search range obtained by segmenting the target search range, and calculate a prescribed cost for the two second candidate positions. That is, by searching for a correction reference position from the outside of the target search range, the correction reference position can be appropriately searched while suppressing the number of pixels for which the prescribed cost is to be calculated. In other words, the processing load caused by the correction of the motion vector can be reduced without reducing the correction accuracy of the motion vector.

[0095] In the image encoding device 100 and the image decoding device 200 according to the scheme of the present invention, a third process is performed after the first process and the second process, that is, a third candidate position is determined based on the reference second candidate position with a smaller prescribed cost among the four calculated candidate positions of the outer frame (for example, the four corners) defining the segmentation search range, and a prescribed cost is calculated for the third candidate position. The third candidate position is a pixel arranged on a straight line between two reference second candidate positions. That is, by calculating the prescribed cost only for the position where the prescribed cost is assumed to be smaller, it is possible to appropriately search for the correction reference position while suppressing the number of pixels for which the prescribed cost is to be calculated.

[0096] [Modification 1]

[0097] Hereinafter, Modification 1 of the embodiment will be described. Hereinafter, differences from the embodiment will be mainly described.

[0098] In the embodiment, the case where two reference second candidate positions are selected as the reference second candidate positions as a result of the second process is exemplified. In contrast, the case where one reference second candidate position is selected as the reference second candidate position as a result of the second process is described.

[0099] Specifically, in the third process, the refinement section (refinement section 111B and refinement section 241B) determines pixels adjacent to the reference position within the segmentation search range as third candidate positions when the reference position is selected as the base second candidate position.

[0100] For example, Fig.10As shown in FIG. 1 , assuming that the prescribed cost of P0 is less than a certain percentage (e.g., 50%) of the representative value (minimum value, maximum value, or average value) of the prescribed costs of P2, P5, and P6, only the reference position (in Fig. 9 In this case, the refinement unit will select pixels adjacent to the reference position within the segmentation search range (in Fig.10 Here, P7 to P9 are 1 / 2 pixels and have values ​​interpolated from integer pixels. That is, the third candidate position includes 1 / 2 pixels.

[0101] exist Fig.10 In the example, the case where the third candidate position is 1 / 2 pixel is described, but the modification 1 is not limited to this. The third candidate position adjacent to the reference position in the segmentation search range may also be an integer pixel.

[0102] [Modification 2]

[0103] Hereinafter, Modification 2 of the embodiment will be described. Hereinafter, differences from the embodiment will be mainly described.

[0104] In the third process, in the embodiment, the 1 / 2 pixel (for example, Figure 6 In contrast, in the second modification, the refinement unit (refinement unit 111B and refinement unit 241B) calculates the predetermined cost only for integer pixels, and does not calculate the predetermined cost for 1 / 2 pixels.

[0105] For example, Fig.11 As shown in FIG. 1 , when the prescribed costs of both P0 and P2 are less than a certain percentage (e.g., 50%) of the representative value (minimum value, maximum value, or average value) of the prescribed costs of P5 and P6, the cost of integer pixels (in the straight line from P0 to P2) may be calculated instead of the cost of 1 / 2 pixels. Fig.11 Alternatively, when the prescribed costs of both P5 and P6 are less than a certain percentage (e.g., 50%) of the representative value (minimum, maximum, or average) of the prescribed costs of P0 and P2, the cost of integer pixels (in the straight line from P5 to P6) may be calculated instead of calculating the cost of 1 / 2 pixels. Fig.11 The specified cost is P7).

[0106] [Variation 3]

[0107] Hereinafter, Modification 3 of the embodiment will be described. Hereinafter, differences from the embodiment will be mainly described.

[0108] In the embodiment, the case where the search range is the same as the target search range is described as an example. In contrast, in Modification 3, the case where the search range (for example, a range of ±4 integer pixels in the horizontal direction and the vertical direction) is larger than the target search range is described. In such a case, the refinement unit (refinement unit 111B and refinement unit 241B) performs a process of determining the target search range from the search range during the refinement process. Such a process is performed before the above-mentioned first process.

[0109] For example, Fig.12 As shown, the refinement unit may divide the search range into target search range A to target search range D, and perform the same processing as in the embodiment for each target search range.

[0110] Alternatively, the refinement unit may also determine the target search range with the smallest specified cost from the target search range A to the target search range D, and perform the same processing as the implementation method on the determined target search range. The specified cost of the target search range may be the specified cost of the representative point of each target search range (for example, the center point (P4, P5, P8, P9), or the four corners (P1, P3, P10, P12)). Alternatively, the specified cost of the target search range may also be the sum of the specified costs of two or more extraction points of each target search range (for example, if it is target search range A, P0, P1, P2, P4, P6 are extracted).

[0111] As described above, by determining the target search range from the search range, the above embodiment can be applied even when the search range is larger than the target search range. In other words, the above embodiment can be applied even if the search range is a range of integer pixels greater than ±4 in the horizontal and vertical directions.

[0112] [Variation 4]

[0113] Hereinafter, Modification 4 of the embodiment will be described. Hereinafter, differences from the embodiment will be mainly described.

[0114] When the block size of the prediction block is larger than the predetermined block size, the prediction block can be divided into sub-block groups smaller than the predetermined size, and the motion vector of each sub-block is refined. As a division method, in the case of a square, it can be 4 divisions, and in other cases, it can be 2 divisions.

[0115] [Other embodiments]

[0116] The present invention has been described through the above embodiments, but the description and drawings constituting part of this disclosure should not be understood to limit the present invention. Based on such disclosure, various alternative embodiments, examples, and application technologies will be apparent to those skilled in the art.

[0117] In the embodiment, the case where the target search range is a range of integer pixels of ±2 in the horizontal direction and the vertical direction is described. However, the embodiment is not limited to this. The target search range may also be a range of integer pixels of ±1 in the horizontal direction and the vertical direction. In this case, the prescribed cost of all integer pixels included in the segmented search range is calculated through the first process and the second process. The prescribed cost of one 1 / 2 pixel selected from five 1 / 2 pixel candidates is calculated through the third process. Alternatively, the target search range may also be a range of integer pixels of ±3 in the horizontal direction and the vertical direction.

[0118] In the embodiment, the displacement of the motion vector in the past direction and the future direction in the refinement process is described. However, the embodiment is not limited to this. The displacement of the motion vector in the past direction and the future direction may not be linked to each other.

[0119] Although not specifically mentioned in the embodiment, a program that causes a computer to execute each process performed by the image encoding device 100 and the image decoding device 200 may be provided. In addition, the program may be recorded in a computer-readable medium. Using a computer-readable medium, the program can be installed in a computer. Here, the computer-readable medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not specifically limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM.

[0120] Alternatively, a chip including a memory storing a program for executing each process performed by the image encoding device 100 and the image decoding device 200 and a processor executing the program stored in the memory may be provided.

[0121] Explanation of symbols

[0122] 10: image processing system; 100: image encoding device; 111: inter-frame prediction unit; 111A: motion vector search unit; 111B: refinement unit; 111C: prediction signal generation unit; 112: intra-frame prediction unit; 121: subtractor; 122: adder; 131: transformation and quantization unit; 132: inverse transformation and inverse quantization unit; 140: encoding unit; 150: loop filter processing unit; 160: frame buffer; 200: image decoding device; 210: decoding unit; 220: inverse transformation and inverse quantization unit; 230: adder; 241: inter-frame prediction unit; 241A: motion vector decoding unit; 241B: refinement unit; 241C: prediction signal generation unit; 242: intra-frame prediction unit; 250: loop filter processing unit; 260: frame buffer.

Claims

1. An image decoding device, It is characterized in that The image decoding device includes: a prediction unit that generates a prediction signal included in a prediction block based on a motion vector; The prediction unit performs a refinement process when a predetermined condition is satisfied, the refinement process comprising setting a search range based on a reference position determined by the motion vector, determining a modified reference position with a minimum predetermined cost from the search range, and modifying the motion vector according to the modified reference position. The predetermined cost is the sum of absolute values ​​of differences between pixel values ​​in a reference block in one reference frame and pixel values ​​in a reference block in the other reference frame, In the refinement process, the prediction unit Selecting a first candidate position from the search range, selecting a second candidate position from the first candidate position and the reference position, If the prescribed cost of the reference position is less than a certain percentage of the minimum value of the prescribed costs of the first candidate positions, only the reference position is selected as the second candidate position, determining pixels adjacent to the reference position selected as the second candidate position as a third candidate position, determining the position with the smallest prescribed cost among the first candidate position, the second candidate position and the third candidate position as the corrected reference position, The prescribed condition is that the prediction block is a block for bidirectional prediction, one reference frame is a frame that is in the past in time relative to the target frame, and the other reference frame is a frame that is in the future in time relative to the target frame, The displacement between the first candidate position and the reference position in the reference frame of the one party is obtained by inverting the sign of the displacement between the first candidate position and the reference position in the reference frame of the other party, The displacement between the second candidate position and the reference position in the reference frame of the one party is obtained by inverting the sign of the displacement between the second candidate position and the reference position in the reference frame of the other party, The displacement between the third candidate position and the reference position in the reference frame of the one party is obtained by inverting the sign of the displacement between the third candidate position and the reference position in the reference frame of the other party, When the block size of the prediction block is larger than a predetermined block size, the prediction unit divides the prediction block into sub-block groups and performs the refinement process on each of the sub-blocks.

2. An image encoding device, It is characterized in that The image encoding device includes: a prediction unit that generates a prediction signal included in a prediction block based on a motion vector; The prediction unit performs a refinement process when a predetermined condition is satisfied, the refinement process comprising setting a search range based on a reference position determined by the motion vector, determining a modified reference position with a minimum predetermined cost from the search range, and modifying the motion vector according to the modified reference position. The predetermined cost is the sum of absolute values ​​of differences between pixel values ​​in a reference block in one reference frame and pixel values ​​in a reference block in the other reference frame, In the refinement process, the prediction unit Selecting a first candidate position from the search range, selecting a second candidate position from the first candidate position and the reference position, If the prescribed cost of the reference position is less than a certain percentage of the minimum value of the prescribed costs of the first candidate positions, only the reference position is selected as the second candidate position, determining pixels adjacent to the reference position selected as the second candidate position as a third candidate position, determining the position with the smallest prescribed cost among the first candidate position, the second candidate position and the third candidate position as the corrected reference position, The prescribed condition is that the prediction block is a block for bidirectional prediction, one reference frame is a frame that is in the past in time relative to the target frame, and the other reference frame is a frame that is in the future in time relative to the target frame, The displacement between the first candidate position and the reference position in the reference frame of the one party is obtained by inverting the sign of the displacement between the first candidate position and the reference position in the reference frame of the other party, The displacement between the second candidate position and the reference position in the reference frame of the one party is obtained by inverting the sign of the displacement between the second candidate position and the reference position in the reference frame of the other party, The displacement between the third candidate position and the reference position in the reference frame of the one party is obtained by inverting the sign of the displacement between the third candidate position and the reference position in the reference frame of the other party, When the block size of the prediction block is larger than a predetermined block size, the prediction unit divides the prediction block into sub-block groups and performs the refinement process on each of the sub-blocks.

3. An image processing system comprising an image encoding device and an image decoding device, It is characterized in that The image encoding device and the image decoding device include a prediction unit that generates a prediction signal included in a prediction block based on a motion vector. The prediction unit performs a refinement process when a predetermined condition is satisfied, the refinement process comprising setting a search range based on a reference position determined by the motion vector, determining a modified reference position with a minimum predetermined cost from the search range, and modifying the motion vector according to the modified reference position. The predetermined cost is the sum of absolute values ​​of differences between pixel values ​​in a reference block in one reference frame and pixel values ​​in a reference block in the other reference frame, In the refinement process, the prediction unit Selecting a first candidate position from the search range, selecting a second candidate position from the first candidate position and the reference position, If the prescribed cost of the reference position is less than a certain percentage of the minimum value of the prescribed costs of the first candidate positions, only the reference position is selected as the second candidate position, determining pixels adjacent to the reference position selected as the second candidate position as a third candidate position, determining the position with the smallest prescribed cost among the first candidate position, the second candidate position and the third candidate position as the corrected reference position, The prescribed condition is that the prediction block is a block for bidirectional prediction, one reference frame is a frame that is in the past in time relative to the target frame, and the other reference frame is a frame that is in the future in time relative to the target frame, The displacement between the first candidate position and the reference position in the reference frame of the one party is obtained by inverting the sign of the displacement between the first candidate position and the reference position in the reference frame of the other party, The displacement between the second candidate position and the reference position in the reference frame of the one party is obtained by inverting the sign of the displacement between the second candidate position and the reference position in the reference frame of the other party, The displacement between the third candidate position and the reference position in the reference frame of the one party is obtained by inverting the sign of the displacement between the third candidate position and the reference position in the reference frame of the other party, When the block size of the prediction block is larger than a predetermined block size, the prediction unit divides the prediction block into sub-block groups and performs the refinement process on each of the sub-blocks.

4. A program product, It is characterized in that The program product includes a program that causes a computer to execute a prediction step of generating a prediction signal included in a prediction block based on a motion vector. The prediction step includes the step of performing a refinement process when a prescribed condition is satisfied, the refinement process comprising setting a search range based on a reference position determined by the motion vector, determining a modified reference position with a minimum prescribed cost from the search range, and modifying the motion vector according to the modified reference position. The predetermined cost is the sum of absolute values ​​of differences between pixel values ​​in a reference block in one reference frame and pixel values ​​in a reference block in the other reference frame, In the step of performing the refinement processing, Selecting a first candidate position from the search range, selecting a second candidate position from the first candidate position and the reference position, If the prescribed cost of the reference position is less than a certain percentage of the minimum value of the prescribed costs of the first candidate positions, only the reference position is selected as the second candidate position, determining pixels adjacent to the reference position selected as the second candidate position as a third candidate position, determining the position with the smallest prescribed cost among the first candidate position, the second candidate position and the third candidate position as the corrected reference position, The prescribed condition is that the prediction block is a block for bidirectional prediction, one reference frame is a frame that is in the past in time relative to the target frame, and the other reference frame is a frame that is in the future in time relative to the target frame, The displacement between the first candidate position and the reference position in the reference frame of the one party is obtained by inverting the sign of the displacement between the first candidate position and the reference position in the reference frame of the other party, The displacement between the second candidate position and the reference position in the reference frame of the one party is obtained by inverting the sign of the displacement between the second candidate position and the reference position in the reference frame of the other party, The displacement between the third candidate position and the reference position in the reference frame of the one party is obtained by inverting the sign of the displacement between the third candidate position and the reference position in the reference frame of the other party, In the step, when the block size of the prediction block is larger than a predetermined block size, the prediction block is divided into sub-block groups, and the refinement process is performed on each of the sub-blocks.

5. An image processing method, It is characterized in that The image processing method comprises a prediction step of generating a prediction signal included in a prediction block according to a motion vector, The prediction step includes the step of performing a refinement process when a prescribed condition is satisfied, the refinement process comprising setting a search range based on a reference position determined by the motion vector, determining a modified reference position with a minimum prescribed cost from the search range, and modifying the motion vector according to the modified reference position. The predetermined cost is the sum of absolute values ​​of differences between pixel values ​​in a reference block in one reference frame and pixel values ​​in a reference block in the other reference frame, In the step of performing the refinement processing, Selecting a first candidate position from the search range, selecting a second candidate position from the first candidate position and the reference position, If the prescribed cost of the reference position is less than a certain percentage of the minimum value of the prescribed costs of the first candidate positions, only the reference position is selected as the second candidate position, determining pixels adjacent to the reference position selected as the second candidate position as a third candidate position, determining the position with the smallest prescribed cost among the first candidate position, the second candidate position and the third candidate position as the corrected reference position, The prescribed condition is that the prediction block is a block for bidirectional prediction, one reference frame is a frame that is in the past in time relative to the target frame, and the other reference frame is a frame that is in the future in time relative to the target frame, The displacement between the first candidate position and the reference position in the reference frame of the one party is obtained by inverting the sign of the displacement between the first candidate position and the reference position in the reference frame of the other party, The displacement between the second candidate position and the reference position in the reference frame of the one party is obtained by inverting the sign of the displacement between the second candidate position and the reference position in the reference frame of the other party, The displacement between the third candidate position and the reference position in the reference frame of the one party is obtained by inverting the sign of the displacement between the third candidate position and the reference position in the reference frame of the other party, In the step, when the block size of the prediction block is larger than a predetermined block size, the prediction block is divided into sub-block groups, and the refinement process is performed on each of the sub-blocks.