Image encoding device and method, image decoding device and method

Abstract

The present invention relates to an image encoding device and method, and an image decoding device and method. A technology for improving encoding efficiency by performing block division suitable for image encoding and decoding is provided. The image encoding device includes: an encoding information storage unit that adds inter-frame prediction information used in inter-frame prediction to a historical prediction motion vector candidate list; a historical prediction motion vector candidate derivation unit that derives historical prediction motion vector candidates from the historical prediction motion vector candidate list; and a historical merge candidate derivation unit that derives historical merge candidates from the historical prediction motion vector candidate list, wherein the historical prediction motion vector candidate derivation unit and the historical merge candidate derivation unit refer to candidates included in the historical prediction motion vector candidate list in different orders as historical prediction motion vector candidates or historical merge candidates.

Description

[0001] This divisional application is a divisional application of the invention patent with application number 201980044863.X, application date December 11, 2019, and applicant JVC Kenwood Corporation. The invention name of the invention patent application is "Image encoding device, image encoding method, image encoding program, image decoding device, image decoding method and image decoding program". Technical Field

[0002] The present invention relates to image encoding and decoding technology for dividing an image into blocks and performing prediction. Background Art

[0003] In image encoding and decoding, the image being processed is divided into blocks, a set of a predetermined number of pixels, and processed in blocks. By dividing the image into appropriate blocks and appropriately setting intra-frame prediction (intra-prediction) and inter-frame prediction (inter-frame prediction), encoding efficiency is improved.

[0004] In the encoding and decoding of moving images, inter-frame prediction, which uses predictions from already encoded or decoded pictures, is used to improve coding efficiency. Patent Document 1 discloses a technique for applying affine transformation to inter-frame prediction. Objects in moving images often deform, such as enlargement, reduction, and rotation. By applying the technique in Patent Document 1, efficient encoding is possible.

[0005] Prior art literature

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Laid-Open No. 9-172644 Summary of the Invention

[0008] Problems to be solved by the invention

[0009] However, the technique of Patent Document 1 involves image transformation, which results in a high processing load. In view of the above problem, the present invention provides a low-load and efficient encoding technique.

[0010] Means of solving technical problems

[0011] In order to solve the above problems, the image encoding device of the first aspect of the present invention includes: an encoding information storage unit, which adds the inter-frame prediction information used in inter-frame prediction to a historical prediction motion vector candidate list; a historical prediction motion vector candidate derivation unit, which derives historical prediction motion vector candidates from the historical prediction motion vector candidate list; and a historical merge candidate derivation unit, which derives historical merge candidates from the historical prediction motion vector candidate list, and the historical prediction motion vector candidate derivation unit and the historical merge candidate derivation unit refer to the candidates included in the historical prediction motion vector candidate list in different orders as historical prediction motion vector candidates or historical merge candidates.

[0012] The image coding method of the second aspect of the present invention includes: a coding information saving step, adding the inter-frame prediction information used in inter-frame prediction to a historical prediction motion vector candidate list; a historical prediction motion vector candidate derivation step, deriving historical prediction motion vector candidates from the historical prediction motion vector candidate list; and a historical merge candidate derivation step, deriving historical merge candidates from the historical prediction motion vector candidate list, in which the candidates included in the historical prediction motion vector candidate list are referenced in different orders as historical prediction motion vector candidates or historical merge candidates.

[0013] The image encoding program of the third aspect of the present invention enables a computer to execute the following steps: an encoding information saving step, adding the inter-frame prediction information used in the inter-frame prediction to the historical prediction motion vector candidate list; a historical prediction motion vector candidate derivation step, deriving the historical prediction motion vector candidate from the historical prediction motion vector candidate list; and a historical merge candidate derivation step, deriving the historical merge candidate from the historical prediction motion vector candidate list, in which the candidates included in the historical prediction motion vector candidate list are referenced in different orders from each other as historical prediction motion vector candidates or historical merge candidates.

[0014] The image decoding device of the fourth aspect of the present invention includes: a coding information storage unit, which adds the inter-frame prediction information used in the inter-frame prediction to a historical prediction motion vector candidate list; a historical prediction motion vector candidate derivation unit, which derives historical prediction motion vector candidates from the historical prediction motion vector candidate list; and a historical merge candidate derivation unit, which derives historical merge candidates from the historical prediction motion vector candidate list, and the historical prediction motion vector candidate derivation unit and the historical merge candidate derivation unit refer to the candidates included in the historical prediction motion vector candidate list in different orders as historical prediction motion vector candidates or historical merge candidates.

[0015] According to the fifth aspect of the present invention, the image decoding method includes: a coding information saving step, adding the inter-frame prediction information used in inter-frame prediction to the historical prediction motion vector candidate list; a historical prediction motion vector candidate derivation step, deriving the historical prediction motion vector candidate from the historical prediction motion vector candidate list; and a historical merge candidate derivation step, deriving the historical merge candidate from the historical prediction motion vector candidate list, in which the candidates included in the historical prediction motion vector candidate list are referenced in different orders as historical prediction motion vector candidates or historical merge candidates.

[0016] The image decoding program of the sixth aspect of the present invention enables a computer to execute the following steps: a coding information saving step, adding the inter-frame prediction information used in the inter-frame prediction to the historical prediction motion vector candidate list; a historical prediction motion vector candidate derivation step, deriving the historical prediction motion vector candidate from the historical prediction motion vector candidate list; and a historical merge candidate derivation step, deriving the historical merge candidate from the historical prediction motion vector candidate list; in the historical prediction motion vector candidate derivation step and the historical merge candidate derivation step, the candidates included in the historical prediction motion vector candidate list are referenced in different orders as historical prediction motion vector candidates or historical merge candidates.

[0017] Effects of the Invention

[0018] According to the present invention, highly efficient image encoding / decoding processing can be realized with a low load. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a block diagram of an image encoding apparatus according to an embodiment of the present invention;

[0020] Figure 2 is a block diagram of an image decoding apparatus according to an embodiment of the present invention;

[0021] Figure 3 is a flowchart for explaining the action of splitting a tree block;

[0022] Figure 4 1 is a diagram showing a situation where an input image is divided into tree blocks;

[0023] Figure 5 is a diagram illustrating a z-scan;

[0024] Figure 6A is a diagram showing the divided shape of a block;

[0025] Figure 6B is a diagram showing the divided shape of a block;

[0026] Figure 6C is a diagram showing the divided shape of a block;

[0027] Figure 6D is a diagram showing the divided shape of a block;

[0028] Figure 6E is a diagram showing the divided shape of a block;

[0029] Figure 7 This is a flowchart for explaining the operation of dividing a block into four.

[0030] Figure 8 This is a flowchart for explaining the operation of dividing a block into two or three.

[0031] Figure 9 It is a syntax used to describe the shape of a block segmentation;

[0032] Figure 10A is a diagram for explaining intra-frame prediction;

[0033] Figure 10B is a diagram for explaining intra-frame prediction;

[0034] Figure 11 is a diagram for explaining a reference block for inter-frame prediction;

[0035] Figure 12 Syntax for describing the prediction mode of a coding block;

[0036] Figure 13 is a diagram showing the correspondence between syntax elements and modes related to inter-frame prediction;

[0037] Figure 14 is a diagram for explaining motion compensation of an affine transformation with two control points;

[0038] Figure 15 is a diagram for explaining motion compensation of an affine transformation with three control points;

[0039] Figure 16 yes Figure 1 A block diagram showing a detailed structure of the inter-frame prediction unit 102;

[0040] Figure 17 yes Figure 16 A block diagram showing a detailed structure of the normal motion vector prediction mode derivation unit 301;

[0041] Figure 18 yes Figure 16 A block diagram showing a detailed structure of the normal merge mode derivation unit 302;

[0042] Figure 19 Is used to illustrate Figure 16Flowchart of the normal motion vector prediction mode derivation process of the normal motion vector prediction mode derivation unit 301;

[0043] Figure 20 is a flowchart showing the processing steps of a normal motion vector prediction mode derivation process;

[0044] Figure 21 is a flowchart illustrating the processing steps of a normal merge mode export process;

[0045] Figure 22 yes Figure 2 A block diagram showing a detailed structure of the inter-frame prediction unit 203;

[0046] Figure 23 yes Figure 22 A block diagram showing a detailed structure of the normal motion vector prediction mode derivation unit 401;

[0047] Figure 24 yes Figure 22 A block diagram showing a detailed structure of the normal merge mode derivation unit 402;

[0048] Figure 25 Is used to illustrate Figure 22 Flowchart of the normal motion vector prediction mode derivation process of the normal motion vector prediction mode derivation unit 401;

[0049] Figure 26 1 is a diagram illustrating the steps of initializing / updating a historical motion vector prediction candidate list;

[0050] Figure 27 This is a flowchart of the same element confirmation processing step in the historical motion vector predictor candidate list initialization / update processing step;

[0051] Figure 28 This is a flowchart of an element shifting process step in a historical motion vector prediction candidate list initialization / update process step;

[0052] Figure 29 is a flowchart illustrating the steps of deriving a historical motion vector prediction candidate;

[0053] Figure 30 is a flowchart illustrating the steps of the historical merge candidate derivation process;

[0054] Figure 31A 1 is a diagram for explaining an example of a process of updating a historical motion vector predictor candidate list;

[0055] Figure 31B 1 is a diagram for explaining an example of a process of updating a historical motion vector predictor candidate list;

[0056] Figure 31C1 is a diagram for explaining an example of a process of updating a historical motion vector predictor candidate list;

[0057] Figure 32 1 is a diagram for explaining motion compensation prediction in a case where the L0 reference picture (RefL0Pic) is located before the current picture (CurPic) in L0 prediction.

[0058] Figure 33 A diagram for explaining motion compensation prediction in a case where a reference picture for L0 prediction is at a time subsequent to a current picture in L0 prediction;

[0059] Figure 34 A diagram for explaining the prediction direction of motion compensation prediction in a case where the reference picture for L0 prediction is before the current picture and the reference picture for L1 prediction is after the current picture in bi-prediction;

[0060] Figure 35 A diagram for explaining the prediction direction of motion compensation prediction in a case where a reference picture for L0 prediction and a reference picture for L1 prediction are located before a current picture in bi-prediction;

[0061] Figure 36 A diagram for explaining the prediction direction of motion compensation prediction in a case where the reference picture for L0 prediction and the reference picture for L1 prediction are at a time subsequent to the current picture in bi-prediction;

[0062] Figure 37 This is a diagram for explaining an example of the hardware configuration of a coding / decoding device according to an embodiment of the present invention;

[0063] Figure 38 is a table showing another example of historical motion vector predictor candidates added by initialization of the historical motion vector predictor candidate list;

[0064] Figure 39 is a table showing another example of historical motion vector predictor candidates added by initialization of the historical motion vector predictor candidate list;

[0065] Figure 40 is a table showing another example of historical motion vector predictor candidates added by initialization of the historical motion vector predictor candidate list;

[0066] Figure 41 is a diagram for explaining the structure of a historical motion vector prediction candidate list;

[0067] Figure 42 A diagram for explaining a case where a leading element is deleted when adding to a historical motion vector predictor candidate list;

[0068] Figure 43 A diagram for explaining how each element is moved within the list when adding to the historical motion vector predictor candidate list;

[0069] Figure 44 A diagram for explaining the case where a new element is added to the historical motion vector predictor candidate list;

[0070] Figure 45 is a diagram for explaining the order of reference to the historical motion vector predictor candidate list in the normal merge mode according to the second embodiment;

[0071] Figure 46 A diagram for explaining the order of reference to the historical motion vector predictor candidate list in the normal motion vector predictor mode according to the second embodiment;

[0072] Figure 47 This is a flowchart illustrating the steps of a historical motion vector predictor candidate derivation process according to the second embodiment. DETAILED DESCRIPTION

[0073] The technologies and technical terms used in this embodiment are defined.

[0074] <tree block>

[0075] In the embodiment, the encoding / decoding target image is divided equally into predetermined sizes. This unit is defined as a tree block. Figure 4 In the example, the tree block size is set to 128×128 pixels, but the tree block size is not limited to this and can be set to any size. The tree blocks that are the processing objects (corresponding to the encoding object in the encoding process and the decoding object in the decoding process) are switched in raster scan order, that is, from left to right and from top to bottom. Each tree block can be further recursively divided. The blocks that become the encoding / decoding objects after the tree blocks are recursively divided are defined as coding blocks. In addition, tree blocks and coding blocks are collectively referred to as blocks. By performing appropriate block division, efficient encoding can be achieved. The size of the tree block can be set to a fixed value determined in advance in the encoding device and the decoding device, or a structure can be adopted in which the size of the tree block determined by the encoding device is transmitted to the decoding device. Here, the maximum size of the tree block is set to 128×128 pixels, and the minimum size of the tree block is set to 16×16 pixels. In addition, the maximum size of the coding block is set to 64×64 pixels, and the minimum size of the coding block is set to 4×4 pixels.

[0076] <Prediction Mode>

[0077] Intra-frame prediction (MODE_INTRA) and inter-frame prediction (MODE_INTER) are switched in units of the processing target coding block. The intra-frame prediction (MODE_INTRA) performs prediction based on the processed image signal of the processing target image, and the inter-frame prediction performs prediction based on the image signal of the processed image.

[0078] The processed image is used to obtain an image, image signal, tree block, block, coding block, etc. by decoding the signal that has been encoded in the encoding process, and is used to obtain an image, image signal, tree block, block, coding block, etc. that has been decoded in the decoding process.

[0079] The mode for distinguishing the intra prediction (MODE_INTRA) and the inter prediction (MODE_INTER) is defined as a prediction mode (PredMode). The prediction mode (PredMode) represents the intra prediction (MODE_INTRA) or the inter prediction (MODE_INTER) in the form of a value.

[0080] <Inter-frame prediction>

[0081] In inter-frame prediction, which performs prediction based on an image signal of a processed image, multiple processed images can be used as reference pictures. To manage multiple reference pictures, two reference lists, L0 (reference list 0) and L1 (reference list 1), are defined, and reference pictures are identified using reference indices in each list. In a P slice, L0 prediction (Pred_L0) can be used. In a B slice, L0 prediction (Pred_L0), L1 prediction (Pred_L1), and bi-prediction (Pred_BI) can be used. L0 prediction (Pred_L0) is inter-frame prediction that refers to a reference picture managed by L0, and L1 prediction (Pred_L1) is inter-frame prediction that refers to a reference picture managed by L1. Bi-prediction (Pred_BI) is inter-frame prediction that performs L0 and L1 predictions simultaneously and refers to individual reference pictures managed by each of L0 and L1. Information that determines L0 prediction, L1 prediction, and bi-prediction is defined as an inter-frame prediction mode. In subsequent processing, constants and variables with the subscript LX added to their output are presumed to be processed as L0 and L1.

[0082] <Predicted Motion Vector Mode>

[0083] The motion vector prediction mode is a mode that transmits an index for determining a motion vector prediction, a differential motion vector, an inter-prediction mode, and a reference index, thereby determining inter-prediction information for the current block. The motion vector prediction is derived from motion vector prediction candidates derived from processed blocks adjacent to the current block or blocks in a processed image that are co-located with or near (adjacent to) the current block, and the index used to determine the motion vector prediction.

[0084] <Merge Mode>

[0085] The merge mode is a mode in which the inter-frame prediction information of the processing target block is derived based on the inter-frame prediction information of processed blocks adjacent to the processing target block or blocks belonging to the processed image that are located at the same position as or near (adjacent to) the processing target block, without transmitting the differential motion vector or reference index.

[0086] The spatial merge candidates are defined as the processed blocks adjacent to the current block and their inter-frame prediction information. The temporal merge candidates are defined as the blocks in the processed image that are co-located with or near (adjacent to) the current block and their inter-frame prediction information derived from the inter-frame prediction information. Each merge candidate is registered in a merge candidate list, and the merge candidate used for predicting the current block is identified by a merge index.

[0087] <adjacent blocks>

[0088] Figure 11 This diagram illustrates the reference blocks used to derive inter-frame prediction information in motion vector prediction mode and merge mode. A0, A1, A2, B0, B1, B2, and B3 are processed blocks adjacent to the current block. T0 is a block in the processed image that is co-located with or near (adjacent to) the current block in the current image.

[0089] A1 and A2 are blocks located to the left of the current coding block and adjacent to it. B1 and B3 are blocks located above and adjacent to it. A0, B0, and B2 are blocks located to the lower left, upper right, and upper left of the current coding block, respectively.

[0090] Details of how adjacent blocks are processed in the predicted motion vector mode and the merge mode are described later.

[0091] <Affine Transformation Motion Compensation>

[0092] Affine transformation motion compensation divides the coding block into predetermined sub-blocks and determines a motion vector for each sub-block. The motion vector for each sub-block is derived based on one or more control points derived from inter-frame prediction information for processed blocks adjacent to the current block or blocks in the processed image that are co-located with or near the current block. In this embodiment, the sub-block size is set to 4×4 pixels, but this size is not limited to this; motion vectors can also be derived on a pixel-by-pixel basis.

[0093] Figure 14 This example shows an affine transformation motion compensation example with two control points. In this case, the two control points have two parameters: a horizontal component and a vertical component. Therefore, an affine transformation with two control points is called a four-parameter affine transformation. Figure 14 CP1 and CP2 are control points.

[0094] Figure 15 This example shows affine transformation motion compensation when there are three control points. In this case, the three control points have two parameters: a horizontal component and a vertical component. Therefore, an affine transformation with three control points is called a six-parameter affine transformation. Figure 15 CP1, CP2, and CP3 are control points.

[0095] Affine motion compensation can be used in either MV prediction mode or merge mode. The mode in which affine motion compensation is applied in MV prediction mode is defined as sub-block MV prediction mode, and the mode in which affine motion compensation is applied in merge mode is defined as sub-block merge mode.

[0096] <Syntax of Inter-frame Prediction>

[0097] use Figure 12 and Figure 13 The syntax related to inter-frame prediction is explained.

[0098] Figure 12 The merge_flag is a flag indicating whether the processing target coding block is set to merge mode or predicted motion vector mode. The merge_affine_flag is a flag indicating whether the sub-block merge mode is applied to the processing target coding block in merge mode. The inter_affine_flag is a flag indicating whether the sub-block predicted motion vector mode is applied to the processing target coding block in predicted motion vector mode. The cu_affine_type_flag is a flag used to determine the number of control points in sub-block predicted motion vector mode.

[0099] Figure 13The values ​​of each syntax element and the corresponding prediction method are shown. merge_flag = 1, merge_affine_flag = 0 corresponds to normal merge mode. Normal merge mode is a merge mode that is not a sub-block merge. mergeflag = 1, merge affine flag = 1 corresponds to sub-block merge mode. merge_flag = 0, inter_affine_flag = 0 corresponds to normal motion vector prediction mode. Normal motion vector prediction mode is a merge of predicted motion vectors that is not a sub-block motion vector prediction mode. merge_flag = 0, inter_affine_flag = 1 corresponds to sub-block motion vector prediction mode. When merge_flag = 0, inter_affine_flag = 1, cu_affine_type_flag is further transmitted to determine the number of control points.

[0100] <poc>

[0101] The POC (Picture Order Count) is a variable associated with the picture to be encoded, and is set to a value that increments by 1 according to the output order of the picture. Based on the POC value, it is possible to determine whether it is the same picture, the relationship between pictures in the output order, and the distance between pictures. For example, if the POC values ​​of two pictures are the same, they can be determined to be the same picture. In the case where the POC values ​​of two pictures are different, it can be determined that the picture with the smaller POC value is the picture output first. The difference between the POC values ​​of the two pictures represents the distance between the pictures in the time axis direction.

[0102] (First embodiment)

[0103] An image encoding device 100 and an image decoding device 200 according to the first embodiment of the present invention will be described.

[0104] Figure 1 This is a block diagram of an image coding apparatus 100 according to the first embodiment. The image coding apparatus 100 according to the embodiment includes a block division unit 101, an inter-frame prediction unit 102, an intra-frame prediction unit 103, a decoded image memory 104, a prediction method determination unit 105, a residual generation unit 106, an orthogonal transform / quantization unit 107, a bit string encoding unit 108, an inverse quantization / inverse orthogonal transform unit 109, a decoded image signal superposition unit 110, and a coding information storage memory 111.

[0105] The block division unit 101 recursively divides the input image to generate coding blocks. The block division unit 101 includes a 4-division unit that divides the block to be divided horizontally and vertically, and a 2-3 division unit that divides the block to be divided horizontally or vertically. The block division unit 101 sets the generated coding block as the processing target coding block and provides the image signal of the processing target coding block to the inter-frame prediction unit 102, the intra-frame prediction unit 103, and the residual generation unit 106. Furthermore, the block division unit 101 provides information indicating the determined recursive division structure to the bitstream encoding unit 108. The detailed operation of the block division unit 101 will be described later.

[0106] The inter-frame prediction unit 102 performs inter-frame prediction on the current coding block. Based on the inter-frame prediction information stored in the coding information storage memory 111 and the decoded image signal stored in the decoded image memory 104, the inter-frame prediction unit 102 derives multiple inter-frame prediction information candidates, selects an appropriate inter-frame prediction mode from the derived multiple candidates, and provides the selected inter-frame prediction mode and the predicted image signal corresponding to the selected inter-frame prediction mode to the prediction method determination unit 105. The detailed structure and operation of the inter-frame prediction unit 102 will be described later.

[0107] The intra-frame prediction unit 103 performs intra-frame prediction on the current coding block. Using the decoded image signal stored in the decoded image memory 104 as reference pixels, the intra-frame prediction unit 103 generates a predicted image signal through intra-frame prediction based on coding information such as the intra-frame prediction mode stored in the coding information storage memory 111. During intra-frame prediction, the intra-frame prediction unit 103 selects an appropriate intra-frame prediction mode from a plurality of intra-frame prediction modes and provides the selected intra-frame prediction mode and the predicted image signal corresponding to the selected intra-frame prediction mode to the prediction method determination unit 105.

[0108] Figure 10A and Figure 10B An example of intra prediction is shown. Figure 10A This diagram shows the correspondence between intra-prediction prediction directions and intra-prediction mode numbers. For example, intra-prediction mode 50 generates an intra-predicted image by vertically replicating reference pixels. Intra-prediction mode 1 is a DC mode, in which all pixel values ​​of the processing target block are set to the average value of the reference pixels. Intra-prediction mode 0 is a Planar mode (two-dimensional mode), in which a two-dimensional intra-predicted image is generated based on reference pixels in the vertical and horizontal directions. Figure 10B This is an example of an intra-prediction image generated in intra-prediction mode 40. The intra-prediction unit 103 copies the value of a reference pixel in the direction indicated by the intra-prediction mode to each pixel in the processing target block. If the reference pixel in the intra-prediction mode is not an integer position, the intra-prediction unit 103 determines the reference pixel value by interpolation based on the values ​​of reference pixels at surrounding integer positions.

[0109] The decoded image memory 104 stores the decoded image generated by the decoded image signal superimposing unit 110. The decoded image memory 104 supplies the stored decoded image to the inter prediction unit 102 and the intra prediction unit 103.

[0110] The prediction method determination unit 105 determines the optimal prediction mode for each of intra-frame prediction and inter-frame prediction by evaluating the coding information, the residual coding amount, the amount of distortion between the predicted image signal and the processing target image signal, and so on. In the case of intra-frame prediction, the prediction method determination unit 105 provides intra-frame prediction information, such as the intra-frame prediction mode, as coding information to the bitstream coding unit 108. In the case of merge mode for inter-frame prediction, the prediction method determination unit 105 provides inter-frame prediction information, such as the merge index and information indicating whether the mode is sub-block merge mode (sub-block merge flag), as coding information to the bitstream coding unit 108. In the case of motion vector prediction mode for inter-frame prediction, the prediction method determination unit 105 provides inter-frame prediction information, such as the inter-frame prediction mode, motion vector prediction index, reference indices for L0 and L1, differential motion vector, and information indicating whether the mode is sub-block motion vector prediction mode (sub-block motion vector prediction flag), as coding information to the bitstream coding unit 108. Furthermore, the prediction method determination unit 105 provides the determined coding information to the coding information storage memory 111. The prediction method determination unit 105 supplies the predicted image signal to the residual generation unit 106 and the decoded image signal superposition unit 110 .

[0111] The residual generation unit 106 generates a residual by subtracting the predicted image signal from the image signal to be processed, and supplies the residual to the orthogonal transformation and quantization unit 107 .

[0112] The orthogonal transform / quantization section 107 performs orthogonal transform and quantization on the residual according to the quantization parameter to generate an orthogonal transformed / quantized residual, and supplies the generated residual to the bit string encoding section 108 and the inverse quantization / inverse orthogonal transform section 109 .

[0113] The bitstream encoder 108 encodes, for each coding block, coding information corresponding to the prediction method determined by the prediction method determiner 105, in addition to information for the sequence, picture, slice, and coding block units. Specifically, the bitstream encoder 108 encodes the prediction mode PredMode for each coding block. When the prediction mode is inter-frame prediction (MODE_INTER), the bitstream encoder 108 encodes coding information (inter-frame prediction information), including a flag for identifying whether the prediction mode is merge mode, a sub-block merge flag, a merge index if the prediction mode is merge mode, the inter-frame prediction mode if not merge mode, a predicted motion vector index, information related to differential motion vectors, and a sub-block predicted motion vector flag, according to a predetermined syntax (bitstream syntax rules), generating a first bitstream. When the prediction mode is intra-frame prediction (MODE_INTRA), the bitstream encoder encodes coding information (intra-frame prediction information), including the intra-frame prediction mode, according to a predetermined syntax (bitstream syntax rules), generating a first bitstream. The bitstream encoding unit 108 performs entropy encoding on the orthogonally transformed and quantized residual according to a predetermined syntax to generate a second bitstream. The bitstream encoding unit 108 multiplexes the first bitstream and the second bitstream according to a predetermined syntax and outputs a bitstream.

[0114] The inverse quantization / inverse orthogonal transform section 109 inversely quantizes and inversely orthogonally transforms the orthogonally transformed / quantized residual supplied from the orthogonal transform / quantization section 107 to calculate a residual, and supplies the calculated residual to the decoded image signal superposition section 110 .

[0115] The decoded image signal superimposing unit 110 superimposes the predicted image signal corresponding to the decision made by the prediction method determining unit 105 with the residual obtained by the inverse quantization and inverse orthogonal transformation performed by the inverse quantization / inverse orthogonal transformation unit 109 to generate a decoded image, which is then stored in the decoded image memory 104. Alternatively, the decoded image signal superimposing unit 110 may perform filtering processing on the decoded image to reduce distortion such as block distortion caused by coding, and then store the decoded image in the decoded image memory 104.

[0116] The coding information storage memory 111 stores coding information such as the prediction mode (inter prediction or intra prediction) determined by the prediction method determination unit 105. In the case of inter prediction, the coding information stored in the coding information storage memory 111 includes inter prediction information such as the determined motion vector, reference indices of the reference lists L0 and L1, and a list of historical motion vector prediction candidates. Furthermore, in the case of merge mode for inter prediction, the coding information stored in the coding information storage memory 111 includes, in addition to the aforementioned information, inter prediction information including a merge index and information indicating whether the mode is sub-block merge mode (sub-block merge flag). Furthermore, in the case of motion vector prediction mode for inter prediction, the coding information stored in the coding information storage memory 111 includes, in addition to the aforementioned information, inter prediction information including the inter prediction mode, predicted motion vector index, differential motion vector, and information indicating whether the mode is sub-block motion vector prediction mode (sub-block motion vector prediction flag). In the case of intra prediction, the coding information stored in the coding information storage memory 111 includes intra prediction information such as the determined intra prediction mode.

[0117] Figure 2 Is to express Figure 1 This is a block diagram showing the structure of an image decoding device according to an embodiment of the present invention corresponding to an image encoding device. The image decoding device according to the embodiment includes a bitstream decoding unit 201, a block division unit 202, an inter-frame prediction unit 203, an intra-frame prediction unit 204, a coding information storage memory 205, an inverse quantization / inverse orthogonal transformation unit 206, a decoded image signal superposition unit 207, and a decoded image memory 208.

[0118] Figure 2 The decoding process of the image decoding device and Figure 1 The decoding process of the internal setting of the image encoding device corresponds to Figure 2 The structures of the coding information storage memory 205, the inverse quantization / inverse orthogonal transformation unit 206, the decoded image signal superposition unit 207 and the decoded image memory 208 are similar to those of the decoded image signal storage memory 205. Figure 1 The encoding information storage memory 111, the inverse quantization / inverse orthogonal transformation unit 109, the decoded image signal superposition unit 110 and the decoded image memory 104 of the image encoding device respectively correspond to functions.

[0119] The bitstream provided to the bitstream decoder 201 is separated according to specified syntax rules. The bitstream decoder 201 decodes the separated first bitstream to obtain information on the sequence, picture, slice, and coding block units, as well as coding information for each coding block. Specifically, the bitstream decoder 201 decodes the prediction mode PredMode for each coding block. The prediction mode PredMode identifies whether it is inter-frame prediction (MODE_INTER) or intra-frame prediction (MODE_INTRA). If the prediction mode is inter-frame prediction (MODE_INTER), the bitstream decoder 201 decodes the coding information (inter-frame prediction information) related to the merge mode flag, the merge index in merge mode, the sub-block merge flag, the inter-frame prediction mode, the prediction motion vector index, the differential motion vector, and the sub-block prediction motion vector flag in the case of predicted motion vector mode according to the specified syntax. The bitstream decoder 201 then provides the coding information (inter-frame prediction information) to the coding information storage memory 205 via the inter-frame prediction unit 203 and the block division unit 202. When the prediction mode is intra prediction (MODE_INTRA), the intra prediction mode and other coded information (intra prediction information) are decoded according to a predetermined syntax and supplied to the coded information storage memory 205 via the inter prediction unit 203 or the intra prediction unit 204 and the block division unit 202. The bitstream decoding unit 201 decodes the separated second bitstream, calculates the residual after orthogonal transformation and quantization, and supplies the residual after orthogonal transformation and quantization to the inverse quantization and inverse orthogonal transformation unit 206.

[0120] When the prediction mode PredMode of the coding block being processed is a motion vector predictor mode in inter-frame prediction (MODE_INTER), the inter-frame prediction unit 203 derives multiple motion vector predictor candidates using the coding information of the decoded image signal stored in the coding information storage memory 205, and registers the derived multiple motion vector predictor candidates in a motion vector predictor candidate list described later. The inter-frame prediction unit 203 selects a motion vector predictor corresponding to the motion vector predictor index decoded and provided by the bitstream decoding unit 201 from the multiple motion vector predictor candidates registered in the motion vector predictor candidate list, calculates a motion vector based on the differential motion vector decoded by the bitstream decoding unit 201 and the selected motion vector predictor, and stores the calculated motion vector in the coding information storage memory 205 along with other coding information. Here, the coding information of the coding block to be provided / stored includes the prediction mode PredMode, flags predFlagL0[xP][yP] and predFlagL1[xP][yP] indicating whether L0 prediction and L1 prediction are used, reference indices refIdxL0[xP][yP] and refIdxL1[xP][yP] of L0 and L1, and motion vectors mvL0[xP][yP] and mvL1[xP][yP] of L0 and L1, respectively. Here, xP and yP are indices indicating the position of the upper left pixel of the coding block within the picture. When the prediction mode PredMode is inter-frame prediction (MODE_INTER) and the inter-frame prediction mode is L0 prediction (Pred_L0), the flag predFlagL0 indicating whether L0 prediction is used is 1, and the flag predFlagL1 indicating whether L1 prediction is used is 0. When the inter prediction mode is L1 prediction (Pred_L1), the flag predFlagL0 indicating whether L0 prediction is used is 0, and the flag predFlagL1 indicating whether L1 prediction is used is 1. When the inter prediction mode is bi-prediction (Pred_BI), the flag predFlagL0 indicating whether L0 prediction is used and the flag predFlagL1 indicating whether L1 prediction is used are both 1. Furthermore, when the prediction mode PredMode of the coding block to be processed is the merge mode in the inter prediction (MODE_INTER), a merge candidate is derived.Using the coding information of the decoded coding blocks stored in the coding information storage memory 205, multiple merge candidates are derived and registered in a merge candidate list, which will be described later. A merge candidate corresponding to the merge index decoded and provided by the bitstream decoding unit 201 is selected from the multiple merge candidates registered in the merge candidate list. Inter-frame prediction information, such as flags predFlagL0[xP][yP] and predFlagL1[xP][yP] indicating whether to use the L0 prediction and L1 prediction of the selected merge candidate, reference indices refIdxL0[xP][yP] and refIdxL1[xP][yP] for L0 and L1, and motion vectors mvL0[xP][yP] and mvL1[xP][yP] for L0 and L1, is stored in the coding information storage memory 205. Here, xP and yP are indices indicating the position of the upper-left pixel of the coding block within the picture. The detailed structure and operation of the inter-frame prediction unit 203 will be described later.

[0121] When the prediction mode PredMode of the coding block being processed is intra prediction (MODE_INTRA), the intra prediction unit 204 performs intra prediction. The intra prediction mode is included in the coding information decoded by the bitstream decoding unit 201. Based on the intra prediction mode included in the decoded information decoded by the bitstream decoding unit 201, the intra prediction unit 204 generates a predicted image signal using intra prediction and the decoded image signal stored in the decoded image memory 208, and supplies the generated predicted image signal to the decoded image signal superimposition unit 207. Since the intra prediction unit 204 corresponds to the intra prediction unit 103 of the image coding device 100, it performs the same processing as the intra prediction unit 103.

[0122] The inverse quantization / inverse orthogonal transformation unit 206 performs inverse orthogonal transformation and inverse quantization on the orthogonally transformed / quantized residual decoded by the bit string decoding unit 201 to obtain an inverse orthogonally transformed / inverse quantized residual.

[0123] The decoded image signal superimposing unit 207 decodes the decoded image signal by superimposing the predicted image signal obtained by the inter-frame prediction unit 203 or the predicted image signal obtained by the intra-frame prediction unit 204 with the residual after inverse orthogonal transformation and inverse quantization by the inverse quantization / inverse orthogonal transformation unit 206, and stores the decoded image signal in the decoded image memory 208. When storing the decoded image signal in the decoded image memory 208, the decoded image signal superimposing unit 207 may perform filtering processing on the decoded image to reduce block distortion caused by encoding, etc., and then store the decoded image signal in the decoded image memory 208.

[0124] Next, the operation of the block division unit 101 in the image coding apparatus 100 will be described. Figure 3 This flowchart illustrates the process of segmenting an image into treeblocks and further segmenting each treeblock. First, the input image is segmented into treeblocks of a predetermined size (step S1001). Each treeblock is scanned in a predetermined order, i.e., a raster scan order (step S1002), and the interior of the target treeblock is segmented (step S1003).

[0125] Figure 7 1 is a flowchart showing the detailed operation of the division process of step S1003. First, it is determined whether the block to be processed is divided into four (step S1101).

[0126] If it is determined that the processing block is to be divided into four, the processing block is divided into four (step S1102). Each block obtained by dividing the processing block is scanned in the Z scanning order, that is, in the order of upper left, upper right, lower left, and lower right (step S1103). Figure 5 is an example of a Z-scan order, Figure 6A 601 is an example of dividing the processing target block into four. Figure 6A The numbers 0 to 3 of 601 represent the order of processing. Then, for each block divided in step S1101, recursively execute Figure 7 Segmentation processing (step S1104).

[0127] When it is determined that the processing target block is not to be divided into 4, 2-3 division is performed (step S1105).

[0128] Figure 8 This is a flowchart showing the detailed operation of the 2-3 division process in step S1105. First, it is determined whether the block to be processed is to be divided into 2-3, that is, whether to divide it into 2 or 3 (step S1201).

[0129] If it is determined that the processing target block is not to be divided into 2-3 blocks, that is, if it is determined that the block is not to be divided, the division is terminated (step S1211). In other words, no further recursive division is performed on the block divided by the recursive division process.

[0130] When it is determined that the processing target block is to be divided into 2-3, it is determined whether the processing target block is to be further divided into 2 (step S1202).

[0131] If it is determined that the processing target block is to be divided into two, it is determined whether to divide the processing target block into two vertically (in the vertical direction) (step S1203). Based on the result, the processing target block is divided into two vertically (in the vertical direction) (step S1204), or the processing target block is divided into two horizontally (in the horizontal direction) (step S1205). As a result of step S1204, the processing target block is divided into two vertically (in the vertical direction) (step S1206). Figure 6B As shown in 602 in FIG, it is divided into two parts, upper and lower (vertical direction). As a result of step S1205, the processing object block is as follows Figure 6D As shown in 604 in FIG, it is divided into two parts, left and right (horizontally).

[0132] In step S1202, if it is not determined that the processing target block is to be divided into two, that is, if it is determined that it is to be divided into three, it is determined whether the processing target block is to be divided into the top, center, and bottom (vertical direction) (step S1206). Based on the result, the processing target block is divided into three parts in the top, center, and bottom (vertical direction) (step S1207), or the processing target block is divided into three parts in the left, center, and right (horizontal direction) (step S1208). In the result of step S1207, if the processing target block is divided into three parts in the top, center, and bottom (vertical direction), the processing target block is divided into three parts in the left, center, and right (horizontal direction) (step S1208). Figure 6C As shown in 603, it is divided into three parts (upper, middle and lower (vertical direction). In the result of step S1208, the processing object block is as follows Figure 6E As shown in 605, it is divided into three parts: left, center and right (horizontally).

[0133] After executing any one of step S1204 , step S1205 , step S1207 , and step S1208 , each block obtained by dividing the processing target block is scanned in order from left to right and from top to bottom (step S1209 ). Figures 6B to 6E The numbers 0 to 2 from 602 to 605 indicate the order of processing. For each divided block, recursively execute Figure 8 2-3 splitting processing (step S1210).

[0134] The recursive block division described here can also limit the need for division based on the number of divisions or the size of the block being processed. The information limiting the need for division can be implemented in a configuration where the information is agreed upon in advance between the encoding device and the decoding device without being transmitted, or the encoding device determines the information limiting the need for division, records it in a bit string, and transmits it to the decoding device.

[0135] When a block is split, the block before the split is called a parent block, and each block after the split is called a child block.

[0136] Next, the operation of the block division unit 202 in the image decoding device 200 will be described. The block division unit 202 divides the treeblocks according to the same processing steps as the block division unit 101 in the image coding device 100. However, the difference is that, whereas the block division unit 101 in the image coding device 100 determines the optimal block division shape by applying an optimization method such as optimal shape estimation based on image recognition or distortion rate optimization, the block division unit 202 in the image decoding device 200 determines the block division shape by decoding block division information recorded in a bit string.

[0137] Figure 9 The syntax (syntactic rules of bit strings) related to the block division of the first embodiment is shown. coding_quadtree() represents the syntax involved in the block division process of 4. multi_type_tree() represents the syntax involved in the block division process of 2 or 3. qt_split is a flag indicating whether the block is divided into 4. When the block is divided into 4, set qt_split = 1, and when the block is not divided into 4, set qt_split = 0. In the case of 4 division (qt_split = 1), the 4-division process is recursively performed on each block after the 4 division (coding_quadtree (0), coding_quadtree (1), coding_quadtree (2), coding_quadtree (3), and the 0 to 3 of the independent variables correspond to Figure 6A 601). In the case of no 4-part split (qt_split=0), the subsequent split is determined according to multi_type_tree(). mtt_split is a flag indicating whether to perform further splitting. In addition, in the case of splitting (mtt_split=1), a flag indicating whether to split in the vertical direction or the horizontal direction, namely mtt_split_vertical, and a flag determining whether to split in 2 or 3, namely mtt_split_binary, are transmitted. mtt_split_vertical=1 indicates splitting in the vertical direction, and mtt_split_vertical=0 indicates splitting in the horizontal direction. mtt_split_binary=1 indicates splitting in 2, and mtt_split_binary=0 indicates splitting in 3. In the case of 2-part splitting (mtt_split_binary=1), each block after the 2-part split is recursively split (multi_type_tree(0), multi_type_tree(1), the 0 to 1 of the independent variable corresponds to Figures 6B to 6D In the case of 3-way split (mtt_split_binary=0), each block after 3-way split is recursively split (multi_type_tree(0), multi_type_tree(1), multi_type_tree(2), 0 to 2 correspond to Figure 6B 603 or Figure 6E 605). Hierarchical block segmentation is performed by recursively calling multi_type_tree until mtt_split = 0.

[0138] <Inter-frame prediction>

[0139] The inter-frame prediction method of the embodiment is Figure 1 The inter-frame prediction unit 102 of the image encoding device and Figure 2 The inter-frame prediction unit 203 of the image decoding device is implemented.

[0140] The inter-frame prediction method according to the embodiment will be described with reference to the drawings. The inter-frame prediction method is performed in units of coding blocks in either encoding or decoding.

[0141] <Description of the Inter-frame Prediction Unit 102 on the Encoding Side>

[0142] Figure 16 It shows Figure 1 The figure shows the detailed structure of the inter-frame prediction unit 102 of the image encoding device. The normal motion vector prediction mode derivation unit 301 derives a plurality of normal motion vector prediction candidates to select a motion vector prediction, and calculates a differential motion vector between the selected motion vector prediction and the detected motion vector. The detected inter-frame prediction mode, reference index, motion vector, and calculated differential motion vector constitute inter-frame prediction information for the normal motion vector prediction mode. This inter-frame prediction information is provided to the inter-frame prediction mode determination unit 305. The detailed structure and processing of the normal motion vector prediction mode derivation unit 301 will be described later.

[0143] The normal merge mode derivation unit 302 derives multiple normal merge candidates, selects a normal merge candidate, and obtains inter-frame prediction information for the normal merge mode. This inter-frame prediction information is provided to the inter-frame prediction mode determination unit 305. The detailed structure and processing of the normal merge mode derivation unit 302 will be described later.

[0144] The sub-block motion vector predictor mode derivation unit 303 derives multiple sub-block motion vector predictor candidates to select a sub-block motion vector predictor and calculates a motion vector difference between the selected sub-block motion vector predictor and the detected motion vector. The detected inter-frame prediction mode, reference index, motion vector, and calculated motion vector difference constitute inter-frame prediction information for the sub-block motion vector predictor mode. This inter-frame prediction information is provided to the inter-frame prediction mode determination unit 305.

[0145] The sub-block merging mode derivation unit 304 derives a plurality of sub-block merging candidates, selects a sub-block merging candidate, and obtains inter-frame prediction information of the sub-block merging mode. This inter-frame prediction information is provided to the inter-frame prediction mode determination unit 305 .

[0146] The inter-prediction mode determination unit 305 determines inter-prediction information based on the inter-prediction information supplied from the normal motion vector prediction mode derivation unit 301, the normal merge mode derivation unit 302, the sub-block motion vector prediction mode derivation unit 303, and the sub-block merge mode derivation unit 304. The inter-prediction information corresponding to the determination result is supplied from the inter-prediction mode determination unit 305 to the motion-compensated prediction unit 306.

[0147] The motion-compensated prediction unit 306 performs inter-frame prediction on the reference image signal stored in the decoded image memory 104 based on the determined inter-frame prediction information. The detailed structure and processing of the motion-compensated prediction unit 306 will be described later.

[0148] <Description of the Inter-frame Prediction Unit 203 on the Decoding Side>

[0149] Figure 22 It shows Figure 2 FIG. 2 is a diagram showing a detailed structure of the inter-frame prediction unit 203 of the image decoding device.

[0150] The normal motion vector predictor mode derivation unit 401 derives multiple normal motion vector predictor candidates to select a motion vector predictor. It then calculates the sum of the selected motion vector predictor and the decoded differential motion vector as the motion vector. The decoded inter-frame prediction mode, reference index, and motion vector constitute the inter-frame prediction information for the normal motion vector predictor mode. This inter-frame prediction information is provided to the motion-compensated prediction unit 406 via the switch 408. The detailed structure and processing of the normal motion vector predictor mode derivation unit 401 will be described later.

[0151] The normal merge mode derivation unit 402 derives multiple normal merge candidates, selects a normal merge candidate, and obtains inter-frame prediction information for the normal merge mode. This inter-frame prediction information is provided to the motion-compensated prediction unit 406 via the switch 408. The detailed structure and processing of the normal merge mode derivation unit 402 will be described later.

[0152] The sub-block motion vector prediction mode derivation unit 403 derives multiple sub-block motion vector prediction candidates to select a sub-block motion vector prediction. The sum of the selected sub-block motion vector prediction and the decoded differential motion vector is calculated as the motion vector. The decoded inter-frame prediction mode, reference index, and motion vector form the inter-frame prediction information for the sub-block motion vector prediction mode. This inter-frame prediction information is provided to the motion-compensated prediction unit 406 via the switch 408.

[0153] The sub-block merging mode derivation unit 404 derives a plurality of sub-block merging candidates, selects a sub-block merging candidate, and obtains inter-frame prediction information of the sub-block merging mode. This inter-frame prediction information is provided to the motion compensation prediction unit 406 via the switch 408 .

[0154] The motion compensation prediction unit 406 performs inter-frame prediction on the reference image signal stored in the decoded image memory 208 based on the determined inter-frame prediction information. The detailed structure and processing of the motion compensation prediction unit 406 are the same as those of the motion compensation prediction unit 306 on the encoding side.

[0155] <Normal Motion Vector Prediction Mode Derivation Unit (Normal AMVP)>

[0156] Figure 17 The normal prediction motion vector mode derivation unit 301 includes a spatial prediction motion vector candidate derivation unit 321, a temporal prediction motion vector candidate derivation unit 322, a historical prediction motion vector candidate derivation unit 323, a prediction motion vector candidate supplement unit 325, a normal motion vector detection unit 326, a prediction motion vector candidate selection unit 327, and a motion vector subtraction unit 328.

[0157] Figure 23 The normal prediction motion vector mode derivation unit 401 includes a spatial prediction motion vector candidate derivation unit 421, a temporal prediction motion vector candidate derivation unit 422, a historical prediction motion vector candidate derivation unit 423, a prediction motion vector candidate supplement unit 425, a prediction motion vector candidate selection unit 426, and a motion vector addition unit 427.

[0158] Use separately Figure 19 、 Figure 25 The flowchart of will explain the processing steps of the normal motion vector prediction mode derivation unit 301 on the encoding side and the normal motion vector prediction mode derivation unit 401 on the decoding side. Figure 19 1 is a flowchart showing the normal motion vector prediction mode derivation processing steps by the normal motion vector mode derivation unit 301 on the encoding side. Figure 25 3 is a flowchart showing the normal motion vector predictor mode derivation processing procedure by the normal motion vector mode derivation unit 401 on the decoding side.

[0159] <Normal Motion Vector Prediction Mode Derivation Unit (Normal AMVP): Description on the Encoding Side>

[0160] refer to Figure 19 The following describes the steps for deriving the normal motion vector prediction mode on the encoding side. Figure 19 Sometimes the description of the processing steps is omitted Figure 19 The word "usually" is shown.

[0161] First, the normal motion vector detection unit 326 detects a normal motion vector ( Figure 19 Step S100).

[0162] Next, the spatial motion vector predictor candidate derivation unit 321, the temporal motion vector predictor candidate derivation unit 322, the historical motion vector predictor candidate derivation unit 323, the motion vector predictor candidate supplementation unit 325, the motion vector predictor candidate selection unit 327, and the motion vector subtraction unit 328 calculate, for each of L0 and L1, a motion vector difference ( Figure 19 Specifically, when the prediction mode PredMode of the processing target block is inter prediction (MODE_INTER) and the inter prediction mode is L0 prediction (Pred_L0), a prediction motion vector candidate list mvpListL0 of L0 is calculated, the prediction motion vector mvpL0 is selected, and a differential motion vector mvdL0 of the motion vector mvL0 of L0 is calculated. When the inter prediction mode of the processing target block is L1 prediction (Pred_L1), a prediction motion vector candidate list mvpListL1 of L1 is calculated, the prediction motion vector mvpL1 is selected, and a differential motion vector mvdL1 of the motion vector mvL1 of L1 is calculated. When the inter-frame prediction mode of the object block is dual prediction (Pred_BI), L0 prediction and L1 prediction are performed simultaneously, the predicted motion vector candidate list mvpListL0 of L0 is calculated, the predicted motion vector mvpL0 of L0 is selected, the differential motion vector mvdL0 of the motion vector mvL0 of L0 is calculated, and the predicted motion vector candidate list mvpListL1 of L1 is calculated, the predicted motion vector mvpL1 of L1 is calculated, and the differential motion vector mvdL1 of the motion vector mvL1 of L1 is calculated respectively.

[0163] Differential motion vector calculations are performed for L0 and L1 separately, but the process is common to both. Therefore, in the following description, L0 and L1 are collectively referred to as LX. When calculating the differential motion vector for L0, X in LX is 0, and when calculating the differential motion vector for L1, X in LX is 1. Furthermore, when calculating the differential motion vector for LX, if information from another list is referenced instead of LX, the other list is referred to as LY.

[0164] In the case of using the motion vector mvLX of LX ( Figure 19 Step S102: Yes), calculate the candidates for the predicted motion vector of LX, and construct the predicted motion vector candidate list mvpListLX of LX ( Figure 19 Step S103). The spatial prediction motion vector candidate derivation unit 321, the temporal prediction motion vector candidate derivation unit 322, the historical prediction motion vector candidate derivation unit 323, and the prediction motion vector candidate supplementation unit 325 in the normal prediction motion vector mode derivation unit 301 derive multiple prediction motion vector candidates to construct a prediction motion vector candidate list mvpListLX. Figure 19 The detailed processing steps of step S103 are as follows: Figure 20 The flowchart is described later.

[0165] Next, the motion vector predictor candidate selection unit 327 selects the motion vector predictor mvpLX of LX from the motion vector predictor candidate list mvpListLX of LX ( Figure 19 Step S104). Here, in the predicted motion vector candidate list mvpListLX, a certain element (the i-th element counted from 0) is represented as mvpListLX[i]. Each differential motion vector is calculated as the difference between the motion vector mvLX and the candidate mvpListLX[i] of each predicted motion vector stored in the predicted motion vector candidate list mvpListLX. For each element (predicted motion vector candidate) of the predicted motion vector candidate list mvpListLX, the coding amount when encoding these differential motion vectors is calculated. Then, among the elements registered in the predicted motion vector candidate list mvpListLX, the candidate mvpListLX[i] of the predicted motion vector with the smallest coding amount for each candidate of the predicted motion vector is selected as the predicted motion vector mvpLX, and the index i is obtained. When there are multiple candidates for the predicted motion vector that generates the smallest amount of code in the predicted motion vector candidate list mvpListLX, the predicted motion vector candidate mvpListLX[i] represented by the number with the smallest index i in the predicted motion vector candidate list mvpListLX is selected as the optimal predicted motion vector mvpLX, and the index i is obtained.

[0166] Next, the motion vector subtraction unit 328 subtracts the selected predicted motion vector mvpLX of LX from the motion vector mvLX of LX, setting mvdLX=mvLX-mvpLX, and calculates the differential motion vector mvdLX of LX ( Figure 19 Step S105).

[0167] <Normal Motion Vector Prediction Mode Derivation Unit (Normal AMVP): Description on the Decoding Side>

[0168] Next, refer to Figure 25 The following describes the normal motion vector prediction mode processing steps on the decoding side. On the decoding side, the spatial motion vector prediction candidate derivation unit 421, the temporal motion vector prediction candidate derivation unit 422, the historical motion vector prediction candidate derivation unit 423, and the motion vector prediction candidate supplementation unit 425 calculate the motion vector ( L0 , L1 ) used for inter-frame prediction in the normal motion vector prediction mode for each of L0 and L1. Figure 25 Specifically, when the prediction mode PredMode of the processing target block is inter prediction (MODE_INTER) and the inter prediction mode of the processing target block is L0 prediction (Pred_L0), a prediction motion vector candidate list mvpListL0 of L0 is calculated, the prediction motion vector mvpL0 is selected, and the motion vector mvL0 of L0 is calculated. When the inter prediction mode of the processing target block is L1 prediction (Pred_L1), a prediction motion vector candidate list mvpListL1 of L1 is calculated, the prediction motion vector mvpL1 is selected, and the motion vector mvL1 of L1 is calculated. When the inter-frame prediction mode of the object block is dual prediction (Pred_BI), L0 prediction and L1 prediction are performed simultaneously, the predicted motion vector candidate list mvpListL0 of L0 is calculated, the predicted motion vector mvpL0 of L0 is selected, the motion vector mvL0 of L0 is calculated, and the predicted motion vector candidate list mvpListL1 of L1 is calculated, the predicted motion vector mvpL1 of L1 is calculated, and the motion vector mvL1 of L1 is calculated respectively.

[0169] Similar to the encoding side, the decoding side also performs motion vector calculation processing for L0 and L1 separately, but the processing is common to both L0 and L1. Therefore, in the following description, L0 and L1 are collectively referred to as LX. LX represents the inter-frame prediction mode used for inter-frame prediction of the coding block being processed. When calculating the motion vector for L0, X is 0, and when calculating the motion vector for L1, X is 1. Furthermore, when calculating the motion vector for LX, if information from a different reference list is referenced instead of the same reference list as the one being calculated, the other reference list is referred to as LY.

[0170] In the case of using the motion vector mvLX of LX ( Figure 25 Step S202: Yes), calculate the candidates for the predicted motion vector of LX, and construct the predicted motion vector candidate list mvpListLX of LX ( Figure 25 The spatial motion vector prediction candidate derivation unit 421, the temporal motion vector prediction candidate derivation unit 422, the historical motion vector prediction candidate derivation unit 423, and the motion vector prediction candidate supplementation unit 425 in the normal motion vector prediction mode derivation unit 401 calculate multiple motion vector prediction candidates and construct a motion vector prediction candidate list mvpListLX. Figure 25 The detailed processing steps of step S203 are as follows: Figure 20 The flowchart is described later.

[0171] Next, the motion vector predictor candidate selection unit 426 extracts the motion vector predictor candidate mvpListLX[mvpIdxLX] corresponding to the motion vector predictor index mvpIdxLX decoded and provided by the bitstream decoding unit 201 from the motion vector predictor candidate list mvpListLX as the selected motion vector predictor mvpLX ( Figure 25 Step S204).

[0172] Next, the motion vector addition unit 427 adds the differential motion vector mvdLX of LX decoded and provided by the bitstream decoding unit 201 and the predicted motion vector mvpLX of LX, and calculates the motion vector mvLX of LX ( mvLX = mvpLX + mvdLX ). Figure 25 Step S205).

[0173] <Normal Motion Vector Prediction Mode Derivation Unit (Normal AMVP): Motion Vector Prediction Method>

[0174] Figure 20 This is a flowchart showing the processing steps of the normal prediction motion vector mode derivation processing having common functions in the normal prediction motion vector mode derivation unit 301 of the image encoding device involved in the embodiment of the present invention and the normal prediction motion vector mode derivation unit 401 of the image decoding device.

[0175] The normal prediction motion vector mode derivation unit 301 and the normal prediction motion vector mode derivation unit 401 include a prediction motion vector candidate list mvpListLX. The prediction motion vector candidate list mvpListLX constitutes a list structure, and is provided with a storage area for storing a prediction motion vector index indicating a position within the prediction motion vector candidate list and a prediction motion vector candidate corresponding to the index as elements. The number of the prediction motion vector index starts from 0, and the prediction motion vector candidates are stored in the storage area of ​​the prediction motion vector candidate list mvpListLX. In this embodiment, it is assumed that the prediction motion vector candidate list mvpListLX can register at least two prediction motion vector candidates (inter-frame prediction information). In addition, the variable numCurrMvpCand indicating the number of prediction motion vector candidates registered in the prediction motion vector candidate list mvpListLX is set to 0.

[0176] The spatial motion vector prediction candidate derivation units 321 and 421 derive candidates for the motion vector prediction from the block adjacent to the left. In this process, the block adjacent to the left ( Figure 11 The inter-frame prediction information of A0 or A1), that is, the flag indicating whether the predicted motion vector candidate can be used, the motion vector, the reference index, etc., is used to derive the predicted motion vector mvLXA, and the derived mvLXA is added to the predicted motion vector candidate list mvpListLX ( Figure 20 In step S301 of FIG. 2 , X is 0 in the case of L0 prediction and 1 in the case of L1 prediction (the same applies below). Next, the spatial motion vector prediction candidate derivation units 321 and 421 derive candidates for the motion vector prediction from the block adjacent to the upper side. In this process, the block adjacent to the upper side ( Figure 11 The inter-frame prediction information of B0, B1 or B2), that is, the flag indicating whether the predicted motion vector candidate can be used, as well as the motion vector, reference index, etc., is used to derive the predicted motion vector mvLXB. If the respectively derived mvLXA and mvLXB are not equal, mvLXB is added to the predicted motion vector candidate list mvpListLX ( Figure 20 Step S302). Figure 20 The processing of steps S301 and S302 is the same except that the position and number of the referenced adjacent blocks are different. The flag avai lableFlagLXN indicating whether the predicted motion vector candidate of the coding block can be used, as well as the motion vector mvLXN and the reference index refIdxN (N represents A or B, the same below) are derived.

[0177] Next, the temporal motion vector predictor candidate derivation units 322 and 422 derive motion vector predictor candidates for blocks in a temporally different picture from the current processing target picture. In this process, the flag availableFlagLXCol indicating whether a motion vector predictor candidate of a coding block in a temporally different picture can be used, the motion vector mvLXCol, the reference index refIdxCol, and the reference list listCol are derived, and mvLXCol is added to the motion vector predictor candidate list mvpListLX ( Figure 20 Step S303).

[0178] Furthermore, it is assumed that the processing of the temporal motion vector predictor candidate derivers 322 and 422 in units of a sequence (SPS), a picture (PPS), or a slice can be omitted.

[0179] Next, the historical motion vector predictor candidate derivation units 323 and 423 add the historical motion vector predictor candidates registered in the historical motion vector predictor candidate list HmvpCandList to the motion vector predictor candidate list mvpListLX ( Figure 20 For details on the registration process of step S304, see Figure 29 The flowchart is described later.

[0180] Next, the motion vector predictor candidate supplementing units 325 and 425 add motion vector predictor candidates with predetermined values ​​such as (0, 0) until the motion vector predictor candidate list mvpListLX is satisfied ( Figure 20 S305).

[0181] <Normal Merge Mode Export Section (Normal Merge)>

[0182] Figure 18 The normal merge mode derivation unit 302 includes a spatial merge candidate derivation unit 341, a temporal merge candidate derivation unit 342, an average merge candidate derivation unit 344, a historical merge candidate derivation unit 345, a merge candidate supplementation unit 346, and a merge candidate selection unit 347.

[0183] Figure 24 The normal merge mode derivation unit 402 includes a spatial merge candidate derivation unit 441, a temporal merge candidate derivation unit 442, an average merge candidate derivation unit 444, a historical merge candidate derivation unit 445, a merge candidate supplementation unit 446, and a merge candidate selection unit 447.

[0184] Figure 21 This is a flowchart illustrating the steps of a normal merging mode derivation process having a common function in the normal merging mode derivation unit 302 of the image encoding device and the normal merging mode derivation unit 402 of the image decoding device according to the embodiment of the present invention.

[0185] The following describes each process in sequence. Unless otherwise specified, the following description is for a B-slice, but is also applicable to a P-slice. However, when the slice type slice_type is a P-slice, only L0 prediction (Pred_L0) exists as an inter-prediction mode, and L1 prediction (Pred_L1) and bi-prediction (Pred_BI) do not. Therefore, processing related to L1 can be omitted.

[0186] In the normal merge mode derivation unit 302 and the normal merge mode derivation unit 402, there is a merge candidate list mergeCandList. The merge candidate list mergeCandList constitutes a list structure, and is provided with a merge index indicating the position inside the merge candidate list, and a storage area for storing the merge candidate corresponding to the index as an element. The number of the merge index starts from 0, and the merge candidate is stored in the storage area of ​​the merge candidate list mergeCandList. In subsequent processing, it is assumed that the merge candidate of the merge index i registered in the merge candidate list mergeCandList is represented by mergeCandList[i]. In this embodiment, it is assumed that the merge candidate list mergeCandList can register at least six merge candidates (inter-frame prediction information). In addition, the variable numCurrMergeCand indicating the number of merge candidates registered in the merge candidate list mergeCandList is set to 0.

[0187] In the spatial merge candidate derivation unit 341 and the spatial merge candidate derivation unit 441, based on the coding information stored in the coding information storage memory 111 of the image coding device or the coding information storage memory 205 of the image decoding device, spatial merge candidates A and B are derived from the blocks adjacent to the left and above the processing target block, and the derived spatial merge candidates are registered in the merge candidate list mergeCandList ( Figure 21 Step S401). Here, N is defined as any one of the spatial merge candidates A, B, or temporal merge candidate Col. The availableFlagN flag indicating whether the inter-frame prediction information of block N can be used as a spatial merge candidate, the reference index refIdxL0N of L0 and the reference index refIdxL1N of L1 of the spatial merge candidate N, the L0 prediction flag predFlagL0N indicating whether L0 prediction is performed, the L1 prediction flag predFlagL1N indicating whether L1 prediction is performed, the motion vector mvL0N of L0, and the motion vector mvL1N of L1 are derived. However, in this embodiment, since the merge candidate is derived without reference to the inter-frame prediction information of the blocks included in the coding block to be processed, the spatial merge candidate using the inter-frame prediction information of the blocks included in the coding block to be processed is not derived.

[0188] Next, in the temporal merge candidate derivation unit 342 and the temporal merge candidate derivation unit 442, temporal merge candidates from pictures at different times are derived, and the derived temporal merge candidates are registered in the merge candidate list mergeCandList ( Figure 21 Step S402). A flag availableFlagCol indicating whether the temporal merging candidate can be used, an L0 prediction flag predFlagL0Col indicating whether L0 prediction of the temporal merging candidate is performed, an L1 prediction flag predFlagL1Col indicating whether L1 prediction is performed, and L0 motion vector mvL0Col and L1 motion vector mvL1Col are derived.

[0189] Furthermore, the processing of the temporal merging candidate derivation unit 342 and the temporal merging candidate derivation unit 442 in units of a sequence (SPS), a picture (PPS), or a slice can be omitted.

[0190] Next, in the historical merge candidate derivation unit 345 and the historical merge candidate derivation unit 445, the historical motion vector predictor candidates registered in the historical motion vector predictor candidate list HmvpCandList are registered in the merge candidate list mergeCandList ( Figure 21 Step S403).

[0191] In addition, when the number of merge candidates numCurrMergeCand registered in the merge candidate list mergeCandList is less than the maximum number of merge candidates MaxNumMergeCand, the number of merge candidates numCurrMergeCand registered in the merge candidate list mergeCandList is used to derive historical merge candidates with the maximum number of merge candidates MaxNumMergeCand as the upper limit, and is registered in the merge candidate list mergeCandList.

[0192] Next, in the average merge candidate derivation unit 344 and the average merge candidate derivation unit 444, the average merge candidate is derived from the merge candidate list mergeCandList, and the derived average merge candidate is added to the merge candidate list mergeCandList ( Figure 21 Step S404).

[0193] In addition, when the number of merge candidates numCurrMergeCand registered in the merge candidate list mergeCandList is less than the maximum number of merge candidates MaxNumMergeCand, the number of merge candidates numCurrMergeCand registered in the merge candidate list mergeCandList is upper bounded by the maximum number of merge candidates MaxNumMergeCand, and the average merge candidate is derived and registered in the merge candidate list mergeCandList.

[0194] Here, the average merge candidate is a new merge candidate having a motion vector obtained by averaging the motion vectors of the first merge candidate and the second merge candidate registered in the merge candidate list mergeCandList for each of the L0 prediction and the L1 prediction.

[0195] Next, in the merge candidate supplement unit 346 and the merge candidate supplement unit 446, when the number of merge candidates numCurrMergeCand registered in the merge candidate list mergeCandList is less than the maximum number of merge candidates MaxNumMergeCand, the number of merge candidates numCurrMergeCand registered in the merge candidate list mergeCandList is derived with the maximum number of merge candidates MaxNumMergeCand as the upper limit to add merge candidates, and register them in the merge candidate list mergeCandList ( Figure 21 Step S405). With the maximum number of merge candidates MaxNumMergeCand as the upper limit, a merge candidate with a motion vector value of (0, 0) and a prediction mode of L0 prediction (Pred_L0) is added to the P slice. A merge candidate with a motion vector value of (0, 0) and a prediction mode of bi-prediction (Pred_BI) is added to the B slice. The reference index when adding the merge candidate is different from the reference index that has already been added.

[0196] Next, the merge candidate selection units 347 and 447 select a merge candidate from the merge candidates registered in the merge candidate list mergeCandList. The merge candidate selection unit 347 on the encoding side selects a merge candidate by calculating the code size and distortion amount, and provides a merge index indicating the selected merge candidate and inter-frame prediction information of the merge candidate to the motion compensation prediction unit 306 via the inter-frame prediction mode determination unit 305. On the other hand, the merge candidate selection unit 447 on the decoding side selects a merge candidate based on the decoded merge index and provides the selected merge candidate to the motion compensation prediction unit 406.

[0197] In the normal merge mode derivation unit 302 and the normal merge mode derivation unit 402, when the size (product of width and height) of a coding block is less than 32, merge candidates are derived from the parent block of the coding block. The merge candidates derived from the parent block are then used in all child blocks. However, this is limited to cases where the size of the parent block is greater than 32 and fits within the frame.

[0198] <Update Historical Motion Vector Prediction Candidate List>

[0199] Next, a method for initializing and updating the historical motion vector predictor candidate list HmvpCandList included in the encoding information storage memory 111 on the encoding side and the encoding information storage memory 205 on the decoding side will be described in detail. Figure 26 This is a flowchart used to illustrate the steps of initializing / updating the historical motion vector prediction candidate list.

[0200] In this embodiment, it is assumed that the updating of the historical motion vector predictor candidate list HmvpCandList is performed in the coding information storage memory 111 and the coding information storage memory 205. Alternatively, a historical motion vector predictor candidate list updating unit may be provided in the inter-frame prediction unit 102 and the inter-frame prediction unit 203 to update the historical motion vector predictor candidate list HmvpCandList.

[0201] The historical prediction motion vector candidate list HmvpCandList is initially set at the beginning of the slice. On the encoding side, when the normal prediction motion vector mode or the normal merge mode is selected by the prediction method determination unit 105, the historical prediction motion vector candidate list HmvpCandList is updated. On the decoding side, when the prediction information decoded by the bit string decoding unit 201 is the normal prediction motion vector mode or the normal merge mode, the historical prediction motion vector candidate list HmvpCandList is updated.

[0202] Inter-frame prediction information used when performing inter-frame prediction in normal motion vector prediction mode or normal merge mode is registered as inter-frame prediction information candidate hMvpCand in the historical motion vector prediction candidate list hmvpCandList. The inter-frame prediction information candidate hMvpCand includes the reference index refIdxL0 of L0 and the reference index refIdxL1 of L1, an L0 prediction flag predFlagL0 indicating whether to perform L0 prediction, an L1 prediction flag predFlagL1 indicating whether to perform L1 prediction, and the motion vector mvL0 of L0 and the motion vector mvL1 of L1.

[0203] If there is inter-frame prediction information having the same value as the inter-frame prediction information candidate hMvpCand among the elements (i.e., inter-frame prediction information) registered in the historical prediction motion vector candidate list HmvpCandList stored in the encoding information storage memory 111 on the encoding side and the encoding information storage memory 205 on the decoding side, the element is deleted from the historical prediction motion vector candidate list HmvpCandList. On the other hand, if there is no inter-frame prediction information having the same value as the inter-frame prediction information candidate hMvpCand, the leading element of the historical prediction motion vector candidate list HmvpCandList is deleted and the inter-frame prediction information candidate hMvpCand is added to the end of the historical prediction motion vector candidate list HmvpCandList.

[0204] The number of elements of the historical motion vector predictor candidate list HmvpCandList included in the encoding information storage memory 111 on the encoding side and the encoding information storage memory 205 on the decoding side of the present invention is set to six.

[0205] First, the historical motion vector prediction candidate list HmvpCandList ( Figure 26 At the beginning of the slice, historical motion vector prediction candidates are added to all elements of the historical motion vector prediction candidate list HmvpCandList, and the value of the historical motion vector prediction candidate number NumHmvpCand registered in the historical motion vector prediction candidate list HmvpCandList is set to 6. Alternatively, all elements of the historical motion vector prediction candidate list HmvpCandList may be cleared at the beginning of the slice, and the value of the historical motion vector prediction candidate number (current candidate number) NumHmvpCand registered in the historical motion vector prediction candidate list HmvpCandList may be set to 0.

[0206] Furthermore, although the initialization of the historical motion vector predictor candidate list HmvpCandList is assumed to be performed in slice units (the first coding block of the slice), it may also be performed in picture units, tile units, or treeblock row units.

[0207] Figure 38 This table shows an example of historical motion vector prediction candidates added by initializing the historical motion vector prediction candidate list HmvpCandList. This example shows the case where the slice type is a B slice and the number of reference pictures is 4. The historical motion vector prediction index hMvpIdx ranges from (the number of historical motion vector prediction candidates NumHmvpCand-1) to 0, and inter-frame prediction information with the value (0, 0) of the motion vector mvLXHmvp (X is 0 or 1) is added as a historical motion vector prediction candidate to the historical motion vector prediction candidate list HmvpCandList, filling the historical motion vector prediction candidate list with historical candidates. At this time, the historical motion vector prediction index hMvpIdx starts from (the number of historical motion vector prediction candidates NumHmvpCand-1), and the reference index refIdxLX (X is 0 or 1) is set to a value incremented by 1 from 0 to (the number of reference pictures numRefIdx-1). Thereafter, duplication between historical motion vector predictor candidates is permitted, and the reference index refIdxLX is set to 0. By setting all values ​​for the number of historical motion vector predictor candidates NumHmvpCand, the value of the number of historical motion vector predictor candidates NumHmvpCand is fixed, thereby eliminating invalid historical motion vector predictor candidates. In this way, by assigning a small reference index refIdxLX value with a high general selection rate to candidates with a high probability of being added to the motion vector predictor candidate list or the merge candidate list and having a large historical motion vector predictor index hMvpIdx, encoding efficiency can be improved.

[0208] In addition, by filling the historical prediction motion vector candidate list with historical prediction motion vector candidates in units of slices, the number of historical prediction motion vector candidates can be treated as a fixed value, thereby simplifying, for example, the historical prediction motion vector candidate derivation process and the historical merge candidate derivation process.

[0209] Here, although the motion vector value is generally assumed to be (0, 0), which has a high probability of being selected, other values ​​may be used as long as they are predetermined values. For example, values ​​such as (4, 4), (0, 32), and (-128, 0) may be used to improve the encoding efficiency of the differential motion vector, or multiple predetermined values ​​may be set to improve the encoding efficiency of the differential motion vector.

[0210] In addition, although the historical prediction motion vector index hMvpIdx starts from (the number of historical prediction motion vector candidates NumHmvpCand-1) and the reference index refIdxLX (X is 0 or 1) is set to a value incremented by 1 from 0 to (the number of reference pictures numRefIdx-1), the historical prediction motion vector index hMvpIdx can start from 0.

[0211] Figure 39 This table shows another example of historical motion vector prediction candidates added by initializing the historical motion vector prediction candidate list HmvpCandList. This example shows the case where the slice type is B slice and the number of reference pictures is 2. In this example, inter-frame prediction information with different values ​​for either the reference index refIdxLX (X is 0 or 1) or the motion vector mvLXHmvp (X is 0 or 1) is added as a historical motion vector prediction candidate, so that there is no duplication between historical motion vector prediction candidates in each element of the historical motion vector prediction candidate list HmvpCandList, and the historical motion vector prediction candidate list is populated. At this time, the historical motion vector prediction index hMvpIdx starts at (the number of historical motion vector prediction candidates NumHmvpCand-1), and the reference index refIdxLX (X is 0 or 1) is set to values ​​that increase by 1 from 0 to (the number of reference pictures numRefIdx-1). Then, when the reference index refIdxLX is 0, motion vectors mvLXHmvp with different values ​​are added as historical motion vector prediction candidates. By setting all values ​​for the number of historical motion vector prediction candidates NumHmvpCand, the value of the number of historical motion vector prediction candidates NumHmvpCand is set to a fixed value, thereby eliminating invalid historical motion vector prediction candidates.

[0212] In this way, by filling the historical prediction motion vector candidate list with non-repeated historical prediction motion vector candidates in slice units, the processing of the merge candidate supplement unit 346 that is later than the historical merge candidate derivation unit 345 in the normal merge mode derivation unit 302 described later, which is implemented in coding block units, can be omitted, thereby reducing the processing amount.

[0213] Here, although the absolute value of each component of the motion vector mvLXHmvp is set to 0 or 1, the absolute value of each component of the motion vector mvLXHmvp may be set to be greater than 1 as long as there is no duplication between historical prediction motion vector candidates.

[0214] In addition, although the historical prediction motion vector index hmvpIdx starts from (the number of historical prediction motion vector candidates NumHmvpCand-1) and the reference index refIdxLX (X is 0 or 1) is set to a value incremented by 1 from 0 to (the number of reference pictures numRefIdx-1), the historical prediction motion vector index hMvpIdx can also start from 0.

[0215] Figure 40 This is a table showing another example of historical motion vector predictor candidates added by initialization of the historical motion vector predictor candidate list HmvpCandList.

[0216] An example is shown for a case where the slice type is B slice and the number of reference pictures is 1. In this example, inter-frame prediction information with a reference index refIdxLX of 0 and different motion vector mvLXHmvp values ​​is added as a historical motion vector prediction candidate so that there is no duplication between the historical motion vector prediction candidates in each element of the historical motion vector prediction candidate list HmvpCandList, and the historical motion vector prediction candidate list is filled. In this case, the historical motion vector prediction index hMvpIdx starts at (the number of historical motion vector prediction candidates NumHmvp_and-1), and inter-frame prediction information with a reference index refIdxLX (X is 0 or 1) of 0 and different motion vector mvLXHmvp values ​​is set. By setting all values ​​for the number of historical motion vector prediction candidates NumHmvpCand, the value of the number of historical motion vector prediction candidates NumHmvpCand is set to a fixed value, thereby eliminating invalid historical motion vector prediction candidates.

[0217] In this manner, by setting the reference index refIdxLX to 0, the reference picture can be initialized regardless of the number of reference pictures, so that the process can be simplified.

[0218] Here, although the absolute value of each component of the motion vector mvLXHmvp is set to a power of 0, 1, or 2. However, if the reference index is 0 and there is no repetition between historical prediction motion vector candidates, the absolute value of each component of the motion vector mvLXHmvp may be a power of a value other than 2, such as 3, 4, or may be any other arbitrary value.

[0219] In addition, although the historical prediction motion vector index hMvpIdx starts from (the number of historical prediction motion vector candidates NumHmvpCand-1) and the inter-frame prediction information in which the reference index refIdxLX (X is 0 or 1) is 0 and the value of the motion vector mvLXHmvp is different is set, the historical prediction motion vector index hMvpIdx can also start from 0.

[0220] Next, the following updating process of the historical motion vector prediction candidate list HmvpCandList is repeated for each coding block in the slice ( Figure 26 Steps S2102 to S2111).

[0221] First, the initial setting is performed in units of coding blocks. The flag identicalCandExist indicating whether there is an identical candidate is set to FALSE (false), and the deletion object index removeIdx indicating the candidate for deletion is set to "0" ( Figure 26 Step S2103).

[0222] Determine whether there is an inter-frame prediction information candidate hMvpCand ( Figure 26 Step S2104). In the case where the prediction method determination unit 105 on the encoding side determines that it is the normal prediction motion vector mode or the normal merge mode, or in the case where the bit string decoding unit 201 on the decoding side decodes it as the normal prediction motion vector mode or the normal merge mode, the inter-frame prediction information is set as the inter-frame prediction information candidate hMvpCand of the registration object. In the case where the prediction method determination unit 105 on the encoding side determines that it is the intra-frame prediction mode, the sub-block prediction motion vector mode or the sub-block merge mode, or in the case where the bit string decoding unit 201 on the decoding side decodes it as the intra-frame prediction mode, the sub-block prediction motion vector mode or the sub-block merge mode, the updating process of the historical prediction motion vector candidate list HmvpCandList is not performed, and there is no inter-frame prediction information candidate hMvpCand of the registration object. In the case where there is no inter-frame prediction information candidate hMvpCand of the registration object, steps S2105 to S2106 are skipped ( Figure 26 Step S2104: No). If there is an inter-frame prediction information candidate hMvpCand to be registered, the processing after step S2105 is performed ( Figure 26 Step S2104: Yes).

[0223] Next, it is determined whether there is an element (inter-frame prediction information) with the same value as the inter-frame prediction information candidate hMvpCand of the registration object in each element of the historical prediction motion vector candidate list HmvpCandList, that is, whether there is an identical element ( Figure 26 Step S2105). Figure 27 This is a flowchart of the same element confirmation processing steps. When the value of the number of historical motion vector prediction candidates NumHmvpCand is 0 ( Figure 27 Step S2121: No), the historical motion vector prediction candidate list HmvpCandList is empty, and since there is no identical candidate, the process is skipped. Figure 27 Steps S2122 to S2125 are performed to end the same element confirmation processing step. When the value of the number of historical motion vector prediction candidates NumHmvpCand is greater than 0 ( Figure 27 Step S2121: Yes), the historical predicted motion vector index hMvpIdx is from 0 to NumHmvpCand-1, and the process of step S2123 is repeated ( Figure 27 First, compare whether the hMvpIdxth element HmvpCandList[hMvpIdx] of the historical motion vector prediction candidate list, counted from 0, is the same as the inter-frame prediction information candidate hMvpCand ( Figure 27 Step S2123). In the same case ( Figure 27 Step S2123: Yes), set the flag identicalCandExist indicating whether there is an identical candidate to TRUE (true), set the deletion object index removeIdx indicating the position of the element to be deleted to the value of the current historical predicted motion vector index hMvpIdx, and end the identical element confirmation process. In the case of non-identical ( Figure 27 Step S2123: No), increase hMvpIdx by 1. If the historical predicted motion vector index hMvpIdx is less than NumHmvpCand-1, perform the processing after step S2123.

[0224] Return again Figure 26 Flowchart of the present invention, shifting and adding elements of the historical motion vector prediction candidate list HmvpCandList ( Figure 26 Step S2106). Figure 28 yes Figure 26 Flowchart of the element shifting / adding processing steps of the historical motion vector prediction candidate list HmvpCandList in step S2106. First, it is determined whether to add a new element after removing the element stored in the historical motion vector prediction candidate list HmvpCandList or to add a new element without removing the element. Specifically, it is compared whether the flag identicalCandExist indicating whether there is an identical candidate is TRUE or whether NumHmvvpCand is 6 ( Figure 28 Step S2141). If either the flag identicalCandExist indicating whether there is an identical candidate is TRUE or the current number of candidates NumHmvpCand is 6 is satisfied ( Figure 28 (Step S2141: Yes), after removing the elements stored in the historical motion vector prediction candidate list HmvpCandList, add new elements. Set the initial value of index i to the value of removeIdx+1. Repeat the element shifting process of step S2143 from this initial value to NumHmvpCand. Figure 28 By copying the elements of HmvpCandList[i] to HmvpCandList[i-1], the elements are shifted forward ( Figure 28 Step S2143), increase i by 1 ( Figure 28 Next, the inter-frame prediction information candidate hMvpCand is added to the (NumHmvpCand-1)th HmvpCandList[NumHmvpCand-1] ( Figure 28 On the other hand, if the flag identicalCandExist indicating whether there is an identical candidate is TRUE (true) and NumHmvpCand is 6, neither of the conditions is satisfied ( Figure 28 In step S2141: No), the elements stored in the historical motion vector prediction candidate list HmvpCandList are not removed, but the inter prediction information candidate hMvpCand is added to the end of the historical motion vector prediction candidate list ( Figure 28 Here, the end of the historical motion vector prediction candidate list is the NumHmvpCandth HmvpCandList[NumHmvpCand], counting from 0. In addition, NumHmvpCand is incremented by 1, and the shifting and adding processing of the elements of the historical motion vector prediction candidate list HmvpCandList is completed.

[0225] Figures 31A to 31C This is a diagram for explaining an example of updating the historical motion vector prediction candidate list. When a new element is added to the historical motion vector prediction candidate list HmvpCandList in which six elements (inter-frame prediction information) are registered, the new inter-frame prediction information is compared with the element at the front of the historical motion vector prediction candidate list HmvpCandList. Figure 31A ), if the new element has the same value as the third element HMVP2 from the beginning of the historical motion vector prediction candidate list HmvpCandList, then delete the element HMVP2 from the historical motion vector prediction candidate list HmvpCandList, and shift (copy) the following elements HMVP3 to HMVP5 one by one to the front, and add the new element to the end of the historical motion vector prediction candidate list HmvpCandList ( Figure 31B ), complete the update of the historical prediction motion vector candidate list HmvpCandList ( Figure 31C ).

[0226] <Historical Motion Vector Prediction Candidate Derivation Process>

[0227] Next, the details of Figure 20 The method of deriving the historical motion vector prediction candidate from the historical motion vector prediction candidate list HmvpCandList in the processing step of step S304, Figure 20 The processing steps of step S304 are common processing in the historical prediction motion vector candidate derivation unit 323 of the normal prediction motion vector mode derivation unit 301 on the encoding side and the historical prediction motion vector candidate derivation unit 423 of the normal prediction motion vector mode derivation unit 401 on the decoding side. Figure 29 This is a flowchart illustrating the steps of deriving historical motion vector prediction candidates.

[0228] When the current number of predicted motion vector candidates numCurrMvpCand is greater than the maximum number of elements in the predicted motion vector candidate list mvpListLX (here 2) or the value of the historical number of predicted motion vector candidates NumHmvpCand is 0 ( Figure 29 Step S2201: No), omitted Figure 29 The processing from step S2202 to step S2209 ends the historical motion vector prediction candidate derivation processing step. When the current number of motion vector prediction candidates numCurrMvpCand is less than the maximum number of elements 2 in the motion vector prediction candidate list mvpListLX and the value of the historical motion vector prediction candidate number NumHmvpCand is greater than 0 ( Figure 29 Step S2201: Yes), execute Figure 29 Processing of steps S2202 to S2209.

[0229] Next, repeat Figure 29 The processing of steps S2203 to S2208 is repeated until the index i is smaller than the value of 1 to 4 and the number of historical prediction motion vector candidates numCheckedHMVPCand ( Figure 29 Steps S2202 to S2209). If the current number of motion vector predictor candidates numCurrMvpCand is greater than or equal to the maximum number of elements in the motion vector predictor candidate list mvpListLX, which is 2 ( Figure 29 Step S2203: No), omitted Figure 29 The processing of steps S2204 to S2209 ends the historical motion vector predictor candidate derivation processing step. Figure 29 In the case of step S2203: Yes), execute Figure 29 Processing after step S2204.

[0230] Next, the processing from step S2205 to step S2207 is performed for Y = 0 and 1 (L0 and L1) respectively ( Figure 29 Steps S2204 to S2208). If the current number of motion vector predictor candidates numCurrMvpCand is greater than or equal to the maximum number of elements in the motion vector predictor candidate list mvpListLX, which is 2 ( Figure 29 Step S2205: No), omitted Figure 29 The processing of steps S2206 to S2209 ends the historical motion vector predictor candidate derivation processing step. When the current number of motion vector predictor candidates numCurrMvpCand is less than the maximum number of elements 2 in the motion vector predictor candidate list mvpListLX ( Figure 29 Step S2205: Yes), execute Figure 29 Processing after step S2206 in .

[0231] Next, in the historical motion vector prediction candidate list HmvpCandList, if the element has the same reference index as the reference index refIdxLX of the encoding / decoding target motion vector and is different from any element in the motion vector prediction list mvpListLX ( Figure 29 Step S2206: Yes), add the motion vector of LY of the historical motion vector prediction candidate HmvpCandList[NumHmvpCand-i] to the numCurrMvpCand-th element mvpListLX[numCurrMvpCand] of the motion vector prediction candidate list ( Figure 29 Step S2207) and the current number of predicted motion vector candidates numCurrMvpCand is increased by 1. When there is no element with the same reference index as the reference index refIdxLX of the encoding / decoding target motion vector in the historical predicted motion vector candidate list HmvpCandList and no element different from any element in the predicted motion vector list mvpListLX ( Figure 29 : No in step S2206), skip the adding process of step S2207.

[0232] In both L0 and L1, the above Figure 29 The processing of steps S2205 to S2207 ( Figure 29 Steps S2204 to S2208). Increase the index i by 1. If the index i is less than or equal to 4 or the number of historical motion vector prediction candidates NumHmvpCand, whichever is smaller, perform the process after step S2203 again ( Figure 29 Steps S2202 to S2209).

[0233] <Historical Merge Candidate Export Processing>

[0234] Next, the details of Figure 21 The method of deriving historical merge candidates from the historical merge candidate list HmvpCandList in the processing step of step S404, Figure 21 The processing steps of step S404 are common processing in the historical merge candidate derivation unit 345 of the normal merge mode derivation unit 302 on the encoding side and the historical merge candidate derivation unit 445 of the normal merge mode derivation unit 402 on the decoding side. Figure 30 This is a flowchart illustrating the steps of the historical merge candidate derivation process.

[0235] First, perform initialization processing ( Figure 30 Step S2301). A value of FALSE is set for each of the elements from 0 to the (numCurrMergeCand-1)th element of isPruned[i], and the number of elements numCurrMergeCand registered in the current merge candidate list is set for the variable numOrigMergeCand.

[0236] Next, the initial value of the index hMvpIdx is set to 1, and the process from the initial value to NumHmvpCand is repeated. Figure 30 Addition processing from step S2303 to step S2310 ( Figure 30 If the number of elements numCurrMergeCand registered in the current merge candidate list is not less than (the maximum number of merge candidates MaxNumMergeCand-1), the merge candidate is added to all elements in the merge candidate list, and thus the historical merge candidate derivation process is terminated ( Figure 30 Step S2303: No). If the number of elements numCurrMergeCand registered in the current merge candidate list is less than (the maximum number of merge candidates MaxNumMergeCand-1), the processing after step S2304 is executed. Set the value of FALSE (false) to sameMotion ( Figure 30 Then, the initial value of index i is set to 0, and the index is merged from the initial value to numOrigMergeCand-1. Figure 30 The processing of steps S2306 and S2307 ( Figure 30 Compare the (NumHmvpCand–hMvpIdx)th element HmvpCandList[NumHmvpCand-hMvpIdx] of the historical motion vector prediction candidate list from 0 to see if it is the same value as the i-th element mergeCandList[i] of the merge candidate list from 0 ( Figure 30 Step S2306).

[0237] The same value of the merge candidate means that the merge candidate has the same value when the values ​​of all the components (inter prediction mode, reference index and motion vector) of the merge candidate are the same. In the case where the merge candidate has the same value and isPruned[i] is FALSE (false) ( Figure 30 Step S2306: Yes), sameMotion and isPruned[i] are both set to TRUE (true) ( Figure 30 Step S2307). In the case of not the same value ( Figure 30 Step S2306: No), skip the processing of step S2307. Figure 30 After the repetitive processing of steps S2305 to S2308 is completed, compare whether sameMotion is FALSE (false) ( Figure 30 Step S2309), if sameMotion is FALSE (false) ( Figure 30 Step S2309: Yes), that is, since the (NumHmvpCand-hMvpIdx)th element HmvpCandList[NumHvpCand-hMvpIdx] counting from 0 of the historical prediction motion vector candidate list does not exist in mergeCandList, the (NumHmvpCand-hMvpIdx)th element HmvpCandList[NumHvpCand-hMvpIdx] counting from 0 of the historical prediction motion vector candidate list is added to the mergeCandList[numCurrMergeCand] of the merge candidate list, and numCurrMergeCand is increased by 1 ( Figure 30 Step S2310). Increase the index hMvpIdx by 1 ( Figure 30 Step S2302), perform Figure 30 Repeat the processing of steps S2302 to S2311.

[0238] After all elements in the historical motion vector prediction candidate list have been confirmed, or merge candidates have been added to all elements in the merge candidate list, the derivation process of the historical merge candidate is completed.

[0239] <Motion Compensated Prediction Processing>

[0240] The motion-compensated prediction unit 306 obtains the position and size of the block currently being predicted during encoding. Furthermore, the motion-compensated prediction unit 306 obtains inter-frame prediction information from the inter-frame prediction mode determination unit 305. Based on the obtained inter-frame prediction information, the motion-compensated prediction unit 306 derives a reference index and motion vector. After obtaining an image signal in which the reference image identified by the reference index in the decoded image memory 104 is shifted from the same position as the image signal of the prediction block by the amount of the motion vector, the prediction signal is generated.

[0241] When the inter-frame prediction mode is prediction from a single reference picture, such as L0 prediction or L1 prediction, the prediction signal obtained from the single reference picture is used as the motion-compensated prediction signal. When the inter-frame prediction mode is prediction from two reference pictures, such as BI prediction, the motion-compensated prediction signal is obtained by weightedly averaging the prediction signals obtained from the two reference pictures. The motion-compensated prediction signal is then provided to the prediction method determination unit 105. Here, the weighted averaging ratio for bi-prediction is set to 1:1, but other weighted averaging ratios may be used. For example, the closer the picture interval between the target picture and the reference picture, the larger the weighting ratio. Alternatively, the weighting ratio may be calculated using a table that corresponds to combinations of picture intervals and weighting ratios.

[0242] The motion-compensated prediction unit 406 has the same function as the motion-compensated prediction unit 306 on the encoding side. The motion-compensated prediction unit 406 obtains inter-frame prediction information from the normal prediction motion vector mode derivation unit 401, the normal merge mode derivation unit 402, the sub-block prediction motion vector mode derivation unit 403, and the sub-block merge mode derivation unit 404 via the switch 408. The motion-compensated prediction unit 406 supplies the obtained motion-compensated prediction signal to the decoded image signal superposition unit 207.

[0243] <About Inter Prediction Mode>

[0244] The process of performing prediction based on a single reference picture is defined as uni-prediction. In the case of uni-prediction, prediction using one of the two reference pictures registered in the reference lists L0 or L1, such as L0 prediction or L1 prediction, is performed.

[0245] Figure 32 The case where the L0 reference picture (RefL0Pic) in uni-prediction is located before the processing target picture (CurPic) is shown. Figure 33 The figure shows the case where the reference picture of L0 prediction in single prediction is at the time after the processing target picture. Figure 32 and Figure 33 The reference picture of L0 prediction is replaced with the reference picture of L1 prediction (RefL1Pic) for single prediction.

[0246] The process of performing prediction based on two reference pictures is defined as bi-prediction. In the case of bi-prediction, both L0 prediction and L1 prediction are used, which is expressed as BI prediction. Figure 34 The diagram shows a case where the reference picture for the L0 prediction in the bi-prediction is at a time before the current picture, and the reference picture for the L1 prediction is at a time after the current picture. Figure 35 The case where the reference picture of the L0 prediction and the reference picture of the L1 prediction in the bi-prediction are at a time point before the processing target picture is shown. Figure 36 The case where the reference picture of the L0 prediction and the reference picture of the L1 prediction in the bi-prediction are at a time point after the current picture is shown.

[0247] In this way, the relationship between the L0 / L1 prediction type and time can be used when L0 is not limited to the past direction and L1 is not limited to the future direction. In addition, in the case of bi-prediction, both L0 prediction and L1 prediction can be performed using the same reference picture. In addition, based on information (e.g., a flag) indicating whether L0 prediction and L1 prediction are used, it is determined whether motion-compensated prediction is performed using uni-prediction or bi-prediction.

[0248] <About Reference Index>

[0249] In an embodiment of the present invention, to improve the accuracy of motion-compensated prediction, the best reference picture can be selected from multiple reference pictures during motion-compensated prediction. Therefore, the reference picture used in motion-compensated prediction is used as a reference index, and the reference index is encoded into the bitstream along with the differential motion vector.

[0250] <Motion Compensation Processing Based on Normal Motion Vector Prediction Mode>

[0251] As in Figure 16 As shown in the inter-frame prediction unit 102 on the encoding side, when the inter-frame prediction information based on the normal prediction motion vector mode derivation unit 301 is selected in the inter-frame prediction mode determination unit 305, the motion compensation prediction unit 306 obtains the inter-frame prediction information from the inter-frame prediction mode determination unit 305, derives the inter-frame prediction mode, reference index, and motion vector of the block currently being processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the prediction method determination unit 105.

[0252] Likewise, as in Figure 22 As also shown in the inter-frame prediction unit 203 on the decoding side, when switch 408 is connected to the normal motion vector prediction mode derivation unit 401 during the decoding process, the motion compensation prediction unit 406 obtains inter-frame prediction information based on the normal motion vector prediction mode derivation unit 401, derives the inter-frame prediction mode, reference index, and motion vector of the block currently being processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the decoded image signal superposition unit 207.

[0253] <Motion Compensation Processing in Normal Merge Mode>

[0254] As in Figure 16 As also shown in the inter-frame prediction unit 102 on the encoding side, when the inter-frame prediction information based on the normal merge mode derivation unit 302 is selected in the inter-frame prediction mode determination unit 305, the motion compensation prediction unit 306 obtains the inter-frame prediction information from the inter-frame prediction mode determination unit 305, derives the inter-frame prediction mode, reference index, and motion vector of the block currently being processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the prediction method determination unit 105.

[0255] Likewise, as in Figure 22 As also shown in the inter-frame prediction unit 203 on the decoding side, when switch 408 is connected to the normal merge mode derivation unit 402 during the decoding process, the motion compensation prediction unit 406 obtains the inter-frame prediction information based on the normal merge mode derivation unit 402, derives the inter-frame prediction mode, reference index, and motion vector of the block currently being processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the decoded image signal superposition unit 207.

[0256] <Motion Compensation Processing Based on Sub-Block Prediction Motion Vector Mode>

[0257] As in Figure 16 As also shown in the inter-frame prediction unit 102 on the encoding side, when the inter-frame prediction information based on the sub-block prediction motion vector mode derivation unit 303 is selected in the inter-frame prediction mode determination unit 305, the motion compensation prediction unit 306 receives the inter-frame prediction information from the inter-frame prediction mode determination unit 305, derives the inter-frame prediction mode, reference index, and motion vector of the block currently being processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the prediction method determination unit 105.

[0258] Likewise, as in Figure 22 As also shown in the inter-frame prediction unit 203 on the decoding side, when switch 408 is connected to the sub-block motion vector prediction mode derivation unit 403 during the decoding process, the motion compensation prediction unit 406 obtains inter-frame prediction information based on the sub-block motion vector prediction mode derivation unit 403, derives the inter-frame prediction mode, reference index, and motion vector of the block currently being processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the decoded image signal superposition unit 207.

[0259] <Motion compensation processing based on sub-block merging mode>

[0260] As in Figure 16 As also shown in the inter-frame prediction unit 102 on the encoding side, when the inter-frame prediction information based on the sub-block merging mode derivation unit 304 is selected by the inter-frame prediction mode determination unit 305, the motion compensation prediction unit 306 obtains the inter-frame prediction information from the inter-frame prediction mode determination unit 305, derives the inter-frame prediction mode, reference index, and motion vector of the block currently being processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the prediction method determination unit 105.

[0261] Likewise, as in Figure 22 As also shown in the inter-frame prediction unit 203 on the decoding side, when switch 408 is connected to the sub-block merging mode derivation unit 404 during the decoding process, the motion compensation prediction unit 406 obtains the inter-frame prediction information based on the sub-block merging mode derivation unit 404, derives the inter-frame prediction mode, reference index, and motion vector of the block currently being processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the decoded image signal superposition unit 207.

[0262] <Motion Compensation Processing Based on Affine Transformation Prediction>

[0263] In normal motion vector prediction mode and normal merge mode, affine model-based motion compensation can be used based on the following flag. The following flag is reflected in the following flag based on the inter-frame prediction conditions determined by the inter-frame prediction mode determination unit 305 during encoding and encoded into the bitstream. During decoding, whether to perform affine model-based motion compensation is determined based on the following flag in the bitstream.

[0264] The sps_affine_enabled_flag indicates whether affine-based motion compensation can be used in inter-frame prediction. If sps_affine_enabled_flag is 0, affine-based motion compensation is disabled on a per-sequence basis. In addition, the inter_affine_flag and cu_affine_type_flag are not transmitted in the CU (coding block) syntax of the coded video sequence. If sps_affine_enabled_flag is 1, affine-based motion compensation can be used in the coded video sequence.

[0265] sps_affine_type_flag indicates whether motion compensation based on the six-parameter affine model can be used in inter-frame prediction. If sps_affine_type_flag is 0, motion compensation based on the six-parameter affine model is suppressed. In addition, cu_affine_type_flag is not transmitted in the CU syntax of the coded video sequence. If sps_affine_type_flag is 1, motion compensation based on the six-parameter affine model can be used in the coded video sequence. If sps_affine_type_flag is not present, it is set to 0.

[0266] When decoding a P slice or B slice, if inter_affine_flag is 1 in the currently processed CU, motion compensation based on an affine model is used to generate a motion compensated prediction signal for the currently processed CU. If inter_affine_flag is 0, the affine model is not used for the currently processed CU. If inter_affine_flag is not present, it is set to 0.

[0267] When decoding a P slice or a B slice, if cu_affine_type_flag is 1 in the current processing target CU, motion compensation based on a six-parameter affine model is used to generate a motion compensated prediction signal for the current processing target CU. If cu_affine_type_flag is 0, motion compensation based on a four-parameter affine model is used to generate a motion compensated prediction signal for the current processing target CU.

[0268] In motion compensation based on an affine model, since a reference index or a motion vector is derived in units of sub-blocks, a motion-compensated prediction signal is generated using the reference index or motion vector to be processed in units of sub-blocks.

[0269] The four-parameter affine model is a mode in which the motion vector of a sub-block is derived from four parameters of the horizontal component and the vertical component of the motion vectors of two control points, and motion compensation is performed on a sub-block basis.

[0270] (Second embodiment)

[0271] In the first embodiment, the order of the elements (candidates) in the reference history motion vector prediction candidate list is the same in the normal motion vector prediction mode and the normal merge mode. On the other hand, in the second embodiment, the following additions and changes are made to the first embodiment. In the normal motion vector prediction mode of the second embodiment, as shown in FIG. Figure 46 As shown, the reference order when using the historical prediction motion vector candidate list is performed in the reverse order of the first embodiment, and in the normal merge mode of the second embodiment, as shown in FIG. Figure 45 As shown, the reference order when using the historical motion vector predictor candidate list is performed in the same order as in the first embodiment. That is, in the normal motion vector predictor mode and the normal merge mode, the order of referencing the candidates included in the historical motion vector predictor candidate list is reversed (inverse order). Therefore, in the normal motion vector predictor mode, the elements in the historical motion vector predictor candidate list are added to the motion vector predictor candidate list in order from the oldest element, while in the normal merge mode, the elements in the historical motion vector predictor candidate list are added to the motion vector predictor candidate list in order from the newest element.

[0272] <Historical Motion Vector Prediction Candidate List>

[0273] Here, the composition and operation of the historical motion vector prediction candidate list will be explained.

[0274] like Figure 41 As shown, the inter-frame prediction information used by the inter-frame prediction in the encoding target block is set as the inter-frame prediction information candidate hMvpCand to be registered, and is registered in the history prediction motion vector candidate list HmvpCandList as a history used in the past. Figure 41 In , the historical prediction motion vector candidate list has a list structure capable of storing six elements, and a basic storage operation is a first-in-first-out (FIFO) method of taking out elements in order starting from the first stored element.

[0275] Here, as an example, the maximum number of elements that can be stored in HmvpCandList is described as 6 as determined by the encoding and decoding sides, but this is not particularly limited and can be more than 6. In addition, the maximum number of elements that can be stored in HmvpCandList can also be configured as 5 or less. For example, HmvpCandList can be configured to have a maximum number of elements equal to the maximum number of elements of inter-frame prediction information candidates, such as the maximum number of elements in the prediction motion vector candidate list mvpListLX, the maximum number of merge candidates, and the maximum number of sub-block merge candidates. In addition, HmvpCandList can be configured to be associated with the maximum number of elements of inter-frame prediction information candidates for each mode.

[0276] By including the maximum number of elements of HmvpCandList in the syntax elements of the bit stream, it is also possible to configure it so as to be transmitted from the encoding side to the decoding side.

[0277] like Figure 41 As shown, HmvpCandList can store 6 elements from position 0 at the beginning of the list to position 5 at the end of the list, and these elements can be filled sequentially from position 0 to position 5. Positions 0 to 5 are managed as historical predicted motion vector indexes hMvpIdx. For example, position 0 can be represented as hMVpIdx[0], and position 5 can be represented as hMVpIdx[5]. The number of stored elements of HmvpCandList is managed by NumHmvpCand, and the increase and decrease of stored elements are managed within the range from 0 to 6, which is the maximum number of elements.

[0278] The case where a new element is added while the maximum number of elements is stored in HmvpCandList is described in detail. Figure 42 As shown in , when the inter-frame prediction information candidate hMvpCand is to be newly registered as history, the element at position 0 is deleted and the position of each element is shifted by one toward the beginning. Figure 43 As shown in , by reducing the number of stored elements by one, a new element can be stored at the last position 5. Therefore, by storing the inter-frame prediction information candidate hMvpCand at the last position 5, as shown in Figure 44 As shown, a new element is added to HmvpCandList.

[0279] <Historical Motion Vector Predictor Candidate Derivation Process in Normal Motion Vector Predictor Mode of Second Embodiment>

[0280] Here, the historical motion vector predictor candidate derivation process in the normal motion vector predictor mode of the second embodiment will be described.

[0281] Figure 47 This is a flowchart illustrating the steps for deriving the historical motion vector prediction candidate list HmvpCandList while referencing it in the reverse order of the reference order of the normal merge mode in the normal motion vector prediction mode of the second embodiment.

[0282] When the current number of motion vector predictor candidates numCurrMvpCand is greater than the maximum number of elements in the motion vector predictor candidate list mvpListLX (here 2) or the value of the historical number of motion vector predictor candidates NumHmvpCand is 0 ( Figure 47 Step S2601: No), omitted Figure 47 The processing of steps S2602 to S2609 ends the historical motion vector predictor candidate derivation processing step. When the current motion vector predictor candidate number numCurrMvpCand is less than 2, which is the maximum number of elements in the motion vector predictor candidate list mvpListLX, and the value of the historical motion vector predictor candidate number NumHmvpCand is greater than 0 ( Figure 47 Step S2601: Yes), execute Figure 47 Processing of steps S2602 to S2609.

[0283] Next, the index i starts from 0 and is repeated until the smaller value of 4 and the number of historical motion vector prediction candidates NumHmvpCand minus 1. Figure 47 The processing in steps S2603 to S2608 ( Figure 47 Steps S2602 to S2609). If the current number of motion vector predictor candidates numCurrMvpCand is greater than or equal to 2, which is the maximum number of elements in the motion vector predictor candidate list mvpListLX ( Figure 47 Step S2603: No), omitted Figure 47 The processing of steps S2604 to S2609 ends the present historical motion vector predictor candidate derivation processing step. If the current number of motion vector predictor candidates numCurrMvpCand is less than 2, which is the maximum number of elements in the motion vector predictor candidate list mvpListLX ( Figure 47 Step S2603: Yes), execute Figure 47 Processing after step S2604.

[0284] Next, the processing from step S2605 to step S2607 is performed for the index Y of 0 and 1 (L0 and L1) respectively ( Figure 47 Steps S2604 to S2608). If the current number of motion vector predictor candidates numCurrMvpCand is greater than or equal to 2, which is the maximum number of elements in the motion vector predictor candidate list mvpListLX ( Figure 47 Step S2605: No), omitted Figure 47 The processing of steps S2606 to S2609 ends the present historical motion vector predictor candidate derivation processing step. If the current number of motion vector predictor candidates numCurrMvpCand is less than 2, which is the maximum number of elements in the motion vector predictor candidate list mvpListLX ( Figure 47 Step S2605: Yes), execute Figure 47 Processing after step S2606.

[0285] Next, in the historical motion vector prediction candidate list HmvpCandList, in the case of an element having the same reference index as the reference index refI_dxLX of the encoding / decoding target motion vector and being different from any element in the motion vector prediction candidate list mvp_ListLX ( Figure 47 Step S2606: Yes), count the numCurrMvpCandth element mvpListLX[numCurrMvpCand] starting from 0 of the predicted motion vector candidate list and add the motion vector of LY of the historical predicted motion vector candidate HmvpCandList[i] ( Figure 47 In step S2607), the current motion vector predictor candidate number numCurrMvpCand is increased by 1. In the historical motion vector predictor candidate list HmvpCandList, if there is no element with the same reference index as the reference index refIdxLX of the encoding / decoding target motion vector and no element is different from any element in the motion vector predictor candidate list mvp_ListLX ( Figure 47 Step S2607: No), skip the adding process of step S2607.

[0286] The above is done in both L0 and L1 Figure 47 The processing of steps S2605 to S2607 ( Figure 47 Steps S2604 to S2608).

[0287] Increment the index i by 1 ( Figure 47 Steps S2602 and S2609), if the index i is less than or equal to the value of 1 minus any smaller value of 4 and the number of historical prediction motion vector candidates NumHmvvpCand, the processing after step S2603 is performed again ( Figure 47 Steps S2602 to S2609).

[0288] As described above, in the normal motion vector predictor mode, the elements in the historical motion vector predictor candidate list are referenced sequentially starting from the element added first (old element), and are added to the motion vector predictor candidate list.

[0289] (Variation 1)

[0290] A historical reference flag indicating the reference order of the historical motion vector prediction candidate list, as in the present embodiment, may be introduced to switch between the reference order of the first embodiment and the reference order of the present embodiment. The historical reference flag may be included in a syntax element of the bitstream and transmitted to the decoding side. The decoding side may obtain the bitstream having the historical reference flag in the syntax element and decode the bitstream.

[0291] (Variation 2)

[0292] In this embodiment, in the normal motion vector prediction mode, as shown in FIG. Figure 46 As shown, the elements 0 to 5 in the historical prediction motion vector candidate list are referenced one by one in the order, in the normal merge mode, as shown in FIG. Figure 45 As shown, the historical motion vector prediction candidate list is referenced one by one in the order from element 5 to element 0. However, this is not limiting as long as the order in which the historical motion vector prediction candidate list is referenced differs between the normal motion vector prediction mode and the normal merge mode. For example, the number of historical motion vector prediction candidate lists referenced can be different between the normal motion vector prediction mode and the normal merge mode, such as referencing elements 0 to 2 in the normal motion vector prediction mode and elements 5 to 1 in the normal merge mode. In this case, it is preferable to have a larger number of references in the normal merge mode, which has a larger number of candidates. Furthermore, in the normal merge mode, the starting point can be changed, such as referencing from element 4 to element 0.

[0293] By adopting the second embodiment described above, by setting the reference order when using the historical motion vector predictor candidate list to be different from that in the normal motion vector predictor mode, it is possible to use candidates that are appropriate for each mode. In particular, when the normal motion vector predictor mode, which encodes differential motion vectors, often requires motion vectors that differ from those surrounding the current block. Therefore, by searching the historical motion vector predictor candidate list and using candidates sequentially, starting with the elements that are considered to be different from the most recent motion vector candidate list and have relatively low correlation, that is, the older elements that were previously stored, coding efficiency can be improved by using candidates that are appropriate for the normal motion vector predictor mode.

[0294] In addition, in the normal merge mode, the higher the correlation around the encoding object block, the higher the encoding efficiency. Therefore, by searching the historical predicted motion vector candidate list, starting from the elements with higher correlation with the most recent motion vector candidate list, that is, the new elements saved later, they are used as candidates in sequence, and the encoding efficiency can be improved by using candidates suitable for the normal merge mode.

[0295] All of the above-described embodiments may also be combined in multiple forms.

[0296] In all the above-described embodiments, the bitstream output by the image encoding device has a specific data format so that it can be decoded according to the encoding method used in the embodiment. In addition, the image decoding device corresponding to the image encoding device can decode the bitstream in the specific data format.

[0297] When using a wired or wireless network to exchange bitstreams between an image encoding device and an image decoding device, the bitstream can be converted into a data format suitable for transmission over the communication line for transmission. In this case, a transmitting device is provided to convert the bitstream output from the image encoding device into encoded data in a data format suitable for transmission over the communication line and transmit the encoded data to the network; and a receiving device is provided to receive the encoded data from the network, convert the encoded data into a bitstream, and provide it to the image decoding device. The transmitting device includes a memory for buffering the bitstream output from the image encoding device; a packet processing unit for packetizing the bitstream; and a transmitting unit for transmitting the packetized encoded data via the network. The receiving device includes a receiving unit for receiving the packetized encoded data via the network; a memory for buffering the received encoded data; and a packet processing unit for packetizing the encoded data to generate a bitstream, and then providing the bitstream to the image decoding device.

[0298] Alternatively, a display unit for displaying images decoded by the image decoding device may be added to the configuration to serve as a display device. In this case, the display unit reads the decoded image signal generated by the decoded image signal superimposing unit 207 and stored in the decoded image memory 208 and displays it on the screen.

[0299] Alternatively, a camera unit may be added to the structure, and the captured image may be input to the image encoding device, thereby serving as the camera device. In this case, the camera unit inputs the captured image signal to the block division unit 101.

[0300] Figure 37 An example of the hardware structure of a coding / decoding device according to this embodiment is shown. The coding / decoding device includes the structure of the image coding device and image decoding device according to the embodiments of the present invention. The coding / decoding device 9000 includes a CPU 9001, a codec IC 9002, an I / O interface 9003, a memory 9004, an optical disc drive 9005, a network interface 9006, and a video interface 9009. These components are connected via a bus 9010.

[0301] The image encoding unit 9007 and the image decoding unit 9008 are typically implemented as a codec IC 9002. The image encoding process in the image encoding device according to the embodiment of the present invention is performed by the image encoding unit 9007, while the image decoding process in the image decoding device according to the embodiment of the present invention is performed by the image decoding unit 9008. The I / O interface 9003 is implemented, for example, as a USB interface and is connected to an external keyboard 9104, mouse 9105, etc. The CPU 9001 controls the encoding and decoding device 9000 based on user operations input through the I / O interface 9003 to perform the user's desired actions. User operations performed through the keyboard 9104, mouse 9105, etc. include selecting whether to perform encoding or decoding functions, setting encoding quality, inputting and outputting bitstreams, and inputting and outputting images.

[0302] When a user wishes to play back an image recorded on disk recording medium 9100, optical disc drive 9005 reads a bitstream from inserted disk recording medium 9100 and transmits the read bitstream to image decoding unit 9008 of codec IC 9002 via bus 9010. Image decoding unit 9008 performs image decoding processing in the image decoding apparatus according to an embodiment of the present invention on the input bitstream and transmits the decoded image to an external monitor 9103 via video interface 9009. Furthermore, codec apparatus 9000 includes a network interface 9006 and is connectable to an external distribution server 9106 or portable terminal 9107 via network 9101. When a user wishes to play back an image recorded on distribution server 9106 or mobile terminal 9107 rather than on disk recording medium 9100, network interface 9006 obtains a bitstream from network 9101 rather than reading the bitstream from input disk recording medium 9100. Furthermore, when a user wishes to reproduce an image recorded in the memory 9004 , the image decoding process in the image decoding apparatus according to the embodiment of the present invention is performed on the bit stream recorded in the memory 9004 .

[0303] If a user wishes to encode an image captured by an external camera 9102 and store it in memory 9004, the video interface 9009 inputs the image from the camera 9102 and transmits it to the image encoding unit 9007 of the codec IC 9002 via bus 9010. The image encoding unit 9007 performs the image encoding processing of the image encoding device according to the embodiment of the present invention on the image input via the video interface 9009, generating a bitstream. The bitstream is then transmitted to the memory 9004 via bus 9010. If the user wishes to record the bitstream on a disk recording medium 9100 instead of in memory 9004, the optical disk drive 9005 writes the bitstream to the inserted disk recording medium 9100.

[0304] A hardware configuration having an image encoding device but not an image decoding device, or a hardware configuration having an image decoding device but not an image encoding device, can also be implemented. Such a hardware configuration can be implemented, for example, by replacing the codec IC 9002 with the image encoding unit 9007 or the image decoding unit 9008, respectively.

[0305] The processes related to the above encoding and decoding can of course be implemented as hardware transmission, storage, and reception devices, and can be implemented by firmware stored in ROM (read-only memory), flash memory, etc., or software on a computer, etc. This firmware program or software program can be provided by being recorded on a computer-readable recording medium, etc., or provided from a server via a wired or wireless network, or provided as a data broadcast via terrestrial or satellite digital broadcasting.

[0306] The present invention has been described above based on the embodiments. The embodiments are merely examples, and those skilled in the art will appreciate that various modifications are possible in the combination of the constituent elements and the processing steps, and that such modifications are also within the scope of the present invention.

[0307] Explanation of symbols

[0308] 100 image encoding device, 101 block division unit, 102 inter-frame prediction unit, 103 intra-frame prediction unit, 104 decoded image memory, 105 prediction method determination unit, 106 residual generation unit, 107 orthogonal transform / quantization unit, 108 bit string encoding unit, 109 inverse quantization / inverse orthogonal transform unit, 110 decoded image signal overlapping unit, 111 coding information storage memory, 200 image decoding device, 201 bit string decoding unit, 202 block division unit, 203 inter-frame prediction unit, 204 intra-frame prediction unit, 205 coding information storage memory, 206 inverse quantization / inverse orthogonal transform unit, 207 decoded image signal overlapping unit, 208 decoded image memory.< / poc>

Claims

1. An image encoding device for encoding a moving image using inter-frame prediction based on inter-frame prediction information in units of blocks obtained by dividing each picture of the moving image, the image encoding device comprising: a coding information storage unit, configured to add the inter-frame prediction information used in the inter-frame prediction to a historical motion vector prediction candidate list; a spatial motion vector predictor candidate deriving unit that derives spatial motion vector predictor candidates from blocks adjacent to the current block and adds the candidates to the motion vector predictor candidate list; a historical motion vector predictor candidate deriving unit configured to derive a historical motion vector predictor candidate from the historical motion vector predictor candidate list and add the candidate to the motion vector predictor candidate list; a spatial merging candidate deriving unit for deriving spatial merging candidates from blocks adjacent to the processing target block and adding the spatial merging candidates to a merging candidate list; a historical merge candidate deriving unit, configured to derive historical merge candidates from the historical motion vector prediction candidate list and add the historical merge candidates to the merge candidate list; as well as an average merge candidate deriving unit for deriving an average merge candidate and adding the average merge candidate to the merge candidate list, wherein the average merge candidate has a motion vector obtained by averaging the motion vectors of two merge candidates in the merge candidate list, The historical motion vector prediction candidate derivation unit and the historical merge candidate derivation unit refer to the candidates included in the historical motion vector prediction candidate list in different orders from each other as historical motion vector prediction candidates or historical merge candidates, The spatial merging candidate is inter-frame prediction information of a processed block adjacent to the processing target block.

2. An image coding method for coding a moving image using inter-frame prediction based on inter-frame prediction information in units of blocks obtained by dividing each picture of the moving image, the image coding method comprising: a coding information saving step of adding the inter-frame prediction information used in the inter-frame prediction to a historical prediction motion vector candidate list; a spatial motion vector prediction candidate deriving step of deriving spatial motion vector prediction candidates from blocks adjacent to the processing target block and adding the candidates to the motion vector prediction candidate list; a historical motion vector prediction candidate deriving step of deriving historical motion vector prediction candidates from the historical motion vector prediction candidate list and adding the candidates to the motion vector prediction candidate list; a spatial merging candidate deriving step of deriving spatial merging candidates from blocks adjacent to the processing target block and adding the spatial merging candidates to a merging candidate list; a historical merging candidate deriving step of deriving historical merging candidates from the historical motion vector prediction candidate list and adding the historical merging candidates to the merging candidate list; as well as an average merge candidate deriving step, deriving an average merge candidate and adding it to the merge candidate list, wherein the average merge candidate has a motion vector obtained by averaging the motion vectors of two merge candidates in the merge candidate list, In the historical motion vector prediction candidate deriving step and the historical merge candidate deriving step, candidates included in the historical motion vector prediction candidate list are referenced in mutually different orders as historical motion vector prediction candidates or historical merge candidates, The spatial merging candidate is inter-frame prediction information of a processed block adjacent to the processing target block.

3. An image decoding device for decoding a bit stream encoding a moving image using inter-frame prediction based on inter-frame prediction information, using blocks obtained by dividing each picture of the moving image as units, the image decoding device comprising: a coding information storage unit, configured to add the inter-frame prediction information used in the inter-frame prediction to a historical motion vector prediction candidate list; a spatial motion vector predictor candidate deriving unit that derives spatial motion vector predictor candidates from blocks adjacent to the current block and adds the candidates to the motion vector predictor candidate list; a historical motion vector predictor candidate deriving unit configured to derive a historical motion vector predictor candidate from the historical motion vector predictor candidate list and add the candidate to the motion vector predictor candidate list; a spatial merging candidate deriving unit for deriving spatial merging candidates from blocks adjacent to the processing target block and adding the spatial merging candidates to a merging candidate list; a historical merge candidate deriving unit, configured to derive historical merge candidates from the historical motion vector prediction candidate list and add the historical merge candidates to the merge candidate list; as well as an average merge candidate deriving unit for deriving an average merge candidate and adding the average merge candidate to the merge candidate list, wherein the average merge candidate has a motion vector obtained by averaging the motion vectors of two merge candidates in the merge candidate list, The historical motion vector prediction candidate derivation unit and the historical merge candidate derivation unit refer to the candidates included in the historical motion vector prediction candidate list in different orders from each other as historical motion vector prediction candidates or historical merge candidates, The spatial merging candidate is inter-frame prediction information of a processed block adjacent to the processing target block.

4. An image decoding method for decoding a bit stream encoding a moving image using inter-frame prediction based on inter-frame prediction information, using blocks obtained by dividing each picture of the moving image as units, the image decoding method comprising: a coding information saving step of adding the inter-frame prediction information used in the inter-frame prediction to a historical prediction motion vector candidate list; a spatial motion vector prediction candidate deriving step of deriving spatial motion vector prediction candidates from blocks adjacent to the processing target block and adding the candidates to the motion vector prediction candidate list; a historical motion vector prediction candidate deriving step of deriving historical motion vector prediction candidates from the historical motion vector prediction candidate list and adding the historical motion vector prediction candidates to the historical motion vector prediction candidate list; a spatial merging candidate deriving step of deriving spatial merging candidates from blocks adjacent to the processing target block and adding the spatial merging candidates to a merging candidate list; a historical merging candidate deriving step of deriving historical merging candidates from the historical motion vector prediction candidate list and adding the historical merging candidates to the merging candidate list; as well as an average merge candidate deriving step, deriving an average merge candidate and adding it to the merge candidate list, wherein the average merge candidate has a motion vector obtained by averaging the motion vectors of two merge candidates in the merge candidate list, In the historical motion vector prediction candidate deriving step and the historical merge candidate deriving step, candidates included in the historical motion vector prediction candidate list are referenced in mutually different orders as historical motion vector prediction candidates or historical merge candidates, The spatial merging candidate is inter-frame prediction information of a processed block adjacent to the processing target block.

5. A storage method for storing a bit stream generated by the image encoding method according to claim 2 in a recording medium.

6. A transmission method for transmitting a bit stream generated by the image encoding method according to claim 2.

Images