Moving picture encoding device and method, and moving picture decoding device and method

Abstract

The invention relates to a moving image encoding device and method, and a moving image decoding device and method. Comprises: a triangular merge candidate list construction unit that constructs a triangular merge candidate list including spatial merge candidates; a first triangulation merge candidate selection unit that selects a first triangulation merge candidate to be a single prediction from the triangulation merge candidate list; and a second triangulation merge candidate selection unit that selects, from the triangulation merge candidate list, a second triangulation merge candidate to be a single prediction, and performs motion compensation on the basis of the weighted average of the first triangulation merge candidate and the second triangulation merge candidate. And storing the single prediction motion information of any one of the first triangular merge candidate and the second triangular merge candidate.

Description

[0001] This application is a divisional application based on the invention with application number 202080003839.4, application date March 6, 2020, applicant JVC KENWOOD CORPORATION, and invention name “Moving image encoding device, moving image encoding method and moving image encoding program, moving image decoding device, moving image decoding method and moving image decoding program”. Technical Field

[0002] The present invention relates to image encoding and decoding technology for dividing an image into blocks and performing prediction. Background Art

[0003] In image encoding and decoding, the image to be processed is divided into a predetermined number of pixel sets, i.e., blocks, and processed in units of blocks. By dividing into appropriate blocks and appropriately setting intra-picture prediction (intra-frame prediction) and inter-picture prediction (inter-frame prediction), the coding efficiency is improved.

[0004] In the encoding / decoding of moving images, the encoding efficiency is improved by performing inter-frame prediction based on the already encoded / decoded pictures. Patent document 1 describes a technique for applying affine transformation in inter-frame prediction. In moving images, deformations such as enlargement, reduction, and rotation of objects are common, and by applying the technique of patent document 1, efficient encoding can be performed.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent document 1: Japanese Patent Application Laid-Open No. 9-172644. Summary of the invention

[0008] However, the technique of Patent Document 1 involves image transformation, so there is a problem of a large processing load. In view of the above problem, the present invention provides a low-load and highly efficient encoding technique.

[0009] In some embodiments of the present invention for solving the above-mentioned problems, the present invention includes: a triangular merge candidate list construction unit, which constructs a triangular merge candidate list including spatial merge candidates; a first triangular merge candidate selection unit, which selects a first triangular merge candidate that becomes a single prediction from the triangular merge candidate list; and a second triangular merge candidate selection unit, which selects a second triangular merge candidate that becomes a single prediction from the triangular merge candidate list, and in an area where motion compensation based on a weighted average of the first triangular merge candidate and the second triangular merge candidate is performed, the single prediction motion information of any one of the first triangular merge candidate or the second triangular merge candidate is used for storage.

[0010] According to the present invention, efficient image encoding / decoding processing can be realized with a low load. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram of an image encoding device according to an embodiment of the present invention;

[0012] Figure 2 is a block diagram of an image decoding device according to an embodiment of the present invention;

[0013] Figure 3 is a flowchart for explaining the action of splitting a tree block;

[0014] Figure 4 is a diagram showing a situation where an input image is divided into tree blocks;

[0015] Figure 5 is a diagram illustrating a z-scan;

[0016] Fig. 6A is a diagram showing the segmented shape of a block;

[0017] Figure 6B is a diagram showing the segmented shape of a block;

[0018] Figure 6C is a diagram showing the segmented shape of a block;

[0019] Fig.6D is a diagram showing the segmented shape of a block;

[0020] Fig. 6E is a diagram showing the segmented shape of a block;

[0021] Figure 7 It is a flowchart for explaining the operation of dividing a block into four;

[0022] Figure 8 It is a flowchart for explaining the operation of dividing a block into two or three.

[0023] Fig. 9 is the syntax used to describe the shape of a block segmentation;

[0024] Fig. 10A is a diagram for explaining intra prediction;

[0025] Fig. 10B is a diagram for explaining intra prediction;

[0026] Fig.11 is a diagram for illustrating a reference block for inter-frame prediction;

[0027] Fig. 12A Syntax for describing the prediction mode of a coding block;

[0028] Fig. 12B Syntax for describing the prediction mode of a coding block;

[0029] Fig.13 is a diagram showing the correspondence between syntax elements and modes related to inter-frame prediction;

[0030] Fig.14 is a diagram for explaining motion compensation of affine transformation with two control points;

[0031] Fig.15 is a diagram for explaining motion compensation of affine transformation with three control points;

[0032] Fig.16 yes Figure 1 A block diagram showing a detailed structure of the inter-frame prediction unit 102;

[0033] Fig.17 yes Fig.16 A block diagram showing a detailed structure of a normal prediction motion vector mode deriving unit 301;

[0034] Fig.18 yes Fig.16 A block diagram showing a detailed structure of a normal merge mode derivation unit 302;

[0035] Fig.19 It is used to illustrate Fig.16 A flowchart of a normal motion vector prediction mode derivation process of the normal motion vector prediction mode derivation unit 301;

[0036] Fig. 20 is a flowchart showing the processing steps of a normal motion vector prediction mode derivation process;

[0037] Fig.21 is a flowchart illustrating the processing steps of a normal merge mode export process;

[0038] Fig. 22 yes Figure 2 A block diagram showing a detailed structure of the inter-frame prediction unit 203;

[0039] Fig.23 yes Fig. 22 A block diagram showing a detailed structure of a normal prediction motion vector mode deriving unit 401;

[0040] Fig.24 yes Fig. 22 A block diagram showing a detailed structure of a normal merge mode derivation unit 402;

[0041] Fig.25 It is used to illustrate Fig. 22 A flowchart of a normal motion vector prediction mode derivation process of the normal motion vector prediction mode derivation unit 401;

[0042] Fig.26 is a diagram illustrating the steps of initializing / updating a historical motion vector prediction candidate list;

[0043] Fig. 27 is a flowchart of a same element confirmation processing step in a historical motion vector prediction candidate list initialization / update processing step;

[0044] Fig.28 is a flowchart of an element shift processing step in a historical motion vector prediction candidate list initialization / update processing step;

[0045] Fig.29 is a flowchart illustrating the steps of deriving a historical motion vector prediction candidate;

[0046] Fig.30 is a flow chart illustrating the steps of the historical merge candidate export process;

[0047] Fig.31A is a diagram for explaining an example of a process of updating a historical motion vector prediction candidate list;

[0048] Fig.31B is a diagram for explaining an example of a process of updating a historical motion vector prediction candidate list;

[0049] Fig.31C is a diagram for explaining an example of a process of updating a historical motion vector prediction candidate list;

[0050] Fig.32 A diagram for explaining motion compensation prediction in a case where a reference picture (RefL0Pic) of L0 is before a processing target picture (CurPic) in L0 prediction;

[0051] Fig.33 A diagram for explaining motion compensation prediction in a case where a reference picture of L0 prediction is at a time point after a processing target picture in L0 prediction;

[0052] Fig.34 A diagram for explaining the prediction direction of motion compensation prediction in a case where a reference picture for L0 prediction is before a processing target picture and a reference picture for L1 prediction is after a processing target picture in bi-prediction;

[0053] Fig.35 A diagram for explaining a prediction direction of motion compensation prediction in a case where a reference picture for L0 prediction and a reference picture for L1 prediction in bi-prediction are at a time before a processing target picture;

[0054] Fig.36A diagram for explaining a prediction direction of motion compensation prediction in a case where a reference picture for L0 prediction and a reference picture for L1 prediction in bi-prediction are at a time point after a processing target picture;

[0055] Fig.37 is a diagram for explaining an example of the hardware structure of the encoding and decoding device according to the embodiment of the present invention;

[0056] Fig.38A is a diagram illustrating the prediction of the triangular merger pattern;

[0057] Fig.38B is a diagram illustrating the prediction of the triangular merger pattern;

[0058] Fig.39 is a flowchart illustrating the steps of the average merge candidate derivation process;

[0059] Fig.40 is a flow chart illustrating the derivation of triangle merge candidates;

[0060] Fig.41 is a flowchart illustrating the derivation of uni-prediction motion information for merging triangular partition 0 according to an embodiment of the present invention;

[0061] Fig.42 is a flowchart illustrating the derivation of single-prediction motion information of merged triangular partition 1 according to an embodiment of the present invention;

[0062] Fig.43A is a diagram illustrating weighting in a triangular merging mode;

[0063] Fig.43B is a diagram illustrating weighting in a triangular merging mode;

[0064] Fig.44A is a diagram illustrating partitioning in a triangle merging pattern;

[0065] Fig.44B is a diagram illustrating partitioning in a triangle merging pattern;

[0066] Fig.44C is a diagram illustrating partitioning in a triangle merging pattern;

[0067] Fig.44D is a diagram illustrating partitioning in a triangle merging pattern;

[0068] Fig.44E is a diagram illustrating partitioning in a triangle merging pattern;

[0069] Fig.44F is a diagram illustrating partitioning in a triangle merging pattern;

[0070] Fig.45A is a diagram illustrating the preservation of information in the triangle merging mode;

[0071] Fig.45B is a diagram illustrating the preservation of information in the triangle merging mode;

[0072] Fig.46A is a diagram illustrating the preservation of information in the triangle merging mode;

[0073] Fig.46B is a diagram illustrating the preservation of information in the triangle merging mode;

[0074] Fig.47A is a diagram illustrating the preservation of information in the triangle merging mode;

[0075] Fig.47B is a diagram illustrating the preservation of information in the triangle merging mode;

[0076] Fig.48A is a diagram illustrating the preservation of information in the triangle merging mode;

[0077] Fig.48B This is a diagram for explaining the storage information in the triangle merging mode. DETAILED DESCRIPTION

[0078] Technologies and technical terms used in this embodiment are defined.

[0079] <tree block>

[0080] In the embodiment, the encoding / decoding processing target image is equally divided into predetermined sizes. This unit is defined as a tree block. Figure 4 In the embodiment, the size of the tree block is set to 128×128 pixels, but the size of the tree block is not limited to this, and any size can be set. The tree blocks that are the processing objects (corresponding to the encoding object in the encoding process and the decoding object in the decoding process) are switched in the raster scanning order, that is, from left to right and from top to bottom. The interior of each tree block can be further recursively divided. The blocks that become the encoding / decoding objects after the tree blocks are recursively divided are defined as encoding blocks. In addition, tree blocks and encoding blocks are collectively defined as blocks. By performing appropriate block division, efficient encoding can be performed. The size of the tree block can be set to a predetermined fixed value in the encoding device and the decoding device, or a structure in which the size of the tree block determined by the encoding device is transmitted to the decoding device can be adopted. Here, the maximum size of the tree block is set to 128×128 pixels, and the minimum size of the tree block is set to 16×16 pixels. In addition, the maximum size of the encoding block is set to 64×64 pixels, and the minimum size of the encoding block is set to 4×4 pixels.

[0081] <Prediction Mode>

[0082] Intra-frame prediction (MODE_INTRA) and inter-frame prediction (MODE_INTER) are switched in units of a processing target coding block. The intra-frame prediction (MODE_INTRA) performs prediction based on a processed image signal of the processing target image, and the inter-frame prediction performs prediction based on an image signal of the processed image.

[0083] The processed image is used to obtain an image, image signal, tree block, block, coding block, etc. by decoding the signal that has been encoded in the encoding process, and is used to obtain an image, image signal, tree block, block, coding block, etc. that has been decoded in the decoding process.

[0084] A mode for identifying the intra prediction (MODE_INTRA) and the inter prediction (MODE_INTER) is defined as a prediction mode (PredMode). The prediction mode (PredMode) expresses the intra prediction (MODE_INTRA) or the inter prediction (MODE_INTER) in the form of a value.

[0085] <Inter-frame prediction>

[0086] In inter-frame prediction for prediction based on an image signal of a processed image, multiple processed images can be used as reference pictures. In order to manage multiple reference pictures, two reference lists, L0 (reference list 0) and L1 (reference list 1), are defined, and reference pictures are determined using reference indexes, respectively. In a P slice, L0 prediction (Pred_L0) can be used. In a B slice, L0 prediction (Pred_L0), L1 prediction (Pred_L1), and dual prediction (Pred_BI) can be used. L0 prediction (Pred_L0) is an inter-frame prediction with reference to a reference picture managed by L0, and L1 prediction (Pred_L1) is an inter-frame prediction with reference to a reference picture managed by L1. Dual prediction (Pred_BI) is an inter-frame prediction that performs L0 prediction and L1 prediction simultaneously and refers to reference pictures managed by each of L0 and L1. Information for determining L0 prediction, L1 prediction, and dual prediction is defined as an inter-frame prediction mode. In subsequent processing, constants and variables with subscript LX added to the output are assumed to be processed as L0 and L1.

[0087] <Prediction Motion Vector Mode>

[0088] The predicted motion vector mode is a mode that transmits an index for determining a predicted motion vector, a differential motion vector, an inter-prediction mode, and a reference index and determines inter-prediction information of a processing target block. The predicted motion vector is derived from a predicted motion vector candidate derived from a processed block adjacent to the processing target block or a block belonging to a processed image that is located at the same position as the processing target block or in the vicinity (neighborhood) of the processing target block and the index for determining the predicted motion vector.

[0089] <Merge Mode>

[0090] The merge mode is a mode in which the inter-frame prediction information of the processing object block is derived based on the inter-frame prediction information of the processed blocks adjacent to the processing object block or the blocks belonging to the processed image that are located at the same position or near (adjacent) the processing object block, without transmitting the differential motion vector and the reference index.

[0091] The processed blocks adjacent to the processing object block and the inter-frame prediction information of the processed blocks are defined as spatial merge candidates. The blocks belonging to the processed image that are located at the same position as the processing object block or in the vicinity (neighborhood) and the inter-frame prediction information derived from the inter-frame prediction information of the block are defined as temporal merge candidates. Each merge candidate is registered in the merge candidate list, and the merge candidate used in the prediction of the processing object block is determined by the merge index.

[0092] <adjacent block>

[0093] Fig.11 This is a diagram for explaining reference blocks used to derive inter-frame prediction information in the prediction motion vector mode and the merge mode. A0, A1, A2, B0, B1, B2, and B3 are processed blocks adjacent to the processing target block. T0 is a block that is located at the same position as or near (adjacent to) the processing target block in the processing target image among the processed image blocks.

[0094] A1 and A2 are blocks located on the left side of the processing target coding block and adjacent to the processing target coding block. B1 and B3 are blocks located on the upper side of the processing target coding block and adjacent to the processing target coding block. A0, B0, and B2 are blocks located at the lower left, upper right, and upper left of the processing target coding block, respectively.

[0095] The details of how neighboring blocks are processed in the predicted motion vector mode and the merge mode are described later.

[0096] <Affine Transformation Motion Compensation>

[0097] Affine transformation motion compensation is to divide the coding block into predetermined units of sub-blocks and determine the motion vector for each divided sub-block separately to perform motion compensation. The motion vector of each sub-block is derived based on one or more control points, and the one or more control points are derived based on the inter-frame prediction information of the processed blocks adjacent to the processing object block or the blocks belonging to the processed image that are located at the same position or near (adjacent) as the processing object block. In this embodiment, the size of the sub-block is set to 4×4 pixels, but the size of the sub-block is not limited to this, and the motion vector can also be derived in units of pixels.

[0098] Fig.14 An example of affine transformation motion compensation when there are two control points is shown. In this case, the two control points have two parameters, a horizontal component and a vertical component. Therefore, the affine transformation when there are two control points is called a four-parameter affine transformation. Fig.14 CP1 and CP2 are control points.

[0099] Fig.15 An example of affine transformation motion compensation when there are three control points is shown. In this case, the three control points have two parameters, a horizontal component and a vertical component. Therefore, the affine transformation when there are three control points is called a six-parameter affine transformation. Fig.15 CP1, CP2, and CP3 are control points.

[0100] Affine transformation motion compensation can be used in either the prediction motion vector mode or the merge mode. A mode in which affine transformation motion compensation is applied in the prediction motion vector mode is defined as the sub-block prediction motion vector mode, and a mode in which affine transformation motion compensation is applied in the merge mode is defined as the sub-block merge mode.

[0101] <Syntax of coded blocks>

[0102] use Fig. 12A , Fig. 12B and Fig.13 , describes the syntax used to express the prediction mode of a coding block. Fig. 12Apred_mode_flag is a flag indicating whether it is inter-frame prediction. If pred_mode_flag is 0, inter-frame prediction is performed, and if pred_mode_flag is 1, intra-frame prediction is performed. In the case of intra-frame prediction, intra-frame prediction information intra_pred_mode is sent, and in the case of inter-frame prediction, merge_flag is sent. merge_flag is a flag indicating whether it is merge mode or predicted motion vector mode. In the case of predicted motion vector mode (merge_flag = 0), a flag inter_affine_flag indicating whether the sub-block prediction motion vector mode is applied is sent. If the sub-block prediction motion vector mode is applied (inter_affine_flag = 1), cu_affine_type_flag is sent. cu_affine_type_flag is a flag used to determine the number of control points in the sub-block prediction motion vector mode.

[0103] On the other hand, in the case of merge mode (merge_flag = 1), the Fig. 12B merge_subblock_flag. merge_subblock_flag is a flag indicating whether the sub-block merge mode is applied. In the case of sub-block merge mode (merge_subblock_flag=1), the merge index merge_subblock_idx is sent. On the other hand, in the case of not sub-block merge mode (merge_subblock_flag=0), the flag merge_triangle_flag indicating whether the triangle merge mode is applied is sent. If the triangle merge mode is applied (merge_triangle_flag=1), the merge triangle index merge_triangle_idx0, merge_triangle_idx1 are sent according to the direction of the partition block merge_triangle_split_dir and each of the two partitions divided. On the other hand, if the triangle merge mode is not applied (merge_triangle_flag=0), the merge index merge_idx is sent.

[0104] Fig.13Each syntax element value and the prediction mode corresponding thereto are shown. merge_flag=0, inter_affine_flag=0 corresponds to the normal prediction motion vector mode (Inter Pred Mode). merge_flag=0, inter_affine_flag=1 corresponds to the sub-block prediction motion vector mode (Inter Affine Mode). merge_flag=1, merge_subblock_flag=0, merge_trianlge_flag=0 corresponds to the normal merge mode (MergeMode). merge_flag=1, merge_subblock_flag=0, merge_trianlge_flag=1 corresponds to the triangle merge mode (Triangle Merge Mode). merge_flag=1, merge_subblock_flag=1 corresponds to the sub-block merge mode (Affine Merge Mode).

[0105] <poc>

[0106] POC (Picture Order Count) is a variable associated with the picture to be encoded, and is set to a value that increases by 1 corresponding to the output order of the picture. According to the value of POC, it is possible to distinguish whether it is the same picture, distinguish the front-and-back relationship between pictures in the output order, and derive the distance between pictures. For example, if the POCs of two pictures have the same value, they can be determined to be the same picture. In the case where the POCs of two pictures have different values, it can be determined that the picture with a smaller POC value is the picture output first, and the difference between the POCs of the two pictures represents the distance between the pictures in the time axis direction.

[0107] (First Embodiment)

[0108] An image encoding device 100 and an image decoding device 200 according to the first embodiment of the present invention will be described.

[0109] Figure 1 1 is a block diagram of an image coding apparatus 100 according to the first embodiment. The image coding apparatus 100 according to the embodiment includes a block division unit 101, an inter-frame prediction unit 102, an intra-frame prediction unit 103, a decoded image memory 104, a prediction method determination unit 105, a residual generation unit 106, an orthogonal transformation / quantization unit 107, a bit string coding unit 108, an inverse quantization / inverse orthogonal transformation unit 109, a decoded image signal superposition unit 110, and a coding information storage memory 111.

[0110] The block division unit 101 recursively divides the input image to generate coding blocks. The block division unit 101 includes a 4-division unit and a 2-3 division unit. The 4-division unit divides the block to be divided in the horizontal direction and the vertical direction respectively, and the 2-3 division unit divides the block to be divided in either the horizontal direction or the vertical direction. The block division unit 101 sets the generated coding block as the processing target coding block, and provides the image signal of the processing target coding block to the inter-frame prediction unit 102, the intra-frame prediction unit 103 and the residual generation unit 106. In addition, the block division unit 101 provides information indicating the determined recursive division structure to the bit string encoding unit 108. The detailed operation of the block division unit 101 will be described later.

[0111] The inter-frame prediction unit 102 performs inter-frame prediction of the processing target coding block. The inter-frame prediction unit 102 derives a plurality of candidates of inter-frame prediction information based on the inter-frame prediction information stored in the coding information storage memory 111 and the decoded image signal stored in the decoded image memory 104, selects an appropriate inter-frame prediction mode from the derived plurality of candidates, and provides the selected inter-frame prediction mode and the predicted image signal corresponding to the selected inter-frame prediction mode to the prediction method determination unit 105. The detailed structure and operation of the inter-frame prediction unit 102 will be described later.

[0112] The intra prediction unit 103 performs intra prediction of the processing target coding block. The intra prediction unit 103 refers to the decoded image signal stored in the decoded image memory 104 as a reference pixel, and generates a predicted image signal through intra prediction based on the coding information such as the intra prediction mode stored in the coding information storage memory 111. In the intra prediction, the intra prediction unit 103 selects an appropriate intra prediction mode from a plurality of intra prediction modes, and provides the selected intra prediction mode and the predicted image signal corresponding to the selected intra prediction mode to the prediction method determination unit 105.

[0113] Fig. 10A and Fig. 10B An example of intra prediction is shown. Fig. 10A : is a diagram showing the correspondence between the prediction direction of intra prediction and the intra prediction mode number. For example, intra prediction mode 50 generates an intra prediction image by copying reference pixels in the vertical direction. Intra prediction mode 1 is a DC mode, and is a mode in which all pixel values ​​of the processing object block are set to the average value of the reference pixels. Intra prediction mode 0 is a Planar mode (two-dimensional mode), and is a mode in which a two-dimensional intra prediction image is generated based on reference pixels in the vertical direction and the horizontal direction. Fig. 10B 4 is an example of an intra-prediction image generated in the intra-prediction mode 40. The intra-prediction unit 103 copies the value of the reference pixel in the direction indicated by the intra-prediction mode to each pixel of the processing target block. When the reference pixel in the intra-prediction mode is not an integer position, the intra-prediction unit 103 determines the reference pixel value by interpolation based on the reference pixel values ​​at the surrounding integer positions.

[0114] The decoded image memory 104 stores the decoded image generated by the decoded image signal superimposing unit 110. The decoded image memory 104 supplies the stored decoded image to the inter prediction unit 102 and the intra prediction unit 103.

[0115] The prediction method determination unit 105 determines the best prediction mode by evaluating each of intra prediction and inter prediction using the coding information and the coding amount of the residual, the distortion amount between the predicted image signal and the processing target image signal, etc. In the case of intra prediction, the prediction method determination unit 105 supplies intra prediction information such as the intra prediction mode as coding information to the bit string coding unit 108. In the case of the merge mode of the inter prediction, the prediction method determination unit 105 supplies inter prediction information such as the merge index and information indicating whether it is the sub-block merge mode (sub-block merge flag) as coding information to the bit string coding unit 108. In the case of the predicted motion vector mode of the inter prediction, the prediction method determination unit 105 supplies inter prediction information such as the inter prediction mode, the predicted motion vector index, the reference index of L0 and L1, the differential motion vector, and the information indicating whether it is the sub-block predicted motion vector mode (sub-block predicted motion vector flag) as coding information to the bit string coding unit 108. In addition, the prediction method determination unit 105 supplies the determined coding information to the coding information storage memory 111. The prediction method determination unit 105 supplies the predicted image signal to the residual generation unit 106 and the decoded image signal superposition unit 110 .

[0116] The residual generation unit 106 generates a residual by subtracting the predicted image signal from the image signal to be processed, and supplies the residual to the orthogonal transformation / quantization unit 107 .

[0117] The orthogonal transform / quantization section 107 orthogonally transforms and quantizes the residual according to the quantization parameter to generate an orthogonally transformed / quantized residual, and supplies the generated residual to the bit string encoding section 108 and the inverse quantization / inverse orthogonal transform section 109 .

[0118] The bitstream coding unit 108 encodes the coding information corresponding to the prediction method determined by the prediction method determination unit 105 for each coding block in addition to the information of the sequence, picture, slice, and coding block units. Specifically, the bitstream coding unit 108 encodes the prediction mode PredMode for each coding block. When the prediction mode is inter-frame prediction (MODE_INTER), the bitstream coding unit 108 encodes the coding information (inter-frame prediction information) such as the flag for distinguishing whether it is a merge mode, the sub-block merge flag, the merge index when it is a merge mode, the inter-frame prediction mode when it is not a merge mode, the predicted motion vector index, the information related to the differential motion vector, and the sub-block predicted motion vector flag according to the prescribed syntax (syntactic rule of the bitstream) to generate a first bitstream. When the prediction mode is intra-frame prediction (MODE_INTRA), the coding information (intra-frame prediction information) such as the intra-frame prediction mode is encoded according to the prescribed syntax (syntactic rule of the bitstream) to generate a first bitstream. In addition, the bit string encoding unit 108 performs entropy encoding on the residual after orthogonal transformation and quantization according to a predetermined syntax to generate a second bit string. The bit string encoding unit 108 multiplexes the first bit string and the second bit string according to a predetermined syntax and outputs a bit stream.

[0119] The inverse quantization / inverse orthogonal transform section 109 inversely quantizes and inversely orthogonally transforms the orthogonally transformed / quantized residual supplied from the orthogonal transform / quantization section 107 to calculate a residual, and supplies the calculated residual to the decoded image signal superposition section 110 .

[0120] The decoded image signal superimposing unit 110 superimposes the predicted image signal corresponding to the decision of the prediction method determining unit 105 and the residual obtained by the inverse quantization and inverse orthogonal transformation performed by the inverse quantization / inverse orthogonal transformation unit 109 to generate a decoded image, and stores it in the decoded image memory 104. In addition, the decoded image signal superimposing unit 110 may perform filtering processing on the decoded image to reduce distortion such as block distortion caused by encoding, and then store it in the decoded image memory 104.

[0121] The coding information storage memory 111 stores coding information such as the prediction mode (inter prediction or intra prediction) determined by the prediction method determination unit 105. In the case of inter prediction, the coding information stored in the coding information storage memory 111 includes inter prediction information such as the determined motion vector, reference indexes of the reference lists L0 and L1, and the historical prediction motion vector candidate list. In addition, in the case of the merge mode of the inter prediction, the coding information stored in the coding information storage memory 111 includes inter prediction information such as the merge index and information indicating whether it is a sub-block merge mode (sub-block merge flag) in addition to the above information. In addition, in the case of the predicted motion vector mode of the inter prediction, the coding information stored in the coding information storage memory 111 includes inter prediction information such as the inter prediction mode, the predicted motion vector index, the differential motion vector, and information indicating whether it is a sub-block predicted motion vector mode (sub-block predicted motion vector flag) in addition to the above information. In the case of intra prediction, the coding information stored in the coding information storage memory 111 includes intra prediction information such as the determined intra prediction mode.

[0122] Figure 2 It means Figure 1 The image decoding device of the embodiment of the present invention includes a bit string decoding unit 201, a block division unit 202, an inter-frame prediction unit 203, an intra-frame prediction unit 204, a coding information storage memory 205, an inverse quantization / inverse orthogonal transformation unit 206, a decoded image signal superposition unit 207, and a decoded image memory 208.

[0123] Figure 2 Decoding processing of an image decoding device and Figure 1 The decoding process of the internal setting of the image encoding device corresponds to Figure 2 The encoding information storage memory 205, the inverse quantization / inverse orthogonal transformation unit 206, the decoded image signal superposition unit 207 and the decoded image memory 208 have the same structure as Figure 1 The encoding information storage memory 111, the inverse quantization / inverse orthogonal transformation unit 109, the decoded image signal superposition unit 110 and the decoded image memory 104 of the image encoding device respectively correspond to the functions.

[0124] The bit stream provided to the bitstream decoding unit 201 is separated according to the prescribed syntax rules. The bitstream decoding unit 201 decodes the separated first bitstream to obtain information of the sequence, picture, slice, coding block unit, and coding information of the coding block unit. Specifically, the bitstream decoding unit 201 decodes the prediction mode PredMode in units of coding blocks, and the prediction mode PredMode distinguishes whether it is inter-frame prediction (MODE_INTER) or intra-frame prediction (MODE_INTRA). When the prediction mode is inter-frame prediction (MODE_INTER), the bitstream decoding unit 201 decodes the coding information (inter-frame prediction information) related to the flag for distinguishing whether it is a merge mode, the merge index in the case of the merge mode, the sub-block merge flag, the inter-frame prediction mode in the case of the prediction motion vector mode, the prediction motion vector index, the differential motion vector, the sub-block prediction motion vector flag, etc. according to the prescribed syntax, and provides the coding information (inter-frame prediction information) to the coding information storage memory 205 via the inter-frame prediction unit 203 and the block division unit 202. When the prediction mode is intra prediction (MODE_INTRA), the intra prediction mode and other coded information (intra prediction information) are decoded according to a predetermined syntax, and the coded information (intra prediction information) is supplied to the coded information storage memory 205 via the inter prediction unit 203 or the intra prediction unit 204 and the block division unit 202. The bit string decoding unit 201 decodes the separated second bit string, calculates the orthogonal transformation / quantization residual, and supplies the orthogonal transformation / quantization residual to the inverse quantization / inverse orthogonal transformation unit 206.

[0125] When the prediction mode PredMode of the coding block to be processed is the prediction motion vector mode in the inter-frame prediction (MODE_INTER), the inter-frame prediction unit 203 uses the coding information of the decoded image signal stored in the coding information storage memory 205 to derive multiple prediction motion vector candidates, and registers the derived multiple prediction motion vector candidates in the prediction motion vector candidate list described later. The inter-frame prediction unit 203 selects a prediction motion vector corresponding to the prediction motion vector index provided by the bit string decoding unit 201 from the multiple prediction motion vector candidates registered in the prediction motion vector candidate list, calculates a motion vector based on the differential motion vector decoded by the bit string decoding unit 201 and the selected prediction motion vector, and stores the calculated motion vector in the coding information storage memory 205 together with other coding information. Here, the coding information of the coding block to be provided / saved is the prediction mode PredMode, the flags predFlagL0[xP][yP] and predFlagL1[xP][yP] indicating whether to use L0 prediction and L1 prediction, the reference indexes refIdxL0[xP][yP][yP] and refIdxL1[xP][yP] of L0 and L1, the motion vectors mvL0[xP][yP] and mvL1[xP][yP] of L0 and L1, etc. Here, xP and yP are indexes indicating the position of the upper left pixel of the coding block in the picture. When the prediction mode PredMode is inter-frame prediction (MODE_INTER) and the inter-frame prediction mode is L0 prediction (Pred_L0), the flag predFlagL0 indicating whether L0 prediction is used is 1, and the flag predFlagL1 indicating whether L1 prediction is used is 0. When the inter prediction mode is L1 prediction (Pred_L1), the flag predFlagL0 indicating whether L0 prediction is used is 0, and the flag predFlagL1 indicating whether L1 prediction is used is 1. When the inter prediction mode is bi-prediction (Pred_BI), the flag predFlagL0 indicating whether L0 prediction is used and the flag predFlagL1 indicating whether L1 prediction is used are both 1. And, when the prediction mode PredMode of the coding block of the processing object is the merge mode in the inter prediction (MODE_INTER), the merge candidate is derived.Using the coding information of the decoded coding block stored in the coding information storage memory 205, a plurality of merge candidates are derived and registered in a merge candidate list described later. A merge candidate corresponding to the merge index decoded and provided by the bit string decoding unit 201 is selected from the plurality of merge candidates registered in the merge candidate list, and inter-frame prediction information such as flags predFlagL0[xP][yP], predFlagL1[xP][yP] indicating whether to use the L0 prediction and L1 prediction of the selected merge candidate, reference indices refIdxL0[xP][yP], refIdxL1[xP][yP] of L0 and L1, and motion vectors mvL0[xP][yP], mvL1[xP][yP] of L0 and L1 are stored in the coding information storage memory 205. Here, xP and yP are indices indicating the position of the upper left pixel of the coding block in the picture. The detailed structure and operation of the inter-frame prediction unit 203 will be described later.

[0126] When the prediction mode PredMode of the coding block to be processed is intra prediction (MODE_INTRA), the intra prediction unit 204 performs intra prediction. The intra prediction mode is included in the coding information decoded by the bit string decoding unit 201. The intra prediction unit 204 generates a predicted image signal by intra prediction and based on the decoded image signal stored in the decoded image memory 208 according to the intra prediction mode included in the decoded information decoded by the bit string decoding unit 201, and provides the generated predicted image signal to the decoded image signal superposition unit 207. Since the intra prediction unit 204 corresponds to the intra prediction unit 103 of the image encoding device 100, it performs the same processing as the intra prediction unit 103.

[0127] The inverse quantization / inverse orthogonal transformation unit 206 performs inverse orthogonal transformation and inverse quantization on the orthogonal transformation / quantization residual decoded by the bit string decoding unit 201 to obtain the inverse orthogonal transformation / inverse quantization residual.

[0128] The decoded image signal superimposing unit 207 decodes the decoded image signal by superimposing the predicted image signal obtained by the inter-frame prediction unit 203 or the predicted image signal obtained by the intra-frame prediction unit 204 and the residual after the inverse orthogonal transformation / inverse quantization by the inverse quantization / inverse orthogonal transformation unit 206, and stores the decoded image signal in the decoded image memory 208. When storing in the decoded image memory 208, the decoded image signal superimposing unit 207 may perform filtering processing on the decoded image to reduce block distortion caused by encoding, and then store it in the decoded image memory 208.

[0129] Next, the operation of the block division unit 101 in the image encoding device 100 will be described. Figure 3 The flowchart shows the operation of dividing an image into tree blocks and further dividing each tree block. First, the input image is divided into tree blocks of a predetermined size (step S1001). Each tree block is scanned in a predetermined order, i.e., a raster scan order (step S1002), and the inside of the tree block to be processed is divided (step S1003).

[0130] Figure 7 1 is a flowchart showing the detailed operation of the division process of step S1003. First, it is determined whether the block to be processed is divided into four (step S1101).

[0131] If it is determined that the processing target block is to be divided into four, the processing target block is divided into four (step S1102). Each block obtained by dividing the processing target block is scanned in the Z scanning order, that is, in the order of upper left, upper right, lower left, and lower right (step S1103). Figure 5 is an example of a Z scan sequence, Fig. 6A 601 is an example of dividing the processing object block into four. Fig. 6A The numbers 0 to 3 of 601 indicate the order of processing. Then, for each block divided in step S1101, recursively execute Figure 7 segmentation processing (step S1104).

[0132] When it is determined that the processing target block is not to be divided into four, a 2-3 division is performed (step S1105).

[0133] Figure 8 1 is a flowchart showing the detailed operation of the 2-3 division process of step S1105. First, it is determined whether the block to be processed is to be divided into 2-3, that is, whether it is to be divided into 2 or 3 (step S1201).

[0134] If it is determined that the processing target block is not to be divided into 2-3 blocks, that is, if it is determined that the block is not to be divided, the division is terminated (step S1211). That is, no further recursive division is performed on the block divided by the recursive division process.

[0135] When it is determined that the processing target block is divided into 2-3, it is determined whether to further divide the processing target block into 2 (step S1202).

[0136] When it is determined that the processing target block is to be divided into two, it is determined whether to divide the processing target block up and down (vertically) (step S1203), and based on the result, the processing target block is divided into two up and down (vertically) (step S1204), or the processing target block is divided into two left and right (horizontally) (step S1205). As a result of step S1204, the processing target block is divided into two Figure 6B As shown in 602 in FIG. 1204 , the processing object block is divided into two parts, the upper and the lower (vertical direction). As a result of step S1205, the processing object block is as follows Fig.6D As shown in 604 in FIG. 1 , it is divided into two parts, left and right (horizontally).

[0137] In step S1202, if it is not determined that the processing object block is to be divided into two, that is, if it is determined that it is to be divided into three, it is determined whether to divide the processing object block into the top, middle and bottom (vertical direction) (step S1206), and based on the result, the processing object block is divided into three into the top, middle and bottom (vertical direction) (step S1207), or the processing object block is divided into three into the left, middle and right (horizontal direction) (step S1208). In the result of step S1207, if the processing object block is divided into three into the top, middle and bottom (vertical direction) (step S1208), the processing object block is divided into three into the left, middle and right (horizontal direction) (step S1209). Figure 6C As shown in 603, it is divided into three parts (upper, middle and lower (vertical direction). In the result of step S1208, the processing object block is as follows Fig. 6E As shown in 605, it is divided into three parts: left, center and right (horizontal direction).

[0138] After any one of step S1204, step S1205, step S1207, and step S1208 is executed, each block obtained by dividing the processing target block is scanned in order from left to right and from top to bottom (step S1209). Figure 6B to Figure 6E The numbers 0 to 2 from 602 to 605 indicate the order of processing. For each divided block, recursively execute Figure 8 2-3 segmentation processing (step S1210).

[0139] The recursive block division described here may also limit the need for division according to the number of divisions or the size of the block to be processed, etc. The information limiting the need for division may be implemented in a structure where the information is agreed upon in advance between the encoding device and the decoding device without transmitting the information, or may be implemented in a structure where the encoding device determines the information limiting the need for division and records it in a bit string and transmits it to the decoding device.

[0140] When a block is split, the block before the split is called a parent block, and each block after the split is called a child block.

[0141] Next, the operation of the block division unit 202 in the image decoding device 200 will be described. The block division unit 202 divides the tree blocks according to the same processing steps as the block division unit 101 of the image encoding device 100. However, the difference is that, in the block division unit 101 of the image encoding device 100, an optimization method such as estimation of the optimal shape based on image recognition or distortion rate optimization is applied to determine the optimal block division shape, while the block division unit 202 in the image decoding device 200 determines the block division shape by decoding the block division information recorded in the bit string.

[0142] Fig. 9 The syntax (syntactic rules of bit strings) related to the block division of the first embodiment is shown. coding_quadtree() represents the syntax involved in the block division process of 4. multi_type_tree() represents the syntax involved in the block division process of 2 or 3. qt_split is a flag indicating whether the block is divided into 4. When the block is divided into 4, set qt_split = 1, and when not divided into 4, set qt_split = 0. In the case of 4 division (qt_split = 1), the 4 division process is recursively performed on each of the divided blocks (coding_quadtree (0), coding_quadtree (1), coding_quadtree (2), coding_quadtree (3), and the independent variables 0 to 3 correspond to Fig. 6A 601). When no 4-partition is performed (qt_split=0), the subsequent partition is determined according to multi_type_tree(). mtt_split is a flag indicating whether to perform further partitioning. Furthermore, when partitioning is performed (mtt_split=1), a flag indicating whether to perform the partitioning in the vertical direction or the horizontal direction, namely mtt_split_vertical, and a flag determining whether to perform the partitioning in the 2 or 3 directions, namely mtt_split_binary, are transmitted. mtt_split_vertical=1 indicates the partitioning in the vertical direction, and mtt_split_vertical=0 indicates the partitioning in the horizontal direction. mtt_split_binary=1 indicates the partitioning in the 2, and mtt_split_binary=0 indicates the partitioning in the 3. In the case of 2-partition (mtt_split_binary=1), the partitioning process is performed recursively on each block after the 2-partition (multi_type_tree(0), multi_type_tree(1), and the 0 to 1 of the independent variable corresponds to Figure 6B to Figure 6D In the case of three-way split (mtt_split_binary=0), each block after the three-way split is recursively split (multi_type_tree(0), multi_type_tree(1), multi_type_tree(2), 0 to 2 correspond to Figure 6B 603 or Fig. 6E 605 of ). Hierarchical block segmentation is performed by recursively calling multi_type_tree until mtt_split=0.

[0143] <Inter-frame prediction>

[0144] The inter-frame prediction method of the embodiment is Figure 1 The inter-frame prediction unit 102 of the image encoding device and Figure 2 The method is implemented in the inter-frame prediction unit 203 of the image decoding device.

[0145] The inter-frame prediction method according to the embodiment will be described with reference to the drawings. The inter-frame prediction method is performed in either the encoding process or the decoding process in units of coding blocks.

[0146] <Description of the Inter-frame Prediction Unit 102 on the Encoding Side>

[0147] Fig.16 It is shown Figure 1 FIG. 1 is a diagram showing a detailed structure of the inter-frame prediction unit 102 of the image encoding device of FIG. The normal prediction motion vector mode deriving unit 301 derives a plurality of normal prediction motion vector candidates to select a prediction motion vector, and calculates a differential motion vector between the selected prediction motion vector and the detected motion vector. The detected inter-frame prediction mode, reference index, motion vector, and calculated differential motion vector are inter-frame prediction information of the normal prediction motion vector mode. The inter-frame prediction information is provided to the inter-frame prediction mode determination unit 305. The detailed structure and processing of the normal prediction motion vector mode deriving unit 301 will be described later.

[0148] In the normal merge mode derivation unit 302, a plurality of normal merge candidates are derived, a normal merge candidate is selected, and inter prediction information of the normal merge mode is obtained. The inter prediction information is provided to the inter prediction mode determination unit 305. The detailed structure and processing of the normal merge mode derivation unit 302 will be described later.

[0149] In the sub-block prediction motion vector mode derivation unit 303, multiple sub-block prediction motion vector candidates are derived to select a sub-block prediction motion vector, and a differential motion vector between the selected sub-block prediction motion vector and the detected motion vector is calculated. The detected inter-frame prediction mode, reference index, motion vector and calculated differential motion vector are inter-frame prediction information of the sub-block prediction motion vector mode. The inter-frame prediction information is provided to the inter-frame prediction mode determination unit 305.

[0150] The sub-block merging mode derivation unit 304 derives a plurality of sub-block merging candidates, selects a sub-block merging candidate, and obtains inter-frame prediction information of the sub-block merging mode. The inter-frame prediction information is provided to the inter-frame prediction mode determination unit 305 .

[0151] The inter prediction mode determination unit 305 determines inter prediction information based on the inter prediction information supplied from the normal prediction motion vector mode derivation unit 301, the normal merge mode derivation unit 302, the sub-block prediction motion vector mode derivation unit 303, and the sub-block merge mode derivation unit 304. The inter prediction information corresponding to the determination result is supplied from the inter prediction mode determination unit 305 to the motion compensation prediction unit 306.

[0152] The motion compensation prediction unit 306 performs inter-frame prediction on the reference image signal stored in the decoded image memory 104 based on the determined inter-frame prediction information. The detailed structure and processing of the motion compensation prediction unit 306 will be described later.

[0153] <Description of the Inter-frame Prediction Unit 203 on the Decoding Side>

[0154] Fig. 22 It is shown Figure 2 FIG. 2 is a diagram showing a detailed structure of the inter-frame prediction unit 203 of the image decoding device.

[0155] The normal prediction motion vector mode deriving unit 401 derives a plurality of normal prediction motion vector candidates to select a prediction motion vector, and calculates the sum of the selected prediction motion vector and the decoded differential motion vector as a motion vector. The decoded inter-frame prediction mode, reference index, and motion vector are inter-frame prediction information of the normal prediction motion vector mode. The inter-frame prediction information is provided to the motion compensation prediction unit 406 via the switch 408. The detailed structure and processing of the normal prediction motion vector mode deriving unit 401 will be described later.

[0156] In the normal merge mode derivation unit 402, a plurality of normal merge candidates are derived to select a normal merge candidate, and inter-frame prediction information of the normal merge mode is obtained. The inter-frame prediction information is provided to the motion compensation prediction unit 406 via the switch 408. The detailed structure and processing of the normal merge mode derivation unit 402 will be described later.

[0157] In the sub-block prediction motion vector mode derivation unit 403, multiple sub-block prediction motion vector candidates are derived to select a sub-block prediction motion vector, and the sum of the selected sub-block prediction motion vector and the decoded differential motion vector is calculated as a motion vector. The decoded inter-frame prediction mode, reference index, and motion vector become inter-frame prediction information of the sub-block prediction motion vector mode. The inter-frame prediction information is provided to the motion compensation prediction unit 406 via the switch 408.

[0158] The subblock merging mode derivation unit 404 derives a plurality of subblock merging candidates, selects a subblock merging candidate, and obtains inter-frame prediction information of the subblock merging mode. The inter-frame prediction information is provided to the motion compensation prediction unit 406 via the switch 408 .

[0159] The motion compensation prediction unit 406 performs inter-frame prediction on the reference image signal stored in the decoded image memory 208 based on the determined inter-frame prediction information. The detailed structure and processing of the motion compensation prediction unit 406 are the same as those of the motion compensation prediction unit 306 on the encoding side.

[0160] <Normal Motion Vector Prediction Mode Derivation Unit (Normal AMVP)>

[0161] Fig.17 The normal prediction motion vector mode extraction unit 301 includes a spatial prediction motion vector candidate extraction unit 321, a temporal prediction motion vector candidate extraction unit 322, a historical prediction motion vector candidate extraction unit 323, a prediction motion vector candidate supplement unit 325, a normal motion vector detection unit 326, a prediction motion vector candidate selection unit 327, and a motion vector subtraction unit 328.

[0162] Fig.23 The normal prediction motion vector mode extraction unit 401 includes a spatial prediction motion vector candidate extraction unit 421, a temporal prediction motion vector candidate extraction unit 422, a historical prediction motion vector candidate extraction unit 423, a prediction motion vector candidate supplement unit 425, a prediction motion vector candidate selection unit 426, and a motion vector addition unit 427.

[0163] Use respectively Fig.19 , Fig.25 The flowchart illustrates the processing steps of the normal prediction motion vector mode derivation unit 301 on the encoding side and the normal prediction motion vector mode derivation unit 401 on the decoding side. Fig.19 1 is a flowchart showing the normal motion vector prediction mode derivation processing steps by the normal motion vector mode derivation unit 301 on the encoding side. Fig.25 It is a flowchart showing the normal motion vector mode derivation processing procedure by the normal motion vector mode derivation unit 401 on the decoding side.

[0164] <Normal Motion Vector Prediction Mode Derivation Unit (Normal AMVP): Description on the Encoding Side>

[0165] refer to Fig.19 The following describes the steps for deriving the normal motion vector prediction mode on the encoding side. Fig.19 Sometimes the processing steps are omitted. Fig.19 The word "usually" is shown.

[0166] First, the normal motion vector detection unit 326 detects a normal motion vector ( Fig.19 Step S100).

[0167] Next, the spatial motion vector prediction candidate derivation unit 321, the temporal motion vector prediction candidate derivation unit 322, the historical motion vector prediction candidate derivation unit 323, the motion vector prediction candidate supplementation unit 325, the motion vector prediction candidate selection unit 327 and the motion vector subtraction unit 328 calculate, for each of L0 and L1, a differential motion vector ( Fig.19 Steps S101 to S106). Specifically, when the prediction mode PredMode of the processing object block is inter-frame prediction (MODE_INTER) and the inter-frame prediction mode is L0 prediction (Pred_L0), the prediction motion vector candidate list mvpListL0 of L0 is calculated, the prediction motion vector mvpL0 is selected, and the differential motion vector mvdL0 of the motion vector mvL0 of L0 is calculated. When the inter-frame prediction mode of the processing object block is L1 prediction (Pred_L1), the prediction motion vector candidate list mvpListL1 of L1 is calculated, the prediction motion vector mvpL1 is selected, and the differential motion vector mvdL1 of the motion vector mvL1 of L1 is calculated. When the inter-frame prediction mode of the processing object block is dual prediction (Pred_BI), L0 prediction and L1 prediction are performed simultaneously, the predicted motion vector candidate list mvpListL0 of L0 is calculated, the predicted motion vector mvpL0 of L0 is selected, the differential motion vector mvdL0 of L0's motion vector mvL0 is calculated, and the predicted motion vector candidate list mvpListL1 of L1 is calculated, the predicted motion vector mvpL1 of L1 is calculated, and the differential motion vector mvdL1 of L1's motion vector mvL1 is calculated.

[0168] The differential motion vector calculation process is performed on L0 and L1 respectively, but the process is common to L0 and L1. Therefore, in the following description, L0 and L1 are represented as a common LX. In the process of calculating the differential motion vector of L0, X of LX is 0, and in the process of calculating the differential motion vector of L1, X of LX is 1. In addition, in the process of calculating the differential motion vector of LX, when the information of another list is referred to without referring to LX, the other list is represented as LY.

[0169] In the case of using the motion vector mvLX of LX ( Fig.19 Step S102: Yes), calculate the candidates for the predicted motion vector of LX, and construct a predicted motion vector candidate list mvpListLX of LX ( Fig.19 Step S103). The spatial prediction motion vector candidate derivation unit 321, the temporal prediction motion vector candidate derivation unit 322, the historical prediction motion vector candidate derivation unit 323, and the prediction motion vector candidate supplementation unit 325 in the normal prediction motion vector mode derivation unit 301 derive multiple prediction motion vector candidates to construct a prediction motion vector candidate list mvpListLX. Fig.19 The detailed processing steps of step S103 are as follows: Fig. 20 The flowchart is described later.

[0170] Next, the motion vector predictor candidate selection unit 327 selects a motion vector predictor mvpLX of LX from the motion vector predictor candidate list mvpListLX of LX ( Fig.19 Step S104). Here, in the predicted motion vector candidate list mvpListLX, a certain element (the i-th element counting from 0) is represented as mvpListLX[i]. Each differential motion vector that is the difference between the motion vector mvLX and the candidates mvpListLX[i] of each predicted motion vector stored in the predicted motion vector candidate list mvpListLX is calculated. For each element (prediction motion vector candidate) of the predicted motion vector candidate list mvpListLX, the amount of code when encoding these differential motion vectors is calculated. Then, among the elements registered in the predicted motion vector candidate list mvpListLX, the candidate mvpListLX[i] of the predicted motion vector with the smallest amount of code for each candidate of the predicted motion vector is selected as the predicted motion vector mvpLX, and the index i is obtained. When there are multiple candidates for the predicted motion vector with the smallest amount of generated code in the predicted motion vector candidate list mvpListLX, the predicted motion vector candidate mvpListLX[i] represented by the number with the smallest index i in the predicted motion vector candidate list mvpListLX is selected as the optimal predicted motion vector mvpLX, and the index i is obtained.

[0171] Next, the motion vector subtraction unit 328 subtracts the selected predicted motion vector mvpLX of LX from the motion vector mvLX of LX, and sets mvdLX=mvLX-mvpLX to calculate the differential motion vector mvdLX of LX ( Fig.19 Step S105).

[0172] <Normal Prediction Motion Vector Mode Derivation Unit (Normal AMVP): Description on the Decoding Side>

[0173] Next, refer to Fig.25 The processing steps of the normal prediction motion vector mode on the decoding side are described. On the decoding side, the spatial prediction motion vector candidate derivation unit 421, the temporal prediction motion vector candidate derivation unit 422, the historical prediction motion vector candidate derivation unit 423 and the prediction motion vector candidate supplementation unit 425 calculate the motion vector ( Fig.25 Specifically, when the prediction mode PredMode of the processing object block is inter prediction (MODE_INTER) and the inter prediction mode of the processing object block is L0 prediction (Pred_L0), the prediction motion vector candidate list mvpListL0 of L0 is calculated, the prediction motion vector mvpL0 is selected, and the motion vector mvL0 of L0 is calculated. When the inter prediction mode of the processing object block is L1 prediction (Pred_L1), the prediction motion vector candidate list mvpListL1 of L1 is calculated, the prediction motion vector mvpL1 is selected, and the motion vector mvL1 of L1 is calculated. When the inter-frame prediction mode of the processing object block is dual prediction (Pred_BI), L0 prediction and L1 prediction are performed simultaneously, the predicted motion vector candidate list mvpListL0 of L0 is calculated, the predicted motion vector mvpL0 of L0 is selected, the motion vector mvL0 of L0 is calculated, and the predicted motion vector candidate list mvpListL1 of L1 is calculated, the predicted motion vector mvpL1 of L1 is calculated, and the motion vector mvL1 of L1 is calculated respectively.

[0174] Similar to the encoding side, the decoding side also performs motion vector calculation processing on L0 and L1 respectively, but L0 and L1 are common processes. Therefore, in the following description, L0 and L1 are represented as a common LX. LX represents the inter-frame prediction mode used for inter-frame prediction of the coding block of the processing object. In the process of calculating the motion vector of L0, X is 0, and in the process of calculating the motion vector of L1, X is 1. In addition, in the process of calculating the motion vector of LX, when the same reference list as the calculation object LX is not referenced but the information of another reference list is referenced, the other reference list is represented as LY.

[0175] In the case of using the motion vector mvLX of LX ( Fig.25 Step S202: Yes), calculate the candidates for the predicted motion vector of LX, and construct a predicted motion vector candidate list mvpListLX of LX ( Fig.25 Step S203). The spatial prediction motion vector candidate derivation unit 421, the temporal prediction motion vector candidate derivation unit 422, the historical prediction motion vector candidate derivation unit 423, and the prediction motion vector candidate supplementation unit 425 in the normal prediction motion vector mode derivation unit 401 calculate multiple prediction motion vector candidates and construct a prediction motion vector candidate list mvpListLX. Fig.25 The detailed processing steps of step S203 are as follows: Fig. 20 The flowchart is described later.

[0176] Next, the motion vector predictor candidate selection unit 426 extracts the motion vector predictor candidate mvpListLX[mvpIdxLX] corresponding to the motion vector predictor index mvpIdxLX decoded and provided by the bit string decoding unit 201 from the motion vector predictor candidate list mvpListLX as the selected motion vector predictor mvpLX ( Fig.25 Step S204).

[0177] Next, the motion vector addition unit 427 adds the differential motion vector mvdLX of LX decoded by the bitstream decoding unit 201 and the predicted motion vector mvpLX of LX, and calculates the motion vector mvLX of LX by setting mvLX=mvpLX+mvdLX. Fig.25 Step S205).

[0178] <Normal prediction motion vector mode derivation unit (normal AMVP): motion vector prediction method>

[0179] Fig. 20 It is a flowchart showing the processing steps of the normal prediction motion vector mode derivation processing having a common function in the normal prediction motion vector mode derivation unit 301 of the image encoding device and the normal prediction motion vector mode derivation unit 401 of the image decoding device involved in the embodiment of the present invention.

[0180] In the normal prediction motion vector mode export unit 301 and the normal prediction motion vector mode export unit 401, there is a prediction motion vector candidate list mvpListLX. The prediction motion vector candidate list mvpListLX constitutes a list structure, and is provided with a storage area for storing a prediction motion vector index indicating a position inside the prediction motion vector candidate list and a prediction motion vector candidate corresponding to the index as elements. The number of the prediction motion vector index starts from 0, and the prediction motion vector candidates are stored in the storage area of ​​the prediction motion vector candidate list mvpListLX. In this embodiment, it is assumed that the prediction motion vector candidate list mvpListLX can register at least two prediction motion vector candidates (inter-frame prediction information). In addition, the variable numCurrMvpCand indicating the number of prediction motion vector candidates registered in the prediction motion vector candidate list mvpListLX is set to 0.

[0181] The spatial motion vector prediction candidate derivation units 321 and 421 derive candidates for the motion vector prediction from the block adjacent to the left. In this process, the block adjacent to the left ( Fig.11 The inter-frame prediction information of A0 or A1), that is, the flag indicating whether the predicted motion vector candidate can be used, the motion vector, the reference index, etc., is used to derive the predicted motion vector mvLXA, and the derived mvLXA is added to the predicted motion vector candidate list mvpListLX ( Fig. 20 In step S301 of FIG. 10 , X is 0 in the case of L0 prediction and 1 in the case of L1 prediction (the same applies below). Next, the spatial prediction motion vector candidate derivation units 321 and 421 derive candidates for prediction motion vectors from blocks adjacent to the upper side. In this process, the blocks adjacent to the upper side ( Fig.11 The inter-frame prediction information (B0, B1 or B2), that is, the flag indicating whether the predicted motion vector candidate can be used, as well as the motion vector, reference index, etc., is used to derive the predicted motion vector mvLXB. If the respectively derived mvLXA and mvLXB are not equal, mvLXB is added to the predicted motion vector candidate list mvpListLX ( Fig. 20 Step S302). Fig. 20 The processing of steps S301 and S302 is common except that the position and number of referenced adjacent blocks are different, and the flag availableFlagLXN indicating whether the predicted motion vector candidate of the coding block can be used, as well as the motion vector mvLXN and the reference index refIdxN (N represents A or B, the same below) are derived.

[0182] Next, the temporal motion vector predictor candidate derivation units 322 and 422 derive candidates for motion vector predictors of blocks in a picture different in time from the current processing target picture. In this process, a flag avai lableFlagLXCol indicating whether a motion vector predictor candidate of a coding block in a picture at a different time can be used, a motion vector mvLXCol, a reference index refIdxCol, and a reference list listCol are derived, and mvLXCol is added to the motion vector predictor candidate list mvpListLX ( Fig. 20 Step S303).

[0183] Furthermore, it is assumed that the processing of the temporal motion vector predictor candidate derivation units 322 and 422 in units of a sequence (SPS), a picture (PPS), or a slice can be omitted.

[0184] Next, the historical motion vector predictor candidate derivation units 323 and 423 add the historical motion vector predictor candidates registered in the historical motion vector predictor candidate list HmvpCandList to the motion vector predictor candidate list mvpListLX ( Fig. 20 For details of the registration process steps of step S304, see Fig.29 The flowchart is described later.

[0185] Next, the motion vector predictor candidate supplementing units 325 and 425 add motion vector predictor candidates with predetermined values ​​such as (0, 0) until the motion vector predictor candidate list mvpListLX is satisfied ( Fig. 20 S305).

[0186] <Normal Merge Mode Export Section (Normal Merge)>

[0187] Fig.18 The normal merge mode derivation unit 302 includes a spatial merge candidate derivation unit 341, a temporal merge candidate derivation unit 342, an average merge candidate derivation unit 344, a historical merge candidate derivation unit 345, a merge candidate supplementation unit 346, and a merge candidate selection unit 347.

[0188] Fig.24 The normal merge mode derivation unit 402 includes a spatial merge candidate derivation unit 441, a temporal merge candidate derivation unit 442, an average merge candidate derivation unit 444, a historical merge candidate derivation unit 445, a merge candidate supplementation unit 446, and a merge candidate selection unit 447.

[0189] Fig.21 It is a flowchart for explaining the steps of a normal merge mode derivation process having a common function in the normal merge mode derivation unit 302 of the image encoding device and the normal merge mode derivation unit 402 of the image decoding device involved in the embodiment of the present invention.

[0190] Each process is described in sequence below. In addition, in the following description, unless otherwise specified, the case where the slice type slice_type is a B slice is described, but it can also be applied to the case of a P slice. However, when the slice type slice_type is a P slice, since only L0 prediction (Pred_L0) exists as an inter-frame prediction mode, L1 prediction (Pred_L1) and bi-prediction (Pred_BI) do not exist. Therefore, the processing around L1 can be omitted.

[0191] In the normal merge mode derivation unit 302 and the normal merge mode derivation unit 402, there is a merge candidate list mergeCandList. The merge candidate list mergeCandList constitutes a list structure, and is provided with a merge index indicating the position inside the merge candidate list, and a storage area for storing the merge candidate corresponding to the index as an element. The number of the merge index starts from 0, and the merge candidates are stored in the storage area of ​​the merge candidate list mergeCandList. In subsequent processing, it is assumed that the merge candidate of the merge index i registered in the merge candidate list mergeCandList is represented by mergeCandList[i]. In this embodiment, it is assumed that the merge candidate list mergeCandList can register at least six merge candidates (inter-frame prediction information). In addition, the variable numCurrMergeCand indicating the number of merge candidates registered in the merge candidate list mergeCandList is set to 0.

[0192] In the spatial merging candidate deriving unit 341 and the spatial merging candidate deriving unit 441, based on the coding information stored in the coding information storage memory 111 of the image coding device or the coding information storage memory 205 of the image decoding device, blocks ( Fig.11 The derived spatial merge candidates are registered in the merge candidate list mergeCandList ( Fig.21 Step S401). Here, N is defined to represent any one of B1, A1, B0, A0, B2 or temporal merge candidate Col. The flag availableFlagN indicating whether the inter-frame prediction information of block N can be used as a spatial merge candidate, the reference index refIdxL0N of L0 and the reference index refIdxL1N of L1 of the spatial merge candidate N, the L0 prediction flag predFlagL0N indicating whether L0 prediction is performed, the L1 prediction flag predFlagL1N indicating whether L1 prediction is performed, the motion vector mvL0N of L0, and the motion vector mvL1N of L1 are derived. However, in the present embodiment, since the merge candidate is derived without reference to the inter-frame prediction information of the block included in the coding block to be processed, the spatial merge candidate using the inter-frame prediction information of the block included in the coding block to be processed is not derived.

[0193] Next, in the temporal merge candidate derivation unit 342 and the temporal merge candidate derivation unit 442, temporal merge candidates from pictures at different times are derived, and the derived temporal merge candidates are registered in the merge candidate list mergeCandList ( Fig.21 Step S402). A flag availableFlagCol indicating whether the temporal merge candidate can be used, an L0 prediction flag predFlagL0Col indicating whether L0 prediction of the temporal merge candidate is performed, and an L1 prediction flag predFlagL1Col indicating whether L1 prediction is performed, as well as a motion vector mvL0Col of L0 and a motion vector mvL1Col of L1 are derived.

[0194] Furthermore, the processing of the temporal merging candidate deriving section 342 and the temporal merging candidate deriving section 442 in units of a sequence (SPS), a picture (PPS), or a slice can be omitted.

[0195] Next, in the historical merge candidate derivation unit 345 and the historical merge candidate derivation unit 445, the historical motion vector prediction candidates registered in the historical motion vector prediction candidate list HmvpCandList are registered in the merge candidate list mergeCandList ( Fig.21 Step S403).

[0196] In addition, when the number of merge candidates numCurrMergeCand registered in the merge candidate list mergeCandList is less than the maximum number of merge candidates MaxNumMergeCand, the number of merge candidates numCurrMergeCand registered in the merge candidate list mergeCandList is used as the upper limit to derive historical merge candidates and register them in the merge candidate list mergeCandList.

[0197] Next, in the average merge candidate derivation unit 344 and the average merge candidate derivation unit 444, the average merge candidate is derived from the merge candidate list mergeCandList, and the derived average merge candidate is added to the merge candidate list mergeCandList ( Fig.21 Step S404).

[0198] In addition, when the number of merge candidates numCurrMergeCand registered in the merge candidate list mergeCandList is less than the maximum number of merge candidates MaxNumMergeCand, the number of merge candidates numCurrMergeCand registered in the merge candidate list mergeCandList is upper bounded by the maximum number of merge candidates MaxNumMergeCand, and the average merge candidate is derived and registered in the merge candidate list mergeCandList.

[0199] Here, the average merge candidate is a new merge candidate having a motion vector obtained by averaging the motion vectors that the first merge candidate and the second merge candidate registered in the merge candidate list mergeCandList have for each of the L0 prediction and the L1 prediction.

[0200] Next, in the merge candidate supplement unit 346 and the merge candidate supplement unit 446, when the number of merge candidates numCurrMergeCand registered in the merge candidate list mergeCandList is less than the maximum number of merge candidates MaxNumMergeCand, the number of merge candidates numCurrMergeCand registered in the merge candidate list mergeCandList is derived with the maximum number of merge candidates MaxNumMergeCand as the upper limit to add merge candidates, and is registered in the merge candidate list mergeCandList ( Fig.21 Step S405). With the maximum number of merge candidates MaxNumMergeCand as the upper limit, in the P slice, a merge candidate whose prediction mode is L0 prediction (Pred_L0) and whose motion vector has a value of (0, 0) is added. In the B slice, a merge candidate whose prediction mode is bi-prediction (Pred_BI) and whose motion vector has a value of (0, 0) is added. The reference index when adding the merge candidate is different from the reference index that has been added.

[0201] Next, the merge candidate selection unit 347 and the merge candidate selection unit 447 select a merge candidate from the merge candidates registered in the merge candidate list mergeCandList. The merge candidate selection unit 347 on the encoding side selects a merge candidate by calculating the amount of code and the amount of distortion, and provides the merge index representing the selected merge candidate and the inter-frame prediction information of the merge candidate to the motion compensation prediction unit 306 via the inter-frame prediction mode determination unit 305. On the other hand, the merge candidate selection unit 447 on the decoding side selects a merge candidate based on the decoded merge index, and provides the selected merge candidate to the motion compensation prediction unit 406.

[0202] <Update the historical motion vector prediction candidate list>

[0203] Next, a method of initializing and updating the historical motion vector predictor candidate list HmvpCandList included in the encoding information storage memory 111 on the encoding side and the encoding information storage memory 205 on the decoding side will be described in detail. Fig.26 This is a flowchart used to illustrate the steps of initializing / updating the historical prediction motion vector candidate list.

[0204] In this embodiment, it is assumed that the update of the historical motion vector prediction candidate list HmvpCandList is performed in the coding information storage memory 111 and the coding information storage memory 205. Alternatively, a historical motion vector prediction candidate list update unit may be provided in the inter-frame prediction unit 102 and the inter-frame prediction unit 203 to implement the update of the historical motion vector prediction candidate list HmvpCandList.

[0205] The historical prediction motion vector candidate list HmvpCandList is initially set at the beginning of the strip. On the encoding side, when the normal prediction motion vector mode or the normal merge mode is selected by the prediction method determination unit 105, the historical prediction motion vector candidate list HmvpCandList is updated. On the decoding side, when the prediction information decoded by the bit string decoding unit 201 is the normal prediction motion vector mode or the normal merge mode, the historical prediction motion vector candidate list HmvpCandList is updated.

[0206] The inter-frame prediction information used when performing inter-frame prediction in the normal prediction motion vector mode or the normal merge mode is registered in the historical prediction motion vector candidate list hMvpCandList as the inter-frame prediction information candidate hMvpCand. The inter-frame prediction information candidate hMvpCand includes the reference index refIdxL0 of L0 and the reference index refIdxL1 of L1, the L0 prediction flag predFlagL0 indicating whether to perform L0 prediction and the L1 prediction flag predFlagL1 indicating whether to perform L1 prediction, the motion vector mvL0 of L0 and the motion vector mvL1 of L1.

[0207] In the case where there is inter-frame prediction information having the same value as the inter-frame prediction information candidate hMvpCand among the elements (i.e., inter-frame prediction information) registered in the historical prediction motion vector candidate list HmvpCandList in the encoding information storage memory 111 on the encoding side and the encoding information storage memory 205 on the decoding side, the element is deleted from the historical prediction motion vector candidate list HmvpCandList. On the other hand, in the case where there is no inter-frame prediction information having the same value as the inter-frame prediction information candidate hMvpCand, the element at the beginning of the historical prediction motion vector candidate list HmvpCandList is deleted, and the inter-frame prediction information candidate hMvpCand is added to the end of the historical prediction motion vector candidate list HmvpCandList.

[0208] The number of elements of the historical motion vector predictor candidate list HmvpCandList included in the encoding information storage memory 111 on the encoding side and the encoding information storage memory 205 on the decoding side of the present invention is set to 6.

[0209] First, the historical prediction motion vector candidate list HmvpCandList is initialized in units of slices ( Fig.26 At the beginning of the slice, all elements of the historical motion vector prediction candidate list HmvpCandList are set to empty, and the value of the number of historical motion vector prediction candidates (current number of candidates) NumHmvpCand registered in the historical motion vector prediction candidate list HmvpCandList is set to 0.

[0210] Furthermore, although the initialization of the historical motion vector prediction candidate list HmvpCandList is performed in slice units (the first coding block of the slice), it may be performed in picture units, in tile units, or in tree block row units.

[0211] Next, the following update process of the historical motion vector prediction candidate list HmvpCandList is repeated for each coding block in the slice ( Fig.26 Steps S2102 to S2107).

[0212] First, the initial setting is performed in units of coding blocks. The flag identicalCandExist indicating whether there is an identical candidate is set to a value of FALSE (false), and the deletion object index removeIdx indicating the candidate for deletion is set to "0" ( Fig.26 Step S2103).

[0213] Determine whether there is an inter-frame prediction information candidate hMvpCand( Fig.26 Step S2104). When the prediction method determination unit 105 on the encoding side determines that it is the normal prediction motion vector mode or the normal merge mode, or when the bit string decoding unit 201 on the decoding side decodes it as the normal prediction motion vector mode or the normal merge mode, the inter-frame prediction information is set as the inter-frame prediction information candidate hMvpCand of the registration object. When the prediction method determination unit 105 on the encoding side determines that it is the intra-frame prediction mode, the sub-block prediction motion vector mode or the sub-block merge mode, or when the bit string decoding unit 201 on the decoding side decodes it as the intra-frame prediction mode, the sub-block prediction motion vector mode or the sub-block merge mode, the historical prediction motion vector candidate list HmvpCandList is not updated, and there is no inter-frame prediction information candidate hMvpCand of the registration object. When there is no inter-frame prediction information candidate hMvpCand of the registration object, steps S2105 to S2106 are skipped ( Fig.26 Step S2104: No). If there is an inter-frame prediction information candidate hMvpCand to be registered, the processing after step S2105 is performed ( Fig.26 Step S2104: Yes).

[0214] Next, it is determined whether there is an element (inter-frame prediction information) with the same value as the inter-frame prediction information candidate hMvpCand of the registration object in each element of the historical prediction motion vector candidate list HmvpCandList, that is, whether there is an identical element ( Fig.26 Step S2105). Fig. 27 Flowchart of the same element confirmation processing steps. When the value of the historical prediction motion vector candidate number Num HmvpCand is 0 ( Fig. 27 Step S2121: No), the historical motion vector prediction candidate list HmvpCandList is empty, and since there is no identical candidate, the process is skipped. Fig. 27 Steps S2122 to S2125 are performed to terminate the same element confirmation processing step. If the value of the historical motion vector prediction candidate number NumHmvpCand is greater than 0 ( Fig. 27 Step S2121: Yes), the historical predicted motion vector index hMvpIdx increases from 0 to NumHmvpCand-1, and the process of step S2123 is repeated ( Fig. 27 First, compare whether the hMvpIdx-th element HmvpCandList[hMvpId x] counting from 0 in the historical prediction motion vector candidate list is the same as the inter-frame prediction information candidate hMvpCand ( Fig. 27 Step S2123). In the same case ( Fig. 27 Step S2123: Yes), set the flag identicalCandExist indicating whether there is an identical candidate to TRUE, set the deletion object index removeIdx indicating the position of the element to be deleted to the value of the current historical predicted motion vector index hMvpIdx, and end the identical element confirmation process. Fig. 27 Step S2123: No), increase hMvpIdx by 1. If the historical predicted motion vector index hMvpIdx is less than Nu mHmvpCand-1, perform the processing after step S2123.

[0215] Return again Fig.26 Flowchart of the present invention is used to perform the shifting and adding processing of the elements of the historical motion vector prediction candidate list HmvpCand List ( Fig.26 Step S2106). Fig.28 yes Fig.26 Flowchart of the element shifting / adding processing steps of the historical motion vector prediction candidate list HmvpCandList in step S2106. First, it is determined whether to add a new element after removing the element stored in the historical motion vector prediction candidate list HmvpCandList or to add a new element without removing the element. Specifically, it is compared whether the flag identicalCand Exist indicating whether there is an identical candidate is TRUE or whether NumHmvvpCand is 6. Fig.28 Step S2141). If the flag identicalCandExist indicating whether there is an identical candidate is TRUE (true) or the current number of candidates NumHmvpCand is 6, either ( Fig.28 Step S2141: Yes), remove the elements stored in the historical prediction motion vector candidate list HmvpCandL ist, and then add new elements. Set the initial value of index i to the value of removeIdx+1. Repeat the element shifting process of step S2143 from the initial value to NumHmvpCand. Fig.28 Steps S2142 to S2144). By copying the elements of HmvpCandList[i] to HmvpCandList[i-1], the elements are shifted forward ( Fig.28 Step S2143), increase i by 1 ( Fig.28 Next, the inter-frame prediction information candidate hMvpCan d is added to the (NumHmvpCand-1)th HmvpCandList[NumHmvpCand-1] (NumHmvpCand-1)th HmvpCandList[NumHmvpCand-1] (NumHmvpCand-1)th HmvpCandList[NumHmvpCand-1]) starting from 0 corresponding to the end of the historical prediction motion vector candidate list. Fig.28 Step S2145) ends the element shifting / adding process of the historical motion vector prediction candidate list HmvpCandList. On the other hand, if the flag identicalCandExist indicating whether there is an identical candidate is TRUE (true) and NumHmvpCand is 6, neither of the conditions is satisfied ( Fig.28 Step S2141: No), without removing the elements stored in the historical motion vector prediction candidate list HmvpCan dList, the inter-frame prediction information candidate hMvpCand is added to the end of the historical motion vector prediction candidate list ( Fig.28 Here, the end of the historical motion vector prediction candidate list is the NumHmvpCandth HmvpCandList[NumHmvp Cand] counting from 0. In addition, NumHmvpCand is increased by 1, and the shifting and adding processing of the elements of the historical motion vector prediction candidate list HmvpCandList is ended.

[0216] Figures 31A to 31C This is a diagram for explaining an example of updating the historical motion vector prediction candidate list. When a new element is added to the historical motion vector prediction candidate list HmvpCandList in which six elements (inter-frame prediction information) are registered, the elements starting from the front of the historical motion vector prediction candidate list Hm vpCandList are compared with the new inter-frame prediction information in sequence ( Fig.31A ), if the new element has the same value as the third element HMVP2 from the beginning of the historical prediction motion vector candidate list HmvpCandList, then the element HMVP2 is deleted from the historical prediction motion vector candidate list HmvpCandList, and the rear elements HMVP3 to HMVP5 are shifted (copied) one by one to the front, and the new element is added to the end of the historical prediction motion vector candidate list HmvpCandList ( Fig.31B ), complete the update of the historical prediction motion vector candidate list Hm vpCandList ( Fig.31C ).

[0217] <Historical motion vector predictor candidate derivation process>

[0218] Next, we will explain in detail Fig. 20 The method of deriving the historical motion vector prediction candidate from the historical motion vector prediction candidate list HmvpCandList in the processing step of step S304, Fig. 20 The processing procedure of step S304 is a common process in the historical prediction motion vector candidate derivation unit 323 of the normal prediction motion vector mode derivation unit 301 on the encoding side and the historical prediction motion vector candidate derivation unit 423 of the normal prediction motion vector mode derivation unit 401 on the decoding side. Fig.29 This is a flowchart illustrating the steps of deriving historical motion vector prediction candidates.

[0219] When the current number of predicted motion vector candidates numCurrMvpCand is greater than the maximum number of elements in the predicted motion vector candidate list mvpListLX (here 2) or the value of the historical number of predicted motion vector candidates NumHmvpCand is 0 ( Fig.29 Step S2201 of No), omitted Fig.29 The processing from step S2202 to step S2209 ends the historical motion vector prediction candidate derivation processing step. When the current number of motion vector prediction candidates numCurrMvpCand is less than the maximum number of elements 2 in the motion vector prediction candidate list mvpListLX and the value of the historical motion vector prediction candidate number NumHmvpCand is greater than 0 ( Fig.29 Step S2201 is), execute Fig.29 Processing of steps S2202 to S2209.

[0220] Next, repeat Fig.29 The processing of steps S2203 to S2208 is repeated until the index i is smaller than the value of 1 to 4 and the number of historical prediction motion vector candidates numCheckedHMVPCand ( Fig.29 Steps S2202 to S2209). If the current number of predicted motion vector candidates numCurrMvpCand is greater than or equal to the maximum number of elements in the predicted motion vector candidate list mvpListLX, which is 2 ( Fig.29 Step S2203: No), omitted Fig.29 The processing of steps S2204 to S2209 ends the historical motion vector prediction candidate derivation processing step. When the current number of motion vector prediction candidates numCurrMvpCand is less than the maximum number of elements in the motion vector prediction candidate list mvpListLX, 2 ( Fig.29 Step S2203: Yes), execute Fig.29 Processing after step S2204.

[0221] Next, the processing from step S2205 to step S2207 is performed for Y = 0 and 1 (L0 and L1) respectively ( Fig.29 Steps S2204 to S2208). If the current number of predicted motion vector candidates numCurrMvpCand is greater than or equal to the maximum number of elements in the predicted motion vector candidate list mvpListLX, which is 2 ( Fig.29 Step S2205: No), omitted Fig.29 The processing of steps S2206 to S2209 ends the historical motion vector prediction candidate derivation processing step. If the current number of motion vector prediction candidates numCurrMvpCand is less than the maximum number of elements 2 in the motion vector prediction candidate list mvpListLX ( Fig.29 Step S2205: Yes), execute Fig.29 Processing after step S2206 in .

[0222] Next, in the historical motion vector prediction candidate list HmvpCandList, if the element has the same reference index as the reference index refIdxLX of the encoding / decoding target motion vector and is different from any element in the motion vector prediction list mvpListLX ( Fig.29 Step S2206: Yes), add the motion vector of LY of the historical motion vector prediction candidate HmvpCandList[NumHmvpCand-i] to the numCurrMvpCandth element mvpListLX[numCurrMvpCand] of the motion vector prediction candidate list starting from 0 ( Fig.29 Step S2207), and the current number of predicted motion vector candidates numCurrMvpCand is increased by 1. When there is no element with the same reference index as the reference index refIdxLX of the encoding / decoding target motion vector in the historical predicted motion vector candidate list HmvpCandList, and there is no element different from any element in the predicted motion vector list mvpListLX ( Fig.29 Step S2206: No), skip the adding process of step S2207.

[0223] In both L0 and L1, the above Fig.29 The processing of steps S2205 to S2207 ( Fig.29 The index i is increased by 1, and when the index i is less than or equal to 4 and the number of historical motion vector prediction candidates NumHmvpCand, whichever is smaller, the processing after step S2203 is performed again ( Fig.29 Steps S2202~S2209).

[0224] <Historical Merge Candidate Export Processing>

[0225] Next, we will explain in detail Fig.21 The method of deriving historical merge candidates from the historical merge candidate list HmvpCandList in the processing step of step S404, Fig.21 The processing procedure of step S404 is a common process in the historical merge candidate derivation unit 345 of the normal merge mode derivation unit 302 on the encoding side and the historical merge candidate derivation unit 445 of the normal merge mode derivation unit 402 on the decoding side. Fig.30 This is a flowchart illustrating the steps of the historical merge candidate derivation process.

[0226] First, perform initialization processing ( Fig.30 Step S2301). The value of FALSE is set for each of the elements from 0 to the (numCurrMergeCand-1)th element of isPruned[i], and the number of elements numCurrMergeCand registered in the current merge candidate list is set for the variable numOrigMergeCand.

[0227] Next, the initial value of the index hMvpIdx is set to 1, and the process is repeated from the initial value to NumHmvpC and from Fig.30 The addition process from step S2303 to step S2310 ( Fig.30 If the number of elements numCurrMergeCand registered in the current merge candidate list is not less than (the maximum number of merge candidates MaxNumMergeCand-1), the merge candidate is added to all elements in the merge candidate list, and thus the historical merge candidate derivation process is terminated ( Fig.30 If the number of elements numCurrMergeCand registered in the current merge candidate list is less than (the maximum number of merge candidates MaxNumMergeCand-1), the processing after step S2304 is executed. Set the value of FALSE (false) to sameMotion ( Fig.30 Next, the initial value of index i is set to 0, and the index is incremented from the initial value to numOrigMergeCand-1. Fig.30 The processing of steps S2306 and S2307 ( Fig.30 S2305 to S2308). Compare whether the (NumHmvpCand-hMvpIdx)th element HmvpCandList[NumHmvp Cand-hMvpIdx] of the historical motion vector prediction candidate list, which is counted from 0, is the same value as the i-th element mergeCandList[i] of the merge candidate list, which is counted from 0. Fig.30 Step S2306).

[0228] The same value of the merge candidate means that the merge candidate has the same value when the values ​​of all the components (inter-frame prediction mode, reference index and motion vector) of the merge candidate are the same. In the case where the merge candidate has the same value and isPruned[i] is FALSE (false) ( Fig.30 In step S2306, sameMotion and isPruned[i] are both set to TRUE (true) ( Fig.30 Step S2307). In the case where the values ​​are not the same ( Fig.30 If the result of step S2306 is No, the processing of step S2307 is skipped. Fig.30 After the repetitive processing of steps S2305 to S2308 is completed, compare whether sameMotion is FALSE (false) ( Fig.30 Step S2309), if sameMotion is FALSE (false) ( Fig.30 In step S2309, since the (NumHmvpCand-hMvpIdx)th element HmvpCandList[NumHvpCand-hMvpIdx] counting from 0 of the historical prediction motion vector candidate list does not exist in mergeCandList, the (NumHmvpCand-hMvpIdx)th element HmvpCandList[NumHvpCand-hMvpIdx] counting from 0 of the historical prediction motion vector candidate list is added to the mergeCandList[numCurrMergeCand] of the merge candidate list, and numCurrMergeCand is increased by 1 ( Fig.30 Step S2310). Increase the index hMvpIdx by 1 ( Fig.30 Step S2302), perform Fig.30 Repeat the processing of steps S2302 to S2311.

[0229] After all elements in the historical prediction motion vector candidate list have been confirmed, or merge candidates have been added to all elements in the merge candidate list, the derivation process of the historical merge candidates is completed.

[0230] <Average Merge Candidate Derivation Process>

[0231] Next, the common processing in the average merge candidate derivation unit 344 of the normal merge mode derivation unit 302 on the encoding side and the average merge candidate derivation unit 444 of the normal merge mode derivation unit 402 on the decoding side, that is, Fig.21 The processing steps of step S403, that is, the method for deriving the average merge candidate are described in detail. Fig.39 This is a flowchart illustrating the steps of deriving average merge candidate processing.

[0232] First, perform initialization processing ( Fig.39 Step S1301). The variable numOrigMergeCand is set to the number of elements numCurrMergeCand registered in the current merge candidate list.

[0233] Next, scan from the beginning of the merge candidate list in sequence to determine two motion information. Set the index representing the first motion information to i=0, and set the index representing the second motion information to j=1 ( Fig.39 If the number of elements numCurrMergeCand registered in the current merge candidate list is not less than (the maximum number of merge candidates MaxNumMergeCand-1), the merge candidate has been added to all elements in the merge candidate list, so the historical merge candidate derivation process is terminated ( Fig.39 If the number of elements registered in the current merge candidate list is less than (the maximum number of merge candidates MaxNumMergeCand-1), the processing after step S1305 is executed.

[0234] Determine whether the i-th motion information mergeCandList[i] in the merge candidate list and the j-th motion information mergeCandList[j] in the merge candidate list are both invalid ( Fig.39 In step S1305), if both are invalid, the average merge candidate of mergeCandList[i] and mergeCandList[j] is not derived, and the next element is transferred. If both mergeCandList[i] and mergeCandList[j] are not invalid, X is set to 0 and 1, and the following process is repeated ( Fig.39 Steps S1306 to S1314).

[0235] Determine whether the LX prediction of mergeCandList[i] is valid ( Fig.39 If the LX prediction of mergeCandList[i] is valid, then determine whether the LX prediction of mergeCandList[j] is valid ( Fig.39 Step S1308). If the LX prediction of mergeCandList[j] is valid, that is, if both the LX prediction of mergeCandList[i] and the LX prediction of mergeCandList[j] are valid, then the motion vector of the LX prediction obtained by averaging the motion vector of the LX prediction of mergeCandList[i] and the motion vector of the LX prediction of mergeCandList[j] and the average merge candidate of the LX prediction having the reference index of the LX prediction of mergeCandList[i] are derived and set as the LX prediction of averageCand, making the LX prediction of averageCand valid ( Fig.39 Step S1309). Fig.39 In step S1308, when the LX prediction of mergeCandList[j] is not valid, that is, when the LX prediction of mergeCandList[i] is valid and the LX prediction of mergeCandList[j] is invalid, the motion vector of the LX prediction of mergeCandList[i] and the average merge candidate of the LX prediction with the reference index are derived and set as the LX prediction of averageCand, so that the LX prediction of averageCand is valid ( Fig.39 Step S1310). If Fig.39 In step S1307, if the LX prediction of mergeCandList[i] is not valid, it is determined whether the LX prediction of mergeCandList[j] is valid ( Fig.39 Step S1311). If the LX prediction of mergeCandList[j] is valid, that is, if the LX prediction of mergeCandList[i] is invalid and the LX prediction of mergeCandList[j] is valid, the motion vector of the LX prediction of mergeCandList[j] and the average merge candidate of the LX prediction with the reference index are derived, set as the LX prediction of averageCand, and the LX prediction of averageCand is made valid ( Fig.39 Step S1312). Fig.39 In step S1311, if the LX prediction of mergeCandList[j] is not valid, that is, if the LX prediction of mergeCandList[i] and the LX prediction of mergeCandList[j] are both invalid, then the LX prediction of averageCand is invalidated ( Fig.39 Step S1312).

[0236] The average merge candidate averageCand of the L0 prediction, L1 prediction or BI prediction generated as described above is added to the mergeCandList[numCurrMergeCand] of the numCurrMergeCand-th merge candidate list, and numCurrMergeCand is increased by 1 ( Fig.39 The above ends the average merge candidate derivation process.

[0237] Furthermore, the average merge candidate is averaged with the horizontal component of the motion vector and the vertical component of the motion vector, respectively.

[0238] <Motion Compensated Prediction Processing>

[0239] The motion compensation prediction unit 306 obtains the position and size of the block that is the object of the current prediction process in the encoding. In addition, the motion compensation prediction unit 306 obtains the inter-frame prediction information from the inter-frame prediction mode determination unit 305. The reference index and the motion vector are derived from the obtained inter-frame prediction information, and after obtaining the image signal of the reference picture specified by the reference index in the decoded image memory 104, the position is moved by the amount of the motion vector from the same position as the image signal of the block that is the object of the prediction process, and a prediction signal is generated.

[0240] In the case where the inter-frame prediction mode in the inter-frame prediction is a prediction from a single reference picture such as L0 prediction or L1 prediction, a prediction signal obtained from one reference picture is set as a motion compensation prediction signal. In the case where the inter-frame prediction mode is a prediction from two reference pictures such as BI prediction, a signal obtained by weighted averaging the prediction signals obtained from the two reference pictures is set as a motion compensation prediction signal, and the motion compensation prediction signal is provided to the prediction method determination unit 105. Here, the weighted average ratio of the double prediction is set to 1:1, but other ratios can also be used for weighted averaging. For example, the closer the picture interval between the picture to be predicted and the reference picture is, the larger the weighted ratio is. In addition, the weighted ratio can be calculated using a correspondence table of combinations of picture intervals and weighted ratios.

[0241] The motion compensation prediction unit 406 has the same function as the motion compensation prediction unit 306 on the encoding side. The motion compensation prediction unit 406 obtains inter-frame prediction information from the normal prediction motion vector mode derivation unit 401, the normal merge mode derivation unit 402, the sub-block prediction motion vector mode derivation unit 403, and the sub-block merge mode derivation unit 404 through the switch 408. The motion compensation prediction unit 406 supplies the obtained motion compensation prediction signal to the decoded image signal superposition unit 207.

[0242] <About Inter Prediction Mode>

[0243] The process of performing prediction based on a single reference picture is defined as uni-prediction. In the case of uni-prediction, prediction using one of two reference pictures registered in the reference list L0 or L1, such as L0 prediction or L1 prediction, is performed.

[0244] Fig.32 The case where the reference picture (RefL0Pic) of L0 in uni-prediction is before the processing target picture (CurPic) is shown. Fig.33 1 shows the case where the reference picture of the L0 prediction in the single prediction is at a time after the processing target picture. Fig.32 and Fig.33 The reference picture of L0 prediction is replaced with the reference picture of L1 prediction (RefL1Pic) to perform single prediction.

[0245] The process of performing prediction based on two reference pictures is defined as bi-prediction. In the case of bi-prediction, both L0 prediction and L1 prediction are used to express BI prediction. Fig.34 The case where the reference picture of the L0 prediction in the bi-prediction is at a time before the processing target picture and the reference picture of the L1 prediction is at a time after the processing target picture is shown. Fig.35 The case where the reference picture of the L0 prediction and the reference picture of the L1 prediction in the bi-prediction are at a time point before the processing target picture is shown. Fig.36 The case where the reference picture of the L0 prediction and the reference picture of the L1 prediction in the bi-prediction are at a time point after the processing target picture is shown.

[0246] In this way, the relationship between the prediction type and time of L0 / L1 can be used when L0 is not limited to the past direction and L1 is not limited to the future direction. In addition, in the case of dual prediction, each of L0 prediction and L1 prediction can be performed using the same reference picture. In addition, based on information (e.g., a flag) indicating whether L0 prediction is used and whether L1 prediction is used, it is determined whether to perform motion compensation prediction in single prediction or in dual prediction.

[0247] <About Reference Index>

[0248] In an embodiment of the present invention, in order to improve the accuracy of motion compensation prediction, the best reference picture can be selected from multiple reference pictures in motion compensation prediction. Therefore, the reference picture used in motion compensation prediction is used as a reference index, and the reference index is encoded into the bit stream together with the differential motion vector.

[0249] <Motion compensation processing based on normal motion vector prediction mode>

[0250] As in Fig.16 As also shown in the inter-frame prediction unit 102 on the encoding side, when the inter-frame prediction information based on the normal prediction motion vector mode deriving unit 301 is selected in the inter-frame prediction mode determination unit 305, the motion compensation prediction unit 306 obtains the inter-frame prediction information from the inter-frame prediction mode determination unit 305, derives the inter-frame prediction mode, reference index and motion vector of the block to be currently processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the prediction method determination unit 105.

[0251] Similarly, as in Fig. 22 As also shown in the inter-frame prediction unit 203 on the decoding side of the decoder, when the switch 408 is connected to the normal prediction motion vector mode deriving unit 401 during the decoding process, the motion compensation prediction unit 406 obtains the inter-frame prediction information based on the normal prediction motion vector mode deriving unit 401, derives the inter-frame prediction mode, reference index and motion vector of the block to be currently processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the decoded image signal superposition unit 207.

[0252] <Motion compensation processing based on normal merge mode>

[0253] As in Fig.16 As also shown in the inter-frame prediction unit 102 in the encoding side of , when the inter-frame prediction information based on the normal merge mode deriving unit 302 is selected in the inter-frame prediction mode determination unit 305, the motion compensation prediction unit 306 obtains the inter-frame prediction information from the inter-frame prediction mode determination unit 305, derives the inter-frame prediction mode, reference index and motion vector of the block to be currently processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the prediction method determination unit 105.

[0254] Similarly, as in Fig. 22 As also shown in the inter-frame prediction unit 203 on the decoding side of the decoder, when the switch 408 is connected to the normal merge mode deriving unit 402 during the decoding process, the motion compensation prediction unit 406 obtains the inter-frame prediction information based on the normal merge mode deriving unit 402, derives the inter-frame prediction mode, reference index, and motion vector of the block currently being processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the decoded image signal superposition unit 207.

[0255] <Motion compensation processing based on sub-block prediction motion vector mode>

[0256] As in Fig.16 As also shown in the inter-frame prediction unit 102 on the encoding side, when the inter-frame prediction information based on the sub-block prediction motion vector mode deriving unit 303 is selected in the inter-frame prediction mode determination unit 305, the motion compensation prediction unit 306 obtains the inter-frame prediction information from the inter-frame prediction mode determination unit 305, derives the inter-frame prediction mode, reference index and motion vector of the block to be currently processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the prediction method determination unit 105.

[0257] Similarly, as in Fig. 22 As also shown in the inter-frame prediction unit 203 on the decoding side of the decoder, when the switch 408 is connected to the sub-block prediction motion vector mode deriving unit 403 during the decoding process, the motion compensation prediction unit 406 obtains the inter-frame prediction information based on the sub-block prediction motion vector mode deriving unit 403, derives the inter-frame prediction mode, reference index and motion vector of the block to be currently processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the decoded image signal superposition unit 207.

[0258] <Motion compensation processing based on sub-block merging mode>

[0259] As in Fig.16 As also shown in the inter-frame prediction unit 102 on the encoding side, when the inter-frame prediction information based on the sub-block merging mode deriving unit 304 is selected in the inter-frame prediction mode determination unit 305, the motion compensation prediction unit 306 obtains the inter-frame prediction information from the inter-frame prediction mode determination unit 305, derives the inter-frame prediction mode, reference index and motion vector of the block to be currently processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the prediction method determination unit 105.

[0260] Similarly, as in Fig. 22 As also shown in the inter-frame prediction unit 203 on the decoding side of the decoder, when the switch 408 is connected to the sub-block merging mode deriving unit 404 during the decoding process, the motion compensation prediction unit 406 obtains the inter-frame prediction information based on the sub-block merging mode deriving unit 404, derives the inter-frame prediction mode, reference index and motion vector of the block to be currently processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the decoded image signal superposition unit 207.

[0261] <Motion compensation processing based on affine transformation prediction>

[0262] In the normal prediction motion vector mode and the normal merge mode, motion compensation based on the affine model can be used based on the following flag. The following flag is reflected in the following flag based on the inter-frame prediction condition determined by the inter-frame prediction mode determination unit 305 in the encoding process and is encoded in the bit stream. In the decoding process, whether to perform motion compensation based on the affine model is determined based on the following flag in the bit stream.

[0263] sps_affine_enabled_flag indicates whether affine model-based motion compensation can be used in inter-frame prediction. If sps_affine_enabled_flag is 0, it is suppressed in sequence units so that affine model-based motion compensation is not used. In addition, inter affine flag and cu affine type_flag are not transmitted in the CU (coding block) syntax of the coded video sequence. If sps_affine_enabled_flag is 1, affine model-based motion compensation can be used in the coded video sequence.

[0264] sps_affine_type_flag indicates whether motion compensation based on the six-parameter affine model can be used in inter-frame prediction. If sps_affine_type_flag is 0, it is suppressed as motion compensation not based on the six-parameter affine model. In addition, cu_affine_type_flag is not transmitted in the CU syntax of the encoded video sequence. If sps_affine_type_flag is 1, motion compensation based on the six-parameter affine model can be used in the encoded video sequence. In the absence of sps_affine_type_flag, it is set to 0.

[0265] When decoding a P slice or a B slice, in the CU currently being processed, if inter_affine_flag is 1, motion compensation based on an affine model is used to generate a motion compensation prediction signal for the CU currently being processed. If inter_affine_flag is 0, the affine model is not used for the CU currently being processed. If inter_affine_flag does not exist, it is set to 0.

[0266] When decoding a P slice or a B slice, in the CU that is currently being processed, if cu_affine_type_flag is 1, motion compensation based on a six-parameter affine model is used to generate a motion compensation prediction signal for the CU that is currently being processed. If cu_affine_type_flag is 0, motion compensation based on a four-parameter affine model is used to generate a motion compensation prediction signal for the CU that is currently being processed.

[0267] In motion compensation based on an affine model, since a reference index or a motion vector is derived in units of sub-blocks, a motion compensation prediction signal is generated using the reference index or motion vector to be processed in units of sub-blocks.

[0268] The four-parameter affine model is a mode in which the motion vector of a sub-block is derived from four parameters of the horizontal component and the vertical component of the motion vectors of two control points, and motion compensation is performed in units of sub-blocks.

[0269] <Triangle Merge Mode>

[0270] Triangle merge mode is a type of merge mode, which is a mode that divides the coding / decoding block into diagonal partitions for motion compensation prediction. Triangle merge mode is a type of geometric partition merge mode that divides the coding / decoding block into non-rectangular blocks. In geometric partition merge mode, it is equivalent to dividing the coding / decoding block into two right triangles along the diagonal line.

[0271] The geometric partitioning merge mode is represented by a combination of two parameters, for example, an index representing the partitioning angle (angleIdx) and an index representing the distance from the center of the coding block (distance Idx). In one example, 64 patterns are defined as geometric partitioning merge modes and fixed-length encoded. Among the 64 patterns, the index representing the partitioning angle represents the angle of the diagonal constituting the coding block (for example, 45 degrees (in a structure representing 360 degrees by 32 partitions, angleIdx=4) or 135 degrees (in a structure representing 360 degrees by 32 partitions, angleIdx=12)), and the two modes with the smallest index representing the distance from the center of the coding block (distanceIdx=0, indicating that the partitioning boundary passes through the center of the coding block) represent the partitioning of the coding block by diagonal lines, which is equivalent to the triangle merge mode.

[0272] use Fig.38A and Fig.38B The triangle merging mode is described. Fig.38A and 38B An example of prediction of a coding / decoding block in 16×16 triangle merge mode is shown. The coding / decoding block in triangle merge mode is divided into 4×4 sub-blocks, and each sub-block is assigned to three partitions: uni-prediction partition 0 (UNI0), uni-prediction partition 1 (UNI1), and bi-prediction partition 2 (BI). Here, the sub-blocks located on the upper side of the diagonal are assigned to partition 0, the sub-blocks located on the lower side of the diagonal are assigned to partition 1, and the sub-blocks located on the diagonal are assigned to partition 2. If merge_triangle_split_dir is 0, then Fig.38A If merge_triangle_split_dir is 1, then Fig.38B Allocate partitions that way.

[0273] In the motion compensation prediction of partition 0, the motion information of the uni-prediction specified by the merge triangle index 0 is used. In the motion compensation prediction of partition 1, the motion information of the uni-prediction specified by the merge triangle index 1 is used. In the motion compensation prediction of partition 2, the motion information of the bi-prediction combining the motion information of the uni-prediction specified by the merge triangle index 0 and the motion information of the uni-prediction specified by the merge triangle index 1 is used.

[0274] Here, uni-predicted motion information refers to one set of motion vector and reference index, and bi-predicted motion information is composed of two sets of motion vector and reference index. In addition, motion information refers to uni-predicted motion information or bi-predicted motion information.

[0275] The merge candidate selection units 347 and 447 use the derived merge candidate list mergeCandList as the triangle merge candidate list triangleMergeCandList.

[0276] Description of the triangle merging candidate export Fig.40 Flowchart of the process.

[0277] First, as the triangle merge candidate list triangleMergeCandList, the merge candidate list mergeCandList is used (step S3501).

[0278] Next, the merge candidate having the motion information list L0 is prioritized, and the uni-prediction motion information of the merged triangle partition 0 is derived (step S3502 ).

[0279] Next, the merge candidate having the motion information list L1 is prioritized, and the uni-prediction motion information of the merged triangle partition 1 is derived (step S3503 ).

[0280] In addition, step S3502 and step S3503 can be exported in different orders or processed in parallel.

[0281] Fig.41 1 is a flowchart illustrating the derivation of uni-prediction motion information for merging triangle partition 0 according to an embodiment of the present invention.

[0282] First, for the Mth candidate in the derived merge candidate list mergeCandList, determine whether the candidate M has motion information from the motion information list L0 (step S3601). If the candidate M has motion information from the motion information list L0, the motion information from the motion information list L0 of the candidate M is used as a delta merge candidate (step S3602). For the candidate M (M=0, 1, ..., numMergeCand-1), steps S3601 and S3602 are performed in ascending order to add and derive the delta merge candidate.

[0283] Next, for the Mth candidate in the derived merge candidate list mergeCandList, determine whether the candidate M has the motion information of the motion information list L1 (step S3603). If the candidate M has the motion information of the motion information list L1, the motion information of the motion information list L1 of the candidate M is used as the triangle merge candidate (step S3604). For the candidate M (M=numMergeCand-1, ..., 1, 0), steps S3603 and S3604 are performed in descending order to add and derive the triangle merge candidate.

[0284] Fig.42 It is a flowchart for explaining the derivation of single-prediction motion information of the merged triangular partition 1 according to an embodiment of the present invention.

[0285] First, for the Mth candidate in the derived merge candidate list mergeCandList, determine whether the candidate M has the motion information of the motion information list L1 (step S3701). If the candidate M has the motion information of the motion information list L1, the motion information of the motion information list L1 of the candidate M is used as the triangle merge candidate (step S3702). For the candidate M (M=0, 1, ..., numMergeCand-1), steps S3701 and S3702 are performed in ascending order to add and derive the triangle merge candidate.

[0286] Next, for the Mth candidate in the derived merge candidate list mergeCandList, determine whether the candidate M has the motion information of the motion information list L0 (step S3703). If the candidate M has the motion information of the motion information list L0, the motion information of the motion information list L0 of the candidate M is used as the triangle merge candidate (step S3704). For the candidate M (M=numMergeCand-1, ..., 1, 0), steps S3703 and S3704 are performed in descending order to add and derive the triangle merge candidate.

[0287] The merge candidate selection unit 347 on the encoding side obtains motion information from the derived triangle merge candidate list triangleMergeCandList, and calculates the amount of coding and the amount of distortion.

[0288] By comparing the calculated multiple coding amounts and distortion amounts, the direction of splitting the coding block and the triangular merge candidates of each partition after the split are selected. When the triangular merge mode is used for encoding, the selected information (the direction of splitting the coding block merge_triangle_split_dir and the merge triangle indexes merge_triangle_idx0, merge_triangle_idx1 indicating the triangular merge candidates of each partition after the split) and the inter-frame prediction information of the triangular merge candidate are provided to the motion compensation prediction unit 306. In addition, the bit string encoding unit 108 encodes the selected information.

[0289] On the other hand, in the case of the triangle merge mode, the merge candidate selection unit 447 on the decoding side selects a triangle merge candidate based on the decoded information (the direction of splitting the coding block merge_triangle_split_dir and the merge triangle indices merge_triangle_idx0, merge_triangle_idx1 representing the triangle merge candidates of each partition after the split), and provides the inter-frame prediction information of the selected triangle merge candidate to the motion compensation prediction unit 406.

[0290] In the case of delta merge mode, the motion compensation prediction units 306 and 406 perform weighted averaging as follows. In the case of luminance, for the width nCbW and height nCbH of the coding block, calculate

[0291] nCbR=(nCbW>nCbH)? (nCbW / nCbH): (nCbH / nCbW).

[0292] Thus, at position (x, y) within the coding block, Fig.38A The weight wValue in this case is:

[0293] wValue=(nCbW>nCbH)?

[0294] (Clip3(0,8,x / nCbR-y+4)):

[0295] (Clip3(0,8,y / nCbR-x+4)).

[0296] on the other hand, Fig.38B The weight wValue in this case is:

[0297] wValue=(nCbW>nCbH)?

[0298] (Clip3(0,8,nCbH-1-x / nCbR-y+4)):

[0299] (Clip3(0,8,nCbW-1-y / nCbR-x+4)).

[0300] In addition, for the number of bits bitDepth, calculate

[0301] shift1 = max(5, 17-bitDepth)

[0302] offset1=1<<(shift1-1).

[0303] Therefore, the weighted average result pbSamples is:

[0304] pbSamples = Clip3 (0, (1 < <bitDepth)-1,

[0305] (predSamplesLA*wValue+

[0306] predSamplesLB*(8-wValue)+offset1)>>shift1).

[0307] Here, predSamplesLA is a pixel value that has been motion compensated using the motion vector mvLA, and predSamplesLB is a pixel value that has been motion compensated using mvLB.

[0308] <Storage Processing in Coded Information Storage Memory>

[0309] When encoding and decoding, the inter-frame prediction information obtained in the triangle merge mode is stored in the coding information memory so that it can be referenced as the inter-frame prediction information around the object block. The storage process to the coding information storage memory is performed in units of 4×4 sub-blocks, and the inter-frame prediction information determined by each partition is stored.

[0310] The determined inter-frame prediction information is the inter-frame prediction information of the uni-prediction partition 0 (UNI0) and the uni-prediction partition 1 (UNI1), and the bi-prediction partition 2 (BI) becomes a partition using the inter-frame prediction information of UNI0 and UNI1.

[0311] In addition, for the sub-blocks on the diagonal line divided as partitions, Fig.43A and 43B In such a weighted case, consider Figures 44A to 44F Partition 2 like that. Fig.44A It will be carried out Fig.43A The weighted area is set to partition 2. Fig.44B The same corresponds to Fig.43B . Fig.44C It will be Fig.43A The weighted sub-block in is set to partition 2. Fig.44D The same corresponds to Fig.43B . Fig.44E Yes Fig.43A The sub-blocks in which the weighting is performed as a whole are regarded as the case of partition 2. Fig.44F The same corresponds to Fig.43B .

[0312] like Fig.45A and Fig.45B As shown, it can be Figures 44A to 44F The area of ​​partition 2 in is saved as BI, but in this embodiment, as Fig.46A and Fig.46B As shown, for a sub-block that is weighted for a portion of a sub-block, the inter-frame prediction information belonging to a partition having a larger weight value is saved as a single prediction (UNIY). Here, Y is 0 or 1.

[0313] And, if Figure 47A to Figure 47B and Figure 48A to Figure 48B As shown, the sub-block originally saved as BI is saved as single prediction by utilizing the inter-frame prediction information of the predetermined partition. Fig.47A and Fig.47B The case where UNI0 is the predetermined partition is shown. Fig.48A and Fig.48B The case where UNI1 is the predetermined partition is shown.

[0314] As shown in this embodiment, by saving the bi-predicted partition 2 (BI) as a single-predicted UNIY in the coding information storage memory, the amount of memory used for storage can be reduced, and each motion information of the motion information list L0 and the motion information list L1 is not converted into BI prediction and saved in the coding information storage memory, so the processing amount can be reduced. In addition, when the inter-frame prediction information determined by the delta merge mode is referenced and used in subsequent encoding and decoding, the processing amount can be reduced.

[0315] Through Fig.48A and Fig.48B Select UNI1 as the scheduled partition. Fig.48B The right adjacent block of the triangle merging pattern becomes Fig.48A In the case of the triangle merging mode, the continuity of the saved motion information can also be maintained in the area where two triangle merging modes are continuous, so it is similar to Fig.47A and Fig.47B Compared with selecting UNI0 as the predetermined partition, the coding efficiency is improved.

[0316] In addition, regarding Fig.48A and Fig.48B In this way, UNI1 is selected as the predetermined partition. Since the motion information of the partition is stored after encoding / decoding, the continuity of processing can be maintained, which is similar to the case of Fig.47A and Fig.47B Compared with selecting UNI0 as the predetermined partition, the processing volume is reduced.

[0317] Furthermore, by fixing the uni-prediction stored as UNI1 to the L0 prediction, it can be processed in the same manner as the P slice, thereby reducing the amount of processing compared to the case of storing it as the L1 prediction.

[0318] All the above-described embodiments may also be combined in multiples.

[0319] In all the above-mentioned embodiments, the bit stream output by the image encoding device has a specific data format so that it can be decoded according to the encoding method used in the embodiment. In addition, the image decoding device corresponding to the image encoding device can decode the bit stream of the specific data format.

[0320] In the case of using a wired or wireless network for exchanging a bit stream between an image encoding device and an image decoding device, the bit stream may be converted into a data format suitable for the transmission form of the communication line for transmission. In this case, a transmitting device is provided, which converts the bit stream output from the image encoding device into coded data in a data format suitable for the transmission form of the communication line and transmits the coded data to the network; and a receiving device is provided, which receives the coded data from the network and restores the coded data to a bit stream and provides it to the image decoding device. The transmitting device includes: a memory that buffers the bit stream output from the image encoding device; a packet processing unit that packets the bit stream; and a transmitting unit that transmits the packetized coded data via the network. The receiving device includes: a receiving unit that receives the packetized coded data via the network; a memory that buffers the received coded data; and a packet processing unit that performs packet processing on the coded data to generate a bit stream, and provides the bit stream to the image decoding device.

[0321] In addition, a display unit for displaying the image decoded by the image decoding device may be added to the configuration to serve as a display device. In this case, the display unit reads the decoded image signal generated by the decoded image signal superimposing unit 207 and stored in the decoded image memory 208, and displays it on the screen.

[0322] In addition, a photographing unit may be added to the structure, and the photographed image may be input to the image encoding device, thereby serving as the photographing device. In this case, the photographing unit inputs the photographed image signal to the block division unit 101 .

[0323] Fig.37 An example of the hardware structure of the coding and decoding device of this embodiment is shown. The coding and decoding device includes the structure of the image coding device and the image decoding device involved in the embodiment of the present invention. The coding and decoding device 9000 includes a CPU 9001, a codec IC 9002, an I / O interface 9003, a memory 9004, an optical disk drive 9005, a network interface 9006, and a video interface 9009, and each part is connected via a bus 9010.

[0324] The image coding unit 9007 and the image decoding unit 9008 are typically implemented as a codec IC 9002. The image coding process of the image coding device according to the embodiment of the present invention is performed by the image coding unit 9007, and the image decoding process in the image decoding device according to the embodiment of the present invention is performed by the image decoding unit 9008. The I / O interface 9003 is realized by, for example, a USB interface, and is connected to an external keyboard 9104, a mouse 9105, etc. The CPU 9001 controls the coding and decoding device 9000 based on the user operation input through the I / O interface 9003 so that the user's desired action is performed. As the operation performed by the user through the keyboard 9104, the mouse 9105, etc., there are selection of which function of coding and decoding is to be performed, setting of coding quality, input and output destination of bit stream, input and output destination of image, etc.

[0325] When a user wishes to reproduce an image recorded in the disk recording medium 9100, the optical disk drive 9005 reads a bit stream from the inserted disk recording medium 9100 and transmits the read bit stream to the image decoding unit 9008 of the codec IC 9002 via the bus 9010. The image decoding unit 9008 performs image decoding processing in the image decoding device according to the embodiment of the present invention on the input bit stream, and transmits the decoded image to the external monitor 9103 via the video interface 9009. In addition, the coding and decoding device 9000 has a network interface 9006 and can be connected to an external distribution server 9106 and a portable terminal 9107 via a network 9101. When a user wishes to reproduce an image recorded on the distribution server 9106 or the mobile terminal 9107 instead of the image recorded on the disk recording medium 9100, the network interface 9006 obtains a bit stream from the network 9101 instead of reading the bit stream from the input disk recording medium 9100. Furthermore, when a user wishes to reproduce an image recorded in the memory 9004 , the image decoding process in the image decoding device according to the embodiment of the present invention is performed on the bit stream recorded in the memory 9004 .

[0326] In the case of an operation in which a user wishes to encode an image captured by an external camera 9102 and record it in the memory 9004, the video interface 9009 inputs the image from the camera 9102 and sends it to the image encoding unit 9007 of the codec IC 9002 via the bus 9010. The image encoding unit 9007 performs the image encoding process in the image encoding device involved in the embodiment of the present invention on the image input via the video interface 9009, and generates a bit stream. Then, the bit stream is sent to the memory 9004 via the bus 9010. When the user wishes to record the bit stream on the disk recording medium 9100 instead of the memory 9004, the optical disk drive 9005 writes the bit stream to the inserted disk recording medium 9100.

[0327] A hardware configuration having an image encoding device but not an image decoding device or a hardware configuration having an image decoding device but not an image encoding device may be realized. Such a hardware configuration may be realized by replacing the codec IC 9002 with the image encoding unit 9007 or the image decoding unit 9008, for example.

[0328] The processing related to the above encoding and decoding can of course be implemented as a transmission, storage and reception device using hardware, and can be implemented by firmware stored in a ROM (read only memory), flash memory, etc. or software of a computer, etc. The firmware program or software program can be provided by recording it in a recording medium readable by a computer, etc., or provided from a server via a wired or wireless network, or provided as a data broadcast of a terrestrial wave or satellite digital broadcast.

[0329] The present invention has been described above based on the embodiments. The embodiments are for illustration only, and it is understood by those skilled in the art that various modifications may be made to the combinations of the constituent elements and the processing steps, and such modifications are also within the scope of the present invention.

[0330] Industrial Applicability

[0331] The present invention can be used in image encoding and decoding technology for dividing an image into blocks and performing prediction.

[0332] Explanation of symbols

[0333] 100…image encoding device, 101…block division unit, 102…inter-frame prediction unit, 103…intra-frame prediction unit, 104…decoded image memory, 105…prediction method determination unit, 106…residual generation unit, 107…orthogonal transformation / quantization unit, 108…bit string encoding unit, 109…inverse quantization / inverse orthogonal transformation unit, 110…decoded image signal superposition unit, 111…coding information storage memory, 200…image decoding device, 201…bit string decoding unit, 202…block division unit, 203 inter-frame prediction unit, 204…intra-frame prediction unit, 205…coding information storage memory, 206…inverse quantization / inverse orthogonal transformation unit, 207…decoded image signal superposition unit, 208…decoded image memory.< / poc>

Claims

1. A moving picture encoding device, the moving picture encoding device using a delta merging mode, characterized in that: include: A triangle merging candidate list building unit, which builds a triangle merging candidate list including spatial merging candidates; A first triangle merge candidate selection unit, which selects a first triangle merge candidate that becomes a single prediction from the triangle merge candidate list, wherein the first triangle merge candidate is used for a triangle partition on the upper side of a diagonal line of the divided coding block; A second triangle merge candidate selection unit, which selects a second triangle merge candidate that becomes a single prediction from the triangle merge candidate list, wherein the second triangle merge candidate is used for a triangle partition below a diagonal line of the divided coding block; as well as an encoding unit, encoding a first triangle merging index indicating the selected first triangle merging candidate and a second triangle merging index indicating the selected second triangle merging candidate, In a dual prediction partition which is an area for performing motion compensation based on a weighted average of the first triangle merge candidate and the second triangle merge candidate and which is a sub-block on a diagonal of a coding block, single prediction motion information of the second triangle merge candidate is stored in a coding information storage memory, so that the single prediction motion information can be used as inter-frame prediction information around the coding block without storing dual prediction information transformed by a first motion information list and a second motion information list.

2. A moving picture encoding method, the moving picture encoding method using a delta merging mode, characterized in that: include: A triangle merging candidate list building step, building a triangle merging candidate list including spatial merging candidates; A first triangle merge candidate selection step, selecting a first triangle merge candidate for single prediction from the triangle merge candidate list, wherein the first triangle merge candidate is used for a triangle partition on the upper side of the diagonal line of the divided coding block; A second triangle merge candidate selection step, selecting a second triangle merge candidate that becomes a single prediction from the triangle merge candidate list, wherein the second triangle merge candidate is used for a triangle partition below a diagonal line of the divided coding block; as well as an encoding step of encoding a first triangle merging index representing the selected first triangle merging candidate and a second triangle merging index representing the selected second triangle merging candidate, In a dual prediction partition which is an area for performing motion compensation based on a weighted average of the first triangle merge candidate and the second triangle merge candidate and which is a sub-block on a diagonal of a coding block, single prediction motion information of the second triangle merge candidate is stored in a coding information storage memory, so that the single prediction motion information can be used as inter-frame prediction information around the coding block without storing dual prediction information transformed by a first motion information list and a second motion information list.

3. A moving picture decoding device, the moving picture decoding device using a delta merging mode, characterized in that: include: A decoding unit, decoding a first triangle merge index indicating a first triangle merge candidate and a second triangle merge index indicating a second triangle merge candidate; A triangle merging candidate list building unit, which builds a triangle merging candidate list including spatial merging candidates; A first triangle merge candidate selection unit, which selects a first triangle merge candidate that becomes a single prediction from the triangle merge candidate list, wherein the first triangle merge candidate is used to divide the triangle partition on the upper side of the diagonal line of the coded block; as well as A second triangle merge candidate selection unit selects a second triangle merge candidate that becomes a single prediction from the triangle merge candidate list, wherein the second triangle merge candidate is used for a triangle partition below a diagonal line of the segmented coding block. In a dual prediction partition which is an area for performing motion compensation based on a weighted average of the first triangle merge candidate and the second triangle merge candidate and which is a sub-block on a diagonal of a coding block, single prediction motion information of the second triangle merge candidate is stored in a coding information storage memory, so that the single prediction motion information can be used as inter-frame prediction information around the coding block without storing dual prediction information transformed by a first motion information list and a second motion information list.

4. A moving picture decoding method, the moving picture decoding method using a delta merging mode, characterized in that: include: A decoding step of decoding a first triangle merge index representing a first triangle merge candidate and a second triangle merge index representing a second triangle merge candidate; A triangle merging candidate list building step, building a triangle merging candidate list including spatial merging candidates; A first triangle merge candidate selection step, selecting a first triangle merge candidate for single prediction from the triangle merge candidate list, wherein the first triangle merge candidate is used for a triangle partition on the upper side of the diagonal line of the divided coding block; as well as A second triangle merge candidate selection step is performed, wherein a second triangle merge candidate that becomes a single prediction is selected from the triangle merge candidate list, wherein the second triangle merge candidate is used for a triangle partition below the diagonal line of the segmented coding block. In a dual prediction partition which is an area for performing motion compensation based on a weighted average of the first triangle merge candidate and the second triangle merge candidate and which is a sub-block on a diagonal of a coding block, single prediction motion information of the second triangle merge candidate is stored in a coding information storage memory, so that the single prediction motion information can be used as inter-frame prediction information around the coding block without storing dual prediction information transformed by a first motion information list and a second motion information list.

5. A storage method, wherein: The bit stream generated by the image encoding method according to claim 2 is stored in a storage medium.

6. A transmission method, wherein: The bit stream generated by the image encoding method according to claim 2 is transmitted.

Images