Image encoding apparatus, image encoding method, image decoding apparatus, image decoding method

Abstract

The present application relates to image encoding apparatus, image encoding method, and image decoding apparatus, image decoding method, including: merge candidate construction unit, constructing merge candidate list, merge candidate list has the maximum number of merge candidates including spatial merge candidate; normal merge candidate selection unit, according to merge index, select normal merge candidate from merge candidate list, which becomes single prediction or double prediction; and triangle merge candidate selection unit, according to first triangle merge index, select first triangle merge candidate from merge candidate list, which becomes single prediction, according to second triangle merge index, select second triangle merge candidate, which becomes single prediction, the first maximum number of triangle merge candidates related to first triangle merge index is the same as the maximum number of merge candidates.

Description

[0001] This application is a divisional application based on an application No. 202080019439.2, filed on March 6, 2020, applicant Intellectual Property Asset Management Co., Ltd., with an invention name of "Image encoding device, image encoding method, and image encoding program, image decoding device, image decoding method, and image decoding program". TECHNICAL FIELD

[0002] The present application relates to an image encoding and decoding technology that divides an image into blocks and performs prediction. BACKGROUND

[0003] In image encoding and decoding, an image as a processing target is divided into a predetermined number of pixel sets, that is, blocks, and processing is performed in units of blocks. By dividing into appropriate blocks, intra prediction (intra prediction), inter prediction (inter prediction) is appropriately set, and the encoding efficiency is improved.

[0004] In encoding / decoding of moving images, the encoding efficiency is improved by inter prediction that predicts from an already encoded / decoded picture. Patent Literature 1 describes a technology that applies an affine transformation at the time of inter prediction. In moving images, there are many cases where a deformation such as enlargement, reduction, and rotation of an object occurs, and by applying the technology of Patent Literature 1, high-efficiency encoding can be performed.

[0005] PRIOR ART DOCUMENT

[0006] PATENT LITERATURE

[0007] Patent Literature 1: Japanese Patent Application Laid-Open No. 9-172644 SUMMARY

[0008] However, since the technology of Patent Literature 1 involves a transformation of an image, there is a problem that the processing load is large. The present application provides a low-load and high-efficiency encoding technology in view of the above problem.

[0009] In a certain aspect of the present application that solves the above problem, there is provided an image encoding device including: a merge candidate construction section that constructs a merge candidate list having a maximum number of merge candidates including a spatial merge candidate; a normal merge candidate selection section that selects a normal merge candidate that becomes a uni-prediction or bi-prediction from the merge candidate list according to a merge index; and a triangle merge candidate selection section that selects a first triangle merge candidate that becomes a uni-prediction from the merge candidate list according to a first triangle merge index, and selects a second triangle merge candidate that becomes a uni-prediction according to a second triangle merge index, and a first maximum number of triangle merge candidates related to the first triangle merge index is the same as the maximum number of merge candidates.

[0010] According to the present application, high-efficiency image encoding / decoding processing can be realized at low load. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a block diagram of an image encoding apparatus to which an embodiment of the present application relates;

[0012] Figure 2 is a block diagram of an image decoding apparatus to which an embodiment of the present application relates;

[0013] Figure 3 is a flowchart for explaining the action of dividing a tree block;

[0014] Figure 4 is a diagram showing a case where an input image is divided into tree blocks;

[0015] Figure 5 is a diagram explaining z-scan;

[0016] Figure 6A is a diagram showing a division shape of a block;

[0017] Figure 6B is a diagram showing a division shape of a block;

[0018] Fig. 6C is a diagram showing a division shape of a block;

[0019] Figure 6D is a diagram showing a division shape of a block;

[0020] Figure 6E is a diagram showing a division shape of a block;

[0021] Figure 7 is a flowchart for explaining the action of 4-dividing a block;

[0022] Figure 8 is a flowchart for explaining the action of 2-dividing or 3-dividing a block;

[0023] Figure 9 is a syntax for describing a shape of block division;

[0024] Figure 10A is a diagram for explaining intra prediction;

[0025] Figure 10B is a diagram for explaining intra prediction;

[0026] Figure 11 is a diagram for explaining a reference block of inter prediction;

[0027] Figure 12A is a syntax for describing a prediction mode of an encoding block;

[0028] Figure 12Bis a syntax for describing a prediction mode of a coding block;

[0029] Figure 13 is a diagram showing a correspondence between syntax elements related to inter prediction and modes;

[0030] Figure 14 is a diagram for explaining affine transform motion compensation in which the control points are two points;

[0031] Figure 15 is a diagram for explaining affine transform motion compensation in which the control points are three points;

[0032] Figure 16 is Figure 1 a block diagram of a detailed structure of the inter prediction section 102 of

[0033] Figure 17 is Figure 16 a block diagram of a detailed structure of the normal prediction motion vector mode derivation section 301 of

[0034] Figure 18 is Figure 16 a block diagram of a detailed structure of the normal merge mode derivation section 302 of

[0035] Figure 19 is a flowchart for explaining a normal prediction motion vector mode derivation process of the normal prediction motion vector mode derivation section 301 of Figure 16

[0036] Figure 20 is a flowchart showing processing steps of the normal prediction motion vector mode derivation process;

[0037] Figure 21 is a flowchart showing processing steps of the normal merge mode derivation process;

[0038] Figure 22 is a block diagram of a detailed structure of the inter prediction section 203 of Figure 2

[0039] Figure 23 is a block diagram of a detailed structure of the normal prediction motion vector mode derivation section 401 of Figure 22

[0040] Figure 24 is a block diagram of a detailed structure of the normal merge mode derivation section 402 of Figure 22

[0041] Figure 25 is a flowchart for explaining a normal prediction motion vector mode derivation process of the normal prediction motion vector mode derivation section 401 of Figure 22

[0042] ​​​​​Figure 26 is a diagram illustrating a history-based motion vector predictor candidate list initialization / update process step;

[0043] Figure 27 is a flowchart of a same element check process step in the history-based motion vector predictor candidate list initialization / update process step;

[0044] Figure 28 is a flowchart of an element shift process step in the history-based motion vector predictor candidate list initialization / update process step;

[0045] Figure 29 is a flowchart illustrating a history-based motion vector predictor derivation process step;

[0046] Figure 30 is a flowchart illustrating a history-based merge candidate derivation process step;

[0047] Figure 31A is a diagram for explaining an example of history-based motion vector predictor candidate list update process;

[0048] Figure 31B is a diagram for explaining an example of history-based motion vector predictor candidate list update process;

[0049] Figure 31C is a diagram for explaining an example of history-based motion vector predictor candidate list update process;

[0050] Figure 32 is a diagram for explaining motion-compensated prediction in a case where a reference picture of L0 (RefL0Pic) in L0 prediction is at a time point before a processing target picture (CurPic) in L0 prediction;

[0051] Figure 33 is a diagram for explaining motion-compensated prediction in a case where a reference picture of L0 prediction is at a time point after a processing target picture in L0 prediction;

[0052] Figure 34 is a diagram for explaining a prediction direction of motion-compensated prediction in a case where a reference picture of L0 prediction is at a time point before a processing target picture and a reference picture of L1 prediction is at a time point after the processing target picture in bi-prediction;

[0053] Figure 35 is a diagram for explaining a prediction direction of motion-compensated prediction in a case where a reference picture of L0 prediction and a reference picture of L1 prediction are at a time point before a processing target picture in bi-prediction;

[0054] Figure 36is a diagram for explaining a prediction direction of motion compensated prediction in a case where a reference picture of L0 prediction and a reference picture of L1 prediction are after a time point of a picture being processed;

[0055] Figure 37 is a diagram for explaining an example of a hardware structure of an encoding and decoding apparatus of an embodiment of the present application;

[0056] Figure 38A is a diagram for explaining prediction of a triangle merge mode;

[0057] Figure 38B is a diagram for explaining prediction of a triangle merge mode;

[0058] Figure 39 is a flowchart for explaining an average merge candidate derivation processing step;

[0059] Figure 40 is a flowchart for explaining a triangle merge candidate derivation;

[0060] Figure 41 is a flowchart for explaining derivation of single prediction motion information of a merge triangle partition 0 of the first embodiment;

[0061] Figure 42 is a flowchart for explaining derivation of single prediction motion information of a merge triangle partition 1 of the first embodiment;

[0062] Figure 43A is a diagram for explaining an example of motion information of a triangle merge candidate of the first embodiment;

[0063] Figure 43B is a diagram for explaining an example of motion information of a triangle merge candidate of the first embodiment;

[0064] Figure 44 is a flowchart for explaining derivation of single prediction motion information of a merge triangle partition 0 of the second embodiment;

[0065] Figure 45 is a flowchart for explaining derivation of single prediction motion information of a merge triangle partition 1 of the second embodiment;

[0066] Figure 46A is a diagram for explaining an example of motion information of a triangle merge candidate of the second embodiment;

[0067] Figure 46B is a diagram for explaining an example of motion information of a triangle merge candidate of the second embodiment. DETAILED DESCRIPTION

[0068] Definitions of technical terms and technical terms used in the present embodiment are defined.

[0069] <tree block>

[0070] In an embodiment, an encoding / decoding processing target image is equally divided in a predetermined size. This unit is defined as a tree block. In Figure 4 In an embodiment, the size of a tree block is set to 128 x 128 pixels, but the size of a tree block is not limited to this, and an arbitrary size can be set. A tree block as a processing target (corresponding to an encoding target in an encoding process, and corresponding to a decoding target in a decoding process) is switched in a raster scan order, i.e., from left to right and from top to bottom. The inside of each tree block can be further recursively divided. A block that becomes an encoding / decoding target after a tree block is recursively divided is defined as an encoding block. In addition, a tree block and an encoding block are collectively defined as a block. By performing appropriate block division, efficient encoding can be performed. The size of a tree block can be a fixed value decided in advance in an encoding apparatus and a decoding apparatus, or a structure in which the size of a tree block decided by an encoding apparatus is transmitted to a decoding apparatus can be adopted. Here, the maximum size of a tree block is set to 128 x 128 pixels, and the minimum size of a tree block is set to 16 x 16 pixels. In addition, the maximum size of an encoding block is set to 64 x 64 pixels, and the minimum size of an encoding block is set to 4 x 4 pixels.

[0071] <Prediction mode>

[0072] Intra prediction (MODE_INTRA) that is predicted from a processed image signal of a processing target image and inter prediction (MODE_INTER) that is predicted from an image signal of a processed image are switched in units of a processing target encoding block.

[0073] A processed image is an image, an image signal, a tree block, a block, an encoding block, and the like obtained by decoding a signal for which encoding is completed in an encoding process, and is used for an image, an image signal, a tree block, a block, an encoding block, and the like for which decoding is completed in a decoding process.

[0074] A mode that identifies the intra prediction (MODE_INTRA) and the inter prediction (MODE_INTER) is defined as a prediction mode (PredMode). The prediction mode (PredMode) represents the intra prediction (MODE_INTRA) or the inter prediction (MODE_INTER) in the form of a value.

[0075] <Inter prediction>

[0076] In inter prediction in which a picture signal of a processed picture is used for prediction, a plurality of processed pictures can be used as reference pictures. In order to manage the plurality of reference pictures, two kinds of reference lists, L0 (reference list 0) and Ll (reference list 1), are defined, and a reference index is used to specify a reference picture. In a P slice, L0 prediction (Pred_L0) can be used. In a B slice, L0 prediction (Pred_L0), Ll prediction (Pred_Ll), and bi-prediction (Pred_BI) can be used. The L0 prediction (Pred_L0) is inter prediction that refers to a reference picture managed by L0, and the Ll prediction (Pred_Ll) is inter prediction that refers to a reference picture managed by Ll. The bi-prediction (Pred_BI) is inter prediction that simultaneously performs L0 prediction and Ll prediction and refers to one by one reference pictures managed by each of L0 and Ll. Information that specifies the L0 prediction, the Ll prediction, and the bi-prediction is defined as an inter prediction mode. With respect to constants, variables, and the like to which a subscript LX is attached in output in later processing, it is assumed that processing is performed on L0, Ll.

[0077] <Prediction Motion Vector Mode>

[0078] The prediction motion vector mode is a mode that transmits an index for specifying a prediction motion vector, a difference motion vector, an inter prediction mode, a reference index, and decides inter prediction information of a processing target block. The prediction motion vector is derived from a prediction motion vector candidate derived from a processed block adjacent to the processing target block or a block belonging to a processed picture and located at the same position as or in the vicinity (neighbor) of the processing target block, and the index for specifying the prediction motion vector.

[0079] <Merge Mode>

[0080] The merge mode is a mode that does not transmit a difference motion vector and a reference index, and derives inter prediction information of a processing target block from inter prediction information of a processed block adjacent to the processing target block or a block belonging to a processed picture and located at the same position as or in the vicinity (neighbor) of the processing target block.

[0081] A processed block adjacent to a processing target block and inter prediction information of the processed block are defined as a spatial merge candidate. A block belonging to a processed picture and located at the same position as or in the vicinity (neighbor) of the processing target block, and inter prediction information derived from the inter prediction information of the block are defined as a temporal merge candidate. Each of the merge candidates is registered in a merge candidate list, and a merge candidate used in prediction of the processing target block is specified by a merge index.

[0082] <Neighbor Block>

[0083] Figure 11is a diagram illustrating a reference block referred to for deriving inter prediction information in the predictive motion vector mode and the merge mode. A0, A1, A2, B0, B1, B2, B3 are processed blocks adjacent to the processing target block. T0 is a block located in the same position as the processing target block in the processing target image or in the vicinity (adjacent) thereof among the processed blocks belonging to the processed image.

[0084] A1, A2 are blocks located on the left side of the processing target coding block and adjacent to the processing target coding block. B1, B3 are blocks located on the upper side of the processing target coding block and adjacent to the processing target coding block. A0, B0, B2 are blocks located on the lower left, upper right, and upper left of the processing target coding block, respectively.

[0085] Details of how to process the adjacent blocks in the predictive motion vector mode and the merge mode are described later.

[0086] <Affine Motion Compensation>

[0087] The affine motion compensation is a method of dividing a coding block into sub-blocks of a predetermined unit and individually determining a motion vector for each of the divided sub-blocks to perform motion compensation. The motion vector of each sub-block is derived based on one or more control points derived from inter prediction information of a processed block adjacent to the processing target block or a block located in the same position as the processing target block or in the vicinity (adjacent) thereof among the blocks belonging to the processed image. In the present embodiment, the size of the sub-block is set to 4 x 4 pixels, but the size of the sub-block is not limited thereto and the motion vector can be derived in units of pixels.

[0088] Figure 14 An example of the affine motion compensation when the number of control points is two is shown. In this case, the two control points have two parameters of the horizontal direction component and the vertical direction component. Therefore, the affine transformation when the number of control points is two is referred to as four-parameter affine transformation. Figure 14 CP1, CP2 of are control points.

[0089] Figure 15 An example of the affine motion compensation when the number of control points is three is shown. In this case, the three control points have two parameters of the horizontal direction component and the vertical direction component. Therefore, the affine transformation when the number of control points is three is referred to as six-parameter affine transformation. Figure 15 CP1, CP2, CP3 of are control points.

[0090] The affine motion compensation can be used in either of the predictive motion vector mode and the merge mode. The mode in which the affine motion compensation is applied in the predictive motion vector mode is defined as the sub-block predictive motion vector mode, and the mode in which the affine motion compensation is applied in the merge mode is defined as the sub-block merge mode.

[0091] < Syntax of a coding block >

[0092] The syntax for describing the prediction mode of a coding block is explained using Figure 12A , Figure 12B and Figure 13 . Figure 12A pred_mode_flag is a flag indicating whether it is inter prediction or not. If pred_mode_flag is 0, it is inter prediction, and if pred_mode_flag is 1, it is intra prediction. In the case of intra prediction, information of intra prediction, intra_pred_mode, is transmitted, and in the case of inter prediction, merge_flag is transmitted. Merge_flag is a flag indicating whether it is merge mode or predicted motion vector mode. In the case of predicted motion vector mode (merge_flag = 0), inter_affine_flag is transmitted, which is a flag indicating whether sub-block predicted motion vector mode is applied or not. If sub-block predicted motion vector mode is applied (inter_affine_flag = 1), cu_affine_type_flag is transmitted. Cu_affine_type_flag is a flag for deciding the number of control points in sub-block predicted motion vector mode.

[0093] On the other hand, in the case of merge mode (merge_flag = 1), merge_subblock_flag of Figure 12B is transmitted. Merge_subblock_flag is a flag indicating whether sub-block merge mode is applied or not. In the case of sub-block merge mode (merge_subblock_flag = 1), merge_subblock_idx is transmitted. On the other hand, in the case of not sub-block merge mode (merge_subblock_flag = 0), merge_triangle_flag is transmitted, which is a flag indicating whether triangle merge mode is applied or not. If triangle merge mode is applied (merge_triangle_flag = 1), merge_triangle_idx0, merge_triangle_idx1 of merge triangle index for the direction of split block merge_triangle_split_dir and two partitions which are split are transmitted. On the other hand, if triangle merge mode is not applied (merge_triangle_flag = 0), merge_idx of merge index is transmitted.

[0094] Figure 13The values of the syntax elements and the corresponding prediction modes are shown. merge_flag = 0, inter affine flag = 0 corresponds to the normal prediction motion vector mode (Inter Pred Mode). merge_flag = 0, inter affine flag = 1 corresponds to the subblock prediction motion vector mode (Inter Affine Mode). merge_flag = 1, merge_subblock_flag = 0, merge_trianlge_flag = 0 corresponds to the normal merge mode (Merge Mode). merge_flag = 1, merge_subblock_flag = 0, merge_trianlge_flag = 1 corresponds to the triangle merge mode (Triangle Merge Mode). merge_flag = 1, merge_subblock_flag = 1 corresponds to the subblock merge mode (Affine Merge Mode).

[0095] <poc>

[0096] POC (Picture Order Count, picture sequence number) is a variable associated with a picture to be encoded, and is set to an incrementally 1 value corresponding to the output order of the picture. According to the value of the POC, it is possible to discriminate whether it is the same picture, discriminate the order of the pictures in the output order, derive the distance between the pictures. For example, if the POCs of two pictures have the same value, it can be judged that it is the same picture. In the case where the POCs of two pictures have different values, it is possible to judge that the picture whose value of the POC is small is the picture which is output first, and the difference between the POCs of the two pictures represents the distance between the pictures in the time axis direction.

[0097] (First Embodiment)

[0098] An image encoding apparatus 100 and an image decoding apparatus 200 according to a first embodiment of the present application will be described.

[0099] Figure 1 is a block diagram of the image encoding apparatus 100 according to the first embodiment. The image encoding apparatus 100 according to the embodiment includes a block division section 101, an inter prediction section 102, an intra prediction section 103, a decoded image memory 104, a prediction method decision section 105, a residual generation section 106, an orthogonal transform / quantization section 107, a bit string encoding section 108, an inverse quantization / inverse orthogonal transform section 109, a decoded image signal superimposition section 110, and an encoding information holding memory 111.

[0100] The block division section 101 recursively divides an image which has been input to generate a coding block. The block division section 101 includes a 4-division section which divides a block which is a division object in the horizontal direction and the vertical direction respectively, and a 2-3-division section which divides a block which is a division object in either one of the horizontal direction and the vertical direction. The block division section 101 sets the generated coding block as a processing target coding block, and supplies an image signal of the processing target coding block to the inter prediction section 102, the intra prediction section 103, and the residual generation section 106. In addition, the block division section 101 supplies information indicating a decided recursive division structure to the bit string encoding section 108. The detailed action of the block division section 101 will be described later.

[0101] The inter prediction section 102 performs inter prediction of the processing target coding block. The inter prediction section 102 derives a plurality of candidates of inter prediction information from the inter prediction information stored in the coding information storage memory 111 and the decoded image signal stored in the decoded image storage memory 104, selects an appropriate inter prediction mode from the derived plurality of candidates, and supplies the selected inter prediction mode and a prediction image signal corresponding to the selected inter prediction mode to the prediction method decision section 105. The detailed structure and operation of the inter prediction section 102 will be described later.

[0102] The intra prediction section 103 performs intra prediction of the processing target coding block. The intra prediction section 103 generates a prediction image signal by intra prediction based on the coding information such as the intra prediction mode stored in the coding information storage memory 111, with reference to the decoded image signal stored in the decoded image storage memory 104 as a reference pixel. In the intra prediction, the intra prediction section 103 selects an appropriate intra prediction mode from a plurality of intra prediction modes, and supplies the selected intra prediction mode and a prediction image signal corresponding to the selected intra prediction mode to the prediction method decision section 105.

[0103] Figure 10A and Figure 10B An example of intra prediction is shown. Figure 10A is a diagram showing the correspondence relationship between the prediction direction of intra prediction and the intra prediction mode number. For example, the intra prediction mode 50 generates an intra prediction image by copying the reference pixel in the vertical direction. The intra prediction mode 1 is a DC mode, and is a mode in which all the pixel values of the processing target block are set to the average value of the reference pixel. The intra prediction mode 0 is a Planar mode (two-dimensional mode), and is a mode in which a two-dimensional intra prediction image is generated from the reference pixels in the vertical and horizontal directions. Figure 10B is an example of an intra prediction image in the case where the intra prediction mode 40 is generated. The intra prediction section 103 copies the value of the reference pixel in the direction indicated by the intra prediction mode to each pixel of the processing target block. In the case where the reference pixel in the intra prediction mode is not an integer position, the intra prediction section 103 decides the reference pixel value by interpolation from the reference pixel values at the surrounding integer positions.

[0104] The decoded image storage memory 104 stores the decoded image generated by the decoded image signal superimposition section 110. The decoded image storage memory 104 supplies the stored decoded image to the inter prediction section 102 and the intra prediction section 103.

[0105] The prediction method decision section 105 decides the optimum prediction mode by evaluating each of the intra prediction and the inter prediction using the code amount of the encoding information and the residual, the distortion amount between the prediction image signal and the processing target image signal, and the like. In the case of the intra prediction, the prediction method decision section 105 provides the intra prediction information such as the intra prediction mode to the bit string encoding section 108 as the encoding information. In the case of the merge mode of the inter prediction, the prediction method decision section 105 provides the inter prediction information such as the merge index, information indicating whether it is the sub-block merge mode (sub-block merge flag), and the like to the bit string encoding section 108 as the encoding information. In the case of the prediction motion vector mode of the inter prediction, the prediction method decision section 105 provides the inter prediction information such as the inter prediction mode, the prediction motion vector index, the reference index of L0 and L1, the difference motion vector, and information indicating whether it is the sub-block prediction motion vector mode (sub-block prediction motion vector flag) to the bit string encoding section 108 as the encoding information. Further, the prediction method decision section 105 provides the decided encoding information to the encoding information holding memory 111. The prediction method decision section 105 provides the prediction image signal to the residual generation section 106 and the decoded image signal superimposition section 110.

[0106] The residual generation section 106 generates a residual by subtracting the prediction image signal from the processing target image signal, and provides it to the orthogonal transform / quantization section 107.

[0107] The orthogonal transform / quantization section 107 performs orthogonal transform and quantization on the residual according to the quantization parameter, to generate an orthogonal transform / quantized residual, and provides the generated residual to the bit string encoding section 108 and the inverse quantization / inverse orthogonal transform section 109.

[0108] The bit string encoding section 108 encodes, for each coding block, the coding information corresponding to the prediction method decided by the prediction method decision section 105, in addition to the information of the sequence, picture, slice, and coding block unit. Specifically, the bit string encoding section 108 encodes the prediction mode PredMode for each coding block. In the case where the prediction mode is inter prediction (MODE INTER), the bit string encoding section 108 encodes, in accordance with a prescribed syntax (syntax rule of bit string), the coding information (inter prediction information) such as a flag that discriminates whether it is a merge mode, a sub-block merge flag, a merge index in the case of the merge mode, an inter prediction mode in the case of not the merge mode, a prediction motion vector index, information related to a differential motion vector, a sub-block prediction motion vector flag, and the like, and generates a first bit string. In the case where the prediction mode is intra prediction (MODE INTRA), the bit string encoding section 108 encodes, in accordance with a prescribed syntax (syntax rule of bit string), the coding information (intra prediction information) such as an intra prediction mode, and generates a first bit string. In addition, the bit string encoding section 108 entropy-encodes, in accordance with a prescribed syntax, the residual after orthogonal transform and quantization, and generates a second bit string. The bit string encoding section 108 multiplexes the first bit string and the second bit string in accordance with a prescribed syntax, and outputs a bit stream.

[0109] The inverse quantization / inverse orthogonal transform section 109 inverse-quantizes and inverse-orthogonally transforms the orthogonal-transformed / quantized residual provided from the orthogonal transform / quantization section 107, to calculate a residual, and provides the calculated residual to the decoded image signal superimposition section 110.

[0110] The decoded image signal superimposition section 110 superimposes the prediction image signal corresponding to the decision of the prediction method decision section 105 and the residual inverse-quantized and inverse-orthogonally transformed by the inverse quantization / inverse orthogonal transform section 109, to generate a decoded image, and stores the decoded image in the decoded image memory 104. Further, the decoded image signal superimposition section 110 can perform a filter process for reducing block distortion and the like caused by encoding on the decoded image, and store the decoded image in the decoded image memory 104.

[0111] The encoding information holding memory 111 holds encoding information such as a prediction mode (inter prediction or intra prediction) decided by the prediction method decision section 105. In the case of inter prediction, the encoding information held in the encoding information holding memory 111 includes the decided motion vector, the reference index of the reference list L0, L1, and the inter prediction information such as the history prediction motion vector candidate list. In addition, in the case of the merge mode of inter prediction, the encoding information held in the encoding information holding memory 111 includes the inter prediction information of the merge index, the information indicating whether it is the sub-block merge mode (sub-block merge flag) in addition to the above information. In addition, in the case of the prediction motion vector mode of inter prediction, the encoding information held in the encoding information holding memory 111 includes the inter prediction information of the inter prediction mode, the prediction motion vector index, the difference motion vector, the information indicating whether it is the sub-block prediction motion vector mode (sub-block prediction motion vector flag) in addition to the above information. In the case of intra prediction, the encoding information held in the encoding information holding memory 111 includes the intra prediction information of the decided intra prediction mode.

[0112] Figure 2 is a block of the image corresponding to the image encoding apparatus of Figure 1 The structure of the image decoding apparatus involved in the embodiment of the present application corresponding to the image encoding apparatus of

[0113] Figure 2 The decoding process of the image decoding apparatus of Figure 1 corresponds to the decoding process of the internal setting of the image encoding apparatus of Figure 2 Each structure of the encoding information holding memory 205, the inverse quantization / inverse orthogonal transform section 206, the decoded image signal superimposition section 207, and the decoded image memory 208 of the image decoding apparatus of Figure 1 corresponds to the function of each structure of the encoding information holding memory 111, the inverse quantization / inverse orthogonal transform section 109, the decoded image signal superimposition section 110, and the decoded image memory 104 of the image encoding apparatus of

[0114] The bitstream supplied to the bit string decoding section 201 is separated according to a prescribed syntax rule. The bit string decoding section 201 decodes the separated first bit string to obtain information of a sequence, a picture, a slice, a coding block unit, and coding information of the coding block unit. Specifically, the bit string decoding section 201 decodes a prediction mode PredMode in units of coding blocks, the prediction mode PredMode discriminating whether it is inter prediction (MODE INTER) or intra prediction (MODE INTRA). In the case where the prediction mode is inter prediction (MODE INTER), the bit string decoding section 201 decodes coding information (inter prediction information) on a flag discriminating whether it is merge mode, a merge index in the case of merge mode, a sub-block merge flag, an inter prediction mode in the case of a prediction motion vector mode, a prediction motion vector index, a difference motion vector, a sub-block prediction motion vector flag, and the like, according to a prescribed syntax, and supplies the coding information (inter prediction information) to the coding information holding memory 205 via the inter prediction section 203 and the block division section 202. In the case where the prediction mode is intra prediction (MODE INTRA), coding information (intra prediction information) on an intra prediction mode and the like is decoded according to a prescribed syntax, and the coding information (intra prediction information) is supplied to the coding information holding memory 205 via the inter prediction section 203 or the intra prediction section 204 and the block division section 202. The bit string decoding section 201 decodes the separated second bit string to calculate a residual after orthogonal transform / quantization, and supplies the residual after orthogonal transform / quantization to the inverse quantization / inverse orthogonal transform section 206.

[0115] When the prediction mode PredMode of the processing target coding block is a prediction motion vector mode in inter prediction (MODE INTER), the inter prediction section 203 derives a plurality of prediction motion vector candidates using the encoding information of the decoded image signal stored in the encoding information holding memory 205, and registers the derived plurality of prediction motion vector candidates in a prediction motion vector candidate list to be described later. The inter prediction section 203 selects a prediction motion vector corresponding to the prediction motion vector index decoded by the bit string decoding section 201 from among the plurality of prediction motion vector candidates registered in the prediction motion vector candidate list, calculates a motion vector from the difference motion vector decoded by the bit string decoding section 201 and the selected prediction motion vector, and saves the calculated motion vector in the encoding information holding memory 205 together with other encoding information. Here, the encoding information of the coding block to be provided / saved is the prediction mode PredMode, flags predFlagL0[xP][yP], predFlagL1[xP][yP] indicating whether or not L0 prediction and L1 prediction are used, reference indexes refIdxL0[xP][yP], refIdxL1[xP][yP] of L0, L1, motion vectors mvL0[xP][yP], mvL1[xP][yP] of L0, L1, and the like. Here, xP, yP are indexes indicating the position of the top-left pixel of the coding block in the picture. In the case where the prediction mode PredMode is inter prediction (MODE INTER) and the inter prediction mode is L0 prediction (Pred L0), the flag predFlagL0 indicating whether or not L0 prediction is used is 1, and the flag predFlagL1 indicating whether or not L1 prediction is used is 0. In the case where the inter prediction mode is L1 prediction (Pred L1), the flag predFlagL0 indicating whether or not L0 prediction is used is 0, and the flag predFlagL1 indicating whether or not L1 prediction is used is 1. In the case where the inter prediction mode is bi-prediction (Pred BI), both the flag predFlagL0 indicating whether or not L0 prediction is used and the flag predFlagL1 indicating whether or not L1 prediction is used are 1. Also, when the prediction mode PredMode of the processing target coding block is a merge mode in inter prediction (MODE INTER), a merge candidate is derived.Using the decoded block information stored in the encoding information holding memory 205, a plurality of merge candidates registered in a merge candidate list described later are derived, a merge candidate corresponding to a merge index provided by the bit string decoding section 201 is selected from the plurality of merge candidates registered in the merge candidate list, and the inter prediction information such as flags indicating whether or not L0 prediction and L1 prediction using the selected merge candidate, reference indexes refldxL0[xP][yP], refldxLl[xP][yP] of L0, L1, and motion vectors mvL0[xP][yP], mvLl[xP][yP] of L0, L1 are held in the encoding information holding memory 205. Here, xP, yP are indexes indicating the position of the top-left pixel of the coding block within the picture. The detailed configuration and operation of the inter prediction section 203 described later.

[0116] When the prediction mode PredMode of the coding block being processed is intra prediction (MODE_INTRA), the intra prediction section 204 performs intra prediction. The intra prediction mode is included in the decoded information decoded by the bit string decoding section 201. The intra prediction section 204 generates a prediction image signal by intra prediction in accordance with the intra prediction mode included in the decoded information decoded by the bit string decoding section 201 and in accordance with the decoded image signal held in the decoded image memory 208, and supplies the generated prediction image signal to the decoded image signal superimposition section 207. The intra prediction section 204 performs the same processing as the intra prediction section 103 of the image encoding apparatus 100.

[0117] The inverse quantization / inverse orthogonal transform section 206 performs inverse orthogonal transform and inverse quantization on the orthogonal transform / quantized residual after decoding by the bit string decoding section 201, to obtain an inverse orthogonal transform / inverse quantized residual.

[0118] The decoded image signal superimposition section 207 decodes the decoded image signal by superimposing the prediction image signal obtained by the inter prediction performed by the inter prediction section 203 or the prediction image signal obtained by the intra prediction performed by the intra prediction section 204 and the inverse orthogonal transform / inverse quantized residual obtained by the inverse quantization / inverse orthogonal transform section 206, and holds the decoded decoded image signal in the decoded image memory 208. The decoded image signal superimposition section 207 can also perform filtering processing such as reduction of block distortion due to encoding on the decoded image and hold the decoded image in the decoded image memory 208 after the filtering processing.

[0119] Next, the operation of the block division section 101 in the image encoding apparatus 100 will be described. Figure 3 is a flowchart showing the action of dividing an image into tree blocks and further dividing each tree block. First, an input image is divided into tree blocks of a predetermined size (step S1001). For each tree block, scanning is performed in a predetermined order, i.e., a raster scan order (step S1002), and the inside of the tree block that is the processing target is divided (step S1003).

[0120] Figure 7 is a flowchart showing the detailed action of the division processing of step S1003. First, it is determined whether the processing target block is to be divided by 4 (step S1101).

[0121] In the case where it is determined that the processing target block is to be divided by 4, the processing target block is divided by 4 (step S1102). For each block obtained by dividing the processing target block, scanning is performed in a Z scan order, i.e., an order of top left, top right, bottom left, and bottom right (step S1103). Figure 5 is an example of a Z scan order, Figure 6A 601 of is an example after the processing target block is divided by 4. Figure 6A The numbers 0 to 3 of 601 of indicate the order of processing. Then, for each block divided in step S1101, the division processing of is recursively performed (step S1104). Figure 7

[0122] In the case where it is determined that the processing target block is not to be divided by 4, 2-3 division is performed (step S1105).

[0123] Figure 8 is a flowchart showing the detailed action of the 2-3 division processing of step S1105. First, it is determined whether the processing target block is to be 2-3 divided, i.e., whether one of 2 division and 3 division is to be performed (step S1201).

[0124] In the case where it is determined that the processing target block is not to be 2-3 divided, i.e., in the case where it is determined that no division is to be performed, the division is ended (step S1211). That is, for a block obtained by the recursive division processing, no further recursive division processing is performed.

[0125] In the case where it is determined that the processing target block is to be 2-3 divided, it is determined whether the processing target block is to be further 2 divided (step S1202).

[0126] In the case where it is determined that the processing target block is to be 2 divided, it is determined whether the processing target block is to be divided in the up-down (vertical) direction (step S1203), and based on the result thereof, the processing target block is 2 divided in the up-down (vertical) direction (step S1204) or 2 divided in the left-right (horizontal) direction (step S1205). As a result of step S1204, the processing target block is​ Figure 6B Figure 6D

[0127] In the case where it is determined in step S1202 that the processing target block is not to be 2-divided, that is, in the case where it is determined that the processing target block is to be 3-divided, it is determined whether the processing target block is to be divided in the upper-middle-lower (vertical direction) (step S1206), and based on the result, the processing target block is 3-divided in the upper-middle-lower (vertical direction) (step S1207) or 3-divided in the left-middle-right (horizontal direction) (step S1208). In the result of step S1207, the processing target block is divided into three in the upper-middle-lower (vertical direction) as shown in 603 of FIG. 6C, and in the result of step S1208, the processing target block is divided into three in the left-middle-right (horizontal direction) as shown in 605 of FIG. 6D. Figure 6E

[0128] After any one of step S1204, step S1205, step S1207, and step S1208 is executed, the blocks divided from the processing target block are scanned in the order from left to right and from top to bottom (step S1209). Figure 6B-6E The numbers 0 to 2 from 602 to 605 of FIG. 6B indicate the order of processing. For each of the divided blocks, the 2-3 division processing of FIG. 6A (step S1210) is recursively executed. Figure 8

[0129] The recursive block division described here can also limit whether division is needed depending on the number of divisions or the size of the block of the processing target, and the like. The information to limit whether division is needed can be realized in a structure in which it is agreed beforehand between the encoding apparatus and the decoding apparatus without information transmission, or in a structure in which the encoding apparatus decides the information to limit whether division is needed and records it in the bit string to be transmitted to the decoding apparatus.

[0130] When a certain block is divided, the block before division is called a parent block, and each of the blocks after division is called a child block.

[0131] Next, the operation of the block division section 202 in the image decoding apparatus 200 will be described. The block division section 202 divides the tree block in the same processing steps as the block division section 101 of the image encoding apparatus 100. However, the difference is that in the block division section 101 of the image encoding apparatus 100, an optimization method such as estimation of the optimal shape based on image recognition or distortion rate optimization is applied to decide the optimal block division shape, and in contrast, the block division section 202 in the image decoding apparatus 200 decides the block division shape by decoding the block division information recorded in the bit string.​​​​

[0132] Figure 9 Syntax (bit string syntax rule) related to the block partitioning of the first embodiment is shown. coding_quadtree() indicates syntax involved in 4-partitioning processing of a block. multi_type_tree() indicates syntax involved in 2-partitioning or 3-partitioning processing of a block. qt_split is a flag indicating whether or not the block is 4-partitioned. In the case where the block is 4-partitioned, qt_split = 1 is set, and in the case where the block is not 4-partitioned, qt_split = 0 is set. In the case of 4-partitioning (qt_split = 1), 4-partitioning processing is recursively performed on each of the blocks partitioned by 4 (coding_quadtree(0), coding_quadtree(1), coding_quadtree(2), coding_quadtree(3), 0 to 3 of the argument correspond to the numbers of 601 of Figure 6A In the case where the block is not 4-partitioned (qt_split = 0), the subsequent partitioning is decided in accordance with multi_type_tree(). mtt_split is a flag indicating whether or not the partitioning is further performed. Also, in the case where the partitioning is performed (mtt_split = 1), a flag indicating whether or not the partitioning is performed in the vertical direction or in the horizontal direction, mtt_split_vertical, and a flag indicating whether or not 2-partitioning or 3-partitioning is performed, mtt_split_binary, are transmitted. mtt_split_vertical = 1 indicates that the partitioning is performed in the vertical direction, and mtt_split_vertical = 0 indicates that the partitioning is performed in the horizontal direction. mtt_split_binary = 1 indicates that 2-partitioning is performed, and mtt_split_binary = 0 indicates that 3-partitioning is performed. In the case of 2-partitioning (mtt_split_binary = 1), partitioning processing is recursively performed on each of the blocks partitioned by 2 (multi_type_tree(0), multi_type_tree(1), 0 to 1 of the argument correspond to the numbers of 602 or 604 of Figure 6B-6D In the case of 3-partitioning (mtt_split_binary = 0), partitioning processing is recursively performed on each of the blocks partitioned by 3 (multi_type_tree(0), multi_type_tree(1), multi_type_tree(2), 0 to 2 of the argument correspond to the numbers of 603 or Figure 6B In the case of 3-partitioning (mtt_split_binary = 0), partitioning processing is recursively performed on each of the blocks partitioned by 3 (multi_type_tree(0), multi_type_tree(1), multi_type_tree(2), 0 to 2 of the argument correspond to the numbers of 603 or Figure 6E In the case of 3-partitioning (mtt_split_binary = 0), partitioning processing is recursively performed on each of the blocks partitioned by 3 (multi_type_tree(0), multi_type_tree(1), multi_type_tree(2), 0 to 2 of the argument correspond to the numbers of 603 or

[0133] <Inter prediction>

[0134] The inter prediction method of the embodiments is implemented in the inter prediction section 102 of the image encoding apparatus and the inter prediction section 203 of the image decoding apparatus. Figure 1 Figure 2

[0135] The inter prediction method related to the embodiments is explained using the drawings. The inter prediction method is implemented in either of the encoding process and the decoding process in units of an encoding block.

[0136] <Explanation of the inter prediction section 102 on the encoding side>

[0137] Figure 16 is a diagram showing the detailed structure of the inter prediction section 102 of the image encoding apparatus of Figure 1 In the normal merge mode derivation section 302, a plurality of normal merge candidates are derived, a normal merge candidate is selected, and inter prediction information of a normal merge mode is obtained. The inter prediction information is supplied to the inter prediction mode decision section 305. The detailed structure and processing of the normal merge mode derivation section 302 will be described later.

[0138] In the sub-block prediction motion vector mode derivation section 303, a plurality of sub-block prediction motion vector candidates are derived to select a sub-block prediction motion vector, and a difference motion vector between the selected sub-block prediction motion vector and the detected motion vector is calculated. The detected inter prediction mode, reference index, motion vector, and calculated difference motion vector are inter prediction information of a sub-block prediction motion vector mode. The inter prediction information is supplied to the inter prediction mode decision section 305.

[0139] In the sub-block merge mode derivation section 304, a plurality of sub-block merge candidates are derived, a sub-block merge candidate is selected, and inter prediction information of a sub-block merge mode is obtained. The inter prediction information is supplied to the inter prediction mode decision section 305.

[0140]

[0141] ​​​The inter prediction mode decision section 305 decides inter prediction information based on the inter prediction information supplied from the normal prediction motion vector mode derivation section 301, the normal merge mode derivation section 302, the sub-block prediction motion vector mode derivation section 303, and the sub-block merge mode derivation section 304. The inter prediction information corresponding to the decision result is supplied from the inter prediction mode decision section 305 to the motion compensated prediction section 306.

[0142] The motion compensated prediction section 306 performs inter prediction on the reference picture signal stored in the decoded picture memory 104 based on the decided inter prediction information. The detailed structure and processing of the motion compensated prediction section 306 will be described later.

[0143] <Description of the inter prediction section 203 on the decoding side>

[0144] Figure 22 is a diagram showing the detailed structure of the inter prediction section 203 of the picture decoding apparatus of Figure 2

[0145] The normal prediction motion vector mode derivation section 401 derives a plurality of normal prediction motion vector candidates to select a prediction motion vector, and calculates the addition value of the selected prediction motion vector and the decoded difference motion vector as a motion vector. The decoded inter prediction mode, the reference index, and the motion vector become the inter prediction information of the normal prediction motion vector mode. This inter prediction information is supplied to the motion compensated prediction section 406 via the switch 408. The detailed structure and processing of the normal prediction motion vector mode derivation section 401 will be described later.

[0146] In the normal merge mode derivation section 402, a plurality of normal merge candidates are derived to select a normal merge candidate, and the inter prediction information of the normal merge mode is obtained. This inter prediction information is supplied to the motion compensated prediction section 406 via the switch 408. The detailed structure and processing of the normal merge mode derivation section 402 will be described later.

[0147] In the sub-block prediction motion vector mode derivation section 403, a plurality of sub-block prediction motion vector candidates are derived to select a sub-block prediction motion vector, and the addition value of the selected sub-block prediction motion vector and the decoded difference motion vector is calculated as a motion vector. The decoded inter prediction mode, the reference index, and the motion vector become the inter prediction information of the sub-block prediction motion vector mode. This inter prediction information is supplied to the motion compensated prediction section 406 via the switch 408.

[0148] In the sub-block merge mode derivation section 404, a plurality of sub-block merge candidates are derived to select a sub-block merge candidate, and the inter prediction information of the sub-block merge mode is obtained. This inter prediction information is supplied to the motion compensated prediction section 406 via the switch 408.

[0149] ​In the motion compensation prediction section 406, the reference picture signal stored in the decoded picture memory 208 is inter-predicted based on the determined inter-prediction information. The detailed structure and processing of the motion compensation prediction section 406 are the same as those of the motion compensation prediction section 306 on the encoding side.

[0150] <General prediction motion vector mode derivation section (General AMVP)>

[0151] Figure 17 The general prediction motion vector mode derivation section 301 includes a spatial prediction motion vector candidate derivation section 321, a temporal prediction motion vector candidate derivation section 322, a history prediction motion vector candidate derivation section 323, a prediction motion vector candidate supplement section 325, a general motion vector detection section 326, a prediction motion vector candidate selection section 327, and a motion vector subtraction section 328.

[0152] Figure 23 The general prediction motion vector mode derivation section 401 includes a spatial prediction motion vector candidate derivation section 421, a temporal prediction motion vector candidate derivation section 422, a history prediction motion vector candidate derivation section 423, a prediction motion vector candidate supplement section 425, a prediction motion vector candidate selection section 426, and a motion vector addition section 427.

[0153] The processing steps of the general prediction motion vector mode derivation section 301 on the encoding side and the general prediction motion vector mode derivation section 401 on the decoding side are explained using the flowcharts of Figure 19 Figure 25 Figure 19 is a flowchart showing the processing steps of the general prediction motion vector mode derivation processing based on the general motion vector mode derivation section 301 on the encoding side, Figure 25 is a flowchart showing the processing steps of the general prediction motion vector mode derivation processing based on the general motion vector mode derivation section 401 on the decoding side.

[0154] <General prediction motion vector mode derivation section (General AMVP): Explanation on the encoding side>

[0155] The processing steps of the general prediction motion vector mode derivation processing on the encoding side are explained with reference to Figure 19 In the explanation of the processing steps of Figure 19 , the word "general" shown in Figure 19 is sometimes omitted.

[0156] First, the general motion vector detection section 326 detects a general motion vector (MvG) for each inter-prediction mode and reference index (step S100 of Figure 19 ).

[0157] ​​Next, the spatial prediction motion vector candidate derivation unit 321, the temporal prediction motion vector candidate derivation unit 322, the historical prediction motion vector candidate derivation unit 323, the prediction motion vector candidate supplementation unit 325, the prediction motion vector candidate selection unit 327, and the motion vector subtraction unit 328 calculate, for each L0 and L1, the differential motion vector of the motion vector used in the inter-frame prediction of the normal prediction motion vector mode. Figure 19 Steps S101 to S106). Specifically, when the prediction mode PredMode of the processed object block is inter-frame prediction (MODE_INTER) and the inter-frame prediction mode is L0 prediction (Pred_L0), the candidate list of predicted motion vectors for L0, mvpListL0, is calculated, the predicted motion vector mvpL0 is selected, and the differential motion vector mvdL0 of the motion vector mvL0 of L0 is calculated. When the inter-frame prediction mode of the processed object block is L1 prediction (Pred_L1), the candidate list of predicted motion vectors for L1, mvpListL1, is calculated, the predicted motion vector mvpL1 is selected, and the differential motion vector mvdL1 of the motion vector mvL1 of L1 is calculated. When the inter-frame prediction mode for processing object blocks is dual prediction (Pred_BI), L0 prediction and L1 prediction are performed simultaneously. The candidate list of predicted motion vectors for L0, mvpListL0, is calculated. The predicted motion vector of L0, mvpL0, is selected. The differential motion vector of L0, mvL0, is calculated. The candidate list of predicted motion vectors for L1, mvpListL1, is calculated. The predicted motion vector of L1, mvpL1, is calculated. The differential motion vector of L1, mvL1, is calculated.

[0158] Differential motion vector calculations are performed separately for L0 and L1, but the process is common to both. Therefore, in the following explanation, L0 and L1 will be represented as a common LX. In the calculation of the differential motion vector for L0, X of LX is 0, and in the calculation of the differential motion vector for L1, X of LX is 1. Furthermore, in the calculation of the differential motion vector for LX, if information from another list is referenced instead of LX, this other list will be represented as LY.

[0159] When using the motion vector mvLX of LX ( Figure 19 Step S102: Yes), calculate the candidate predicted motion vectors of LX, and construct the candidate predicted motion vector list mvpListLX( Figure 19 of step S103). The spatial prediction motion vector candidate derivation section 321, the temporal prediction motion vector candidate derivation section 322, the history prediction motion vector candidate derivation section 323, and the prediction motion vector candidate supplement section 325 in the usual prediction motion vector mode derivation section 301 derive a plurality of candidates of the prediction motion vector, and construct a prediction motion vector candidate list mvpListLX. Regarding Figure 19 The detailed processing steps of step S103 are described using the flowchart of Figure 20 in the following.

[0160] Next, the prediction motion vector candidate selection section 327 selects the prediction motion vector mvpLX of LX from the prediction motion vector candidate list mvpListLX of LX ( Figure 19 of step S104). Here, a certain element (the i-th element counted from 0) in the prediction motion vector candidate list mvpListLX is denoted as mvpListLX[i]. Each difference motion vector that is the difference between the motion vector mvLX and each candidate mvpListLX[i] of the prediction motion vector saved in the prediction motion vector candidate list mvpListLX is calculated. The code amount when these difference motion vectors are encoded is calculated for each element (prediction motion vector candidate) of the prediction motion vector candidate list mvpListLX. Then, among each element registered in the prediction motion vector candidate list mvpListLX, the candidate mvpListLX[i] of the prediction motion vector whose code amount is the smallest is selected as the prediction motion vector mvpLX of each candidate of the prediction motion vector, and the index i is acquired. In the case where a plurality of candidates of the prediction motion vector that become the smallest code amount exist in the prediction motion vector candidate list mvpListLX, the candidate mvpListLX[i] of the prediction motion vector indicated by the number of the index i in the prediction motion vector candidate list mvpListLX that is smaller is selected as the optimal prediction motion vector mvpLX, and the index i is acquired.

[0161] Next, the motion vector subtraction section 328 subtracts the selected prediction motion vector mvpLX of LX from the motion vector mvLX of LX, and sets mvdLX = mvLX - mvpLX to calculate the difference motion vector mvdLX of LX ( Figure 19 of step S105).

[0162] <Usual prediction motion vector mode derivation section (usual AMVP): explanation on decoding side>

[0163] Next, with reference to Figure 25 The typical prediction motion vector mode processing steps on the decoding side are explained. On the decoding side, the spatial prediction motion vector candidate derivation unit 421, the temporal prediction motion vector candidate derivation unit 422, the historical prediction motion vector candidate derivation unit 423, and the prediction motion vector candidate supplementation unit 425 calculate, for each L0 and L1, the motion vectors used in inter-frame prediction in the typical prediction motion vector mode. Figure 25 Steps S201 to S206). Specifically, when the prediction mode PredMode of the processing object block is inter-frame prediction (MODE_INTER) and the inter-frame prediction mode of the processing object block is L0 prediction (Pred_L0), the candidate list of predicted motion vectors for L0, mvpListL0, is calculated, the predicted motion vector mvpL0 is selected, and the motion vector mvL0 of L0 is calculated. When the inter-frame prediction mode of the processing object block is L1 prediction (Pred_L1), the candidate list of predicted motion vectors for L1, mvpListL1, is calculated, the predicted motion vector mvpL1 is selected, and the motion vector mvL1 of L1 is calculated. When the inter-frame prediction mode for processing object blocks is dual prediction (Pred_BI), L0 prediction and L1 prediction are performed simultaneously. The candidate list of predicted motion vectors for L0, mvpListL0, is calculated. The predicted motion vector of L0, mvpL0, is selected, and the motion vector of L0, mvL0, is calculated. The candidate list of predicted motion vectors for L1, mvpListL1, is calculated, and the predicted motion vector of L1, mvpL1, is calculated. The motion vector of L1, mvL1, is calculated separately.

[0164] Similar to the encoding side, motion vector calculations are also performed on L0 and L1 separately on the decoding side, but L0 and L1 are processed in a common manner. Therefore, in the following description, L0 and L1 are denoted as a common LX. LX represents the inter-prediction mode used for inter-frame prediction of the coded block being processed. In the process of calculating the motion vector of L0, X is 0, and in the process of calculating the motion vector of L1, X is 1. In addition, in the process of calculating the motion vector of LX, if information from another reference list is referenced instead of the same reference list as the LX being calculated, this other reference list is denoted as LY.

[0165] When using the motion vector mvLX of LX ( Figure 25 Step S202: Yes), calculate the candidates for predicted motion vectors of LX, and construct the candidate list of predicted motion vectors of LX, mvpListLX( Figure 25 The candidates of the plurality of prediction motion vectors are calculated by the spatial prediction motion vector candidate derivation section 421, the temporal prediction motion vector candidate derivation section 422, the history prediction motion vector candidate derivation section 423, and the prediction motion vector candidate supplement section 425 in the usual prediction motion vector mode derivation section 401, and a prediction motion vector candidate list mvpListLX is constructed. As for the Figure 25 The detailed processing steps of step S203 are described using the flowchart of Figure 20

[0166] Next, the candidate mvpListLX[mvpIdxLX] of the prediction motion vector corresponding to the index mvpIdxLX of the prediction motion vector provided by the decoding of the bit string in the bit string decoding section 201 is taken out from the prediction motion vector candidate list mvpListLX by the prediction motion vector candidate selection section 426 as the selected prediction motion vector mvpLX (step S204). Figure 25

[0167] Next, the differential motion vector mvdLX of LX provided by the decoding of the bit string in the bit string decoding section 201 and the prediction motion vector mvpLX of LX are added by the motion vector addition section 427, and set as mvLX = mvpLX + mvdLX, and the motion vector mvLX of LX is calculated (step S205). Figure 25

[0168] <Usual Prediction Motion Vector Mode Derivation Section (Usual AMVP): Prediction Method of Motion Vector>

[0169] Figure 20 is a flowchart showing the processing steps of the usual prediction motion vector mode derivation processing which has a common function in the usual prediction motion vector mode derivation section 301 of the image encoding apparatus and the usual prediction motion vector mode derivation section 401 of the image decoding apparatus involved in the embodiments of the present application.

[0170] ​​​In the usual prediction motion vector mode derivation section 301 and the usual prediction motion vector mode derivation section 401, there is a prediction motion vector candidate list mvpListLX. The prediction motion vector candidate list mvpListLX constitutes a list structure, and has a storage area in which a prediction motion vector index indicating a position inside the prediction motion vector candidate list and a prediction motion vector candidate corresponding to the index are stored as elements. The number of the prediction motion vector index starts from 0, and the prediction motion vector candidate is stored in the storage area of the prediction motion vector candidate list mvpListLX. In the present embodiment, it is assumed that the prediction motion vector candidate list mvpListLX can register at least two prediction motion vector candidates (inter prediction information). Further, a variable numCurrMvpCand indicating the number of prediction motion vector candidates registered in the prediction motion vector candidate list mvpListLX is set to 0.

[0171] The spatial prediction motion vector candidate derivation section 321 and 421 derives a candidate of a prediction motion vector from a block adjacent to the left. In this processing, the inter prediction information of the block adjacent to the left (A0 or Al) of the block to be coded is referred to, that is, a flag indicating whether or not a prediction motion vector candidate can be used, and a motion vector, a reference index, and the like are referred to, and a prediction motion vector mvLXA is derived. The derived mvLXA is added to the prediction motion vector candidate list mvpListLX (step S301 of FIG. 3). Figure 11 Figure 20 In L0 prediction, X is 0, and in L1 prediction, X is 1 (the same applies hereinafter). Next, the spatial prediction motion vector candidate derivation section 321 and 421 derives a candidate of a prediction motion vector from a block adjacent to the top. In this processing, the inter prediction information of the block adjacent to the top (B0, Bl, or B2) of the block to be coded is referred to, that is, a flag indicating whether or not a prediction motion vector candidate can be used, and a motion vector, a reference index, and the like are referred to, and a prediction motion vector mvLXB is derived. If the derived mvLXA and mvLXB are not equal to each other, the mvLXB is added to the prediction motion vector candidate list mvpListLX (step S302 of FIG. 3). Figure 11 Figure 20 The processing of steps S301 and S302 of FIG. 3 is common except for the position and the number of the adjacent blocks to be referred to, and a flag availableFlagLXN indicating whether or not a prediction motion vector candidate of a coded block can be used, and a motion vector mvLXN, a reference index refIdxN (N indicates A or B, the same applies hereinafter) are derived. Figure 20

[0172] ​​​Next, the temporal prediction motion vector candidate derivation units 322 and 422 derive a candidate of a prediction motion vector from a block in a picture different in time from the current processing target picture. In this processing, a flag availableFlagLXCol indicating whether or not it is possible to use a prediction motion vector of a coded block of a picture different in time as a prediction motion vector candidate, and a motion vector mvLXCol, a reference index refldxCol, a reference list listCol are derived, and mvLXCol is added to the prediction motion vector candidate list mvpListLX (step S303 of FIG. 13). Figure 20

[0173] Further, it is assumed that the processing of the temporal prediction motion vector candidate derivation units 322 and 422 in units of sequence (SPS), picture (PPS), or slice can be omitted.

[0174] Next, the history prediction motion vector candidate derivation units 323 and 423 add the history prediction motion vector candidates registered in the history prediction motion vector candidate list HmvpCandList to the prediction motion vector candidate list mvpListLX (step S304 of FIG. 14). Figure 20 The details of the registration processing step of this step S304 are described later using the flowchart of FIG. 15. Figure 29

[0175] Next, the prediction motion vector candidate supplement units 325 and 425 add a prediction motion vector candidate of a predetermined value such as (0, 0) until the prediction motion vector candidate list mvpListLX is satisfied (step S305 of FIG. 16). Figure 20

[0176] <General Merge Mode Derivation Unit (General Merge)>

[0177] Figure 18 The general merge mode derivation unit 302 includes a spatial merge candidate derivation unit 341, a temporal merge candidate derivation unit 342, an average merge candidate derivation unit 344, a history merge candidate derivation unit 345, a merge candidate supplement unit 346, and a merge candidate selection unit 347.

[0178] Figure 24 The general merge mode derivation unit 402 includes a spatial merge candidate derivation unit 441, a temporal merge candidate derivation unit 442, an average merge candidate derivation unit 444, a history merge candidate derivation unit 445, a merge candidate supplement unit 446, and a merge candidate selection unit 447.

[0179] Figure 21 ​​​is a flowchart illustrating steps of a general merge mode derivation process common to the general merge mode derivation section 302 of the image encoding apparatus and the general merge mode derivation section 402 of the image decoding apparatus involved in the embodiment of the present application.

[0180] The processes will be described below in order. Also, in the following description, the case where the slice type slice_type is a B slice will be described unless specifically described, but the case where the slice type slice_type is a P slice can also be applied. However, in the case where the slice type slice_type is a P slice, since only L0 prediction (Pred_L0) is present as the inter prediction mode, L1 prediction (Pred_L1) and bi-prediction (Pred_BI) are not present. Therefore, the process around L1 can be omitted.

[0181] In the general merge mode derivation section 302 and the general merge mode derivation section 402, a merge candidate list mergeCandList is present. The merge candidate list mergeCandList is configured in a list structure, and is provided with a merge index indicating a position inside the merge candidate list, and a storage area in which a merge candidate corresponding to the index is stored as an element. The number of the merge index starts from 0, and a merge candidate is stored in the storage area of the merge candidate list mergeCandList. In the following process, it is assumed that the merge candidate of the merge index i registered in the merge candidate list mergeCandList is represented by mergeCandList[i]. In the present embodiment, it is assumed that the merge candidate list mergeCandList can register at least six merge candidates (inter prediction information). Also, a variable numCurrMergeCand indicating the number of the merge candidates registered in the merge candidate list mergeCandList is set to 0.

[0182] In the spatial merge candidate derivation section 341 and the spatial merge candidate derivation section 441, spatial merge candidates from the respective blocks (B1, A1, B0, A0, B2) adjacent to the block to be processed are derived in the order of B1, A1, B0, A0, B2 from the encoding information stored in the encoding information storage memory 111 of the image encoding apparatus or the encoding information storage memory 205 of the image decoding apparatus, and the derived spatial merge candidates are registered in the merge candidate list mergeCandList. Figure 11 Figure 21 ​N. The flag availableFlagN indicating whether or not the inter prediction information of the block N can be used as a spatial merge candidate, the reference index refIdxLON of L0 of the spatial merge candidate N, the reference index refIdxL1N of L1 of the spatial merge candidate N, the L0 prediction flag predFlagLON indicating whether or not L0 prediction is performed, the L1 prediction flag predFlagL1N indicating whether or not L1 prediction is performed, the motion vector mvLON of L0, and the motion vector mvL1N of L1 are derived. However, in the present embodiment, since the merge candidate is not derived by referring to the inter prediction information of the block included in the coding block which is a processing target, the spatial merge candidate using the inter prediction information of the block included in the coding block which is a processing target is not derived.

[0183] Next, in the temporal merge candidate derivation section 342 and the temporal merge candidate derivation section 442, the temporal merge candidate from the picture at different time is derived, and the derived temporal merge candidate is registered in the merge candidate list mergeCandList (step S402). Figure 21 The flag availableFlagCol indicating whether or not the temporal merge candidate can be used, the L0 prediction flag predFlagL0Col indicating whether or not L0 prediction of the temporal merge candidate is performed, and the L1 prediction flag predFlagL1Col indicating whether or not L1 prediction is performed, and the motion vector mvL0Col of L0 and the motion vector mvL1Col of L1 are derived.

[0184] Further, the processing of the temporal merge candidate derivation section 342 and the temporal merge candidate derivation section 442 in units of sequence (SPS), picture (PPS), or slice can be omitted.

[0185] Next, in the history merge candidate derivation section 345 and the history merge candidate derivation section 445, the history prediction motion vector candidate registered in the history prediction motion vector candidate list HmvpCandList is registered in the merge candidate list mergeCandList (step S403). Figure 21

[0186] ​Further, in a case where the number of merge candidates numCurrMergeCand registered in the merge candidate list mergeCandList is smaller than the maximum number of merge candidates MaxNumMergeCand, the number of merge candidates numCurrMergeCand registered in the merge candidate list mergeCandList is derived up to the maximum number of merge candidates MaxNumMergeCand, and the history merge candidate is registered in the merge candidate list mergeCandList.

[0187] Next, in the average merge candidate deriving section 344 and the average merge candidate deriving section 444, the average merge candidate is derived from the merge candidate list mergeCandList, the derived average merge candidate is added to the merge candidate list mergeCandList, and the average merge candidate is registered in the merge candidate list mergeCandList (step S404). Figure 21

[0188] Further, in a case where the number of merge candidates numCurrMergeCand registered in the merge candidate list mergeCandList is smaller than the maximum number of merge candidates MaxNumMergeCand, the number of merge candidates numCurrMergeCand registered in the merge candidate list mergeCandList is derived up to the maximum number of merge candidates MaxNumMergeCand, and the average merge candidate is registered in the merge candidate list mergeCandList.

[0189] Here, the average merge candidate is a new merge candidate having a motion vector obtained by averaging motion vectors of the first merge candidate and the second merge candidate registered in the merge candidate list mergeCandList in accordance with each of the L0 prediction and the L1 prediction.

[0190] Next, in the merge candidate supplement section 346 and the merge candidate supplement section 446, in a case where the number of merge candidates numCurrMergeCand registered in the merge candidate list mergeCandList is smaller than the maximum number of merge candidates MaxNumMergeCand, the number of merge candidates numCurrMergeCand registered in the merge candidate list mergeCandList is derived up to the maximum number of merge candidates MaxNumMergeCand, and the addition merge candidate is registered in the merge candidate list mergeCandList (step S404). Figure 21 ​The merge candidate is added in the P slice with the prediction mode of the motion vector having the value (0, 0) of the L0 prediction (Pred_L0) as an upper limit of the maximum number of merge candidates MaxNumMergeCand. In the B slice, the merge candidate is added with the prediction mode of the motion vector having the value (0, 0) of the bi-prediction (Pred_BI). The reference index at the time of adding the merge candidate is different from the already added reference index.

[0191] Next, in the merge candidate selection section 347 and the merge candidate selection section 447, a merge candidate is selected from the merge candidates registered in the merge candidate list mergeCandList. The merge candidate selection section 347 on the encoding side selects a merge candidate by calculating the code amount and the distortion amount, and provides the inter prediction information of the selected merge candidate to the motion compensation prediction section 306 via the inter prediction mode decision section 305. On the other hand, in the merge candidate selection section 447 on the decoding side, a merge candidate is selected based on the decoded merge index, and the selected merge candidate is provided to the motion compensation prediction section 406.

[0192] <Update History Prediction Motion Vector Candidate List>

[0193] Next, the initialization method and the update method of the history prediction motion vector candidate list HmvpCandList possessed by the encoding information storage memory 111 on the encoding side and the encoding information storage memory 205 on the decoding side will be described in detail. Figure 26 is a flowchart for explaining the history prediction motion vector candidate list initialization / update processing step.

[0194] In the present embodiment, it is assumed that the update of the history prediction motion vector candidate list HmvpCandList is performed in the encoding information storage memory 111 and the encoding information storage memory 205. It is also possible to provide a history prediction motion vector candidate list update section in the inter prediction section 102 and the inter prediction section 203, and implement the update of the history prediction motion vector candidate list HmvpCandList.

[0195] The initial setting of the history prediction motion vector candidate list HmvpCandList is performed at the beginning of the slice. On the encoding side, the history prediction motion vector candidate list HmvpCandList is updated in the case where the normal prediction motion vector mode or the normal merge mode is selected by the prediction method decision section 105. On the decoding side, the history prediction motion vector candidate list HmvpCandList is updated in the case where the prediction information decoded by the bit string decoding section 201 is the normal prediction motion vector mode or the normal merge mode.

[0196] Inter-frame prediction information used during inter-frame prediction in either the normal motion vector prediction mode or the normal merging mode is registered in the historical prediction motion vector candidate list hmvpCandList as inter-frame prediction information candidate hMvpCand. Inter-frame prediction information candidate hMvpCand includes the reference index refIdxL0 for L0 and the reference index refIdxL1 for L1, the L0 prediction flag predFlagL0 indicating whether L0 prediction is performed, the L1 prediction flag predFlagL1 indicating whether L1 prediction is performed, the motion vector mvL0 for L0, and the motion vector mvL1 for L1.

[0197] If, among the elements (i.e., inter-frame prediction information) registered in the historical prediction motion vector candidate list HmvpCandList stored in the encoding information storage memory 111 on the encoding side and the encoding information storage memory 205 on the decoding side, there exists inter-frame prediction information with the same value as the inter-frame prediction information candidate hMvpCand, then that element is deleted from the historical prediction motion vector candidate list HmvpCandList. On the other hand, if there is no inter-frame prediction information with the same value as the inter-frame prediction information candidate hMvpCand, then the element at the beginning of the historical prediction motion vector candidate list HmvpCandList is deleted, and the inter-frame prediction information candidate hMvpCand is added to the end of the historical prediction motion vector candidate list HmvpCandList.

[0198] The number of elements in the historical predicted motion vector candidate list HmvpCandList possessed by the encoding information storage memory 111 on the encoding side and the encoding information storage memory 205 on the decoding side of the present invention is set to 6.

[0199] First, initialize the historical predicted motion vector candidate list HmvpCandList, which is based on strips. Figure 26 Step S2101). At the beginning of the strip, all elements of the historical predicted motion vector candidate list HmvpCandList are made empty, and the value of NumHmvpCand, the number of historical predicted motion vector candidates registered in the historical predicted motion vector candidate list HmvpCandList (the current number of candidates), is set to 0.

[0200] In addition, although the initialization of the historical predicted motion vector candidate list HmvpCandList is set to be performed on a strip-by-strip basis (the initial encoded block of the strip), it can also be performed on an image-by-image basis, a tile-by-tile basis, or a tree block-by-tree basis.

[0201] Next, for each coded block in the strip, the following update process for the historical predicted motion vector candidate list HmvpCandList is repeated ( Figure 26 Steps S2102 to S2107).

[0202] First, initial settings are performed on a block-by-block basis. The flag `identicalCandExist`, indicating the existence of identical candidates, is set to `FALSE`, and the index `removeIdx`, representing the candidate to be deleted, is set to 0. Figure 26 Step S2103).

[0203] Determine whether there are inter-frame prediction information candidates hMvpCand for the registered object. Figure 26 Step S2104). If the prediction method determination unit 105 on the encoding side determines it to be a normal prediction motion vector mode or a normal merging mode, or if the bit string decoding unit 201 on the decoding side decodes it to be a normal prediction motion vector mode or a normal merging mode, the inter-frame prediction information is set as the inter-frame prediction information candidate hMvpCand for the registered object. If the prediction method determination unit 105 on the encoding side determines it to be an intra-frame prediction mode, a sub-block prediction motion vector mode, or a sub-block merging mode, or if the bit string decoding unit 201 on the decoding side decodes it to be an intra-frame prediction mode, a sub-block prediction motion vector mode, or a sub-block merging mode, the historical prediction motion vector candidate list HmvpCandList is not updated, and there is no inter-frame prediction information candidate hMvpCand for the registered object. If there is no inter-frame prediction information candidate hMvpCand for the registered object, steps S2105 to S2106 are skipped. Figure 26 Step S2104: No). If there is a candidate hMvpCand for inter-frame prediction information of the registered object, proceed with the processing after step S2105. Figure 26 Step S2104: Yes).

[0204] Next, it is determined whether there exists an element in each element of the historical predicted motion vector candidate list HmvpCandList that has the same value as the inter-frame prediction information candidate hMvpCand of the registered object (inter-frame prediction information), that is, whether there is a matching element. Figure 26 Step S2105). Figure 27 This is a flowchart of the same element confirmation process. When the value of the historical predicted motion vector candidate number NumHmvpCand is 0 ( Figure 27 Step S2121: No), the historical predicted motion vector candidate list HmvpCandList is empty. Since there are no identical candidates, this step is skipped. Figure 27 Steps S2122 to S2125 conclude the same feature confirmation process. This applies when the value of the historical predicted motion vector candidate number NumHmvpCand is greater than 0 ( Figure 27 Step S2121: Yes), the historical predicted motion vector index hMvpIdx is from 0 to NumHmvpCand-1, and the processing of step S2123 is repeated. Figure 27 Steps S2122 to S2125). First, compare whether the hMvpIdx-th element HmvpCandList[hMvpIdx] in the historical predicted motion vector candidate list (starting from 0) is the same as the inter-frame prediction information candidate hMvpCand. Figure 27 Step S2123). Under the same circumstances ( Figure 27 Step S2123: If yes, set the flag `identicalCandExist`, indicating whether there are identical candidates, to TRUE; set the current historical predicted motion vector index `hMvpIdx`, representing the location of the deleted object's index `removeIdx`, to the value of the index `hMvpIdx`; and end the identical feature confirmation process. In the case of different features ( Figure 27 Step S2123: No), increase hMvpIdx by 1. If the historical predicted motion vector index hMvpIdx is below NumHmvpCand-1, then proceed with the processing after step S2123.

[0205] Return again Figure 26 The flowchart describes the process of shifting and adding elements to the historical predicted motion vector candidate list HmvpCandList. Figure 26 Step S2106). Figure 28 yes Figure 26 The flowchart for the feature shifting / addition process in step S2106 of the historical predicted motion vector candidate list HmvpCandList is as follows: First, it is determined whether to add new features after removing features stored in the historical predicted motion vector candidate list HmvpCandList, or to add new features without removing existing features. Specifically, it compares whether the flag identicalCandExist, indicating the existence of identical candidates, is TRUE or whether NumHmvvpCand is 6. Figure 28 Step S2141). Under the condition that either the flag `identicalCandExist` indicating the existence of identical candidates is TRUE or the current number of candidates `NumHmvpCand` is 6 (…), Figure 28 Step S2141: Yes), after removing the features stored in the historical predicted motion vector candidate list HmvpCandList, add new features. Set the initial value of index i to the value of removeIdx+1. Repeat the feature shifting process of step S2143 from this initial value to NumHmvpCand. Figure 28 Steps S2142 to S2144). By copying the elements of HmvpCandList[i] to HmvpCandList[i-1], the elements are shifted forward ( Figure 28 Step S2143), increment i by 1 ( Figure 28 Steps S2142 to S2144). Next, add the inter-frame prediction information candidate hMvpCand to the (NumHmvpCand-1)th HmvpCandList[NumHmvpCand-1], which is the last corresponding to the historical prediction motion vector candidate list, starting from 0. Figure 28 Step S2145) ends the feature shifting / addition process of the historical predicted motion vector candidate list HmvpCandList. On the other hand, if neither of the conditions that the flag identicalCandExist indicating the existence of identical candidates is TRUE nor NumHmvpCand is 6 is met ( Figure 28 Step S2141: No), instead of removing the features stored in the historical predicted motion vector candidate list HmvpCandList, add the inter-frame prediction information candidate hMvpCand to the end of the historical predicted motion vector candidate list. Figure 28 Step S2146). Here, the last element of the historical predicted motion vector candidate list is the NumHmvpCand-th HmvpCandList[NumHmvpCand], counting from 0. Additionally, NumHmvpCand is incremented by 1, ending the shifting and addition process for elements in the historical predicted motion vector candidate list HmvpCandList.

[0206] Figure 31A-31C This diagram illustrates an example of the update process for the historical predicted motion vector candidate list. When adding a new feature to the historical predicted motion vector candidate list HmvpCandList, which already contains six registered features (inter-frame prediction information), the new inter-frame prediction information is compared sequentially with the features preceding those in the historical predicted motion vector candidate list HmvpCandList. Figure 31A ), if the new element is the same value as the third element HMVP2 from the beginning of the history-based motion vector predictor candidate list HmvpCandList, then the element HMVP2 is deleted from the history-based motion vector predictor candidate list HmvpCandList and the elements HMVP3 ~ HMVP5 behind are shifted one by one to the front (copy), the new element is added to the end of the history-based motion vector predictor candidate list HmvpCandList Figure 31B ), the update of the history-based motion vector predictor candidate list HmvpCandList is completed Figure 31C

[0207] <History-based motion vector predictor derivation process>

[0208] Next, the method of deriving the history-based motion vector predictor from the history-based motion vector predictor candidate list HmvpCandList in the processing step of step S304 as Figure 20 Figure 20 The processing step of step S304 is a process common to the history-based motion vector predictor derivation section 323 of the normal motion vector prediction mode derivation section 301 on the encoding side, and the history-based motion vector predictor derivation section 423 of the normal motion vector prediction mode derivation section 401 on the decoding side. Figure 29 is a flowchart illustrating the history-based motion vector predictor derivation process step.

[0209] In the case where the current number of motion vector predictor candidates numCurrMvpCand is 2 or more than the maximum number of elements of the motion vector predictor list mvpListLX (here, 2) or the value of the history-based number of motion vector predictor candidates NumHmvpCand is 0 (NO of step S2201 of Figure 29 the processing of steps S2202 to S2209 of Figure 29 is omitted, and the history-based motion vector predictor derivation process step is ended. In the case where the current number of motion vector predictor candidates numCurrMvpCand is less than 2, the maximum number of elements of the motion vector predictor list mvpListLX, and the value of the history-based number of motion vector predictor candidates NumHmvpCand is greater than 0 (YES of step S2201 of Figure 29 the processing of steps S2202 to S2209 of Figure 29 is performed.

[0210] Next, the processing of steps S2203 to S2208 of Figure 29 is repeated until either the index i is from 1 to 4 or the history-based number of motion vector predictor candidates numCheckedHMVPCand is smaller (YES of step S2208 of Figure 29 ​​S2202 to S2209). In a case where the current number of prediction motion vector candidates numCurrMvpCand is 2 or more (NO in step S2203), the processes of steps S2204 to S2209 are omitted, and the historical prediction motion vector candidate derivation process step is ended. In a case where the current number of prediction motion vector candidates numCurrMvpCand is less than 2 (YES in step S2203), the processes after step S2204 are executed. Figure 29 S2204 to S2209 are omitted, and the historical prediction motion vector candidate derivation process step is ended. In a case where the current number of prediction motion vector candidates numCurrMvpCand is less than 2 (YES in step S2203), the processes after step S2204 are executed. Figure 29 S2204 to S2209 are omitted, and the historical prediction motion vector candidate derivation process step is ended. In a case where the current number of prediction motion vector candidates numCurrMvpCand is less than 2 (YES in step S2203), the processes after step S2204 are executed. Figure 29 S2204 to S2209 are omitted, and the historical prediction motion vector candidate derivation process step is ended. In a case where the current number of prediction motion vector candidates numCurrMvpCand is less than 2 (YES in step S2203), the processes after step S2204 are executed. Figure 29 S2204 to S2209 are omitted, and the historical prediction motion vector candidate derivation process step is ended. In a case where the current number of prediction motion vector candidates numCurrMvpCand is less than 2 (YES in step S2203), the processes after step S2204 are executed.

[0211] Next, the processes from step S2205 to S2207 are executed for Y being 0 and 1 (L0 and L1) respectively (steps S2204 to S2208). In a case where the current number of prediction motion vector candidates numCurrMvpCand is 2 or more (NO in step S2205), the processes of steps S2206 to S2209 are omitted, and the historical prediction motion vector candidate derivation process step is ended. In a case where the current number of prediction motion vector candidates numCurrMvpCand is less than 2 (YES in step S2205), the processes after step S2206 are executed. Figure 29 S2206 to S2209 are omitted, and the historical prediction motion vector candidate derivation process step is ended. In a case where the current number of prediction motion vector candidates numCurrMvpCand is less than 2 (YES in step S2203), the processes after step S2204 are executed. Figure 29 S2206 to S2209 are omitted, and the historical prediction motion vector candidate derivation process step is ended. In a case where the current number of prediction motion vector candidates numCurrMvpCand is less than 2 (YES in step S2203), the processes after step S2204 are executed. Figure 29 S2206 to S2209 are omitted, and the historical prediction motion vector candidate derivation process step is ended. In a case where the current number of prediction motion vector candidates numCurrMvpCand is less than 2 (YES in step S2203), the processes after step S2204 are executed. Figure 29 S2206 to S2209 are omitted, and the historical prediction motion vector candidate derivation process step is ended. In a case where the current number of prediction motion vector candidates numCurrMvpCand is less than 2 (YES in step S2203), the processes after step S2204 are executed. Figure 29 S2206 to S2209 are omitted, and the historical prediction motion vector candidate derivation process step is ended. In a case where the current number of prediction motion vector candidates numCurrMvpCand is less than 2 (YES in step S2203), the processes after step S2204 are executed.

[0212] Next, in a case where there is an element in the historical prediction motion vector candidate list HmvpCandList having the same reference index as the reference index refIdxLX of the encoding / decoding target motion vector and different from any element of the prediction motion vector list mvpListLX (YES in step S2206), the motion vector of LY of the historical prediction motion vector candidate HmvpCandList[NumHmvpCand-i] is added to the 0th element mvpListLX[0] of the prediction motion vector candidate list (step S2207). Figure 29 S2206 to S2209 are omitted, and the historical prediction motion vector candidate derivation process step is ended. In a case where the current number of prediction motion vector candidates numCurrMvpCand is less than 2 (YES in step S2203), the processes after step S2204 are executed. Figure 29 Step S2207) increments the current number of predicted motion vector candidates numCurrMvpCand by 1. This occurs when there are no elements in the historical predicted motion vector candidate list HmvpCandList with the same reference index refIdxLX as the motion vector of the encoded / decoded object, and which are also different from any element in the predicted motion vector list mvpListLX. Figure 29 Step S2206: No), skip the addition process in step S2207.

[0213] Both L0 and L1 are performed as above Figure 29 Processing steps S2205 to S2207 ( Figure 29 Steps S2204 to S2208). Increment index i by 1. If index i is less than or equal to 4 and either the historical predicted motion vector candidate number NumHmvpCand is smaller, repeat the processing after step S2203. Figure 29 Steps S2202 to S2209).

[0214] <Historical Merge Candidate Export Processing>

[0215] Next, a detailed explanation will be provided as... Figure 29 Step S404 is a method for deriving historical merge candidates from the historical merge candidate list HmvpCandList. Figure 29 The processing step S404 is a common process in the historical merge candidate derivation section 345 of the normal merge mode derivation section 302 on the encoding side and the historical merge candidate derivation section 445 of the normal merge mode derivation section 402 on the decoding side. Figure 21 This is a flowchart illustrating the steps involved in exporting historical merged candidates.

[0216] First, perform initialization processing ( Figure 21 Step S2301). Set the value of FALSE for each element from 0 to (numCurrMergeCand-1)th element in isPruned[i], and set the variable numOrigMergeCand to the number of elements registered in the current merge candidate list, numCurrMergeCand.

[0217] Next, the initial value of index hMvpIdx is set to 1, and the process is repeated from this initial value to NumHmvpCand. Figure 30 The addition process from step S2303 to step S2310 ( Figure 30 Steps S2302 to S2311). If the number of features registered in the current merge candidate list, numCurrMergeCand, is not below (MaxNumMergeCand-1), then the historical merge candidate export process ends because all features have been added to the merge candidate list as merge candidates. Figure 30 If the number of features registered in the current merge candidate list, numCurrMergeCand, is less than or equal to (MaxNumMergeCand-1), then proceed with the processing after step S2304. Set the value of sameMotion to FALSE. Figure 30 (Step S2304). Next, the initial value of index i is set to 0, and the process continues from this initial value to numOrigMergeCand-1. Figure 30 Processing of steps S2306 and S2307 ( Figure 30 (S2305~S2308). Compare whether the (NumHmvpCand-hMvpIdx)th element HmvpCandList[NumHmvpCand-hMvpIdx] in the historical motion vector prediction candidate list (starting from 0) is the same as the i-th element mergeCandList[i] in the merged candidate list (starting from 0). Figure 30 Step S2306).

[0218] The term "same value" for merge candidates refers to the fact that all the constituent elements (inter-frame prediction mode, reference index, and motion vector) of a merge candidate have the same value. This applies when the merge candidates have the same value and isPruned[i] is FALSE. Figure 30 Step S2306 is that both sameMotion and isPruned[i] are set to TRUE. Figure 30 Step S2307). In the case that the values ​​are not the same ( Figure 30 If step S2306 is not passed (no), skip step S2307. Figure 30 After the repeated processing of steps S2305 to S2308 is completed, compare whether sameMotion is FALSE. Figure 30 Step S2309), in the case that sameMotion is FALSE (false) Figure 30 Step S2309 is that, since the (NumHmvpCand-hMvpIdx)th element HmvpCandList[NumHvpCand-hMvpIdx] from the historical predicted motion vector candidate list (starting from 0) does not exist in mergeCandList, therefore, the (NumHmvpCand-hMvpIdx)th element HmvpCandList[NumHmvpCand-hMvpIdx] from the historical predicted motion vector candidate list (starting from 0) is added to mergeCandList[numCurrMergeCand] of the merged candidate list (starting from 0), and numCurrMergeCand is incremented by 1. Figure 30 Step S2310). Increment the index hMvpIdx by 1 ( Figure 30 Step S2302) is performed. Figure 30 Repeat steps S2302 to S2311.

[0219] After confirming all elements in the historical predicted motion vector candidate list, or adding all elements in the merge candidate list to the merge candidate list, the export process of the historical merge candidate is completed.

[0220] <Average Merging Candidate Export Processing>

[0221] Next, a detailed explanation will be provided as... Figure 30 The average merging candidate derivation method in step S403. Figure 30 The processing step S403 is a common process in the average merge candidate derivation unit 344 of the normal merge mode derivation unit 302 on the encoding side and the average merge candidate derivation unit 444 of the normal merge mode derivation unit 402 on the decoding side. Figure 21 This is a flowchart used to illustrate the average merge candidate export processing steps.

[0222] First, perform initialization processing ( Figure 21 Step S1301). Set the variable numOrigMergeCand to the number of features registered in the current merge candidate list, numCurrMergeCand.

[0223] Next, starting from the beginning of the candidate list to scan sequentially, two motion information pieces are determined. The index i representing the first motion information is set to 0, and the index j representing the second motion information is set to 1. Figure 39 Steps S1302 to S1303). If the number of features registered in the current merge candidate list, numCurrMergeCand, is not below (MaxNumMergeCand-1), then all features for which merge candidates have been added to the merge candidate list are merge candidates, and thus the historical merge candidate export process ends. Figure 39 (Step S1304). If the number of elements registered in the current merge candidate list, numCurrMergeCand, is less than (MaxNumMergeCand-1), then proceed with the processing after step S1305.

[0224] Determine whether both the i-th motion information mergeCandList[i] and the j-th motion information mergeCandList[j] in the candidate list are invalid. Figure 39 If both mergeCandList[i] and mergeCandList[j] are invalid, then the average merge candidate of mergeCandList[i] and mergeCandList[j] is not derived, and the process moves to the next feature. If mergeCandList[i] and mergeCandList[j] are not both invalid, then X is set to 0 and 1, and the following process is repeated ( Figure 39 Steps S1306 to S1314).

[0225] Determine whether the LX prediction of mergeCandList[i] is valid. Figure 39 Step S1307). If the LX prediction of mergeCandList[i] is valid, then determine whether the LX prediction of mergeCandList[j] is valid ( Figure 39 Step S1308). If the LX prediction of mergeCandList[j] is valid, that is, if both the LX predictions of mergeCandList[i] and mergeCandList[j] are valid, then derive the average merge candidate of the LX predictions with the motion vectors of the LX predictions and the reference index of the LX predictions of mergeCandList[i], and set it as the LX prediction of averageCand, making the LX prediction of averageCand valid. The motion vector of the LX prediction is obtained by averaging the motion vectors of the LX predictions of mergeCandList[i] and mergeCandList[j]. Figure 39 Step S1309). In Figure 39 If the LX predictor of mergeCandList[j] is not valid, i.e., if the LX predictor of mergeCandList[i] is valid and the LX predictor of mergeCandList[j] is not valid, the average merge candidate having the average of the motion vector and the reference index of the LX predictor of mergeCandList[i] is derived and set as the LX predictor of averageCand, and the LX predictor of averageCand is made valid (step S1308). In this case, the average of the motion vector and the reference index of the LX predictor of mergeCandList[i] is derived. Figure 39 If the LX predictor of mergeCandList[i] is not valid, it is determined whether the LX predictor of mergeCandList[j] is valid (step S1307). If the LX predictor of mergeCandList[j] is valid, i.e., if the LX predictor of mergeCandList[i] is not valid and the LX predictor of mergeCandList[j] is valid, the average merge candidate having the average of the motion vector and the reference index of the LX predictor of mergeCandList[j] is derived and set as the LX predictor of averageCand, and the LX predictor of averageCand is made valid (step S1311). In this case, the average of the motion vector and the reference index of the LX predictor of mergeCandList[j] is derived. Figure 39 Figure 39 If the LX predictor of mergeCandList[j] is not valid, i.e., if the LX predictor of mergeCandList[i] is not valid and the LX predictor of mergeCandList[j] is not valid, the LX predictor of averageCand is made not valid (step S1311). In this case, the LX predictor of averageCand is made not valid. Figure 39 Figure 39 Figure 39

[0226] The average merge candidate averageCand of the L0 predictor, the L1 predictor or the BI predictor generated as described above is added to the numCurrMergeCandth mergeCandList[numCurrMergeCand] of the merge candidate list, and numCurrMergeCand is increased by 1 (step S1315). Above, the derivation process of the average merge candidate is ended. Figure 39

[0227] In addition, the average merge candidate is averaged with the horizontal component of the motion vector and the vertical component of the motion vector, respectively.

[0228]

[0229] ​​​​​​The motion compensation prediction section 306 acquires the position and size of a block that is an object of prediction processing in encoding. In addition, the motion compensation prediction section 306 acquires inter prediction information from the inter prediction mode decision section 305. A reference index and a motion vector are derived from the acquired inter prediction information, and after acquiring an image signal of a reference picture determined by the reference index from the position of the image signal of the block that is an object of prediction processing moved by the amount of the motion vector from the same position, a prediction signal is generated.

[0230] In the case of inter prediction mode in inter prediction being prediction from a single reference picture like L0 prediction or L1 prediction, a prediction signal acquired from one reference picture is set as a motion compensation prediction signal, and in the case of inter prediction mode being prediction mode being prediction from two reference pictures like BI prediction, a signal obtained by weighted averaging of prediction signals acquired from two reference pictures is set as a motion compensation prediction signal, and the motion compensation prediction signal is supplied to the prediction method decision section 105. Here, the ratio of weighted averaging of bi-prediction is set to 1:1, but other ratios can be used for weighted averaging. For example, it can be set so that the closer the picture interval between the picture that is an object of prediction and the reference picture, the larger the ratio of weighting. In addition, a table of correspondence of combinations of picture intervals and ratios of weighting can be used for calculation of the ratio of weighting.

[0231] The motion compensation prediction section 406 has the same function as the motion compensation prediction section 306 on the encoding side. The motion compensation prediction section 406 acquires inter prediction information from the normal prediction motion vector mode derivation section 401, the normal merge mode derivation section 402, the sub-block prediction motion vector mode derivation section 403, and the sub-block merge mode derivation section 404 through the switch 408. The motion compensation prediction section 406 supplies the obtained motion compensation prediction signal to the decoded image signal superimposition section 207.

[0232] <Regarding Inter Prediction Mode>

[0233] Processing that performs prediction from a single reference picture is defined as single prediction. In the case of single prediction, L0 prediction or L1 prediction that uses either of two reference pictures registered in the reference list L0 or L1 is performed.

[0234] Figure 39 A case where the reference picture (RefL0Pic) of L0 in single prediction is at a time point before the processing object picture (CurPic) is shown. Figure 39 A case where the reference picture of L0 prediction in single prediction is at a time point after the processing object picture is shown. Similarly, it is also possible to perform prediction using a reference picture that is at a time point after the processing object picture by Figure 32 and Figure 33 The reference picture for L0 prediction is replaced with a reference picture for L1 prediction (RefL1Pic) to perform single prediction.

[0235] The processing to be performed according to the prediction from two reference pictures is defined as double prediction, and in the case of double prediction, both L0 prediction and L1 prediction are expressed as BI prediction. Figure 32 The case where the reference picture for L0 prediction in double prediction is at a time point before the processing target picture and the reference picture for L1 prediction is at a time point after the processing target picture is shown. Figure 33 The case where the reference picture for L0 prediction and the reference picture for L1 prediction in double prediction are at a time point before the processing target picture is shown. Figure 34 The case where the reference picture for L0 prediction and the reference picture for L1 prediction in double prediction are at a time point after the processing target picture is shown.

[0236] Thus, the prediction category of L0 / L1 and the relationship with time can be used in a case where L0 is not limited to the past direction and L1 is not limited to the future direction. In addition, in the case of double prediction, each of L0 prediction and L1 prediction can be performed using the same reference picture. Furthermore, whether to perform motion compensation prediction in single prediction or in double prediction is determined based on information (e.g., a flag) indicating whether to use L0 prediction and whether to use L1 prediction.

[0237] <About Reference Index>

[0238] In the embodiment of the present application, in order to improve the accuracy of motion compensation prediction, the best reference picture can be selected from a plurality of reference pictures in motion compensation prediction. Therefore, the reference picture to be used in motion compensation prediction is used as a reference index, and the reference index is encoded into a bitstream together with a differential motion vector.

[0239] <Motion Compensation Processing Based on Normal Prediction Motion Vector Mode>

[0240] As shown in the inter prediction section 102 on the encoding side in Figure 35 As shown in the inter prediction section 102 on the encoding side in

[0241] Likewise, as shown in the inter prediction section 102 on the encoding side in Figure 36 As also shown in the inter-frame prediction unit 203 on the decoding side, when switch 408 is connected to the normal prediction motion vector mode derivation unit 401 during decoding, motion compensation prediction unit 406 acquires the inter-frame prediction information based on the normal prediction motion vector mode derivation unit 401, derives the inter-frame prediction mode, reference index, and motion vector of the block to be processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the decoded image signal overlay unit 207.

[0242] <Motion compensation processing based on the usual merging pattern>

[0243] As in Figure 16 As also shown in the inter-frame prediction unit 102 on the encoding side, when inter-frame prediction information based on the normal merging mode derivation unit 302 is selected in the inter-frame prediction mode determination unit 305, the motion compensation prediction unit 306 obtains the inter-frame prediction information from the inter-frame prediction mode determination unit 305, derives the inter-frame prediction mode, reference index, and motion vector of the block to be processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the prediction method determination unit 105.

[0244] Similarly, as in Figure 22 As also shown in the inter-frame prediction unit 203 on the decoding side, when switch 408 is connected to the normal merging mode derivation unit 402 during decoding, motion compensation prediction unit 406 acquires inter-frame prediction information based on the normal merging mode derivation unit 402, derives the inter-frame prediction mode, reference index, and motion vector of the block currently being processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the decoded image signal overlay unit 207.

[0245] <Motion Compensation Processing Based on Sub-Block Predicted Motion Vector Patterns>

[0246] As in Figure 16 As also shown in the inter-frame prediction unit 102 on the encoding side, when the inter-frame prediction information based on the sub-block prediction motion vector mode derivation unit 303 is selected in the inter-frame prediction mode determination unit 305, the motion compensation prediction unit 306 obtains the inter-frame prediction information from the inter-frame prediction mode determination unit 305, derives the inter-frame prediction mode, reference index, and motion vector of the block to be processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the prediction method determination unit 105.

[0247] Similarly, as in Figure 22 As also shown in the inter prediction section 203 on the decoding side, when the switch 408 is connected to the sub-block prediction motion vector mode derivation section 403 in the decoding process, the motion compensation prediction section 406 acquires the inter prediction information based on the sub-block prediction motion vector mode derivation section 403, derives the inter prediction mode, the reference index, and the motion vector of the block that is the current processing target, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is supplied to the decoded image signal superimposition section 207.

[0248] <Motion compensation processing based on sub-block merge mode>

[0249] As also shown in the inter prediction section 102 on the encoding side, when the inter prediction information based on the sub-block merge mode derivation section 304 is selected in the inter prediction mode decision section 305, the motion compensation prediction section 306 acquires the inter prediction information from the inter prediction mode decision section 305, derives the inter prediction mode, the reference index, and the motion vector of the block that is the current processing target, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is supplied to the prediction method decision section 105. Figure 16

[0250] Likewise, as also shown in the inter prediction section 203 on the decoding side, when the switch 408 is connected to the sub-block merge mode derivation section 404 in the decoding process, the motion compensation prediction section 406 acquires the inter prediction information based on the sub-block merge mode derivation section 404, derives the inter prediction mode, the reference index, and the motion vector of the block that is the current processing target, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is supplied to the decoded image signal superimposition section 207. Figure 22 <Motion compensation processing based on affine transform prediction>

[0251] In the normal prediction motion vector mode and the normal merge mode, motion compensation based on an affine model can be used based on the following flag. The following flag is reflected in the following flag based on the inter prediction condition decided by the inter prediction mode decision section 305 in the encoding process, and is encoded into the bitstream. In the decoding process, whether or not to perform motion compensation based on an affine model is determined based on the following flag in the bitstream.

[0252]

[0253] ​​sps_affine_enabled_flag indicates whether affine model based motion compensation can be used in inter prediction. If sps_affine_enabled_flag is 0, it is prohibited on a sequence level such that affine model based motion compensation is not used. In addition, inter_affine_flag and cu_affine_type_flag are not signaled in the CU (coding block) syntax of the coded video sequence. If sps_affine_enabled_flag is 1, affine model based motion compensation can be used in the coded video sequence.

[0254] sps_affine_type_flag indicates whether six-parameter affine model based motion compensation can be used in inter prediction. If sps_affine_type_flag is 0, it is prohibited that six-parameter affine model based motion compensation is used. In addition, cu_affine_type_flag is not signaled in the CU syntax of the coded video sequence. If sps_affine_type_flag is 1, six-parameter affine model based motion compensation can be used in the coded video sequence. In the absence of sps_affine_type_flag, it is set to be 0.

[0255] In the case of decoding a P slice or a B slice, in the CU that becomes the current processing target, if inter_affine_flag is 1, affine model based motion compensation is used to generate the motion compensation prediction signal of the CU that becomes the current processing target. If inter_affine_flag is 0, affine model is not used for the CU that becomes the current processing target. In the absence of inter_affine_flag, it is set to be 0.

[0256] In the case of decoding a P slice or a B slice, in the CU that becomes the current processing target, if cu_affine_type_flag is 1, six-parameter affine model based motion compensation is used to generate the motion compensation prediction signal of the CU that becomes the current processing target. If cu_affine_type_flag is 0, four-parameter affine model based motion compensation is used to generate the motion compensation prediction signal of the CU that becomes the current processing target.

[0257] In affine model based motion compensation, because the reference index or the motion vector is derived in a sub-block unit, the motion compensation prediction signal is generated using the reference index or the motion vector that becomes the processing target in a sub-block unit.

[0258] The four-parameter affine model is a mode in which a motion vector of a sub-block is derived from four parameters of horizontal and vertical components of motion vectors of two control points, and motion compensation is performed in units of sub-blocks.

[0259] <Triangle Merge Mode>

[0260] The triangle merge mode is one of the merge modes, and is a mode in which a coding / decoding block is divided into diagonal partitions and motion-compensated prediction is performed. The triangle merge mode is one of the geometric partition merge modes in which a coding / decoding block is divided into blocks of non-rectangular shapes, and is equivalent to a mode in which a coding / decoding block is divided into two right-angled triangles by a diagonal line in the geometric partition merge mode.

[0261] The geometric partition merge mode is expressed by, for example, a combination of an index (angleldx) indicating a partition angle and an index (distanceIdx) indicating a distance from the center of a coding block. In one example, 64 patterns are defined as the geometric partition merge mode, and are fixed-length coded. In the 64 patterns, the index indicating the partition angle indicates an angle of a diagonal line constituting a coding block (for example, 45 degrees (angleldx = 4 in a structure in which 360 degrees are expressed by 32 divisions) or 135 degrees (angleldx = 12 in a structure in which 360 degrees are expressed by 32 divisions)), and the index indicating the distance from the center of the coding block indicates 2 patterns of the minimum (distanceIdx = 0, indicating that the partition boundary passes through the center of the coding block), which are equivalent to a mode in which the coding block is divided by a diagonal line, that is, the triangle merge mode.

[0262] Using Figure 16 and Figure 22 The triangle merge mode is described. Figure 38A and 38B An example of prediction of a coding / decoding block in the triangle merge mode of 16 x 16 is shown. The coding / decoding block in the triangle merge mode is divided into sub-blocks of 4 x 4, and each sub-block is allocated to three partitions of partition 0 of single prediction (UNI0), partition 1 of single prediction (UNI1), and partition 2 of double prediction (BI). Here, the sub-blocks located on the upper side of the diagonal line are allocated to partition 0, the sub-blocks located on the lower side of the diagonal line are allocated to partition 1, and the sub-blocks located on the diagonal line are allocated to partition 2. If merge_triangle_split_dir is 0, the partitions are allocated as in Figure 38B , and if merge_triangle_split_dir is 1, the partitions are allocated as in Figure 38A .

[0263] The single prediction motion information specified by the merge triangle index 0 is used for the motion-compensated prediction of the partition 0. The single prediction motion information specified by the merge triangle index 1 is used for the motion-compensated prediction of the partition 1. For the motion-compensated prediction of the partition 2, the bi-prediction motion information that combines the single prediction motion information specified by the merge triangle index 0 and the single prediction motion information specified by the merge triangle index 1 is used.

[0264] Here, the single prediction motion information is one set of motion vector and reference index, and the bi-prediction motion information is composed of two sets of motion vector and reference index. In addition, the motion information refers to the single prediction motion information or the bi-prediction motion information.

[0265] The merge candidate selection sections 347 and 447 use the derived merge candidate list mergeCandList as the triangle merge candidate list triangleMergeCandList.

[0266] The flowchart relating to the derivation of the triangle merge candidate is described. Figure 38A The flowchart relating to the derivation of the triangle merge candidate is described.

[0267] First, the merge candidate list mergeCandList is used as the triangle merge candidate list triangleMergeCandList (step S3501).

[0268] Next, the single prediction motion information of the merge triangle partition 0 is derived by giving priority to the merge candidate that includes the motion information list L0 (step S3502).

[0269] Next, the single prediction motion information of the merge triangle partition 1 is derived by giving priority to the merge candidate that includes the motion information list L1 (step S3503).

[0270] In addition, the step S3502 and the step S3503 can be derived in different order, or can be processed in parallel.

[0271] Figure 38B The flowchart relating to the derivation of the single prediction motion information of the merge triangle partition 0 of the first embodiment is described.

[0272] First, for the Mth candidate in the derived merge candidate list mergeCandList, it is determined whether the candidate M has the motion information of the motion information list L0 (step S3601). In the case where the candidate M has the motion information of the motion information list L0, the motion information of the motion information list L0 of the candidate M is set as the triangle merge candidate (step S3602). For the candidate M (M = 0, 1,..., numMergeCand - 1), the step S3601 and the step S3602 are performed in ascending order, and the triangle merge candidate is additionally derived.

[0273] Next, for the Mth candidate in the derived merge candidate list mergeCandList, it is determined whether the candidate M has the motion information of the motion information list L1 (step S3603). In the case where the candidate M has the motion information of the motion information list L1, the motion information of the motion information list L1 of the candidate M is set as the triangular merge candidate (step S3604). The steps S3603 and S3604 are performed in descending order for the candidates M (M = numMergeCand - 1,..., 1, 0), and the triangular merge candidate is appended.

[0274] Figure 40 is a flowchart illustrating derivation of the single prediction motion information of the merge triangular partition 1 of the first embodiment.

[0275] First, for the Mth candidate in the derived merge candidate list mergeCandList, it is determined whether the candidate M has the motion information of the motion information list L1 (step S3701). In the case where the candidate M has the motion information of the motion information list L1, the motion information of the motion information list L1 of the candidate M is set as the triangular merge candidate (step S3702). The steps S3701 and S3702 are performed in ascending order for the candidates M (M = 0, 1,..., numMergeCand - 1), and the triangular merge candidate is appended.

[0276] Next, for the Mth candidate in the derived merge candidate list mergeCandList, it is determined whether the candidate M has the motion information of the motion information list L0 (step S3703). In the case where the candidate M has the motion information of the motion information list L0, the motion information of the motion information list L0 of the candidate M is set as the triangular merge candidate (step S3704). The steps S3703 and S3704 are performed in descending order for the candidates M (M = numMergeCand - 1,..., 1, 0), and the triangular merge candidate is appended.

[0277] Figure 41 and Figure 42 is a diagram illustrating an example of the motion information of the triangular merge candidate of the first embodiment.

[0278] Figure 43A An example of the merge candidate list is shown. In the merge candidate of merge index 0, the inter prediction mode is bi-prediction (Pred-BI), the motion information of the motion information list L0 is MV0_L0, and the motion information of the motion information list L1 is MV0_L1. In the merge candidate of merge index 1, the inter prediction mode is uni-prediction (Pred-L0), the motion information of the motion information list L0 is MV1_L0, and there is no motion information of the motion information list L1. In the merge candidate of merge index 2, the inter prediction mode is uni-prediction (Pred-L1), there is no motion information of the motion information list L0, and the motion information of the motion information list L1 is MV2_L1. In the merge candidate of merge index 3, the inter prediction mode is bi-prediction (Pred_BI), the motion information of the motion information list L0 is MV3_L0, and the motion information of the motion information list L1 is MV3_L1. In the merge candidate of merge index 4, the inter prediction mode is uni-prediction (Pred-L0), the motion information of the motion information list L0 is MV4_L0, and there is no motion information of the motion information list L1.

[0279] Figure 43B is a diagram showing the motion information of the merge triangle partition 0 and the merge triangle partition 1 in the case of an example of the merge candidate list of Figure 43A .

[0280] The uni-prediction motion information candidates of the merge triangle partition 0 are constituted in the order of MV0_L0, MV1_L0, MV3_L0, MV4_L0, MV3_L1. The uni-prediction motion information candidates of the merge triangle partition 1 are constituted in the order of MV0_L1, MV2_L1, MV3_L1, MV4_L0, MV3_L0.

[0281] Here, the maximum values of the merge triangle index 0 (merge_triangle_idx0) and the merge triangle index 1 (merge_triangle_idx1) are the same as the merge index (merge_idx). That is, the maximum number of triangle merge candidates of the merge triangle partition 0 and the merge triangle partition 1 is equal to the maximum number of merge candidates MaxNumMergeCand. Thus, the contexts of the CABAC (Context-based Adaptive Binary Arithmetic Coding) of the merge index, the merge triangle index 0, and the merge triangle index 1 can be made common, and the view and the context table can be simplified and the memory can be saved. In addition, the maximum value (the maximum number of merge candidates) of the merge index is coded (decoded) by the SPS (Sequence Parameter Set). Also, the maximum values (the maximum numbers of triangle merge candidates) of the merge triangle index 0 and the merge triangle index 1 are coded (decoded) by the SPS.

[0282] Thus, by changing the order of the motion information of the merge triangle partition 0 and the merge triangle partition 1, the merge candidate list is used again, while the possibility of the repetition of the motion information of the merge triangle partition 0 and the merge triangle partition 1 is suppressed, and the triangle merge candidate with high coding efficiency is derived. The above is a feature of the present application, and the single prediction motion information priority motion information list L1 of the merge triangle partition 0 can be derived, and the single prediction motion information priority motion information list L0 of the merge triangle partition 1 can be derived.

[0283] (Modified Example 1)

[0284] In the modified example 1, unlike the merge triangle index in the first embodiment, the maximum numbers of triangle merge candidates of the merge triangle index 0 and the merge triangle index 1 are different from each other. Here, the maximum number of triangle merge candidates of the merge triangle index 1 is smaller than the maximum number of triangle merge candidates of the merge triangle index 0 by 1. At this time, as the single prediction motion information candidate of the merge triangle index 1, the merge candidate selected in the merge triangle index 0 is excluded. That is, the maximum number of triangle merge candidates of the merge triangle index 0 is the same as the maximum number of merge candidates of the merge index, and the maximum number of triangle merge candidates of the merge triangle index 1 is smaller than the maximum number of merge candidates of the merge index by 1.

[0285] As described above, by making the maximum number of the merge triangle index 1 smaller than the merge triangle index 0 by 1, the coding efficiency is improved in the coding string such as the Truncated Unary which improves the coding efficiency according to the number of candidates. In addition, by excluding the repetition of the merge candidate, the coding efficiency can be improved.

[0286] Here, in encoding of a general image, the utilization rate of the triangle merge mode is lower than that of the general merge mode. Therefore, the maximum number of triangle merge candidates of merge triangle index 0 can also be made smaller than the maximum number of merge candidates of the merge index. In addition, for the merge candidate list, the closer the candidate is to the beginning, the more effective the candidate, and the higher the selection rate, and the farther the candidate is from the beginning, the less effective the candidate, and the lower the selection rate. Therefore, in the triangle merge mode, the selection rate of the candidate farther from the beginning, that is, the larger candidate in merge triangle index 0 or merge triangle index 1, is low, and thus it is possible to improve the processing efficiency while suppressing a decrease in coding efficiency.

[0287] (Second Embodiment)

[0288] In the second embodiment, the order of candidate search within the reference list 1 in the merge triangle partition 0 and the merge triangle partition 1 is different from that of the first embodiment. The other configurations are the same as those of the first embodiment.

[0289] Figure 43B is a flowchart illustrating derivation of the single prediction motion information of the merge triangle partition 0 of the second embodiment.

[0290] First, for the Mth candidate within the derived merge candidate list mergeCandList, it is determined whether the candidate M has motion information of the motion information list L0 (step S3801). In the case where the candidate M has the motion information of the motion information list L0, the motion information of the motion information list L0 of the candidate M is set as a triangle merge candidate (step S3802). For the candidate M (M = 0, 1,..., numMergeCand - 1), steps S3801 and S3802 are performed in ascending order, and a triangle merge candidate is appended to be derived.

[0291] Next, for the Mth candidate within the derived merge candidate list mergeCandList, it is determined whether the candidate M has motion information of the motion information list L1 (step S3803). In the case where the candidate M has the motion information of the motion information list L1, the motion information of the motion information list L1 of the candidate M is set as a triangle merge candidate (step S3804). For the candidate M (M = 0, 1,..., numMergeCand - 1), steps S3803 and S3804 are performed in ascending order, and a triangle merge candidate is appended to be derived.

[0292] Figure 43A is a flowchart illustrating derivation of the single prediction motion information of the merge triangle partition 1 of the second embodiment.

[0293] First, for the Mth candidate in the derived merge candidate list mergeCandList, it is determined whether the candidate M has motion information of the motion information list L1 (step S3901). In the case where the candidate M has motion information of the motion information list L1, the motion information of the motion information list L1 of the candidate M is set as a triangular merge candidate (step S3902). For the candidates M (M = numMergeCand - 1,..., 1, 0), the step S3901 and the step S3902 are performed in descending order, and the triangular merge candidate is appended.

[0294] Next, for the Mth candidate in the derived merge candidate list mergeCandList, it is determined whether the candidate M has motion information of the motion information list L0 (step S3903). In the case where the candidate M has motion information of the motion information list L0, the motion information of the motion information list L0 of the candidate M is set as a triangular merge candidate (step S3904). For the candidates M (M = numMergeCand - 1,..., 1, 0), the step S3903 and the step S3904 are performed in descending order, and the triangular merge candidate is appended.

[0295] Figure 44 and Figure 45 is a diagram illustrating an example of the motion information of the triangular merge candidate of the second embodiment. Figure 46A shows an example of the merge candidate list, and is the same as Figure 46B

[0296] Figure 46A is a diagram illustrating what information the single prediction motion information candidate of the merge triangle partition 0 and the single prediction motion information of the merge triangle partition 1 are, in the case of the example of the merge candidate list of Figure 43A

[0297] The single prediction motion information candidate of the merge triangle partition 0 is constituted by motion information in the order of MV0_L0, MV1_L0, MV3_L0, MV4_L0, MV0_L1. The single prediction motion information candidate of the merge triangle partition 1 is constituted by motion information in the order of MV3_L1, MV2_L1, MV0_L1, MV4_L0, MV3_L0.

[0298] ​​Here, the maximum values of the merge triangle index 0 (merge_triangle_idx0) and the merge triangle index 1 (merge_triangle_idx1) are the same as the merge index (merge_idx). That is, the maximum number of triangle merge candidates of the merge triangle partition 0 and the merge triangle partition 1 is equal to the maximum number of merge candidates MaxNumMergeCand. Thus, the contexts of the CABAC (Context-based Adaptive Binary Arithmetic Coding) of the merge index, the merge triangle index 0, and the merge triangle index 1 can be made common, and the view and the context table can be simplified and the memory can be saved. In addition, the maximum value (the maximum number of merge candidates) of the merge index is coded (decoded) by the SPS (Sequence Parameter Set). Also, the maximum values (the maximum numbers of triangle merge candidates) of the merge triangle index 0 and the merge triangle index 1 are coded (decoded) by the SPS.

[0299] Thus, by changing the priority order of the motion information of the merge triangle partition 0 and the merge triangle partition 1, the merge candidate list is used again, while the possibility of the repetition of the motion information of the merge triangle partition 0 and the merge triangle partition 1 is suppressed, and the triangle merge candidate with high coding efficiency is derived. The above is a feature of the present application, and the single prediction motion information priority motion information list L1 of the merge triangle partition 0 can be derived, and the single prediction motion information priority motion information list L0 of the merge triangle partition 1 can be derived.

[0300] (Modified Example 1)

[0301] In the modified example 1, unlike the merge triangle index in the second embodiment, the maximum number of triangle merge candidates of the merge triangle index 0 and the merge triangle index 1 are different from each other. Here, the maximum number of triangle merge candidates of the merge triangle index 1 is smaller than the maximum number of triangle merge candidates of the merge triangle index 0 by 1. At this time, as the single prediction motion information candidate of the merge triangle index 1, the merge candidate selected in the merge triangle index 0 is excluded. That is, the maximum number of triangle merge candidates of the merge triangle index 0 is the same as the maximum number of merge candidates of the merge index, and the maximum number of triangle merge candidates of the merge triangle index 1 is smaller than the maximum number of merge candidates of the merge index by 1.

[0302] As described above, by making the maximum number of the merge triangle index 1 smaller than the merge triangle index 0 by 1, the coding efficiency is improved in the coding string such as the Truncated Unary which improves the coding efficiency according to the number of candidates. In addition, by excluding the repetition of the merge candidate, the coding efficiency can be improved.

[0303] Here, in encoding of a general image, the utilization rate of the triangle merge mode is lower than that of the general merge mode. Therefore, the maximum number of triangle merge candidates of merge triangle index 0 can also be made smaller than the maximum number of merge candidates of merge index. Further, for the merge candidate list, the closer the candidate is to the beginning, the more effective the candidate, and the higher the selection rate, and the farther the candidate is from the beginning, the less effective the candidate, and the lower the selection rate. Therefore, in the triangle merge mode, the selection rate of the candidate farther from the beginning, that is, the larger candidate in merge triangle index 0 or merge triangle index 1, is low, and thus it is possible to improve the processing efficiency while suppressing a decrease in coding efficiency.

[0304] In all the embodiments described above, the bitstream output from the image encoding apparatus has a specific data format so as to be decodable in accordance with the encoding method used in the embodiment. Further, an image decoding apparatus corresponding to the image encoding apparatus can decode the bitstream of the specific data format.

[0305] In a case where a wired or wireless network is used for exchanging the bitstream between the image encoding apparatus and the image decoding apparatus, the bitstream can be converted into a data format suitable for a transmission form of the communication line and transmitted. In this case, a transmission apparatus that converts the bitstream output from the image encoding apparatus into encoded data of a data format suitable for a transmission form of the communication line and transmits the encoded data to the network, and a reception apparatus that receives the encoded data from the network and restores the encoded data into the bitstream to supply the image decoding apparatus are provided. The transmission apparatus includes a memory that buffers the bitstream output from the image encoding apparatus, a packet processing section that packetizes the bitstream, and a transmission section that transmits the encoded data that has been packetized via the network. The reception apparatus includes a reception section that receives the encoded data that has been packetized via the network, a memory that buffers the received encoded data, and a packet processing section that packetizes the encoded data to generate the bitstream and supplies the bitstream to the image decoding apparatus.

[0306] Further, it is also possible to add a display section that displays an image decoded by the image decoding apparatus to the configuration as a display apparatus. In this case, the display section reads out the decoded image signal generated by the decoded image signal superimposition section 207 and saved in the decoded image memory 208, and displays it on a screen.

[0307] Further, it is also possible to add a photographing section to the configuration, input an image photographed to the image encoding apparatus, and thereby function as a photographing apparatus. In this case, the photographing section inputs the photographed image signal to the block division section 101.

[0308] Figure 46B Figure 46A Figure 37 An example of a hardware structure of an encoding and decoding apparatus of the present embodiment is shown. The encoding and decoding apparatus includes the structures of the image encoding apparatus and the image decoding apparatus according to the present embodiment. The encoding and decoding apparatus 9000 has a CPU 9001, a codec IC 9002, an I / O interface 9003, a memory 9004, an optical disk drive 9005, a network interface 9006, a video interface 9009, and the like, which are connected by a bus 9010.

[0309] The image encoding section 9007 and the image decoding section 9008 are typically mounted as the codec IC 9002. The image encoding processing of the image encoding apparatus according to the present embodiment is performed by the image encoding section 9007, and the image decoding processing in the image decoding apparatus according to the present embodiment is performed by the image decoding section 9008. The I / O interface 9003 is realized by, for example, a USB interface, and is connected to an external keyboard 9104, a mouse 9105, and the like. The CPU 9001 controls the encoding and decoding apparatus 9000 to perform an action desired by a user, based on a user operation input through the I / O interface 9003. As the operation by the user through the keyboard 9104, the mouse 9105, and the like, there are a selection of which function of encoding and decoding to perform, a setting of the encoding quality, an input / output destination of a bitstream, an input / output destination of an image, and the like.

[0310] When the user desires to reproduce an image recorded in the disk recording medium 9100, the optical disk drive 9005 reads out a bitstream from the inserted disk recording medium 9100, and transmits the read-out bitstream to the image decoding section 9008 of the codec IC 9002 via the bus 9010. The image decoding section 9008 performs the image decoding processing in the image decoding apparatus according to the present embodiment on the input bitstream, and transmits a decoded image to an external monitor 9103 via the video interface 9009. In addition, the encoding and decoding apparatus 9000 has the network interface 9006, and can be connected to an external distribution server 9106, a mobile terminal 9107 via a network 9101. When the user desires to reproduce an image recorded on the distribution server 9106 or the mobile terminal 9107, instead of an image recorded on the disk recording medium 9100, the network interface 9006 acquires a bitstream from the network 9101, instead of reading out a bitstream from the input disk recording medium 9100. In addition, in a case where the user desires to reproduce an image recorded in the memory 9004, the image decoding processing in the image decoding apparatus according to the present embodiment is performed on a bitstream recorded in the memory 9004.

[0311] In a case where the user desires to encode and record an image captured by the external camera 9102 in the memory 9004, the video interface 9009 inputs the image from the camera 9102, and sends it to the image encoding section 9007 of the codec IC 9002 via the bus 9010. The image encoding section 9007 performs the image encoding processing in the image encoding apparatus according to the embodiment of the present application on the image input via the video interface 9009, and generates a bitstream. The bitstream is then sent to the memory 9004 through the bus 9010. When the user desires to record the bitstream on the disk recording medium 9100 instead of the memory 9004, the optical disk drive 9005 performs the writing of the bitstream with respect to the disk recording medium 9100 inserted.

[0312] A hardware structure having only the image encoding apparatus or having only the image decoding apparatus can also be realized. Such a hardware structure is realized, for example, by replacing the codec IC 9002 with the image encoding section 9007 or the image decoding section 9008, respectively.

[0313] The processing related to the above-described encoding and decoding can of course be realized as a transmitting, storing, and receiving apparatus using hardware, and can be realized by a firmware or software of a computer or the like stored in a ROM (Read Only Memory), a flash memory, or the like. The firmware program, the software program can be provided recorded in a recording medium readable by a computer or the like, can be provided from a server through a wired or wireless network, and can also be provided as a data broadcast of a terrestrial wave or a satellite digital broadcast.

[0314] The above describes the present application based on the embodiments. The embodiments are examples, and the combination of these respective constituent elements and respective processing steps can have various modifications, and such modifications are also within the scope of the present application, as can be understood by those skilled in the art.

[0315] Industrial Applicability

[0316] The present application can be utilized in image encoding and decoding techniques that divide an image into blocks to perform prediction.

[0317] Explanation of Symbols

[0318] 100 image encoding apparatus, 101 block division section, 102 inter prediction section, 103 intra prediction section, 104 decoded image memory, 105 prediction method decision section, 106 residual generation section, 107 orthogonal transform / quantization section, 108 bit string encoding section, 109 inverse quantization / inverse orthogonal transform section, 110 decoded image signal superimposition section, 111 encoding information holding memory, 200 image decoding apparatus, 201 bit string decoding section, 202 block division section, 203 inter prediction section, 204 intra prediction section, 205 encoding information holding memory, 206 inverse quantization / inverse orthogonal transform section, 207 decoded image signal superimposition section, 208 decoded image memory.< / poc>

Claims

1. An image coding apparatus that codes a moving picture using inter prediction based on inter prediction information in units of blocks, generates a bitstream, characterized by comprising: a general merge candidate list construction section that constructs a general merge candidate list including general merge candidates; a general merge candidate selection section that selects a general merge candidate that becomes a uni-prediction or bi-prediction uni-prediction from the general merge candidate list according to a merge index; a triangle merge candidate selection section that selects a first triangle merge candidate that becomes a uni-prediction from the general merge candidate list according to a first triangle merge index, and selects a second triangle merge candidate that becomes a uni-prediction from the general merge candidate list according to a second triangle merge index; and a coding section that codes (i) the merge index or (ii) the first triangle merge index and the second triangle merge index, the first triangle merge index and the second triangle merge index selecting the first triangle merge candidate and the second triangle merge candidate from the general merge candidate list in different priority order respectively.

2. An image coding method that codes a moving picture using inter prediction based on inter prediction information in units of blocks, generates a bitstream, characterized by comprising: a general merge candidate list construction step that constructs a general merge candidate list including general merge candidates; a general merge candidate selection step that selects a general merge candidate that becomes a uni-prediction or bi-prediction uni-prediction from the general merge candidate list according to a merge index; a triangle merge candidate selection step that selects a first triangle merge candidate that becomes a uni-prediction from the general merge candidate list according to a first triangle merge index, and selects a second triangle merge candidate that becomes a uni-prediction from the general merge candidate list according to a second triangle merge index; and a coding step that codes (i) the merge index or (ii) the first triangle merge index and the second triangle merge index, the first triangle merge index and the second triangle merge index selecting the first triangle merge candidate and the second triangle merge candidate from the general merge candidate list in different priority order respectively.

3. An image decoding apparatus that decodes a bitstream coded using inter prediction in units of blocks, characterized by comprising: a decoding section that decodes (i) a merge index or (ii) a first triangle merge index and a second triangle merge index; a general merge candidate list construction section that constructs a general merge candidate list including general merge candidates; a general merge candidate selection section that selects a general merge candidate that becomes a uni-prediction or bi-prediction uni-prediction from the general merge candidate list according to the merge index; and a triangle merge candidate selection section that selects a first triangle merge candidate that becomes a uni-prediction from the general merge candidate list according to the first triangle merge index, and selects a second triangle merge candidate that becomes a uni-prediction from the general merge candidate list according to the second triangle merge index. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ a triangle merge candidate selection section selects a first triangle merge candidate that is a single prediction from the general merge candidate list according to the first triangle merge index, and selects a second triangle merge candidate that is a single prediction from the general merge candidate list according to the second triangle merge index, the first triangle merge index and the second triangle merge index select the first triangle merge candidate and the second triangle merge candidate from the general merge candidate list in different priority order, respectively.

4. An image decoding method of decoding a bitstream generated by encoding a moving picture using inter prediction in units of blocks, the image decoding method characterized by comprising: a decoding step of decoding (i) a merge index or (ii) a first triangle merge index and a second triangle merge index; a general merge candidate list construction step of constructing a general merge candidate list including general merge candidates; a general merge candidate selection step of selecting a general merge candidate that is a single prediction or a bi-prediction from the general merge candidate list according to the merge index; and a triangle merge candidate selection step of selecting a first triangle merge candidate that is a single prediction from the general merge candidate list according to the first triangle merge index, and selecting a second triangle merge candidate that is a single prediction from the general merge candidate list according to the second triangle merge index, the first triangle merge index and the second triangle merge index select the first triangle merge candidate and the second triangle merge candidate from the general merge candidate list in different priority order, respectively.

5. A storage method of storing a bitstream generated by the image encoding method according to claim 2 in a recording medium.

6. A transmission method of transmitting a bitstream generated by the image encoding method according to claim 2.

Images