Image encoding apparatus and method, image decoding apparatus and method

Abstract

The present invention relates to an image encoding apparatus and method, and an image decoding apparatus and method. A technique for improving encoding efficiency by performing block partitioning suitable for image encoding and decoding is provided. The image encoding apparatus includes an encoding information holding section that adds inter-frame prediction information used in inter-frame prediction to a history prediction motion vector candidate list, a history prediction motion vector candidate derivation section that derives a history prediction motion vector candidate from the history prediction motion vector candidate list, and a history merge candidate derivation section that derives a history merge candidate from the history prediction motion vector candidate list, the history prediction motion vector candidate derivation section and the history merge candidate derivation section referring to candidates included in the history prediction motion vector candidate list as the history prediction motion vector candidate or the history merge candidate in mutually different orders.

Description

[0001] This divisional application is a divisional application of an invention patent filed on December 11, 2019, with application number 201980044863.X and applicant JVC Kenwood Corporation. The invention patent application is entitled "Image Encoding Apparatus, Image Encoding Method, Image Encoding Program, Image Decoding Apparatus, Image Decoding Method and Image Decoding Program". Technical Field

[0002] This invention relates to image encoding and decoding techniques for segmenting images into blocks and making predictions. Background Technology

[0003] In image encoding and decoding, the image to be processed is divided into blocks, which are sets of a predetermined number of pixels, and processed on a block-by-block basis. By dividing the image into appropriate blocks and appropriately setting intra-frame prediction and inter-frame prediction, encoding efficiency is improved.

[0004] In the encoding / decoding of moving images, encoding efficiency is improved by performing inter-frame prediction based on already encoded / decoded images. Patent Document 1 discloses a technique that applies affine transformation during inter-frame prediction. In moving images, objects often undergo deformations such as magnification, reduction, and rotation; by applying the technique in Patent Document 1, efficient encoding can be achieved.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent document 1: Japanese Patent Application Publication No. 9-172644 Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] However, the technology in Patent Document 1 involves image transformations, resulting in a high processing load. In view of the above problems, this invention provides a low-load and efficient encoding technology.

[0010] means of solving technical problems

[0011] To address the aforementioned problems, the image encoding apparatus of the first aspect of the present invention includes: an encoding information storage unit that adds the inter-frame prediction information used in inter-frame prediction to a historical prediction motion vector candidate list; a historical prediction motion vector candidate derivation unit that derives historical prediction motion vector candidates from the historical prediction motion vector candidate list; and a historical merging candidate derivation unit that derives historical merging candidates from the historical prediction motion vector candidate list, wherein the historical prediction motion vector candidate derivation unit and the historical merging candidate derivation unit reference candidates included in the historical prediction motion vector candidate list in different orders as historical prediction motion vector candidates or historical merging candidates.

[0012] The image encoding method of the second aspect of the present invention includes: an encoding information saving step, which adds the inter-frame prediction information used in inter-frame prediction to a historical prediction motion vector candidate list; a historical prediction motion vector candidate derivation step, which derives historical prediction motion vector candidates from the historical prediction motion vector candidate list; and a historical merging candidate derivation step, which derives historical merging candidates from the historical prediction motion vector candidate list, wherein in the historical prediction motion vector candidate derivation step and the historical merging candidate derivation step, candidates included in the historical prediction motion vector candidate list are referenced in different orders as historical prediction motion vector candidates or historical merging candidates.

[0013] The image encoding program of the third aspect of the present invention causes a computer to perform the following steps: an encoding information saving step, which adds the inter-frame prediction information used in inter-frame prediction to a historical prediction motion vector candidate list; a historical prediction motion vector candidate derivation step, which derives historical prediction motion vector candidates from the historical prediction motion vector candidate list; and a historical merging candidate derivation step, which derives historical merging candidates from the historical prediction motion vector candidate list, wherein in the historical prediction motion vector candidate derivation step and the historical merging candidate derivation step, candidates included in the historical prediction motion vector candidate list are referenced in different orders as historical prediction motion vector candidates or historical merging candidates.

[0014] The image decoding apparatus of the fourth aspect of the present invention includes: an encoding information storage unit that adds the inter-frame prediction information used in inter-frame prediction to a historical prediction motion vector candidate list; a historical prediction motion vector candidate derivation unit that derives historical prediction motion vector candidates from the historical prediction motion vector candidate list; and a historical merging candidate derivation unit that derives historical merging candidates from the historical prediction motion vector candidate list, wherein the historical prediction motion vector candidate derivation unit and the historical merging candidate derivation unit reference candidates included in the historical prediction motion vector candidate list in different orders as historical prediction motion vector candidates or historical merging candidates.

[0015] An image decoding method according to a fifth aspect of the present invention includes: an encoding information saving step, which adds the inter-frame prediction information used in inter-frame prediction to a historical prediction motion vector candidate list; a historical prediction motion vector candidate derivation step, which derives historical prediction motion vector candidates from the historical prediction motion vector candidate list; and a historical merging candidate derivation step, which derives historical merging candidates from the historical prediction motion vector candidate list, wherein in the historical prediction motion vector candidate derivation step and the historical merging candidate derivation step, candidates included in the historical prediction motion vector candidate list are referenced in different orders as historical prediction motion vector candidates or historical merging candidates.

[0016] The image decoding program of the sixth aspect of the present invention causes a computer to perform the following steps: an encoding information saving step, which adds the inter-frame prediction information used in inter-frame prediction to a historical prediction motion vector candidate list; a historical prediction motion vector candidate derivation step, which derives historical prediction motion vector candidates from the historical prediction motion vector candidate list; and a historical merging candidate derivation step, which derives historical merging candidates from the historical prediction motion vector candidate list; in the historical prediction motion vector candidate derivation step and the historical merging candidate derivation step, candidates included in the historical prediction motion vector candidate list are referenced in different orders as historical prediction motion vector candidates or historical merging candidates.

[0017] Invention Effects

[0018] According to the present invention, high-efficiency image encoding / decoding processing can be achieved with low overhead. Attached Figure Description

[0019] Figure 1 This is a block diagram of an image encoding apparatus according to an embodiment of the present invention;

[0020] Figure 2 This is a block diagram of an image decoding apparatus according to an embodiment of the present invention;

[0021] Figure 3 This is a flowchart illustrating the actions of splitting tree blocks;

[0022] Figure 4 This is a diagram illustrating the process of segmenting an input image into tree blocks;

[0023] Figure 5 This is a diagram illustrating the z-scan.

[0024] Figure 6A This is a diagram showing the segmented shape of the block;

[0025] Figure 6B This is a diagram showing the segmented shape of the block;

[0026] Figure 6C This is a diagram showing the segmented shape of the block;

[0027] Figure 6D This is a diagram showing the segmented shape of the block;

[0028] Figure 6E This is a diagram showing the segmented shape of the block;

[0029] Figure 7 This is a flowchart illustrating the action of dividing a block into four parts;

[0030] Figure 8 This is a flowchart used to illustrate the action of dividing a block into 2 or 3 parts;

[0031] Figure 9 It is a syntax used to describe the shape of block segmentation;

[0032] Figure 10A This is a diagram used to illustrate intra-frame prediction;

[0033] Figure 10B This is a diagram used to illustrate intra-frame prediction;

[0034] Figure 11 This is a diagram used to illustrate the reference block for inter-frame prediction;

[0035] Figure 12 Syntax used to describe the prediction pattern of coded blocks;

[0036] Figure 13 It is a graph showing the correspondence between syntactic elements and patterns related to inter-frame prediction;

[0037] Figure 14 This is a diagram used to illustrate affine transformation motion compensation with two control points;

[0038] Figure 15 This is a diagram used to illustrate affine transformation motion compensation with three control points;

[0039] Figure 16 yes Figure 1 A block diagram showing the detailed structure of the inter-frame prediction unit 102;

[0040] Figure 17 yes Figure 16 A block diagram showing the detailed structure of the typical predicted motion vector mode derivation unit 301;

[0041] Figure 18 yes Figure 16 A block diagram showing the detailed structure of the typical merging mode derivation section 302;

[0042] Figure 19 It is used for explanation Figure 16A flowchart of the normal prediction motion vector pattern export process of the normal prediction motion vector pattern export unit 301.

[0043] Figure 20 This is a flowchart illustrating the typical processing steps for deriving motion vector patterns from predicted data.

[0044] Figure 21 This is a flowchart illustrating the typical steps involved in exporting data using a merge pattern.

[0045] Figure 22 yes Figure 2 A block diagram showing the detailed structure of the inter-frame prediction unit 203;

[0046] Figure 23 yes Figure 22 A block diagram showing the detailed structure of the typical predicted motion vector mode derivation unit 401;

[0047] Figure 24 yes Figure 22 A block diagram showing the detailed structure of the typical merging mode derivation section 402;

[0048] Figure 25 It is used for explanation Figure 22 A flowchart of the normal prediction motion vector pattern export process of the normal prediction motion vector pattern export unit 401.

[0049] Figure 26 This is a diagram illustrating the initialization / update process of the historical predicted motion vector candidate list;

[0050] Figure 27 This is a flowchart of the same feature confirmation process in the initialization / update process of the historical predicted motion vector candidate list.

[0051] Figure 28 This is a flowchart of the feature shifting process in the initialization / update process of the historical predicted motion vector candidate list.

[0052] Figure 29 This is a flowchart illustrating the steps involved in deriving candidate motion vectors from historical predictions.

[0053] Figure 30 This is a flowchart illustrating the steps involved in exporting historical merge candidates;

[0054] Figure 31A This is a diagram illustrating an example of the historical predicted motion vector candidate list update process;

[0055] Figure 31B This is a diagram illustrating an example of the historical predicted motion vector candidate list update process;

[0056] Figure 31CThis is a diagram illustrating an example of the historical predicted motion vector candidate list update process;

[0057] Figure 32 This is a graph used to illustrate motion compensation prediction when the reference image (RefL0Pic) of L0 is at a time before the processing object image (CurPic) in L0 prediction;

[0058] Figure 33 This is a diagram used to illustrate motion compensation prediction when the reference image for L0 prediction is at a time after the image of the object being processed in L0 prediction.

[0059] Figure 34 This is a diagram used to illustrate the prediction direction of motion compensation prediction when the reference image for L0 prediction is before the object image being processed, and the reference image for L1 prediction is after the object image being processed.

[0060] Figure 35 This is a diagram used to illustrate the prediction direction of motion compensation prediction when the reference image for L0 prediction and the reference image for L1 prediction are at a time before the object image being processed in dual prediction.

[0061] Figure 36 This is a diagram used to illustrate the prediction direction of motion compensation prediction when the reference image for L0 prediction and the reference image for L1 prediction are at a time after the object image being processed in dual prediction.

[0062] Figure 37 This is a diagram illustrating an example of the hardware structure of the encoding / decoding apparatus according to an embodiment of the present invention;

[0063] Figure 38 This is another example of a table representing historical predicted motion vector candidates added through the initialization of the historical predicted motion vector candidate list;

[0064] Figure 39 This is another example of a table representing historical predicted motion vector candidates added through the initialization of the historical predicted motion vector candidate list;

[0065] Figure 40 This is another example of a table representing historical predicted motion vector candidates added through the initialization of the historical predicted motion vector candidate list;

[0066] Figure 41 This is a diagram used to illustrate the composition of the candidate list of historical predicted motion vectors;

[0067] Figure 42 This is a diagram used to illustrate the situation where the leading element is removed when adding to the candidate list of historical predicted motion vectors;

[0068] Figure 43 It is a diagram used to illustrate the movement of elements within the list when adding them to the historical predicted motion vector candidate list;

[0069] Figure 44 This is a diagram used to illustrate the situation when adding new features as they are added to the historical predicted motion vector candidate list;

[0070] Figure 45 This is a diagram illustrating the reference order of the historical predicted motion vector candidate list in the general merging mode of the second embodiment;

[0071] Figure 46 This is a diagram illustrating the reference order of the historical predicted motion vector candidate list under the normal predicted motion vector mode of the second embodiment;

[0072] Figure 47 This is a flowchart illustrating the historical prediction motion vector candidate derivation processing steps of the second embodiment. Detailed Implementation

[0073] Define the technologies and technical terms used in this embodiment.

[0074] <Tree Block>

[0075] In this implementation, the image to be encoded / decoded is divided equally into segments of a predetermined size. This unit is defined as a tree block. Figure 4 In this design, the size of the tree block is set to 128×128 pixels, but the size is not limited to this and can be set to any size. The tree blocks, which are processed (corresponding to encoding objects in encoding processing and decoding objects in decoding processing), are switched according to the raster scan order, i.e., from left to right and from top to bottom. Each tree block can be further recursively divided internally. The block that becomes the encoding / decoding object after recursively dividing the tree block is defined as an encoding block. Furthermore, tree blocks and encoding blocks are collectively referred to as blocks. By performing appropriate block division, efficient encoding can be achieved. The size of the tree block can be a predetermined fixed value set in the encoding and decoding devices, or a structure can be adopted where the size of the tree block determined by the encoding device is transmitted to the decoding device. Here, the maximum size of the tree block is set to 128×128 pixels, and the minimum size is set to 16×16 pixels. Similarly, the maximum size of the encoding block is set to 64×64 pixels, and the minimum size is set to 4×4 pixels.

[0076] <Prediction Pattern>

[0077] The intra-frame prediction (MODE_INTRA) and inter-frame prediction (MODE_INTER) are switched on a per-processing-object-coded-block basis. The intra-frame prediction (MODE_INTRA) predicts based on the processed image signal of the object image, while the inter-frame prediction predicts based on the image signal of the processed image.

[0078] The processed image is used to decode the signal that has been encoded in the encoding process to obtain the image, image signal, tree block, block, coded block, etc., and is used to decode the image, image signal, tree block, block, coded block, etc. in the decoding process.

[0079] The mode that identifies the intra-frame prediction (MODE_INTRA) and inter-frame prediction (MODE_INTER) is defined as the prediction mode (PredMode). The prediction mode (PredMode) is represented by a value for either intra-frame prediction (MODE_INTRA) or inter-frame prediction (MODE_INTER).

[0080] Inter-frame prediction

[0081] In inter-frame prediction, which predicts based on image signals from processed images, multiple processed images can be used as reference pictures. To manage these multiple reference pictures, two reference lists, L0 (reference list 0) and L1 (reference list 1), are defined, each using a reference index to determine the reference picture. In a P-slice, L0 prediction (Pred_L0) can be used. In a B-slice, L0 prediction (Pred_L0), L1 prediction (Pred_L1), and double prediction (Pred_BI) can be used. L0 prediction (Pred_L0) is an inter-frame prediction that references the reference pictures managed by L0, and L1 prediction (Pred_L1) is an inter-frame prediction that references the reference pictures managed by L1. Double prediction (Pred_BI) is an inter-frame prediction that simultaneously performs L0 and L1 predictions and references individual reference pictures managed by each of L0 and L1. The information determining L0 prediction, L1 prediction, and double prediction is defined as the inter-frame prediction mode. Regarding the constants and variables with the subscript LX appended to the output in subsequent processing, the premise is that they will be processed as L0 and L1.

[0082] <Predicting motion vector patterns>

[0083] The predicted motion vector mode is a mode that transmits the index, differential motion vector, inter-frame prediction mode, reference index, and inter-frame prediction information for determining the predicted motion vector and deciding on the target block. The predicted motion vector is derived from the predicted motion vector candidate and the index used to determine the predicted motion vector. The predicted motion vector candidate is derived from the processed blocks adjacent to the target block or from blocks in the processed image that are located at the same position as or near the target block.

[0084] <Merge Mode>

[0085] The merging mode is as follows: without transmitting differential motion vectors or reference indices, the inter-frame prediction information of the processing object block is derived based on the inter-frame prediction information of processed blocks adjacent to the processing object block or blocks in the processed image that are located at the same position or nearby (neighboring) to the processing object block.

[0086] Processed blocks adjacent to the target block, along with their inter-frame prediction information, are defined as spatial merge candidates. Blocks belonging to the processed image that are located at the same position as or near the target block, along with their inter-frame prediction information derived from that block, are defined as temporal merge candidates. Each merge candidate is registered in a merge candidate list, and the merge candidate used in the prediction of the target block is determined by a merge index.

[0087] <Adjacent blocks>

[0088] Figure 11 This diagram illustrates the reference blocks used to derive inter-frame prediction information in both the predicted motion vector mode and the merge mode. A0, A1, A2, B0, B1, B2, and B3 are processed blocks adjacent to the object block. T0 is a block within the processed image blocks that is located at the same position or in the vicinity (nearby) of the object block in the object image.

[0089] A1 and A2 are blocks located to the left of the processing object's encoding block and adjacent to it. B1 and B3 are blocks located above the processing object's encoding block and adjacent to it. A0, B0, and B2 are blocks located to the lower left, upper right, and upper left of the processing object's encoding block, respectively.

[0090] The details of how adjacent blocks are handled in the prediction motion vector mode and the merging mode are described later.

[0091] <Affine Transformation Motion Compensation>

[0092] Affine transformation motion compensation involves dividing the coded block into sub-blocks of predetermined units and determining motion vectors for each sub-block individually. The motion vectors of each sub-block are derived based on one or more control points. These control points are derived from inter-frame prediction information of processed blocks adjacent to the target block or blocks within the processed image that are located at the same position or nearby (neighboring) to the target block. In this embodiment, the size of the sub-block is set to 4×4 pixels, but the size of the sub-block is not limited to this; motion vectors can also be derived in pixels.

[0093] Figure 14 An example of affine transformation motion compensation with two control points is shown. In this case, the two control points have two parameters: a horizontal component and a vertical component. Therefore, the affine transformation with two control points is called a four-parameter affine transformation. Figure 14 CP1 and CP2 are control points.

[0094] Figure 15 An example of affine transformation motion compensation with three control points is shown. In this case, the three control points have two parameters: a horizontal component and a vertical component. Therefore, the affine transformation with three control points is called a six-parameter affine transformation. Figure 15 CP1, CP2, and CP3 are control points.

[0095] Affine transformation motion compensation can be used in either the predictive motion vector mode or the merging mode. A mode in which affine transformation motion compensation is applied in the predictive motion vector mode is defined as the sub-block predictive motion vector mode, and a mode in which affine transformation motion compensation is applied in the merging mode is defined as the sub-block merging mode.

[0096] <Syntax of Inter-Frame Prediction>

[0097] use Figure 12 and Figure 13 The syntax related to inter-frame prediction is explained.

[0098] Figure 12 `merge_flag` indicates whether the processed object coding block is set to merge mode or predictive motion vector mode. `merge_affine_flag` indicates whether to apply sub-block merging mode in the processed object coding block in merge mode. `inter_affine_flag` indicates whether to apply sub-block predictive motion vector mode in the processed object coding block in predictive motion vector mode. `cu_affine_type_flag` is a flag used to determine the number of control points in sub-block predictive motion vector mode.

[0099] Figure 13This shows the values ​​of each syntactic element and their corresponding prediction methods. `merge_flag = 1` and `merge_affine_flag = 0` correspond to the normal merge mode. The normal merge mode is a merge mode that is not a sub-block merge. `merge_flag = 1` and `merge affine_flag = 1` correspond to the sub-block merge mode. `merge_flag = 0` and `inter_affine_flag = 0` correspond to the normal predicted motion vector mode. The normal predicted motion vector mode is a predicted motion vector merge that is not a sub-block predicted motion vector mode. `merge_flag = 0` and `inter_affine_flag = 1` correspond to the sub-block predicted motion vector mode. When `merge_flag = 0` and `inter_affine_flag = 1`, `cu_affine_type_flag` is further passed to determine the number of control points.

[0100] <poc>

[0101] The Picture Order Count (POC) is a variable associated with the picture to be encoded, and it is set to an incrementing value of 1 corresponding to the output order of the pictures. Based on the POC value, it is possible to determine whether the pictures are the same, the order of the pictures in the output sequence, and the distance between the exported pictures. For example, if two pictures have the same POC value, they can be considered the same picture. If two pictures have different POC values, the picture with the smaller POC value is determined to be the first image output, and the difference between the POC values ​​of the two pictures represents the distance between them along the timeline.

[0102] (First Implementation)

[0103] The image encoding apparatus 100 and the image decoding apparatus 200 according to the first embodiment of the present invention will be described.

[0104] Figure 1 This is a block diagram of the image encoding apparatus 100 according to the first embodiment. The image encoding apparatus 100 of the embodiment includes a block segmentation unit 101, an inter-frame prediction unit 102, an intra-frame prediction unit 103, a decoded image memory 104, a prediction method determination unit 105, a residual generation unit 106, an orthogonal transform / quantization unit 107, a bit string encoding unit 108, an inverse quantization / inverse orthogonal transform unit 109, a decoded image signal overlap unit 110, and an encoding information storage memory 111.

[0105] The block segmentation unit 101 recursively segments the input image to generate coded blocks. The block segmentation unit 101 includes a 4-segmentation unit and a 2-3 segmentation unit. The 4-segmentation unit segments the blocks to be segmented in both the horizontal and vertical directions, while the 2-3 segmentation unit segments the blocks to be segmented in either the horizontal or vertical direction. The block segmentation unit 101 sets the generated coded blocks as processing target coded blocks and provides the image signal of the processing target coded blocks to the inter-frame prediction unit 102, the intra-frame prediction unit 103, and the residual generation unit 106. Furthermore, the block segmentation unit 101 provides information representing the determined recursive segmentation structure to the bit string encoding unit 108. Detailed operation of the block segmentation unit 101 will be described later.

[0106] The inter-frame prediction unit 102 performs inter-frame prediction for the target coded block. Based on the inter-frame prediction information stored in the encoded information storage memory 111 and the decoded image signal stored in the decoded image memory 104, the inter-frame prediction unit 102 derives multiple candidate inter-frame prediction information, selects a suitable inter-frame prediction mode from the derived candidates, and provides the selected inter-frame prediction mode and the corresponding prediction image signal to the prediction method determination unit 105. The detailed structure and operation of the inter-frame prediction unit 102 will be described later.

[0107] The intra-prediction unit 103 performs intra-prediction of the target coded block. The intra-prediction unit 103 uses the decoded image signal stored in the decoded image memory 104 as a reference pixel, and generates a predicted image signal based on intra-prediction information such as the intra-prediction mode stored in the encoding information storage memory 111. In intra-prediction, the intra-prediction unit 103 selects a suitable intra-prediction mode from multiple intra-prediction modes and provides the selected intra-prediction mode and the predicted image signal corresponding to the selected intra-prediction mode to the prediction method determination unit 105.

[0108] Figure 10A and Figure 10B An example of intra-frame prediction is shown. Figure 10A This diagram illustrates the correspondence between the prediction direction of intra-prediction and the intra-prediction mode number. For example, intra-prediction mode 50 generates an intra-predicted image by copying reference pixels in the vertical direction. Intra-prediction mode 1 is a DC mode, which sets the pixel values ​​of all pixels in the processed object block to the average of the reference pixels. Intra-prediction mode 0 is a Planar mode (two-dimensional mode), which generates a two-dimensional intra-prediction image based on reference pixels in both the vertical and horizontal directions. Figure 10B This is an example of an intra-prediction image generated in intra-prediction mode 40. The intra-prediction unit 103 copies the value of a reference pixel in the direction indicated by the intra-prediction mode to each pixel of the processing object block. If the reference pixel in the intra-prediction mode is not at an integer position, the intra-prediction unit 103 determines the reference pixel value by interpolation based on the reference pixel values ​​at surrounding integer positions.

[0109] The decoded image memory 104 stores the decoded image generated by the decoded image signal overlap portion 110. The decoded image memory 104 provides the stored decoded image to the inter-frame prediction unit 102 and the intra-frame prediction unit 103.

[0110] The prediction method determination unit 105 evaluates each of intra-frame prediction and inter-frame prediction by using coding information and the amount of coding of the residual, the amount of distortion between the predicted image signal and the image signal to be processed, etc., to determine the optimal prediction mode. In the case of intra-frame prediction, the prediction method determination unit 105 provides intra-frame prediction information such as the intra-frame prediction mode as coding information to the bit string coding unit 108. In the case of inter-frame prediction merging mode, the prediction method determination unit 105 provides inter-frame prediction information such as merging index and information indicating whether it is a sub-block merging mode (sub-block merging flag) as coding information to the bit string coding unit 108. In the case of inter-frame prediction motion vector prediction mode, the prediction method determination unit 105 provides inter-frame prediction information such as inter-frame prediction mode, prediction motion vector index, L0 and L1 reference indices, differential motion vector, and information indicating whether it is a sub-block prediction motion vector mode (sub-block prediction motion vector flag) as coding information to the bit string coding unit 108. In addition, the prediction method determination unit 105 provides the determined coding information to the coding information storage memory 111. The prediction method determination unit 105 provides the predicted image signal to the residual generation unit 106 and the decoded image signal overlap unit 110.

[0111] The residual generation unit 106 generates a residual by subtracting the predicted image signal from the image signal of the object being processed, and provides it to the orthogonal transformation / quantization unit 107.

[0112] The orthogonal transformation / quantization unit 107 performs orthogonal transformation and quantization on the residual according to the quantization parameters to generate the orthogonal transformation / quantization residual, and provides the generated residual to the bit string encoding unit 108 and the inverse quantization / inverse orthogonal transformation unit 109.

[0113] In addition to information about sequences, images, stripes, and coding block units, the bit string encoding unit 108 encodes coding information corresponding to the prediction method determined by the prediction method determination unit 105 for each coding block. Specifically, the bit string encoding unit 108 encodes the prediction mode PredMode for each coding block. When the prediction mode is inter-frame prediction (MODE_INTER), the bit string encoding unit 108 encodes coding information (inter-frame prediction information) such as the flag indicating whether it is a merging mode, the sub-block merging flag, the merging index when it is a merging mode, the inter-frame prediction mode when it is not a merging mode, the predicted motion vector index, information related to the differential motion vector, and the sub-block predicted motion vector flag, according to the prescribed syntax (bit string syntax rules), to generate a first bit string. When the prediction mode is intra-frame prediction (MODE_INTRA), the bit string encoding unit encodes coding information (intra-frame prediction information) such as the intra-frame prediction mode, according to the prescribed syntax (bit string syntax rules), to generate a first bit string. Furthermore, the bit string encoding unit 108 performs entropy encoding on the orthogonal transform and quantized residuals according to a prescribed syntax to generate a second bit string. The bit string encoding unit 108 then multiplexes the first and second bit strings according to a prescribed syntax to output a bit stream.

[0114] The inverse quantization / inverse quadrature transformation unit 109 performs inverse quantization and inverse quadrature transformation on the residual provided by the quadrature transformation / quantization unit 107 to calculate the residual, and provides the calculated residual to the decoded image signal overlap unit 110.

[0115] The decoded image signal overlay unit 110 overlays the predicted image signal corresponding to the decision of the prediction method determination unit 105 with the residual obtained by inverse quantization and inverse quadrature transformation performed by the inverse quantization / inverse quadrature transformation unit 109 to generate a decoded image, which is then stored in the decoded image memory 104. Alternatively, the decoded image signal overlay unit 110 may also perform filtering processing on the decoded image to reduce block distortion and other distortions caused by encoding before storing it in the decoded image memory 104.

[0116] The encoding information storage memory 111 stores encoding information such as the prediction mode (inter-frame prediction or intra-frame prediction) determined by the prediction method determination unit 105. In the case of inter-frame prediction, the encoding information stored in the encoding information storage memory 111 includes the determined motion vector, the reference index of reference lists L0 and L1, and the historical predicted motion vector candidate list. Furthermore, in the case of inter-frame prediction merging mode, the encoding information stored in the encoding information storage memory 111 includes, in addition to the above information, inter-frame prediction information such as a merging index and information indicating whether it is a sub-block merging mode (sub-block merging flag). Furthermore, in the case of inter-frame prediction predicting motion vector mode, the encoding information stored in the encoding information storage memory 111 includes, in addition to the above information, inter-frame prediction information such as the inter-frame prediction mode, predicted motion vector index, differential motion vector, and information indicating whether it is a sub-block predicting motion vector mode (sub-block predicting motion vector flag). In the case of intra-frame prediction, the encoding information stored in the encoding information storage memory 111 includes intra-frame prediction information such as the determined intra-frame prediction mode.

[0117] Figure 2 It means and Figure 1 The image encoding apparatus is shown in the block diagram of the image decoding apparatus according to the embodiments of the present invention. The image decoding apparatus of the embodiments includes a bit string decoding unit 201, a block segmentation unit 202, an inter-frame prediction unit 203, an intra-frame prediction unit 204, an encoded information storage memory 205, an inverse quantization / inverse quadrature transformation unit 206, a decoded image signal overlap unit 207, and a decoded image memory 208.

[0118] Figure 2 Image decoding device decoding processing and Figure 1 The decoding processing is internally configured within the image encoding device, therefore Figure 2 The encoded information storage memory 205, the inverse quantization / inverse quadrature transform unit 206, the decoded image signal overlap unit 207, and the decoded image memory 208 each have structures that are consistent with... Figure 1 The functions of each structure of the image encoding device, including the encoding information storage memory 111, the inverse quantization / inverse quadrature transformation unit 109, the decoded image signal overlap unit 110, and the decoded image memory 104, are respectively defined.

[0119] The bitstream provided to the bitstream decoding unit 201 is separated according to prescribed syntax rules. The bitstream decoding unit 201 decodes the separated first bitstream to obtain information on the sequence, image, stripe, coded block unit, and coded block unit encoding information. Specifically, the bitstream decoding unit 201 decodes the prediction mode PredMode on a coded block basis. The prediction mode PredMode is determined to be either inter-frame prediction (MODE_INTER) or intra-frame prediction (MODE_INTRA). When the prediction mode is inter-frame prediction (MODE_INTER), the bitstream decoding unit 201 decodes the encoding information (inter-frame prediction information) related to the flag indicating whether it is a merge mode, the merge index in the case of merge mode, the sub-block merge flag, the inter-frame prediction mode in the case of prediction motion vector mode, the prediction motion vector index, the differential motion vector, and the sub-block prediction motion vector flag, according to the prescribed syntax. The encoding information (inter-frame prediction information) is then provided to the encoding information storage memory 205 via the inter-frame prediction unit 203 and the block segmentation unit 202. When the prediction mode is intra-prediction (MODE_INTRA), the coded information (intra-prediction information) such as the intra-prediction mode is decoded according to the prescribed syntax, and the coded information (intra-prediction information) is provided to the coded information storage memory 205 via the inter-prediction unit 203 or the intra-prediction unit 204 and the block segmentation unit 202. The bit string decoding unit 201 decodes the separated second bit string, calculates the residual after orthogonal transformation / quantization, and provides the residual after orthogonal transformation / quantization to the inverse quantization / inverse orthogonal transformation unit 206.

[0120] When the prediction mode PredMode of the coded block being processed is the prediction motion vector mode in inter-frame prediction (MODE_INTER), the inter-frame prediction unit 203 uses the coded information of the decoded image signal stored in the coded information storage memory 205 to derive multiple candidate prediction motion vectors, and registers the derived candidate prediction motion vectors in the candidate prediction motion vector list described later. The inter-frame prediction unit 203 selects the prediction motion vector corresponding to the prediction motion vector index provided by the bit string decoding unit 201 from the multiple candidate prediction motion vectors registered in the candidate prediction motion vector list, calculates the motion vector based on the differential motion vector decoded by the bit string decoding unit 201 and the selected prediction motion vector, and stores the calculated motion vector along with other coded information in the coded information storage memory 205. Here, the encoding information of the coded blocks to be provided / saved includes the prediction mode PredMode, flags indicating whether L0 and L1 prediction are used (predFlagL0[xP][yP], predFlagL1[xP][yP]), reference indices for L0 and L1 (refIdxL0[xP][yP], refIdxL1[xP][yP]), motion vectors for L0 and L1 (mvL0[xP][yP], mvL1[xP][yP]), etc. Here, xP and yP represent the indices of the top-left pixel position of the coded block within the image. When the prediction mode PredMode is inter-frame prediction (MODE_INTER) and the inter-frame prediction mode is L0 prediction (Pred_L0), the flag predFlagL0 indicating whether L0 prediction is used is 1, and the flag predFlagL1 indicating whether L1 prediction is used is 0. When the inter-frame prediction mode is L1 prediction (Pred_L1), the flag predFlagL0 indicating whether to use L0 prediction is 0, and the flag predFlagL1 indicating whether to use L1 prediction is 1. When the inter-frame prediction mode is double prediction (Pred_BI), both the flag predFlagL0 indicating whether to use L0 prediction and the flag predFlagL1 indicating whether to use L1 prediction are 1. Furthermore, when the prediction mode PredMode of the coded block being processed is the merge mode in inter-frame prediction (MODE_INTER), merge candidates are derived.Using the encoding information of the decoded coded blocks stored in the encoding information storage memory 205, multiple merging candidates are derived and registered in the merging candidate list (described later). From the multiple merging candidates registered in the merging candidate list, a merging candidate corresponding to the merging index provided by the bit string decoding unit 201 is selected. Inter-frame prediction information, including the flags predFlagL0[xP][yP], predFlagL1[xP][yP] indicating whether to utilize the selected merging candidate for L0 and L1 prediction, the reference indices refIdxL0[xP][yP], refIdxL1[xP][yP] for L0 and L1, and the motion vectors mvL0[xP][yP], mvL1[xP][yP] for L0 and L1, are stored in the encoding information storage memory 205. Here, xP and yP are indices representing the position of the top-left pixel of the coded block within the image. The detailed structure and operation of the inter-frame prediction unit 203 will be described later.

[0121] When the prediction mode PredMode of the encoded block being processed is intra-prediction (MODE_INTRA), the intra-prediction unit 204 performs intra-prediction. The intra-prediction mode is included in the encoded information decoded by the bit string decoding unit 201. Based on the intra-prediction mode included in the decoded information decoded by the bit string decoding unit 201, the intra-prediction unit 204 generates a predicted image signal based on the decoded image signal stored in the decoded image memory 208, and provides the generated predicted image signal to the decoded image signal overlay unit 207. Since the intra-prediction unit 204 corresponds to the intra-prediction unit 103 of the image encoding apparatus 100, it performs the same processing as the intra-prediction unit 103.

[0122] The inverse quantization / inverse quadrature transformation unit 206 performs inverse quadrature transformation and inverse quantization on the residual after quadrature transformation / quantization decoded by the bit string decoding unit 201 to obtain the residual after inverse quadrature transformation / inverse quantization.

[0123] The decoded image signal overlay unit 207 decodes the image signal by overlaying the predicted image signal obtained by inter-frame prediction by inter-frame prediction unit 203 or intra-frame prediction by intra-frame prediction unit 204, and the residual after inverse quadrature transformation / inverse quantization by inverse quantization / inverse quadrature transformation unit 206, and stores the decoded image signal in the decoded image memory 208. When storing the image signal in the decoded image memory 208, the decoded image signal overlay unit 207 may also perform filtering processing on the decoded image to reduce block distortion caused by encoding before storing it in the decoded image memory 208.

[0124] Next, the operation of the block segmentation unit 101 in the image encoding device 100 will be explained. Figure 3 This is a flowchart illustrating the process of segmenting an image into tree blocks and further segmenting each tree block. First, the input image is segmented into tree blocks of a predetermined size (step S1001). For each tree block, it is scanned in a predetermined order, i.e., raster scan order (step S1002), and the interior of the tree block to be processed is segmented (step S1003).

[0125] Figure 7 This is a flowchart illustrating the detailed actions of the segmentation process in step S1003. First, it is determined whether to divide the block of the object to be processed into 4 parts (step S1101).

[0126] If it is determined that the processing object block 4 should be divided, the processing object block 4 is divided (step S1102). For each block obtained by dividing the processing object block, it is scanned in the Z-scan order, that is, the order of top left, top right, bottom left, and bottom right (step S1103). Figure 5 This is an example of Z-scan order. Figure 6A 601 is an example of processing object block 4 after splitting. Figure 6A The numbers 0 to 3 in step 601 indicate the processing order. Then, for each block divided in step S1101, the process is executed recursively. Figure 7 The segmentation process (step S1104).

[0127] If it is determined that the object block to be processed will not be divided into 4 parts, then 2-3 parts will be performed (step S1105).

[0128] Figure 8 This is a flowchart showing the detailed actions of the 2-3 segmentation process in step S1105. First, it is determined whether to perform 2-3 segmentation on the block to be processed, that is, whether to perform either 2-segmentation or 3-segmentation (step S1201).

[0129] If it is determined that the block to be processed will not be split into 2-3 segments, that is, if it is determined that no splitting will be performed, the splitting process ends (step S1211). That is, for blocks obtained by recursive splitting, no further recursive splitting is performed.

[0130] If it is determined that the block of the object to be processed should be divided into 2-3, then it is determined whether to further divide the block of the object to be processed into 2 (step S1202).

[0131] If it is determined that the processing object block should be divided into two parts, it is determined whether to divide the processing object block vertically (step S1203). Based on the result, the processing object block is divided into two parts vertically (step S1204), or the processing object block is divided into two parts horizontally (step S1205). As a result of step S1204, the processing object block is as follows: Figure 6B As shown in 602, it is divided into two parts, upper and lower (vertical direction). As a result of step S1205, the processed object block is as follows: Figure 6D As shown in 604, it is divided into two parts, left and right (horizontal direction).

[0132] In step S1202, if it is not determined that the processing object block is to be divided into two parts, that is, if it is determined that it is to be divided into three parts, then it is determined whether to divide the processing object block into top, middle, and bottom (vertical direction) (step S1206). Based on this result, the processing object block is divided into three parts in the top, middle, and bottom (vertical direction) (step S1207), or the processing object block is divided into three parts in the left, middle, and right (horizontal direction) (step S1208). In the result of step S1207, the processing object block is as follows: Figure 6C As shown in 603, it is divided into three parts: upper, middle, and lower (vertical direction). In the result of step S1208, the processed object block is as follows: Figure 6E As shown in 605, it is divided into three parts: left, middle, and right (horizontal direction).

[0133] After executing any one of steps S1204, S1205, S1207, or S1208, the blocks into which the processing object block is divided are scanned in order from left to right and from top to bottom (step S1209). Figures 6B to 6E The numbers 0 to 2, from 602 to 605, indicate the processing order. For each segmented block, the process is executed recursively. Figure 8 The 2-3 segmentation process (step S1210).

[0134] The recursive block partitioning described here can also limit whether partitioning is necessary based on the number of partitions or the size of the block being processed. The information limiting whether partitioning is necessary can be implemented by pre-agreeing between the encoding and decoding devices without transmitting the information, or by the encoding device determining whether partitioning is necessary and recording the information in a bit string before transmitting it to the decoding device.

[0135] When a block is divided, the block before the division is called the parent block, and the blocks after the division are called child blocks.

[0136] Next, the operation of the block segmentation unit 202 in the image decoding apparatus 200 will be described. The block segmentation unit 202 segments tree blocks according to the same processing steps as the block segmentation unit 101 in the image encoding apparatus 100. However, the difference is that in the block segmentation unit 101 of the image encoding apparatus 100, the optimal block segmentation shape is determined by applying optimization methods such as optimal shape estimation or distortion rate optimization based on image recognition. In contrast, the block segmentation unit 202 in the image decoding apparatus 200 determines the block segmentation shape by decoding the block segmentation information recorded in the bit string.

[0137] Figure 9 The syntax (bit string syntax rules) related to block partitioning in the first embodiment is shown. `coding_quadtree()` represents the syntax involved in the 4-partitioning of the block. `multi_type_tree()` represents the syntax involved in the 2-partitioning or 3-partitioning of the block. `qt_split` is a flag indicating whether the block is 4-partitioned. When the block is 4-partitioned, `qt_split` = 1; otherwise, `qt_split` = 0. In the case of 4-partitioning (`qt_split` = 1), the 4-partitioned blocks are recursively partitioned into 4 parts (`coding_quadtree(0), coding_quadtree(1), coding_quadtree(2), coding_quadtree(3)`, where 0 to 3 correspond to... Figure 6A (601). Without a 4-splitting operation (qt_split = 0), subsequent splits are determined according to multi_type_tree(). mtt_split is a flag indicating whether further splitting is required. Furthermore, if splitting is required (mtt_split = 1), the flag indicating whether the split is vertical or horizontal is transmitted, namely mtt_split_vertical, and the flag indicating whether to perform a 2-splitting or 3-splitting operation, namely mtt_split_binary, is transmitted. mtt_split_vertical = 1 indicates a vertical split, and mtt_split_vertical = 0 indicates a horizontal split. mtt_split_binary = 1 indicates a 2-splitting operation, and mtt_split_binary = 0 indicates a 3-splitting operation. In the case of a 2-splitting operation (mtt_split_binary = 1), the blocks after the 2-splitting are recursively split (multi_type_tree(0), multi_type_tree(1), where the 0 to 1 of the independent variable corresponds to Figures 6B to 6D (Numbers 602 or 604). In the case of 3-partition (mtt_split_binary = 0), the 3-partitioned blocks are recursively split (multi_type_tree(0), multi_type_tree(1), multi_type_tree(2), 0 to 2 correspond to...). Figure 6B 603 or Figure 6E (Number 605). Hierarchical block splitting is performed by recursively calling multi_type_tree until mtt_split = 0.

[0138] Inter-frame prediction

[0139] The inter-frame prediction method in the implementation method Figure 1 The inter-frame prediction unit 102 of the image coding apparatus and Figure 2 It is implemented in the inter-frame prediction unit 203 of the image decoding device.

[0140] The inter-frame prediction method according to the implementation method is described with reference to the accompanying drawings. The inter-frame prediction method is implemented in either the encoding or decoding process on a block-by-block basis.

[0141] <Explanation of the inter-frame prediction unit 102 on the coding side>

[0142] Figure 16 It is shown Figure 1 A diagram showing the detailed structure of the inter-frame prediction unit 102 of the image coding apparatus. The normal prediction motion vector pattern derivation unit 301 derives multiple normal prediction motion vector candidates to select a prediction motion vector and calculates the difference motion vector between the selected prediction motion vector and the detected motion vector. The detected inter-frame prediction pattern, reference index, motion vector, and calculated difference motion vector constitute the inter-frame prediction information for the normal prediction motion vector pattern. This inter-frame prediction information is provided to the inter-frame prediction pattern determination unit 305. The detailed structure and processing of the normal prediction motion vector pattern derivation unit 301 will be described later.

[0143] In the normal merging mode derivation unit 302, multiple normal merging candidates are derived, and a normal merging candidate is selected to obtain inter-frame prediction information for the normal merging mode. This inter-frame prediction information is provided to the inter-frame prediction mode determination unit 305. The detailed structure and processing of the normal merging mode derivation unit 302 will be described later.

[0144] In the sub-block prediction motion vector mode derivation unit 303, multiple sub-block prediction motion vector candidates are derived to select a sub-block prediction motion vector, and the difference motion vector between the selected sub-block prediction motion vector and the detected motion vector is calculated. The detected inter-frame prediction mode, reference index, motion vector, and calculated difference motion vector constitute the inter-frame prediction information of the sub-block prediction motion vector mode. This inter-frame prediction information is provided to the inter-frame prediction mode determination unit 305.

[0145] In the sub-block merging mode derivation unit 304, multiple sub-block merging candidates are derived, and a sub-block merging candidate is selected to obtain inter-frame prediction information for the sub-block merging mode. This inter-frame prediction information is provided to the inter-frame prediction mode determination unit 305.

[0146] The inter-frame prediction mode determination unit 305 determines inter-frame prediction information based on the inter-frame prediction information provided by the normal prediction motion vector mode derivation unit 301, the normal merging mode derivation unit 302, the sub-block prediction motion vector mode derivation unit 303, and the sub-block merging mode derivation unit 304. The inter-frame prediction information corresponding to the determination result is provided from the inter-frame prediction mode determination unit 305 to the motion compensation prediction unit 306.

[0147] The motion compensation prediction unit 306 performs inter-frame prediction on the reference image signal stored in the decoded image memory 104 based on the determined inter-frame prediction information. The detailed structure and processing of the motion compensation prediction unit 306 will be described later.

[0148] <Explanation of the inter-frame prediction unit 203 on the decoding side>

[0149] Figure 22 It is shown Figure 2 A diagram showing the detailed structure of the inter-frame prediction unit 203 of the image decoding device.

[0150] Normally, the predictive motion vector mode derivation unit 401 derives multiple normally predicted motion vector candidates to select a predicted motion vector, and calculates the sum of the selected predicted motion vector and the decoded differential motion vector as the motion vector. The decoded inter-frame prediction mode, reference index, and motion vector constitute the inter-frame prediction information of the normally predicted motion vector mode. This inter-frame prediction information is provided to the motion compensation prediction unit 406 via switch 408. The detailed structure and processing of the normally predicted motion vector mode derivation unit 401 will be described later.

[0151] In the normal merging mode derivation unit 402, multiple normal merging candidates are derived to select a normal merging candidate, thereby obtaining inter-frame prediction information for the normal merging mode. This inter-frame prediction information is provided to the motion compensation prediction unit 406 via switch 408. The detailed structure and processing of the normal merging mode derivation unit 402 will be described later.

[0152] In the sub-block prediction motion vector mode derivation unit 403, multiple sub-block prediction motion vector candidates are derived to select a sub-block prediction motion vector. The selected sub-block prediction motion vector is calculated as the sum of the selected sub-block prediction motion vector and the decoded differential motion vector, which is then used as the motion vector. The decoded inter-frame prediction mode, reference index, and motion vector constitute the inter-frame prediction information of the sub-block prediction motion vector mode. This inter-frame prediction information is provided to the motion compensation prediction unit 406 via switch 408.

[0153] In the sub-block merging mode derivation unit 404, multiple sub-block merging candidates are derived to select a sub-block merging candidate, thereby obtaining inter-frame prediction information for the sub-block merging mode. This inter-frame prediction information is provided to the motion compensation prediction unit 406 via switch 408.

[0154] In the motion compensation prediction unit 406, inter-frame prediction is performed on the reference image signal stored in the decoded image memory 208 based on the determined inter-frame prediction information. The detailed structure and processing of the motion compensation prediction unit 406 are the same as those of the motion compensation prediction unit 306 on the encoding side.

[0155] <Typically predicted motion vector pattern derivation part (usually AMVP)>

[0156] Figure 17 The normal prediction motion vector pattern derivation unit 301 includes a spatial prediction motion vector candidate derivation unit 321, a time prediction motion vector candidate derivation unit 322, a historical prediction motion vector candidate derivation unit 323, a prediction motion vector candidate supplementation unit 325, a normal motion vector detection unit 326, a prediction motion vector candidate selection unit 327, and a motion vector subtraction unit 328.

[0157] Figure 23 The general prediction motion vector pattern derivation unit 401 includes a spatial prediction motion vector candidate derivation unit 421, a time prediction motion vector candidate derivation unit 422, a historical prediction motion vector candidate derivation unit 423, a prediction motion vector candidate supplementation unit 425, a prediction motion vector candidate selection unit 426, and a motion vector addition unit 427.

[0158] Use respectively Figure 19 , Figure 25 The flowchart describes the processing steps of the normal prediction motion vector pattern derivation unit 301 on the encoding side and the normal prediction motion vector pattern derivation unit 401 on the decoding side. Figure 19 This is a flowchart illustrating the normal predicted motion vector pattern derivation processing steps of the normal motion vector pattern derivation unit 301 based on the encoding side. Figure 25 This is a flowchart illustrating the normal predicted motion vector pattern export processing steps of the normal motion vector pattern export unit 401 based on the decoding side.

[0159] <Commonly Predictive Motion Vector Pattern Derivation Unit (typically AMVP): Explanation of the Encoding Side>

[0160] refer to Figure 19 The typical steps for deriving predicted motion vector patterns on the encoding side are explained. Figure 19 The instructions for the processing steps sometimes omit... Figure 19 The word "usually" is shown.

[0161] First, the typical motion vector detection unit 326 detects the typical motion vector for each inter-frame prediction mode and reference index. Figure 19 Step S100).

[0162] Next, the spatial prediction motion vector candidate derivation unit 321, the temporal prediction motion vector candidate derivation unit 322, the historical prediction motion vector candidate derivation unit 323, the prediction motion vector candidate supplementation unit 325, the prediction motion vector candidate selection unit 327, and the motion vector subtraction unit 328 calculate, for each L0 and L1, the differential motion vector of the motion vector used in the inter-frame prediction of the normal prediction motion vector mode. Figure 19 Steps S101 to S106). Specifically, when the prediction mode PredMode of the processed object block is inter-frame prediction (MODE_INTER) and the inter-frame prediction mode is L0 prediction (Pred_L0), the candidate list of predicted motion vectors for L0, mvpListL0, is calculated, the predicted motion vector mvpL0 is selected, and the differential motion vector mvdL0 of the motion vector mvL0 of L0 is calculated. When the inter-frame prediction mode of the processed object block is L1 prediction (Pred_L1), the candidate list of predicted motion vectors for L1, mvpListL1, is calculated, the predicted motion vector mvpL1 is selected, and the differential motion vector mvdL1 of the motion vector mvL1 of L1 is calculated. When the inter-frame prediction mode for processing object blocks is dual prediction (Pred_BI), L0 prediction and L1 prediction are performed simultaneously. The candidate list of predicted motion vectors for L0, mvpListL0, is calculated. The predicted motion vector of L0, mvpL0, is selected. The differential motion vector of L0, mvL0, is calculated. The candidate list of predicted motion vectors for L1, mvpListL1, is calculated. The predicted motion vector of L1, mvpL1, is calculated. The differential motion vector of L1, mvL1, is calculated.

[0163] Differential motion vector calculations are performed separately for L0 and L1, but the process is common to both. Therefore, in the following explanation, L0 and L1 will be represented as a common LX. In the calculation of the differential motion vector for L0, X of LX is 0, and in the calculation of the differential motion vector for L1, X of LX is 1. Furthermore, in the calculation of the differential motion vector for LX, if information from another list is referenced instead of LX, this other list will be represented as LY.

[0164] When using the motion vector mvLX of LX ( Figure 19 Step S102: Yes), calculate the candidate predicted motion vectors of LX, and construct the candidate predicted motion vector list mvpListLX( Figure 19 Step S103). Multiple candidate predicted motion vectors are derived from the spatial predicted motion vector candidate derivation unit 321, the temporal predicted motion vector candidate derivation unit 322, the historical predicted motion vector candidate derivation unit 323, and the predicted motion vector candidate supplementation unit 325 in the normal predicted motion vector pattern derivation unit 301, constructing a predicted motion vector candidate list mvpListLX. (About...) Figure 19 The detailed processing steps of step S103 are as follows: Figure 20 The flowchart is described later.

[0165] Next, the predicted motion vector candidate selection unit 327 selects the predicted motion vector mvpLX of LX from the predicted motion vector candidate list mvpListLX of LX. Figure 19 Step S104). Here, in the candidate list of predicted motion vectors mvpListLX, a certain element (the i-th element counting from 0) is represented as mvpListLX[i]. Calculate each differential motion vector, which is the difference between the motion vector mvLX and the candidate mvpListLX[i] of each predicted motion vector stored in the candidate list of predicted motion vectors mvpListLX. For each element (predicted motion vector candidate) in the candidate list of predicted motion vectors mvpListLX, calculate the coding amount when encoding these differential motion vectors. Then, among the elements registered in the candidate list of predicted motion vectors mvpListLX, select the candidate mvpListLX[i] of the predicted motion vector with the smallest coding amount as the predicted motion vector mvpLX, and obtain the index i. If there are multiple candidates for the predicted motion vector that will become the smallest generated code amount in the candidate list of predicted motion vectors mvpListLX, the candidate mvpListLX[i] represented by the smallest index i in the candidate list of predicted motion vectors mvpListLX is selected as the best predicted motion vector mvpLX, and that index i is obtained.

[0166] Next, the motion vector subtraction unit 328 subtracts the selected predicted motion vector mvpLX of LX from the motion vector mvLX of LX, denoted as mvdLX = mvLX - mvpLX, to calculate the differential motion vector mvdLX of LX. Figure 19 Step S105).

[0167] <Typical Predictive Motion Vector Pattern Derivation Unit (Typical AMVP): Decoding Side Explanation>

[0168] Next, refer to Figure 25 The typical prediction motion vector mode processing steps on the decoding side are explained. On the decoding side, the spatial prediction motion vector candidate derivation unit 421, the temporal prediction motion vector candidate derivation unit 422, the historical prediction motion vector candidate derivation unit 423, and the prediction motion vector candidate supplementation unit 425 calculate, for each L0 and L1, the motion vectors used in inter-frame prediction in the typical prediction motion vector mode. Figure 25 Steps S201 to S206). Specifically, when the prediction mode PredMode of the processing object block is inter-frame prediction (MODE_INTER) and the inter-frame prediction mode of the processing object block is L0 prediction (Pred_L0), the candidate list of predicted motion vectors for L0, mvpListL0, is calculated, the predicted motion vector mvpL0 is selected, and the motion vector mvL0 of L0 is calculated. When the inter-frame prediction mode of the processing object block is L1 prediction (Pred_L1), the candidate list of predicted motion vectors for L1, mvpListL1, is calculated, the predicted motion vector mvpL1 is selected, and the motion vector mvL1 of L1 is calculated. When the inter-frame prediction mode for processing object blocks is dual prediction (Pred_BI), L0 prediction and L1 prediction are performed simultaneously. The candidate list of predicted motion vectors for L0, mvpListL0, is calculated. The predicted motion vector of L0, mvpL0, is selected, and the motion vector of L0, mvL0, is calculated. The candidate list of predicted motion vectors for L1, mvpListL1, is calculated, and the predicted motion vector of L1, mvpL1, is calculated. The motion vector of L1, mvL1, is calculated separately.

[0169] Similar to the encoding side, motion vector calculations are also performed on L0 and L1 separately on the decoding side, but L0 and L1 are processed in a common manner. Therefore, in the following description, L0 and L1 are denoted as a common LX. LX represents the inter-prediction mode used for inter-frame prediction of the coded block being processed. In the process of calculating the motion vector of L0, X is 0, and in the process of calculating the motion vector of L1, X is 1. In addition, in the process of calculating the motion vector of LX, if information from another reference list is referenced instead of the same reference list as the LX being calculated, this other reference list is denoted as LY.

[0170] When using the motion vector mvLX of LX ( Figure 25 Step S202: Yes), calculate the candidates for predicted motion vectors of LX, and construct the candidate list of predicted motion vectors of LX, mvpListLX( Figure 25 Step S203). Multiple candidates for predicted motion vectors are calculated from the spatial predicted motion vector candidate derivation unit 421, the temporal predicted motion vector candidate derivation unit 422, the historical predicted motion vector candidate derivation unit 423, and the predicted motion vector candidate supplementation unit 425 in the normal predicted motion vector pattern derivation unit 401, and a predicted motion vector candidate list mvpListLX is constructed. (About...) Figure 25 The detailed processing steps of step S203, using Figure 20 The flowchart is described later.

[0171] Next, the prediction motion vector candidate selection unit 426 retrieves the candidate mvpListLX[mvpIdxLX] of the prediction motion vector that corresponds to the index mvpIdxLX of the prediction motion vector provided by the bit string decoding unit 201 from the prediction motion vector candidate list mvpListLX, and selects it as the selected prediction motion vector mvpLX. Figure 25 Step S204).

[0172] Next, the motion vector addition unit 427 performs an addition operation on the differential motion vector mvdLX and the predicted motion vector mvpLX of LX provided by the bit string decoding unit 201, denoted as mvLX = mvpLX + mvdLX, to calculate the motion vector mvLX of LX. Figure 25 Step S205).

[0173] <Commonly Predictive Motion Vector Pattern Derivative (AMVP): Motion Vector Prediction Methods>

[0174] Figure 20 This is a flowchart illustrating the processing steps of the general prediction motion vector pattern derivation process, which has a common function in the general prediction motion vector pattern derivation unit 301 of the image encoding apparatus and the general prediction motion vector pattern derivation unit 401 of the image decoding apparatus according to embodiments of the present invention.

[0175] Both the normal prediction motion vector pattern derivation unit 301 and the normal prediction motion vector pattern derivation unit 401 have a prediction motion vector candidate list mvpListLX. The prediction motion vector candidate list mvpListLX forms a list structure and is provided with a storage area that stores prediction motion vector indices representing positions within the prediction motion vector candidate list and the prediction motion vector candidates corresponding to those indices as elements. The prediction motion vector indices start from 0, and the prediction motion vector candidates are stored in the storage area of ​​the prediction motion vector candidate list mvpListLX. In this embodiment, it is assumed that the prediction motion vector candidate list mvpListLX can register at least two prediction motion vector candidates (inter-frame prediction information). Furthermore, the variable numCurrMvpCand, representing the number of prediction motion vector candidates registered in the prediction motion vector candidate list mvpListLX, is set to 0.

[0176] Spatial prediction motion vector candidate derivation units 321 and 421 derive candidates for predicted motion vectors from the block adjacent to the left. In this process, reference is made to the block adjacent to the left ( Figure 11 The inter-frame prediction information (A0 or A1), including flags indicating whether the predicted motion vector candidate can be used, motion vectors, reference indices, etc., is used to derive the predicted motion vector mvLXA, and then the exported mvLXA is added to the predicted motion vector candidate list mvpListLX. Figure 20 Step S301). Additionally, X is 0 during L0 prediction and 1 during L1 prediction (the same applies below). Next, the spatial prediction motion vector candidate derivation units 321 and 421 derive candidates for predicted motion vectors from the block adjacent to the upper side. In this process, reference is made to the block adjacent to the upper side ( Figure 11 The inter-frame prediction information (B0, B1, or B2) indicates whether the predicted motion vector candidate can be used, along with the motion vector, reference index, etc., to derive the predicted motion vector mvLXB. If the derived mvLXA and mvLXB are not equal, then mvLXB is added to the predicted motion vector candidate list mvpListLX. Figure 20 Step S302). Figure 20 The processing steps S301 and S302 are the same except that the positions and number of the referenced adjacent blocks are different. They derive the flag availableFlagLXN indicating whether the predicted motion vector candidate of the coded block can be utilized, the motion vector mvLXN, and the reference index refIdxN (N represents A or B, the same below).

[0177] Next, the temporal prediction motion vector candidate derivation units 322 and 422 derive candidates for predicted motion vectors from blocks in images whose times differ from the current processing object image. In this process, the flag `availableFlagLXCol` indicating whether the coded blocks of images from different times can be utilized, along with the motion vector `mvLXCol`, reference index `refIdxCol`, and reference list `listCol`, are derived, and `mvLXCol` is added to the predicted motion vector candidate list `mvpListLX`. Figure 20 Step S303).

[0178] Furthermore, it is assumed that the processing of time-predicted motion vector candidate deriving units 322 and 422, which are in units of sequence (SPS), image (PPS), or strip, can be omitted.

[0179] Next, the historical predicted motion vector candidate derivation units 323 and 423 add the historical predicted motion vector candidates registered in the historical predicted motion vector candidate list HmvpCandList to the predicted motion vector candidate list mvpListLX. Figure 20 Step S304). Details of the registration processing steps for step S304 are available via [link / reference]. Figure 29 The flowchart is described later.

[0180] Next, the predicted motion vector candidate supplementary units 325 and 425 add predicted motion vector candidates with predetermined values ​​such as (0, 0) until the predicted motion vector candidate list mvpListLX is satisfied. Figure 20 (S305).

[0181] <Normal Merge Pattern Export Section (Normal Merge)>

[0182] Figure 18 The normal merging mode derivation section 302 includes a spatial merging candidate derivation section 341, a temporal merging candidate derivation section 342, an average merging candidate derivation section 344, a historical merging candidate derivation section 345, a merging candidate supplementation section 346, and a merging candidate selection section 347.

[0183] Figure 24 The normal merging mode derivation section 402 includes a spatial merging candidate derivation section 441, a temporal merging candidate derivation section 442, an average merging candidate derivation section 444, a historical merging candidate derivation section 445, a merging candidate supplementation section 446, and a merging candidate selection section 447.

[0184] Figure 21 This is a flowchart illustrating the steps of a normal merging mode export process that have a common function in the normal merging mode export unit 302 of the image encoding apparatus and the normal merging mode export unit 402 of the image decoding apparatus according to embodiments of the present invention.

[0185] The following sections will explain each process in turn. Furthermore, unless otherwise specified, the following explanations will focus on the case where the slice_type is B-slice, but this also applies to the case of P-slice. However, when the slice_type is P-slice, since only L0 prediction (Pred_L0) exists as the inter-frame prediction mode, and L1 prediction (Pred_L1) and double prediction (Pred_BI) do not exist, the processing surrounding L1 can be omitted.

[0186] Both the normal merge mode derivation unit 302 and the normal merge mode derivation unit 402 have a merge candidate list, mergeCandList. The merge candidate list mergeCandList forms a list structure and includes a merge index indicating the position within the merge candidate list, and a storage area for storing the merge candidates corresponding to the index as elements. The merge index number starts from 0, and merge candidates are stored in the storage area of ​​the merge candidate list mergeCandList. In subsequent processing, it is assumed that the merge candidate registered at merge index i in the merge candidate list mergeCandList is represented by mergeCandList[i]. In this embodiment, it is assumed that the merge candidate list mergeCandList can register at least six merge candidates (inter-frame prediction information). Furthermore, the variable numCurrMergeCand, representing the number of merge candidates registered in the merge candidate list mergeCandList, is set to 0.

[0187] In the spatial merging candidate export unit 341 and the spatial merging candidate export unit 441, based on the encoding information stored in the encoding information storage memory 111 of the image encoding device or the encoding information storage memory 205 of the image decoding device, spatial merging candidates A and B from the blocks adjacent to the left and top sides of the processing target block are exported, and the exported spatial merging candidates are registered in the merge candidate list mergeCandList. Figure 21 (Step S401). Here, N is defined to represent any one of spatial merge candidates A, B, or temporal merge candidate Col. The following are derived: availableFlagN indicating whether inter-frame prediction information of block N can be used as a spatial merge candidate; reference indices refIdxL0N and refIdxL1N for L0 and L1 of spatial merge candidate N; L0 prediction flag predFlagL0N indicating whether L0 prediction is performed; L1 prediction flag predFlagL1N indicating whether L1 prediction is performed; motion vector mvL0N for L0; and motion vector mvL1N for L1. However, in this embodiment, since the merge candidates are derived without referencing the inter-frame prediction information of the blocks contained in the coded block to be processed, spatial merge candidates using the inter-frame prediction information of the blocks contained in the coded block to be processed are not derived.

[0188] Next, in the time merging candidate export unit 342 and the time merging candidate export unit 442, time merging candidates from images at different times are exported, and the exported time merging candidates are registered in the merge candidate list mergeCandList. Figure 21 Step S402). Derive the availableFlagCol, which indicates whether time merging candidates can be used; the predFlagL0Col, which indicates whether time merging candidates should be predicted for L0; the predFlagL1Col, which indicates whether L1 prediction should be made; and the motion vectors mvL0Col for L0 and mvL1Col for L1.

[0189] Furthermore, the processing of the time merging candidate export unit 342 and the time merging candidate export unit 442, which are based on sequences (SPS), images (PPS), or stripes, can be omitted.

[0190] Next, in the historical merge candidate derivation unit 345 and the historical merge candidate derivation unit 445, the historical predicted motion vector candidates registered in the historical predicted motion vector candidate list HmvpCandList are registered in the merge candidate list mergeCandList. Figure 21 Step S403).

[0191] Furthermore, if the number of merge candidates registered in the merge candidate list mergeCandList, numCurrMergeCand, is less than the maximum number of merge candidates, MaxNumMergeCand, then the number of merge candidates registered in the merge candidate list mergeCandList numCurrMergeCand is used as the upper limit of the maximum number of merge candidates, MaxNumMergeCand, to derive historical merge candidates, and these candidates are then registered in the merge candidate list mergeCandList.

[0192] Next, in the average merge candidate derivation section 344 and the average merge candidate derivation section 444, average merge candidates are derived from the merge candidate list mergeCandList, and the exported average merge candidates are added to the merge candidate list mergeCandList. Figure 21 Step S404).

[0193] Furthermore, if the number of merge candidates registered in the merge candidate list numCurrMergeCand is less than the maximum number of merge candidates MaxNumMergeCand, the number of merge candidates registered in the merge candidate list numCurrMergeCand is capped at the maximum number of merge candidates MaxNumMergeCand, and an average number of merge candidates is derived and registered in the merge candidate list mergeCandList.

[0194] Here, the average merge candidate is a new merge candidate that has a motion vector obtained by averaging the motion vectors of the first and second merge candidates registered in the merge candidate list mergeCandList according to each L0 prediction and L1 prediction.

[0195] Next, in merge candidate supplementation section 346 and merge candidate supplementation section 446, when the number of merge candidates registered in the merge candidate list mergeCandList, numCurrMergeCand, is less than the maximum number of merge candidates, MaxNumMergeCand, the number of merge candidates registered in the merge candidate list mergeCandList, numCurrMergeCand, is used as the upper limit of the maximum number of merge candidates, MaxNumMergeCand, to derive additional merge candidates, and these candidates are registered in the merge candidate list mergeCandList. Figure 21 Step S405). Using the maximum number of merge candidates (MaxNumMergeCand) as the upper limit, in the P-strip, add merge candidates with a prediction mode of L0 prediction (Pred_L0) where the motion vector has a value of (0,0). In the B-strip, add merge candidates with a prediction mode of double prediction (Pred_BI) where the motion vector has a value of (0,0). The reference index when adding merge candidates is different from the reference index already added.

[0196] Next, in the merge candidate selection unit 347 and merge candidate selection unit 447, merge candidates are selected from those registered in the merge candidate list mergeCandList. On the encoding side, the merge candidate selection unit 347 selects merge candidates by calculating the amount of code and the amount of distortion, and provides the merge index representing the selected merge candidate and the inter-frame prediction information of the merge candidate to the motion compensation prediction unit 306 via the inter-frame prediction mode determination unit 305. On the other hand, on the decoding side, the merge candidate selection unit 447 selects merge candidates based on the decoded merge index and provides the selected merge candidates to the motion compensation prediction unit 406.

[0197] In the normal merge mode export unit 302 and the normal merge mode export unit 402, if the size (product of width and height) of a certain coding block is less than 32, merge candidates are exported from the parent block of that coding block. Then, the merge candidates exported from the parent block are used in all child blocks. However, this is limited to the case where the size of the parent block is 32 or more and is contained within the frame.

[0198] <Update historical prediction motion vector candidate list>

[0199] Next, the initialization and update methods of the historical predicted motion vector candidate list HmvpCandList possessed by the encoding information storage memory 111 on the encoding side and the encoding information storage memory 205 on the decoding side are described in detail. Figure 26 This is a flowchart illustrating the steps involved in initializing / updating the candidate list of historical predicted motion vectors.

[0200] In this embodiment, it is assumed that the update of the historical predicted motion vector candidate list HmvpCandList is performed in the encoding information storage memory 111 and the encoding information storage memory 205. Alternatively, a historical predicted motion vector candidate list update unit may be provided in the inter-frame prediction unit 102 and the inter-frame prediction unit 203 to perform the update of the historical predicted motion vector candidate list HmvpCandList.

[0201] At the beginning of the strip, the historical predicted motion vector candidate list HmvpCandList is initially set. On the encoding side, if the prediction method determination unit 105 selects the normal predicted motion vector mode or the normal merging mode, the historical predicted motion vector candidate list HmvpCandList is updated. On the decoding side, if the prediction information decoded by the bit string decoding unit 201 is the normal predicted motion vector mode or the normal merging mode, the historical predicted motion vector candidate list HmvpCandList is updated.

[0202] Inter-frame prediction information used during inter-frame prediction in either the normal motion vector prediction mode or the normal merging mode is registered in the historical prediction motion vector candidate list hmvpCandList as inter-frame prediction information candidate hMvpCand. Inter-frame prediction information candidate hMvpCand includes the reference index refIdxL0 for L0 and the reference index refIdxL1 for L1, the L0 prediction flag predFlagL0 indicating whether L0 prediction is performed, the L1 prediction flag predFlagL1 indicating whether L1 prediction is performed, the motion vector mvL0 for L0, and the motion vector mvL1 for L1.

[0203] If, among the elements (i.e., inter-frame prediction information) registered in the historical prediction motion vector candidate list HmvpCandList stored in the encoding information storage memory 111 on the encoding side and the encoding information storage memory 205 on the decoding side, there exists inter-frame prediction information with the same value as the inter-frame prediction information candidate hMvpCand, then that element is deleted from the historical prediction motion vector candidate list HmvpCandList. On the other hand, if there is no inter-frame prediction information with the same value as the inter-frame prediction information candidate hMvpCand, then the element at the beginning of the historical prediction motion vector candidate list HmvpCandList is deleted, and the inter-frame prediction information candidate hMvpCand is added to the end of the historical prediction motion vector candidate list HmvpCandList.

[0204] The number of elements in the historical predicted motion vector candidate list HmvpCandList possessed by the encoding information storage memory 111 on the encoding side and the encoding information storage memory 205 on the decoding side of the present invention is set to 6.

[0205] First, initialize the historical predicted motion vector candidate list HmvpCandList( on a strip basis). Figure 26 (Step S2101). At the beginning of the strip, add historical predicted motion vector candidates to all elements in the historical predicted motion vector candidate list HmvpCandList, and set the value of the number of historical predicted motion vector candidates NumHmvpCand registered in the historical predicted motion vector candidate list HmvpCandList to 6. Alternatively, at the beginning of the strip, clear all elements in the historical predicted motion vector candidate list HmvpCandList, and set the value of the number of historical predicted motion vector candidates (the current number of candidates) NumHmvpCand registered in the historical predicted motion vector candidate list HmvpCandList to 0.

[0206] Furthermore, although it is assumed that the initialization of the historical predicted motion vector candidate list HmvpCandList is implemented in strip units (the initial coded blocks of the strip), it can also be implemented in image units, tile units, or tree block rows units.

[0207] Figure 38 This is a table representing an example of historical predicted motion vector candidates added through the initialization of the historical predicted motion vector candidate list HmvpCandList. An example is shown for a stripe of type B and a reference image count of 4. The historical predicted motion vector index hMvpIdx ranges from (historical predicted motion vector candidate count NumHmvpCand-1) to 0. Depending on the stripe type, inter-frame prediction information with a motion vector mvLXHmvp (X is 0 or 1) value of (0, 0) is added as a historical predicted motion vector candidate to the historical predicted motion vector candidate list HmvpCandList, thus populating the historical predicted motion vector candidate list. At this point, the historical predicted motion vector index hMvpIdx starts from (historical predicted motion vector candidate count NumHmvpCand-1), and the reference index refIdxLX (X is 0 or 1) is set to a value incremented by 1 from 0 to (reference image count numRefIdx-1). Subsequently, duplication among historical predicted motion vector candidates is allowed, and the reference index refIdxLX is set to 0. Invalid historical predicted motion vector candidates are eliminated by setting all values ​​for the historical predicted motion vector candidate count NumHmvpCand, thus fixing the value of NumHmvpCand. In this way, coding efficiency can be improved by assigning smaller values ​​of the reference index refIdxLX, which generally have a high selection rate, from candidates with high probabilities of large historical predicted motion vector index hMvpIdx values ​​added to or merged into the predicted motion vector candidate list.

[0208] In addition, by filling the historical predicted motion vector candidate list with historical predicted motion vector candidates in strip units, the number of historical predicted motion vector candidates can be treated as a fixed value, thus simplifying processes such as the export of historical predicted motion vector candidates and the export of historical merged candidates.

[0209] Here, although the value of the motion vector is assumed to be (0, 0), which has a high probability, it can be any other value as long as it is a predetermined value. For example, it can be set to values ​​such as (4, 4), (0, 32), (-128, 0) to improve the coding efficiency of the differential motion vector, or multiple predetermined values ​​can be set to improve the coding efficiency of the differential motion vector.

[0210] Furthermore, although the historical predicted motion vector index hMvpIdx starts from (the number of historical predicted motion vector candidates NumHmvpCand-1) and the reference index refIdxLX (X is 0 or 1) is set to a value that increments by 1 from 0 to (the number of reference images numRefIdx-1), the historical predicted motion vector index hMvpIdx can start from 0.

[0211] Figure 39 This is another example of historical prediction motion vector candidates added through the initialization of the historical prediction motion vector candidate list HmvpCandList. An example is shown where the strip type is B and the number of reference images is 2. In this example, to ensure that there are no duplicate historical prediction motion vector candidates among the elements of the historical prediction motion vector candidate list HmvpCandList, either a different inter-frame prediction value of the reference index refIdxLX (X is 0 or 1) or the motion vector mvLXHmvp (X is 0 or 1) is added as a historical prediction motion vector candidate and the historical prediction motion vector candidate list is populated. At this time, the historical prediction motion vector index hMvpIdx starts from (the number of historical prediction motion vector candidates NumHmvpCand-1), and the reference index refIdxLX (X is 0 or 1) is set to a value incremented by 1 from 0 to (the number of reference images numRefIdx-1). Then, when the reference index refIdxLX is 0, motion vectors mvLXHmvp with different values ​​are added as candidates for historical predicted motion vectors. By setting all values ​​for the number of historical predicted motion vector candidates NumHmvpCand, and setting the value of the number of historical predicted motion vector candidates NumHmvpCand to a fixed value, invalid historical predicted motion vector candidates are eliminated.

[0212] In this way, by filling the historical predicted motion vector candidate list with non-repeating historical predicted motion vector candidates in strip units, the processing of the merge candidate supplementation unit 346, which is located after the historical merge candidate export unit 345 in the general merge mode export unit 302 described later and implemented in coding block units, can be omitted, thereby reducing the amount of processing.

[0213] Here, although the absolute values ​​of each component of the motion vector mvLXHmvp are set to 0 or 1, as long as there is no duplication among historical predicted motion vector candidates, the absolute values ​​of each component of the motion vector mvLXHmvp can also be set to greater than 1.

[0214] Furthermore, although the historical predicted motion vector index hmvpIdx starts from (the number of historical predicted motion vector candidates NumHmvpCand-1) and the reference index refIdxLX (X is 0 or 1) is set to a value that increments by 1 from 0 to (the number of reference images numRefIdx-1), the historical predicted motion vector index hMvpIdx can also start from 0.

[0215] Figure 40 This is another example of a table representing historical prediction motion vector candidates added through the initialization of the historical prediction motion vector candidate list HmvpCandList.

[0216] An example is shown where the strip type is B and the number of reference images is 1. In this example, inter-frame prediction information with a reference index refIdxLX of 0 and different values ​​of motion vector mvLXHmvp is added as historical prediction motion vector candidates to populate the historical prediction motion vector candidate list HmvpCandList, ensuring that there are no duplicates among the historical prediction motion vector candidates in each element. At this point, the historical prediction motion vector index hMvpIdx starts from (historical prediction motion vector candidate number NumHmvp_and-1), and inter-frame prediction information with a reference index refIdxLX (X is 0 or 1) of 0 and different values ​​of motion vector mvLXHmvp is set. By setting all values ​​for the historical prediction motion vector candidate number NumHmvpCand, the value of NumHmvpCand is fixed, thereby eliminating invalid historical prediction motion vector candidates.

[0217] In this way, by setting the reference index refIdxLX to 0, the reference image can be initialized without considering the number of reference images, thus simplifying the process.

[0218] Here, the absolute values ​​of the components of the motion vector mvLXHmvp are set to powers of 0, 1, or 2. However, if the reference index is 0 and there are no duplicates among historical predicted motion vector candidates, the absolute values ​​of the components of the motion vector mvLXHmvp can be powers of values ​​other than 2, such as 3, 4, or any other arbitrary value.

[0219] Furthermore, although the historical predicted motion vector index hMvpIdx starts from (the number of historical predicted motion vector candidates NumHmvpCand-1) and sets inter-frame prediction information where the reference index refIdxLX (X is 0 or 1) is 0 and the values ​​of the motion vectors mvLXHmvp are different, the historical predicted motion vector index hMvpIdx can also start from 0.

[0220] Next, the following update process for the historical predicted motion vector candidate list HmvpCandList is repeated for each coded block within the strip. Figure 26 Steps S2102 to S2111).

[0221] First, initial settings are performed on a block-by-block basis. The flag `identicalCandExist`, indicating the existence of identical candidates, is set to `FALSE`, and the index of the candidate to be deleted, `removeIdx`, is set to "0". Figure 26 Step S2103).

[0222] Determine whether there are inter-frame prediction information candidates hMvpCand for the registered object. Figure 26 Step S2104). If the prediction method determination unit 105 on the encoding side determines it to be a normal prediction motion vector mode or a normal merging mode, or if the bit string decoding unit 201 on the decoding side decodes it to be a normal prediction motion vector mode or a normal merging mode, the inter-frame prediction information is set as the inter-frame prediction information candidate hMvpCand for the registered object. If the prediction method determination unit 105 on the encoding side determines it to be an intra-frame prediction mode, a sub-block prediction motion vector mode, or a sub-block merging mode, or if the bit string decoding unit 201 on the decoding side decodes it to be an intra-frame prediction mode, a sub-block prediction motion vector mode, or a sub-block merging mode, the historical prediction motion vector candidate list HmvpCandList is not updated, and there is no inter-frame prediction information candidate hMvpCand for the registered object. If there is no inter-frame prediction information candidate hMvpCand for the registered object, steps S2105 to S2106 are skipped. Figure 26 Step S2104: No). If there is a candidate hMvpCand for inter-frame prediction information of the registered object, proceed with the processing after step S2105. Figure 26 Step S2104: Yes).

[0223] Next, it is determined whether there exists an element in each element of the historical predicted motion vector candidate list HmvpCandList that has the same value as the inter-frame prediction information candidate hMvpCand of the registered object (inter-frame prediction information), that is, whether there is a matching element. Figure 26 Step S2105). Figure 27 This is a flowchart of the same element confirmation process. When the value of the historical predicted motion vector candidate number NumHmvpCand is 0 ( Figure 27 Step S2121: No), the historical predicted motion vector candidate list HmvpCandList is empty. Since there are no identical candidates, skip this step. Figure 27 Steps S2122 to S2125 conclude the same feature confirmation process. This applies when the value of the historical predicted motion vector candidate number NumHmvpCand is greater than 0 ( Figure 27 Step S2121: Yes), the historical predicted motion vector index hMvpIdx is from 0 to NumHmvpCand-1, and the processing of step S2123 is repeated. Figure 27 Steps S2122 to S2125). First, compare whether the hMvpIdx-th element HmvpCandList[hMvpIdx] in the historical predicted motion vector candidate list (starting from 0) is the same as the inter-frame prediction information candidate hMvpCand. Figure 27 Step S2123). Under the same circumstances ( Figure 27 Step S2123: If yes, set the flag `identicalCandExist`, indicating whether there are identical candidates, to TRUE; set the current historical predicted motion vector index `hMvpIdx`, representing the location of the deleted object's index `removeIdx`, to the value of the index `hMvpIdx`; and end the identical feature confirmation process. In the case of different features ( Figure 27 Step S2123: No), increase hMvpIdx by 1. If the historical predicted motion vector index hMvpIdx is below NumHmvpCand-1, then proceed with the processing after step S2123.

[0224] Return again Figure 26 The flowchart describes the process of shifting and adding elements to the historical predicted motion vector candidate list HmvpCandList. Figure 26 Step S2106). Figure 28 yes Figure 26 The flowchart for the feature shifting / addition process in step S2106 of the historical predicted motion vector candidate list HmvpCandList is as follows: First, it is determined whether to add new features after removing features stored in the historical predicted motion vector candidate list HmvpCandList, or to add new features without removing existing features. Specifically, it compares whether the flag identicalCandExist, indicating the existence of identical candidates, is TRUE or whether NumHmvvpCand is 6. Figure 28 Step S2141). Under the condition that either the flag `identicalCandExist` indicating the existence of identical candidates is TRUE or the current number of candidates `NumHmvpCand` is 6 (…), Figure 28 Step S2141: Yes), after removing the features stored in the historical predicted motion vector candidate list HmvpCandList, add new features. Set the initial value of index i to the value of removeIdx+1. Repeat the feature shifting process of step S2143 from this initial value to NumHmvpCand. Figure 28 Steps S2142 to S2144). By copying the elements of HmvpCandList[i] to HmvpCandList[i-1], the elements are shifted forward ( Figure 28 Step S2143), increment i by 1 ( Figure 28 Steps S2142 to S2144). Next, add the inter-frame prediction information candidate hMvpCand to the (NumHmvpCand-1)th HmvpCandList[NumHmvpCand-1], which is the last corresponding to the historical prediction motion vector candidate list, starting from 0. Figure 28 Step S2145) ends the feature shifting / addition process of the historical predicted motion vector candidate list HmvpCandList. On the other hand, if neither of the conditions that the flag identicalCandExist indicating the existence of identical candidates is TRUE nor NumHmvpCand is 6 is met ( Figure 28 Step S2141: No), instead of removing the features stored in the historical predicted motion vector candidate list HmvpCandList, add the inter-frame prediction information candidate hMvpCand to the end of the historical predicted motion vector candidate list. Figure 28 Step S2146). Here, the last element of the historical predicted motion vector candidate list is the NumHmvpCand-th HmvpCandList[NumHmvpCand], counting from 0. Additionally, NumHmvpCand is incremented by 1, ending the shifting and addition process for elements in the historical predicted motion vector candidate list HmvpCandList.

[0225] Figures 31A to 31C This diagram illustrates an example of the update process for the historical predicted motion vector candidate list. When adding a new feature to the historical predicted motion vector candidate list HmvpCandList, which already contains six registered features (inter-frame prediction information), the new inter-frame prediction information is compared sequentially with the features preceding them in the historical predicted motion vector candidate list HmvpCandList. Figure 31A If the new feature has the same value as the third feature (HMVP2) from the beginning of the historical predicted motion vector candidate list (HmvpCandList), then remove feature HMVP2 from the historical predicted motion vector candidate list (HmvpCandList), shift (copy) the subsequent features HMVP3 to HMVP5 one by one forward, and add the new feature to the end of the historical predicted motion vector candidate list (HmvpCandList). Figure 31B Complete the update of the historical predicted motion vector candidate list HmvpCandList. Figure 31C ).

[0226] <Historical Predicted Motion Vector Candidate Export Processing>

[0227] Next, a detailed explanation will be provided as... Figure 20 Step S304 is a method for deriving historical predicted motion vector candidates from the historical predicted motion vector candidate list HmvpCandList. Figure 20 The processing step S304 is a common process in the historical prediction motion vector candidate derivation unit 323 of the normal prediction motion vector pattern derivation unit 301 on the encoding side and the historical prediction motion vector candidate derivation unit 423 of the normal prediction motion vector pattern derivation unit 401 on the decoding side. Figure 29 This is a flowchart illustrating the steps involved in deriving candidate motion vectors from historical predictions.

[0228] If the current number of predicted motion vector candidates, numCurrMvpCand, is greater than or equal to the maximum number of features in the predicted motion vector candidate list mvpListLX (which is 2 here), or if the historical number of predicted motion vector candidates, NumHmvpCand, is 0 ( Figure 29 Step S2201: No), omitted Figure 29 The processing from steps S2202 to S2209 ends the historical predicted motion vector candidate export processing step. When the current number of predicted motion vector candidates, numCurrMvpCand, is less than the maximum number of features in the predicted motion vector candidate list mvpListLX (2), and the value of the historical predicted motion vector candidate number, NumHmvpCand, is greater than 0 (…), the process continues. Figure 29 Step S2201: Yes), execute Figure 29 The processing steps S2202 to S2209.

[0229] Next, repeat. Figure 29 The processing in steps S2203 to S2208 continues until either index i is between 1 and 4 and either the historical predicted motion vector candidate number numCheckedHMVPCand is a smaller value. Figure 29 Steps S2202 to S2209). When the current number of predicted motion vector candidates, numCurrMvpCand, is greater than or equal to the maximum number of features in the predicted motion vector candidate list mvpListLX (more than 2). Figure 29 Step S2203: No), omitted Figure 29 The processing steps S2204 to S2209 conclude the historical predicted motion vector candidate export processing step. The current number of predicted motion vector candidates, numCurrMvpCand, is less than the maximum number of features in the predicted motion vector candidate list mvpListLX, which is 2 ( Figure 29 If step S2203 is true, execute the following steps: Figure 29 The processing after step S2204.

[0230] Next, for Y values ​​of 0 and 1 (L0 and L1), the processes from steps S2205 to S2207 are performed respectively. Figure 29 Steps S2204 to S2208). When the current number of predicted motion vector candidates, numCurrMvpCand, is greater than or equal to the maximum number of features in the predicted motion vector candidate list mvpListLX (more than 2). Figure 29 Step S2205: No), omitted Figure 29 The processing steps S2206 to S2209 conclude the historical predicted motion vector candidate export processing step. This applies when the current number of predicted motion vector candidates, numCurrMvpCand, is less than the maximum number of features in the predicted motion vector candidate list mvpListLX (2). Figure 29 Step S2205 in the process: Yes, execute Figure 29 The processing after step S2206 in the process.

[0231] Next, in the historical predicted motion vector candidate list HmvpCandList, the element that has the same reference index as the reference index refIdxLX of the encoded / decoded object motion vector, and is different from any element in the predicted motion vector list mvpListLX ( Figure 29 Step S2206: Yes), add the motion vector of the historical predicted motion vector candidate HmvpCandList[NumHmvpCand-i] to the element mvpListLX[numCurrMvpCand] starting from 0 in the predicted motion vector candidate list. Figure 29 Step S2207), and increment the current number of predicted motion vector candidates numCurrMvpCand by 1. This occurs when there are no features in the historical predicted motion vector candidate list HmvpCandList with the same reference index refIdxLX as the motion vector of the encoded / decoded object, and no features that are different from any feature in the predicted motion vector list mvpListLX. Figure 29 Step S2206: No), skip the addition process in step S2207.

[0232] Both L0 and L1 are performed as above Figure 29 Processing steps S2205 to S2207 ( Figure 29 Steps S2204 to S2208). Increment index i by 1. If index i is less than or equal to 4 and either the historical predicted motion vector candidate number NumHmvpCand is smaller, repeat the processing after step S2203. Figure 29 Steps S2202 to S2209).

[0233] <Historical Merge Candidate Export Processing>

[0234] Next, a detailed explanation will be provided as... Figure 21 Step S404 is a method for deriving historical merge candidates from the historical merge candidate list HmvpCandList. Figure 21 The processing step S404 is a common process in the historical merge candidate derivation section 345 of the normal merge mode derivation section 302 on the encoding side and the historical merge candidate derivation section 445 of the normal merge mode derivation section 402 on the decoding side. Figure 30 This is a flowchart illustrating the steps involved in the historical merge candidate export process.

[0235] First, perform initialization processing ( Figure 30 Step S2301). Set the value of FALSE for each element from 0 to (numCurrMergeCand-1)th element in isPruned[i], and set the variable numOrigMergeCand to the number of elements registered in the current merge candidate list, numCurrMergeCand.

[0236] Next, the initial value of index hMvpIdx is set to 1, and the process is repeated from this initial value to NumHmvpCand. Figure 30 The addition process from step S2303 to step S2310 ( Figure 30 Steps S2302 to S2311). If the number of features registered in the current merge candidate list, numCurrMergeCand, is not below (MaxNumMergeCand-1), then the historical merge candidate export process ends because all features have been added to the merge candidate list as merge candidates. Figure 30 Step S2303: No). If the number of features registered in the current merge candidate list, numCurrMergeCand, is less than (MaxNumMergeCand-1), then proceed with the processing after step S2304. Set the value of sameMotion to FALSE. Figure 30 (Step S2304). Next, the initial value of index i is set to 0, and the process continues from this initial value to numOrigMergeCand-1. Figure 30 Processing steps S2306 and S2307 ( Figure 30 (S2305~S2308). Compare whether the (NumHmvpCand-hMvpIdx)th element HmvpCandList[NumHmvpCand-hMvpIdx] in the historical motion vector prediction candidate list (starting from 0) is the same as the i-th element mergeCandList[i] in the merged candidate list (starting from 0). Figure 30 Step S2306).

[0237] The term "same value" for merge candidates refers to the fact that all the constituent elements (inter-frame prediction mode, reference index, and motion vector) of a merge candidate have the same value. This applies when the merge candidates have the same value and isPruned[i] is FALSE. Figure 30 Step S2306: Yes), sameMotion and isPruned[i] are both set to TRUE. Figure 30 Step S2307). In the case that the values ​​are not the same ( Figure 30 If step S2306 is not specified, skip step S2307. Figure 30 After the repeated processing of steps S2305 to S2308 is completed, compare whether sameMotion is FALSE. Figure 30 Step S2309), in the case that sameMotion is FALSE (false) Figure 30 Step S2309: Yes), that is, since the (NumHmvpCand-hMvpIdx)th element HmvpCandList[NumHvpCand-hMvpIdx] from the historical predicted motion vector candidate list does not exist in mergeCandList, therefore, the (NumHmvpCand-hMvpIdx)th element HmvpCandList[NumHmvpCand-hMvpIdx] from the historical predicted motion vector candidate list is added to the mergeCandList[numCurrMergeCand] of the merge candidate list, and numCurrMergeCand is incremented by 1. Figure 30 Step S2310). Increment the index hMvpIdx by 1 ( Figure 30 Step S2302) is performed. Figure 30 Repeat steps S2302 to S2311.

[0238] After confirming all elements in the historical predicted motion vector candidate list, or adding all elements in the merge candidate list to the merge candidate list, the export process of the historical merge candidate is completed.

[0239] <Motion Compensation Predictive Processing>

[0240] The motion compensation prediction unit 306 acquires the position and size of the block of the object currently being predicted during encoding. Additionally, the motion compensation prediction unit 306 acquires inter-frame prediction information from the inter-frame prediction mode determination unit 305. Based on the acquired inter-frame prediction information, a reference index and motion vector are derived. After acquiring an image signal that shifts the reference image determined by the reference index in the decoded image memory 104 from the same position as the image signal of the predicted block by the amount of motion vector, a prediction signal is generated.

[0241] In inter-frame prediction, when the inter-frame prediction mode is L0 or L1 prediction, which involves prediction from a single reference image, the prediction signal obtained from one reference image is set as the motion-compensated prediction signal. When the inter-frame prediction mode is BI prediction, which involves prediction from two reference images, the signal obtained by weighted averaging of the prediction signals obtained from the two reference images is set as the motion-compensated prediction signal, and this motion-compensated prediction signal is provided to the prediction method determination unit 105. Here, the weighted average ratio of the two predictions is set to 1:1, but other ratios can also be used for weighted averaging. For example, the closer the image to be predicted is to the reference image, the larger the weighting ratio. Alternatively, a table corresponding to combinations of image intervals and weighting ratios can be used to calculate the weighting ratio.

[0242] The motion compensation prediction unit 406 has the same function as the motion compensation prediction unit 306 on the encoding side. The motion compensation prediction unit 406 acquires inter-frame prediction information from the normal prediction motion vector pattern derivation unit 401, the normal merging pattern derivation unit 402, the sub-block prediction motion vector pattern derivation unit 403, and the sub-block merging pattern derivation unit 404 via switch 408. The motion compensation prediction unit 406 provides the acquired motion compensation prediction signal to the decoded image signal overlay unit 207.

[0243] <About Inter-Frame Prediction Mode>

[0244] The process of making a prediction based on a single reference image is defined as single prediction. In the case of single prediction, a prediction is made using either an L0 prediction or an L1 prediction, which uses either of the two reference images registered in the reference list L0 or L1.

[0245] Figure 32 This shows the case where the reference image (RefL0Pic) of L0 in a single prediction is at a moment before the processing object image (CurPic). Figure 33 This illustrates the case where the reference image for the L0 prediction in a single prediction is at a time after the processed object image. Similarly, it is also possible to... Figure 32 and Figure 33 The reference image for L0 prediction is replaced with the reference image for L1 prediction (RefL1Pic) for single prediction.

[0246] The process of making predictions based on two reference images is defined as dual prediction. In the case of dual prediction, L0 prediction and L1 prediction are used to describe BI prediction. Figure 34 This illustrates the case where the reference image for L0 prediction is at a time before the object image is processed, and the reference image for L1 prediction is at a time after the object image is processed. Figure 35 This shows the situation where the reference images for L0 prediction and L1 prediction in a dual prediction are at a time before the object image is processed. Figure 36 This shows the case where the reference images for L0 prediction and L1 prediction in a dual prediction scenario are at a time after the processed object image.

[0247] Thus, the relationship between the prediction category and time of L0 / L1 can be used when L0 is not limited to the past direction and L1 is not limited to the future direction. Furthermore, in the case of dual prediction, the same reference image can be used to perform both L0 and L1 predictions. Additionally, it is determined whether motion compensation prediction is performed using single or dual prediction based on information indicating whether L0 or L1 prediction is used (e.g., flags).

[0248] <About the Reference Index>

[0249] In embodiments of the present invention, to improve the accuracy of motion compensation prediction, the optimal reference image can be selected from multiple reference images during motion compensation prediction. Therefore, the reference image used in motion compensation prediction is used as a reference index, and the reference index is encoded into the bitstream along with the differential motion vector.

[0250] <Motion compensation processing based on typical predicted motion vector patterns>

[0251] As in Figure 16 As also shown in the inter-frame prediction unit 102 on the encoding side, when inter-frame prediction information based on the normal prediction motion vector mode derivation unit 301 is selected in the inter-frame prediction mode determination unit 305, the motion compensation prediction unit 306 obtains the inter-frame prediction information from the inter-frame prediction mode determination unit 305, derives the inter-frame prediction mode, reference index, and motion vector of the block to be processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the prediction method determination unit 105.

[0252] Similarly, as in Figure 22 As also shown in the inter-frame prediction unit 203 on the decoding side, when switch 408 is connected to the normal prediction motion vector mode derivation unit 401 during decoding, motion compensation prediction unit 406 acquires the inter-frame prediction information based on the normal prediction motion vector mode derivation unit 401, derives the inter-frame prediction mode, reference index, and motion vector of the block to be processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the decoded image signal overlay unit 207.

[0253] <Motion compensation processing based on the usual merging pattern>

[0254] As in Figure 16 As also shown in the inter-frame prediction unit 102 on the encoding side, when inter-frame prediction information based on the normal merging mode derivation unit 302 is selected in the inter-frame prediction mode determination unit 305, the motion compensation prediction unit 306 obtains the inter-frame prediction information from the inter-frame prediction mode determination unit 305, derives the inter-frame prediction mode, reference index, and motion vector of the block to be processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the prediction method determination unit 105.

[0255] Similarly, as in Figure 22 As also shown in the inter-frame prediction unit 203 on the decoding side, when switch 408 is connected to the normal merging mode derivation unit 402 during decoding, motion compensation prediction unit 406 acquires inter-frame prediction information based on the normal merging mode derivation unit 402, derives the inter-frame prediction mode, reference index, and motion vector of the block currently being processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the decoded image signal overlay unit 207.

[0256] <Motion Compensation Processing Based on Sub-Block Predicted Motion Vector Patterns>

[0257] As in Figure 16 As also shown in the inter-frame prediction unit 102 on the encoding side, when the inter-frame prediction information based on the sub-block prediction motion vector mode derivation unit 303 is selected in the inter-frame prediction mode determination unit 305, the motion compensation prediction unit 306 obtains the inter-frame prediction information from the inter-frame prediction mode determination unit 305, derives the inter-frame prediction mode, reference index, and motion vector of the block to be processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the prediction method determination unit 105.

[0258] Similarly, as in Figure 22 As also shown in the inter-frame prediction unit 203 on the decoding side, when switch 408 is connected to sub-block prediction motion vector mode derivation unit 403 during decoding, motion compensation prediction unit 406 acquires inter-frame prediction information based on sub-block prediction motion vector mode derivation unit 403, derives the inter-frame prediction mode, reference index, and motion vector of the block currently being processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to decoded image signal overlay unit 207.

[0259] <Motion Compensation Processing Based on Sub-Block Merging Pattern>

[0260] As in Figure 16 As also shown in the inter-frame prediction unit 102 on the encoding side, when the inter-frame prediction mode determination unit 305 selects the inter-frame prediction information based on the sub-block merging mode derivation unit 304, the motion compensation prediction unit 306 obtains the inter-frame prediction information from the inter-frame prediction mode determination unit 305, derives the inter-frame prediction mode, reference index, and motion vector of the block to be processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the prediction method determination unit 105.

[0261] Similarly, as in Figure 22 As also shown in the inter-frame prediction unit 203 on the decoding side, when switch 408 is connected to sub-block merging mode derivation unit 404 during decoding, motion compensation prediction unit 406 acquires inter-frame prediction information based on sub-block merging mode derivation unit 404, derives the inter-frame prediction mode, reference index, and motion vector of the block currently being processed, and generates a motion compensation prediction signal. The generated motion compensation prediction signal is provided to the decoded image signal overlay unit 207.

[0262] <Motion Compensation Processing Based on Affine Transform Prediction>

[0263] In both the normal prediction motion vector mode and the normal merging mode, affine model-based motion compensation can be used based on the following flags. These flags are reflected in the following flags based on the inter-frame prediction conditions determined by the inter-frame prediction mode determination unit 305 during the encoding process, and are encoded into the bitstream. During the decoding process, whether to perform affine model-based motion compensation is determined based on the following flags in the bitstream.

[0264] `sps_affine_enabled_flag` indicates whether affine-based motion compensation can be used in inter-frame prediction. If `sps_affine_enabled_flag` is 0, sequence-by-sequence suppression is applied, preventing affine-based motion compensation. Furthermore, `inter_affine_flag` and `cu_affine_type_flag` are not transmitted in the CU (coded block) syntax of the encoded video sequence. If `sps_affine_enabled_flag` is 1, affine-based motion compensation can be used in the encoded video sequence.

[0265] `sps_affine_type_flag` indicates whether motion compensation based on a six-parameter affine model can be used in inter-frame prediction. If `sps_affine_type_flag` is 0, motion compensation based on a six-parameter affine model is suppressed. Additionally, `cu_affine_type_flag` is not transmitted in the CU syntax of the encoded video sequence. If `sps_affine_type_flag` is 1, motion compensation based on a six-parameter affine model can be used in the encoded video sequence. If `sps_affine_type_flag` is not present, it is set to 0.

[0266] When decoding P-strips or B-strips, in the CU that is currently being processed, if `inter_affine_flag` is 1, motion compensation based on an affine model is used to generate the motion compensation prediction signal for the CU that is currently being processed. If `inter_affine_flag` is 0, the affine model is not used for the CU that is currently being processed. If `inter_affine_flag` does not exist, it is set to 0.

[0267] When decoding P-strips or B-strips, in the CU that is currently being processed, if cu_affine_type_flag is 1, motion compensation based on a six-parameter affine model is used to generate the motion compensation prediction signal for the CU that is currently being processed. If cu_affine_type_flag is 0, motion compensation based on a four-parameter affine model is used to generate the motion compensation prediction signal for the CU that is currently being processed.

[0268] In motion compensation based on affine models, since reference indices or motion vectors are derived on a sub-block basis, motion compensation prediction signals are generated using the reference indices or motion vectors that are the objects of processing on a sub-block basis.

[0269] The four-parameter affine model is as follows: the motion vector of the sub-block is derived from the four parameters of the horizontal and vertical components of the motion vectors of the two control points, and motion compensation is performed on a sub-block basis.

[0270] (Second Implementation)

[0271] In the first embodiment, the order of elements (candidates) in the historical predicted motion vector candidate list is the same in both the normal predicted motion vector mode and the normal merging mode. On the other hand, in the second embodiment, the first embodiment is modified and supplemented as follows. In the normal predicted motion vector mode of the second embodiment, as... Figure 46 As shown, the reference order for utilizing the historical predicted motion vector candidate list is executed in the reverse order of the first embodiment, and in the normal merging mode of the second embodiment, as... Figure 45 As shown, the reference order when using the historical predicted motion vector candidate list is executed in the same order as in the first embodiment. That is, in the normal predicted motion vector mode and the normal merging mode, the order of the candidates included in the historical predicted motion vector candidate list is reversed. Therefore, in the normal predicted motion vector mode, the elements in the historical predicted motion vector candidate list are added to the predicted motion vector candidate list sequentially, starting from the oldest elements, while in the normal merging mode, the elements in the historical predicted motion vector candidate list are added to the predicted motion vector candidate list sequentially, starting from the newest elements.

[0272] <Historical Predicted Motion Vector Candidate List>

[0273] Here, the composition and operation of the historical predicted motion vector candidate list will be explained.

[0274] like Figure 41 As shown, the inter-frame prediction information used by inter-frame prediction in the coded object block is set as the inter-frame prediction information candidate hMvpCand to be registered, and as a previously used history, it is registered in the historical prediction motion vector candidate list HmvpCandList. Figure 41 In the historical prediction motion vector candidate list, there is a list structure that can store six elements, and the basic storage operation is a first-in-first-out (FIFO) method that retrieves elements in order from the first stored element.

[0275] Here, as an example, the maximum number of features that can be stored in HmvpCandList is described as 6, which is determined by the encoding and decoding sides, but it is not particularly limited and can be more than 6. Alternatively, the maximum number of features that can be stored in HmvpCandList can be 5 or less. For example, HmvpCandList can be configured to have a maximum number of features equal to the maximum number of inter-frame prediction information candidates, such as the maximum number of features in the prediction motion vector candidate list mvpListLX, the maximum number of merged candidates, and the maximum number of sub-block merged candidates. Furthermore, HmvpCandList can be configured to be associated with the maximum number of features in the inter-frame prediction information candidates for each mode.

[0276] It is also possible to construct a transmission from the encoding side to the decoding side by including the maximum number of elements of the HmvpCandList in the syntactic elements of the bitstream.

[0277] like Figure 41 As shown, HmvpCandList can save 6 features from position 0 at the beginning of the list to position 5 at the end of the list, and these features can be populated sequentially from position 0 to position 5. Positions 0 to 5 are managed as historical predicted motion vector indices hMvpIdx. For example, position 0 can be represented as hMVpIdx[0], and position 5 can be represented as hMVpIdx[5]. The number of features saved in HmvpCandList is managed using NumHmvpCand, and the addition or removal of saved features is managed within a range from 0 to 6, which is the maximum number of features.

[0278] This section provides a detailed explanation of the scenario where new features are added while the maximum number of features is stored in HmvpCandList. For example... Figure 42 As shown, when registering the inter-frame prediction information candidate hMvpCand as new history, the feature at position 0 (the first element) is deleted, and the position of each feature is shifted one position towards the beginning. The shifting result is as follows: Figure 43 As shown, by reducing the number of saved features by one, a new feature can be saved at the end position 5. Therefore, by saving the inter-frame prediction information candidate hMvpCand at the end position 5, as... Figure 44 As shown, add a new element to HmvpCandList.

[0279] <Historical Predicted Motion Vector Candidate Derivation Process under the Normal Predicted Motion Vector Mode in the Second Implementation>

[0280] Here, the historical predicted motion vector candidate derivation process under the normal predicted motion vector mode of the second embodiment will be described.

[0281] Figure 47 This is a flowchart illustrating the historical predicted motion vector candidate derivation process when, in the normal predicted motion vector mode of the second embodiment, the historical predicted motion vector candidate list HmvpCandList is constructed while referencing in the reverse order of the reference order in the normal merging mode.

[0282] When the current number of predicted motion vector candidates, numCurrMvpCand, is greater than or equal to the maximum number of features in the predicted motion vector candidate list mvpListLX (which is 2 in this case), or when the historical number of predicted motion vector candidates, NumHmvpCand, is 0 ( Figure 47 Step S2601: No), omitted Figure 47 The processing steps S2602 to S2609 end the historical predicted motion vector candidate export processing step. This occurs when the current number of predicted motion vector candidates, numCurrMvpCand, is less than 2 times the maximum number of features in the predicted motion vector candidate list mvpListLX, and the value of the historical predicted motion vector candidate number, NumHmvpCand, is greater than 0. Figure 47 Step S2601: Yes), execute Figure 47 The processing steps S2602 to S2609.

[0283] Next, starting with index i from 0, the value is calculated by subtracting 1 from the smaller of 4 and the historical predicted motion vector candidate number NumHmvpCand, and this process is repeated. Figure 47 The processing in steps S2603 to S2608 ( Figure 47 Steps S2602 to S2609). When the current number of predicted motion vector candidates, numCurrMvpCand, is more than twice the maximum number of features in the predicted motion vector candidate list mvpListLX (… Figure 47 Step S2603: No), omitted Figure 47 The processing steps S2604 to S2609 conclude the current historical predicted motion vector candidate export processing steps. This applies when the current number of predicted motion vector candidates, numCurrMvpCand, is less than 2 times the maximum number of features in the predicted motion vector candidate list, mvpListLX. Figure 47 Step S2603: Yes), execute Figure 47 The processing after step S2604.

[0284] Next, for indices Y of 0 and 1 (L0 and L1), the processing from steps S2605 to S2607 is performed respectively. Figure 47 Steps S2604 to S2608). When the current number of predicted motion vector candidates, numCurrMvpCand, is more than twice the maximum number of features in the predicted motion vector candidate list mvpListLX (… Figure 47 Step S2605: No), omitted Figure 47 The processing steps S2606 to S2609 conclude the current historical predicted motion vector candidate export processing steps. This applies when the current number of predicted motion vector candidates, numCurrMvpCand, is less than 2 times the maximum number of features in the predicted motion vector candidate list, mvpListLX. Figure 47 Step S2605: Yes), execute Figure 47 The processing after step S2606.

[0285] Next, in the historical predicted motion vector candidate list HmvpCandList, the element that has the same reference index as the reference index refI_dxLX of the encoded / decoded object motion vector and is different from any element in the predicted motion vector candidate list mvp_ListLX ( Figure 47 Step S2606: Yes), starting from 0 in the predicted motion vector candidate list, add the motion vector of the historical predicted motion vector candidate HmvpCandList[i] to the LY of the element mvpListLX[numCurrMvpCand]. Figure 47 In step S2607, increment the current predicted motion vector candidate number numCurrMvpCand by 1. In the historical predicted motion vector candidate list HmvpCandList, if there are no features with the same reference index as the reference index refIdxLX of the encoded / decoded object's motion vector and different from any feature in the predicted motion vector candidate list mvp_ListLX, then... Figure 47 Step S2607: No), skip the addition process in step S2607.

[0286] The above is performed in both L0 and L1. Figure 47 Processing steps S2605 to S2607 ( Figure 47 Steps S2604 to S2608).

[0287] Increment index i by 1 ( Figure 47 In steps S2602 and S2609, if index i is less than or equal to any small value among 4 and the historical predicted motion vector candidate number NumHmvvpCand minus 1, then the processing after step S2603 is performed again. Figure 47 Steps S2602 to S2609).

[0288] As described above, in the usual motion vector prediction mode, the historical motion vector candidate list is referenced sequentially, starting with the first added feature (old feature), and then added to the motion vector candidate list.

[0289] (One of the variations)

[0290] Alternatively, a historical reference flag representing the reference order of a historical predicted motion vector candidate list as in this embodiment can be imported, switching the reference order of the first embodiment and the reference order of this embodiment. The historical reference flag can be included in the syntactic elements of the bitstream and transmitted to the decoding side. The decoding side can obtain the bitstream with the historical reference flag in the syntactic elements and can decode the bitstream.

[0291] (Variation Example 2)

[0292] In this embodiment, under the normal prediction motion vector mode, such as Figure 46 As shown, elements 0 to 5 in the historical predicted motion vector candidate list are referenced one by one. In the typical merging mode, as... Figure 45 As shown, elements 5 through 0 in the historical predicted motion vector candidate list are referenced sequentially. However, this is not limited to this method as long as the order in which the historical predicted motion vector candidate list is referenced differs between the normal predicted motion vector mode and the normal merging mode. For example, elements 0 through 2 can be referenced in the normal predicted motion vector mode, while elements 5 through 1 can be referenced in the normal merging mode, thus allowing the number of referenced elements in the historical predicted motion vector candidate list to differ between the two modes. In this case, it is preferable to have a larger number of references in the normal merging mode, which has more candidates. Furthermore, in the normal merging mode, the starting point can be changed, as in the case of referencing elements 4 through 0.

[0293] By employing the second embodiment described above, and by setting the reference order when using the historical predicted motion vector candidate list to a different reference order than the usual predicted motion vector pattern, candidates suitable for each pattern can be used. In particular, when selecting the usual predicted motion vector pattern for encoding differential motion vectors, motion vectors that differ from the surrounding conditions of the encoded object block are often required. Therefore, by searching the historical predicted motion vector candidate list and using the previously saved older features—those with relatively low relevance considered different from the most recent motion vector candidate list—as candidates sequentially, encoding efficiency can be improved by using candidates suitable for the usual predicted motion vector pattern.

[0294] In addition, in the normal merging mode, the higher the relevance of the surrounding objects in the coded object block, the higher the coding efficiency. Therefore, by searching the historical predicted motion vector candidate list and using the features that are highly relevant to the most recent motion vector candidate list, i.e. the newly saved features, as candidates in sequence, coding efficiency can be improved by using candidates that are suitable for the normal merging mode.

[0295] All of the above-described embodiments can also be combined in multiple ways.

[0296] In all the embodiments described above, the bitstream output by the image encoding device has a specific data format so that it can be decoded according to the encoding method used in the embodiment. Furthermore, the image decoding device corresponding to the image encoding device is capable of decoding the bitstream of this specific data format.

[0297] When using wired or wireless networks to exchange bitstreams between an image encoding device and an image decoding device, the bitstream can be converted into a data format suitable for transmission over the communication line for transmission. In this case, a transmitting device is provided that converts the bitstream output from the image encoding device into encoded data in a data format suitable for transmission over the communication line and transmits the encoded data to the network; and a receiving device receives the encoded data from the network and restores the encoded data to a bitstream for supply to the image decoding device. The transmitting device includes: a memory for buffering the bitstream output from the image encoding device; a packet processing unit for packetizing the bitstream; and a transmitting unit for transmitting the packetized encoded data via the network. The receiving device includes: a receiving unit for receiving the packetized encoded data via the network; a memory for buffering the received encoded data; and a packet processing unit for packetizing the encoded data to generate a bitstream and supplying the bitstream to the image decoding device.

[0298] Alternatively, the display device can be configured by adding a display unit that displays the image decoded by the image decoding device. In this case, the display unit reads the decoded image signal generated by the decoded image signal overlap unit 207 and stored in the decoded image memory 208, and displays it on the screen.

[0299] Alternatively, an imaging unit can be added to the structure to input the captured image into an image encoding device, thus functioning as an imaging device. In this case, the imaging unit inputs the captured image signal into the block segmentation unit 101.

[0300] Figure 37 An example of the hardware structure of the encoding / decoding apparatus according to this embodiment is shown. The encoding / decoding apparatus includes the structure of the image encoding apparatus and the image decoding apparatus according to embodiments of the present invention. The encoding / decoding apparatus 9000 has a CPU 9001, an encoder / decoder IC 9002, an I / O interface 9003, a memory 9004, an optical disc drive 9005, a network interface 9006, and a video interface 9009, and the various parts are connected via a bus 9010.

[0301] The image encoding unit 9007 and the image decoding unit 9008 are typically installed as a codec IC 9002. In the image encoding apparatus according to embodiments of the present invention, image encoding processing is performed by the image encoding unit 9007, and in the image decoding apparatus according to embodiments of the present invention, image decoding processing is performed by the image decoding unit 9008. The I / O interface 9003 is implemented, for example, via a USB interface, and is connected to an external keyboard 9104, mouse 9105, etc. The CPU 9001 controls the encoding / decoding apparatus 9000 to execute the user-desired action based on user operations input through the I / O interface 9003. These operations, performed by the user via the keyboard 9104, mouse 9105, etc., include selecting which function to perform (encoding or decoding), setting the encoding quality, setting the input / output destination of the bitstream, and setting the input / output destination of the image.

[0302] When a user wishes to reproduce an image recorded on the disc recording medium 9100, the optical disc drive 9005 reads a bitstream from the inserted disc recording medium 9100 and sends the read bitstream to the image decoding unit 9008 of the codec IC 9002 via the bus 9010. The image decoding unit 9008 performs image decoding processing in the image decoding apparatus according to embodiments of the present invention on the input bitstream and sends the decoded image to an external monitor 9103 via the video interface 9009. Additionally, the codec apparatus 9000 has a network interface 9006 and can connect to an external distribution server 9106 and a portable terminal 9107 via the network 9101. When a user wishes to reproduce an image recorded on the distribution server 9106 or the mobile terminal 9107 instead of an image recorded on the disc recording medium 9100, the network interface 9006 obtains the bitstream from the network 9101 instead of reading the bitstream from the input disc recording medium 9100. Furthermore, if a user wishes to reproduce an image recorded in memory 9004, the image decoding process in the image decoding apparatus according to an embodiment of the present invention is performed on the bitstream recorded in memory 9004.

[0303] When a user wishes to encode and record an image captured by an external camera 9102 in memory 9004, the video interface 9009 inputs the image from the camera 9102 and sends it to the image encoding unit 9007 of the codec IC 9002 via bus 9010. The image encoding unit 9007 performs image encoding processing according to the image encoding apparatus of this invention on the image input via the video interface 9009 and generates a bitstream. Then, the bitstream is sent to memory 9004 via bus 9010. When the user wishes to record the bitstream on the disc recording medium 9100 instead of memory 9004, the optical disc drive 9005 writes the bitstream to the inserted disc recording medium 9100.

[0304] It is also possible to implement a hardware structure that has an image encoding device but no image decoding device, or a hardware structure that has an image decoding device but no image encoding device. Such a hardware structure can be implemented, for example, by replacing the codec IC9002 with an image encoding unit 9007 or an image decoding unit 9008, respectively.

[0305] The processing related to the above encoding and decoding can, of course, be implemented using hardware transmission, storage, and reception devices, and can be implemented through firmware stored in ROM (Read-Only Memory), flash memory, or software such as computers. This firmware or software program can be provided by recording it on a readable recording medium such as a computer, or by providing it from a server via wired or wireless networks, or by providing it as data broadcast via terrestrial wave or satellite digital broadcasting.

[0306] The present invention has been described above based on embodiments. The embodiments are illustrative; various modifications can be made to the combination of these constituent elements and processing steps, and such modifications are also within the scope of the present invention, as will be understood by those skilled in the art.

[0307] Symbol Explanation

[0308] 100 Image encoding device, 101 Block segmentation unit, 102 Inter-frame prediction unit, 103 Intra-frame prediction unit, 104 Decoded image memory, 105 Prediction method determination unit, 106 Residual generation unit, 107 Orthogonal transform / quantization unit, 108 Bit string encoding unit, 109 Inverse quantization / inverse orthogonal transform unit, 110 Decoded image signal overlay unit, 111 Encoded information storage memory, 200 Image decoding device, 201 Bit string decoding unit, 202 Block segmentation unit, 203 Inter-frame prediction unit, 204 Intra-frame prediction unit, 205 Encoded information storage memory, 206 Inverse quantization / inverse orthogonal transform unit, 207 Decoded image signal overlay unit, 208 Decoded image memory.< / poc>

Claims

1. An image coding apparatus, wherein the moving image is encoded using inter-frame prediction based on inter-frame prediction information, taking blocks as units of segmentation of each image of the moving image as the unit, the image coding apparatus being characterized in that it comprises: The encoding information storage unit adds the inter-frame prediction information used in inter-frame prediction to the historical prediction motion vector candidate list; The spatial prediction motion vector candidate derivation section derives spatial prediction motion vector candidates from blocks adjacent to the processed object block and adds them to the prediction motion vector candidate list. The time-predicted motion vector candidate derivation unit derives time-predicted motion vector candidates from blocks located at the same position as or near the processed object block, and adds them to the predicted motion vector candidate list; The historical predicted motion vector candidate derivation unit derives historical predicted motion vector candidates from the historical predicted motion vector candidate list and adds them to the predicted motion vector candidate list; The spatial merge candidate export section exports spatial merge candidates from blocks adjacent to the processed object block and adds them to the merge candidate list; The time merging candidate export section exports time merging candidates from blocks located at the same position as or near the processed object block and adds them to the merge candidate list; as well as The historical merge candidate derivation section derives historical merge candidates from the historical predicted motion vector candidate list and adds them to the merge candidate list. The historical predicted motion vector candidate derivation section and the historical merged candidate derivation section reference candidates included in the historical predicted motion vector candidate list in different orders, as either historical predicted motion vector candidates or historical merged candidates. The spatial merging candidate is the inter-frame prediction information of already processed blocks adjacent to the block being processed. The temporal merging candidate is inter-frame prediction information derived from the inter-frame prediction information of blocks that are located at the same position as or near the processing target block in the blocks belonging to the processed image.

2. An image coding method, wherein the moving image is encoded using inter-frame prediction based on inter-frame prediction information, taking the blocks of each image segmented as units, the image coding method is characterized by comprising: The encoding information saving step adds the inter-frame prediction information used in inter-frame prediction to the historical prediction motion vector candidate list; The spatial prediction motion vector candidate export step involves exporting spatial prediction motion vector candidates from blocks adjacent to the processed object block and adding them to the prediction motion vector candidate list. The time-predicted motion vector candidate export step involves exporting time-predicted motion vector candidates from blocks located at the same position as or near the processed object block, and adding them to the predicted motion vector candidate list. The historical predicted motion vector candidate export step involves exporting historical predicted motion vector candidates from the historical predicted motion vector candidate list and adding them to the predicted motion vector candidate list. The spatial merge candidate export step involves exporting spatial merge candidates from blocks adjacent to the processed object block and adding them to the merge candidate list. The time merging candidate export step involves exporting time merging candidates from blocks located at the same position as or near the processed object block, and adding them to the merge candidate list. as well as The historical merge candidate export step involves exporting historical merge candidates from the historical predicted motion vector candidate list and adding them to the merge candidate list. In the historical predicted motion vector candidate derivation step and the historical merged candidate derivation step, candidates included in the historical predicted motion vector candidate list are referenced in different orders as historical predicted motion vector candidates or historical merged candidates. The spatial merging candidate is the inter-frame prediction information of already processed blocks adjacent to the block being processed. The temporal merging candidate is inter-frame prediction information derived from the inter-frame prediction information of blocks that are located at the same position as or near the processing target block in the blocks belonging to the processed image.

3. An image decoding apparatus, comprising, using inter-frame prediction based on inter-frame prediction information to decode a bit string encoding the moving image, wherein the image decoding apparatus comprises: The encoding information storage unit adds the inter-frame prediction information used in inter-frame prediction to the historical prediction motion vector candidate list; The spatial prediction motion vector candidate derivation section derives spatial prediction motion vector candidates from blocks adjacent to the processed object block and adds them to the prediction motion vector candidate list. The time-predicted motion vector candidate derivation unit derives time-predicted motion vector candidates from blocks located at the same position as or near the processed object block, and adds them to the predicted motion vector candidate list; The historical predicted motion vector candidate derivation unit derives historical predicted motion vector candidates from the historical predicted motion vector candidate list and adds them to the predicted motion vector candidate list; The spatial merge candidate export section exports spatial merge candidates from blocks adjacent to the processed object block and adds them to the merge candidate list; The time merging candidate export section exports time merging candidates from blocks located at the same position as or near the processed object block and adds them to the merge candidate list; as well as The historical merging candidate derivation unit derives historical merging candidates from the historical predicted motion vector candidate list and adds them to the merge candidate list. The historical predicted motion vector candidate derivation unit and the historical merging candidate derivation unit reference candidates included in the historical predicted motion vector candidate list in different orders as either historical predicted motion vector candidates or historical merging candidates. The spatial merging candidate is the inter-frame prediction information of already processed blocks adjacent to the block being processed. The temporal merging candidate is inter-frame prediction information derived from the inter-frame prediction information of blocks that are located at the same position as or near the processing target block in the blocks belonging to the processed image.

4. An image decoding method, wherein the bit string encoding the moving image is decoded using inter-frame prediction based on inter-frame prediction information, taking the blocks of each image segmented from the moving image as units, the image decoding method being characterized in that it includes: The encoding information saving step adds the inter-frame prediction information used in inter-frame prediction to the historical prediction motion vector candidate list; The spatial prediction motion vector candidate export step involves exporting spatial prediction motion vector candidates from blocks adjacent to the processed object block and adding them to the prediction motion vector candidate list. The time-predicted motion vector candidate export step involves exporting time-predicted motion vector candidates from blocks located at the same position as or near the processed object block, and adding them to the predicted motion vector candidate list. The historical predicted motion vector candidate export step involves exporting historical predicted motion vector candidates from the historical predicted motion vector candidate list and adding them to the predicted motion vector candidate list. The spatial merge candidate export step involves exporting spatial merge candidates from blocks adjacent to the processed object block and adding them to the merge candidate list. The time merging candidate export step involves exporting time merging candidates from blocks located at the same position as or near the processed object block, and adding them to the merge candidate list. as well as The historical merge candidate export step involves exporting historical merge candidates from the historical predicted motion vector candidate list and adding them to the merge candidate list. In the historical predicted motion vector candidate derivation step and the historical merged candidate derivation step, candidates included in the historical predicted motion vector candidate list are referenced in different orders as historical predicted motion vector candidates or historical merged candidates. The spatial merging candidate is the inter-frame prediction information of already processed blocks adjacent to the block being processed. The temporal merging candidate is inter-frame prediction information derived from the inter-frame prediction information of blocks that are located at the same position as or near the processing target block in the blocks belonging to the processed image.

5. A bitstream generated by the image encoding method according to claim 2.

6. A storage method for storing a bitstream generated by the image encoding method according to claim 2 into a storage medium.

7. A transmission method for transmitting a bitstream generated by the image encoding method according to claim 2.

Images