Image encoding method and image decoding method

Abstract

This provides an image encoding method. [Solution] The method comprises: a first step of selecting at least one motion reference block from encoded pixel blocks having motion information; a second step of selecting at least one available block from the motion reference block that has candidate motion information to be applied to the block to be encoded, and which has different motion information from each other; a third step of selecting one selected block from the available blocks; a fourth step of generating a predicted image of the block to be encoded using the motion information of the selected block; a fifth step of encoding the prediction error between the predicted image and the original image; and a sixth step of encoding selection information that identifies the selected block by referring to a predetermined code table according to the number of available blocks.

Description

Technical Field

[0001] The present invention relates to methods for encoding and decoding moving images and still images.

Background Art

[0002] In recent years, a moving image encoding method with significantly improved encoding efficiency has been recommended jointly by ITU-T and ISO / IEC as ITU-T Rec. H.264 and ISO / IEC 14496-10 (hereinafter referred to as H.264). In H.264, prediction processing, transform processing, and entropy encoding processing are performed in units of rectangular blocks (for example, 16×16 pixel block units, 8×8 pixel block units, etc.). In the prediction processing, for a rectangular block to be encoded (encoding target block), motion compensation for performing temporal prediction is performed by referring to an already encoded frame (reference frame). In such motion compensation, it is necessary to encode motion information including a motion vector as spatial shift information between the encoding target block and the block referred to in the reference frame and send it to the decoding side. Further, when performing motion compensation using a plurality of reference frames, it is necessary to encode the reference frame number together with the motion information. Therefore, the amount of code related to the motion information and the reference frame number may increase.

[0003] As an example of a method for obtaining a motion vector in motion compensation prediction, there is a direct mode in which a motion vector to be assigned to an encoding target block is derived from the motion vectors already assigned to the encoded blocks, and a prediction image is generated based on the derived motion vector (see Patent Document 1 and Patent Document 2). In the direct mode, since the motion vector is not encoded, the amount of code for the motion information can be reduced. The direct mode is adopted, for example, in H.264 / AVC.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

[0005] In direct mode, the motion vector of the block to be encoded is predicted and generated using a fixed method: calculating the motion vector from the median value of the motion vector of an already encoded block adjacent to the block to be encoded. Therefore, the degree of freedom in calculating the motion vector is low.

[0006] To increase the degree of freedom in calculating motion vectors, a method has been proposed in which one of several encoded blocks is selected and assigned to the block to be encoded. In this method, selection information identifying the selected encoded block must always be transmitted so that the decoding side can identify the selected encoded block. Therefore, when determining which motion vector to assign to the block to be encoded by selecting one of several encoded blocks, there is a problem in that the amount of coding related to the selection information increases.

[0007] This invention was made to solve the above-mentioned problems and aims to provide an image coding and image decoding method with high coding efficiency. [Means for solving the problem]

[0008] An image encoding method according to one embodiment of the present invention comprises: a first step of selecting at least one motion reference block from encoded pixel blocks having motion information; a second step of selecting at least one available block from the motion reference block that has candidate motion information to be applied to the block to be encoded, and which has different motion information from each other; a third step of selecting one selected block from the available blocks; a fourth step of generating a predicted image of the block to be encoded using the motion information of the selected block; a fifth step of encoding the prediction error between the predicted image and the original image; and a sixth step of encoding selection information that identifies the selected block by referring to a code table predetermined according to the number of available blocks.

[0009] An image decoding method according to another embodiment of the present invention comprises: a first step of selecting at least one motion reference block from a decoded pixel block having motion information; a second step of selecting at least one available block from the motion reference block that has candidate motion information to be applied to the block to be decoded and has different motion information from each other; a third step of obtaining selection information for identifying the selected block by decoding the input encoded data with reference to a code table predetermined according to the number of available blocks; a fourth step of selecting one selected block from the available blocks according to the selection information; a fifth step of generating a predicted image of the block to be decoded using the motion information of the selected block; a sixth step of decoding the predicted residual of the block to be decoded from the encoded data; and a seventh step of obtaining a decoded image from the predicted image and the predicted residual. [Effects of the Invention]

[0010] According to the present invention, coding efficiency can be improved. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram schematically showing the configuration of an image encoding device according to the first embodiment. [Figure 2A] This figure shows an example of the size of a macroblock, which is the encoding processing unit of the image decoding unit shown in Figure 1. [Figure 2B] This figure shows another example of the size of macroblocks, which are the encoding processing units of the image decoding unit shown in Figure 1. [Figure 3] Figure 1 shows the sequence in which the image encoding unit encodes pixel blocks within the frame to be encoded. [Figure 4] This figure shows an example of a motion information frame held by the motion information memory shown in Figure 1. [Figure 5] This flowchart shows an example of a procedure for processing the input image signal shown in Figure 1. [Figure 6A] This figure shows an example of the inter-prediction processing performed by the motion compensation unit in Figure 1. [Figure 6B] This figure shows another example of the interpretation process performed by the motion compensation unit in Figure 1. [Figure 7A] This figure shows an example of the size of the motion compensation block used in interpretation processing. [Figure 7B] This figure shows another example of the size of motion compensation blocks used in interpretation processing. [Figure 7C] This figure shows yet another example of the size of motion compensation blocks used in interpretation processing. [Figure 7D] This figure shows another example of the size of motion compensation blocks used in interpretation processing. [Figure 8A] This figure shows an example of the arrangement of spatial and temporal motion reference blocks. [Figure 8B] This figure shows another example of the placement of spatial direction motion reference blocks. [Figure 8C] Figure 8B shows the relative position of the spatial motion reference block with respect to the encoding target block. [Figure 8D] This figure shows another example of the placement of time-direction motion reference blocks. [Figure 8E] It is a diagram showing yet another example of the arrangement of the time-direction movement reference blocks. [Figure 8F] It is a diagram showing yet another example of the arrangement of the time-direction movement reference blocks. [Figure 9] It is a flowchart showing an example of a method in which the available block acquisition unit in FIG. 1 selects an available block from the movement reference blocks. [Figure 10] It is a diagram showing an example of an available block selected according to the method in FIG. 9 from the movement reference blocks shown in FIG. 8. [Figure 11] It is a diagram showing an example of the available block information output by the available block acquisition unit in FIG. 1. [Figure 12A] It is a diagram showing an example of the identity determination of the movement information between blocks by the available block acquisition unit in FIG. 1. [Figure 12B] It is a diagram showing another example of the identity determination of the movement information between blocks by the available block acquisition unit in FIG. 1. [Figure 12C] It is a diagram showing yet another example of the identity determination of the movement information between blocks by the available block acquisition unit in FIG. 1. [Figure 12D] It is a diagram showing another example of the identity determination of the movement information between blocks by the available block acquisition unit in FIG. 1. [Figure 12E] It is a diagram showing yet another example of the identity determination of the movement information between blocks by the available block acquisition unit in FIG. 1. [Figure 12F] It is a diagram showing another example of the identity determination of the movement information between blocks by the available block acquisition unit in FIG. 1. [Figure 13] It is a block diagram schematically showing the configuration of the prediction unit in FIG. 1. [Figure 14] It is a diagram showing a group of movement information output by the time-direction movement information acquisition unit in FIG. 13. [Figure 15] It is an explanatory diagram explaining the interpolation process with a few pixel accuracies available in the motion compensation process by the motion compensation unit in FIG. 13. [Figure 16]Figure 13 is a flowchart illustrating an example of the operation of the prediction unit. [Figure 17] Figure 13 shows how the motion compensation unit copies the motion information of the time-direction motion reference block to the block to be encoded. [Figure 18] This is a block diagram schematically showing the configuration of the variable-length coding unit in Figure 1. [Figure 19] This figure shows an example of generating syntax based on available block information. [Figure 20] This figure shows an example of binarizing the selected block information syntax corresponding to the available block information. [Figure 21] This is an explanatory diagram illustrating the scaling of motion information. [Figure 22] This figure shows the syntax structure according to the embodiment. [Figure 23A] This figure shows an example of a macroblock layer thinkus according to the first embodiment. [Figure 23B] This figure shows another example of a macroblock layer thinkus according to the first embodiment. [Figure 24A] This figure shows the mb_type and the corresponding code table for B-slicing in H.264. [Figure 24B] This figure shows an example of a code table according to the embodiment. [Figure 24C] This figure shows the mb_type and the corresponding code table for P-slicing in H.264. [Figure 24D] This figure shows another example of a code table according to the embodiment. [Figure 25A] This figure shows an example of mb_type and a corresponding code table in a B slice according to the embodiment. [Figure 25B] This figure shows another example of mb_type and the corresponding code table in a P slice according to the embodiment. [Figure 26] This is a block diagram schematically showing the configuration of an image encoding device according to the second embodiment. [Figure 27]This is a block diagram schematically showing the configuration of the prediction unit in Figure 26. [Figure 28] This is a block diagram schematically showing the configuration of the second prediction unit in Figure 27. [Figure 29] Figure 26 is a block diagram schematically showing the configuration of the variable-length coding unit. [Figure 30A] This figure shows an example of macroblock layer syntax according to the second embodiment. [Figure 30B] This figure shows another example of macroblock layer syntax according to the second embodiment. [Figure 31] This is a schematic block diagram showing an image decoding device according to the third embodiment. [Figure 32] This is a block diagram showing the coded sequence decoding unit shown in Figure 31 in more detail. [Figure 33] This is a block diagram showing the prediction unit in more detail, as shown in Figure 31. [Figure 34] This is a schematic block diagram showing an image decoding device according to the fourth embodiment. [Figure 35] This is a block diagram showing the coded sequence decoding unit shown in Figure 33 in more detail. [Figure 36] This is a block diagram showing the prediction unit in more detail, as shown in Figure 33. [Modes for carrying out the invention]

[0012] Hereinafter, an image coding and image decoding method and apparatus according to the present invention will be described with reference to the drawings as necessary. In the following embodiments, parts with the same number will be treated as performing the same operation, and therefore repeated explanations will be omitted.

[0013] (First embodiment) Figure 1 schematically shows the configuration of an image encoding device according to a first embodiment of the present invention. As shown in Figure 1, this image encoding device comprises an image encoding unit 100, an encoding control unit 150, and an output buffer 120. This image encoding device may be implemented using hardware such as an LSI chip, or it may be implemented by having a computer execute an image encoding program.

[0014] The image encoding unit 100 receives the original image (input image signal) 10, which is either a moving or still image, for example, in pixel block units obtained by dividing the original image. The image encoding unit 100 compresses and encodes the input image signal 10, as will be described in detail later, to generate encoded data 14. The generated encoded data 14 is temporarily stored in the output buffer 120 and sent to a storage system (storage medium) or transmission system (communication line) (not shown) at an output timing managed by the encoding control unit 150.

[0015] The coding control unit 150 controls the entire coding process of the image coding unit 100, including feedback control of the generated code amount, quantization control, prediction mode control, and entropy coding control. Specifically, the coding control unit 150 provides coding control information 50 to the image coding unit 100 and receives feedback information 51 from the image coding unit 100 as appropriate. The coding control information 50 includes prediction information, motion information 18, and quantization parameter information. The prediction information includes prediction mode information and block size information. The motion information 18 includes motion vectors, reference frame numbers, and prediction directions (unidirectional prediction, bidirectional prediction). The quantization parameter information includes quantization parameters such as quantization width (quantization step size) and a quantization matrix. The feedback information 51 includes the code amount generated by the image coding unit 100 and is used, for example, to determine the quantization parameters.

[0016] The image encoding unit 100 encodes the input image signal 10 using pixel blocks (e.g., macroblocks, subblocks, single pixels, etc.) obtained by dividing the original image as units. Therefore, the input image signal 10 is sequentially input to the image encoding unit 100 in units of pixel blocks obtained by dividing the original image. In this embodiment, the encoding processing unit is defined as a macroblock, and the pixel block (macroblock) to be encoded, corresponding to the input image signal 10, is simply referred to as the encoding target block. Furthermore, the image frame containing the encoding target block, i.e., the image frame to be encoded, is referred to as the encoding target frame.

[0017] Such encoding target blocks may be, for example, 16x16 pixel blocks as shown in Figure 2A, or 64x64 pixel blocks as shown in Figure 2B. Furthermore, encoding target blocks may also be 32x32 pixel blocks, 8x8 pixel blocks, etc. The shape of the macroblock is not limited to the square shape shown in Figures 2A and 2B; it may be set to any shape, such as a rectangle. Moreover, the processing unit described above is not limited to pixel blocks like macroblocks; it may also be a frame or a field.

[0018] Note that the encoding process for each pixel block within the frame to be encoded can be performed in any order. In this embodiment, for the sake of simplicity, as shown in Figure 3, the encoding process is performed on each pixel block row by row, starting from the top-left pixel block to the bottom-right pixel block of the frame to be encoded, that is, in the order of the raster scan.

[0019] The image coding unit 100 shown in Figure 1 includes a prediction unit 101, a subtractor 102, a conversion / quantization unit 103, a variable-length coding unit 104, an inverse quantization / inverse conversion unit 105, an adder 106, a frame memory 107, a motion information memory 108, and an available block acquisition unit 109.

[0020] In the image encoding unit 100, the input image signal 10 is input to the prediction unit 101 and the subtractor 102. The subtractor 102 receives the input image signal 10 and also receives the predicted image signal 11 from the prediction unit 101, which will be described later. The subtractor 102 calculates the difference between the input image signal 10 and the predicted image signal 11 and generates a predicted error image signal 12.

[0021] The transformation / quantization unit 103 receives the prediction error image signal 12 from the subtractor 102 and performs a transformation process on the received prediction error image signal 12 to generate transformation coefficients. The transformation process is, for example, an orthogonal transformation such as the discrete cosine transform (DCT). In other embodiments, the transformation / quantization unit 103 may generate transformation coefficients using methods such as wavelet transform and independent component analysis instead of the discrete cosine transform. Furthermore, the transformation / quantization unit 103 quantizes the generated transformation coefficients based on the quantization parameters given by the coding control unit 150. The quantized transformation coefficients (transformation coefficient information) 13 are output to the variable-length coding unit 104 and the inverse quantization / inverse transformation unit 105.

[0022] The inverse quantization / inverse transform unit 105 inversely quantizes the quantized transformation coefficients 13 according to the quantization parameters given by the coding control unit 150, i.e., the same quantization parameters as the transformation / quantization unit 103. Subsequently, the inverse quantization / inverse transform unit 105 performs an inverse transform on the inversely quantized transformation coefficients to generate the decoding prediction error signal 15. The inverse transform process by the inverse quantization / inverse transform unit 105 is equivalent to the inverse transform process of the transformation process by the transformation / quantization unit 103. For example, the inverse transform process may be an inverse discrete cosine transform (IDCT) or an inverse wavelet transform.

[0023] The adder 106 receives a decoded prediction error signal 15 from the inverse quantization / inverse transform unit 105, and also receives a predicted image signal 11 from the prediction unit 101. The adder 106 adds the decoded prediction error signal 15 and the predicted image signal 11 to generate a local decoded image signal 16. The generated local decoded image signal 16 is stored in the frame memory 107 as a reference image signal 17. The reference image signal 17 stored in the frame memory 107 is read out and referenced by the prediction unit 101 when encoding subsequent blocks to be encoded.

[0024] The prediction unit 101 receives a reference image signal 17 from the frame memory 107 and available block information 30 from the available block acquisition unit 109, which will be described later. Furthermore, the prediction unit 101 receives reference motion information 19 from the motion information memory 108, which will be described later. Based on the reference image signal 17, reference motion information 19, and available block information 30, the prediction unit 101 generates a predicted image signal 11, motion information 18, and selected block information 31 for the block to be encoded. Specifically, the prediction unit 101 includes a motion information selection unit 118 that generates motion information 18 and selected block information 31 based on the available block information 30 and reference motion information 19, and a motion compensation unit 113 that generates a predicted image signal 11 based on the motion information 18. The predicted image signal 11 is sent to the subtractor 102 and the adder 106. The motion information 18 is stored in the motion information memory 108 for subsequent prediction processing of the block to be encoded. The selected block information 31 is sent to the variable-length encoding unit 104. The prediction unit 101 will be explained in detail later.

[0025] Motion information memory 108 temporarily stores motion information 18 as reference motion information 19. Figure 4 shows an example of the configuration of motion information memory 108. As shown in Figure 4, reference motion information 19 is held in frame units in motion information memory 108, and the reference motion information 19 forms motion information frames 25. Motion information 18 related to encoded blocks is sequentially input into motion information memory 108, and as a result, motion information memory 108 holds multiple motion information frames 25 with different encoding times.

[0026] The reference motion information 19 is held within the motion information frame 25 in predetermined block units (for example, 4x4 pixel blocks). The motion vector block 28 shown in Figure 4 represents a pixel block of the same size as the block to be encoded, the available block, and the selected block, for example, a 16x16 pixel block. A motion vector is assigned to each 4x4 pixel block in the motion vector block 28. Interpretation processing using motion vector blocks is called motion vector block prediction processing. The reference motion information 19 held in the motion information memory 108 is read by the prediction unit 101 when generating motion information 18. The motion information 18 held by the available block, as described later, refers to the reference motion information 19 held in the area where the available block is located in the motion information memory 108.

[0027] Furthermore, the motion information memory 108 is not limited to the example of storing the reference motion information 19 in 4x4 pixel block units; it may also store the reference motion information 19 in other pixel block units. For example, the pixel block unit for the reference motion information 19 may be 1 pixel or a 2x2 pixel block. Also, the shape of the pixel block for the reference motion information 19 is not limited to a square shape; it can be any shape.

[0028] The available block acquisition unit 109 in Figure 1 acquires reference motion information 19 from the motion information memory 108 and, based on the acquired reference motion information 19, selects available blocks that can be used for the prediction processing of the prediction unit 101 from among multiple blocks that have already been encoded. The selected available blocks are sent to the prediction unit 101 and the variable-length encoding unit 104 as available block information 30. Encoded blocks that are candidates for selecting available blocks are called motion reference blocks. The method for selecting motion reference blocks and available blocks will be explained in detail later.

[0029] The variable-length coding unit 104 receives, in addition to the conversion coefficient information 13, selected block information 31 from the prediction unit 101, coding parameters such as prediction information and quantization parameters from the coding control unit 150, and available block information 30 from the available block acquisition unit 109. The variable-length coding unit 104 entropy codes (e.g., isochromatic coding, Huffman coding, or arithmetic coding) the quantized conversion coefficients 13, selected block information 31, available block information 30, and coding parameters to generate coded data 14. The coded parameters, along with the selected block information 31 and prediction information, include all parameters necessary for decoding, such as information about the conversion coefficients and information about quantization. The generated coded data 14 is temporarily stored in the output buffer 120 and sent to a storage system or transmission system (not shown).

[0030] Figure 5 shows the processing procedure for the input image signal 10. As shown in Figure 5, first, a predicted image signal 11 is generated by the prediction unit 101 (step S501). In the generation of the predicted image signal 11 in step S501, one of the available blocks, which will be described later, is selected as a selected block, and the predicted image signal 11 is created using the selected block information 31, the motion information of the selected block, and the reference image signal 17. The difference between the predicted image signal 11 and the input image signal 10 is calculated by the subtractor 102, and a predicted error image signal 12 is generated (step S502).

[0031] Next, the prediction error image signal 12 is subjected to orthogonal transformation and quantization by the transformation / quantization unit 103 to generate transformation coefficient information 13 (step S503). The transformation coefficient information 13 and the selected block information 31 are sent to the variable-length coding unit 104, where variable-length coding is performed to generate coded data 14 (step S504). In step S504, the code table is switched to have a number of entries equal to the number of available blocks in the code table according to the selected block information 31, and the selected block information 31 is coded in variable length. The bitstream 20 of the coded data is sent to a storage system or transmission line (not shown).

[0032] The conversion coefficient information 13 generated in step S503 is inversely quantized and inversely transformed by the inverse quantization / inverse transformation unit 105 to obtain the decoded prediction error signal 15 (step S505). The decoded prediction error signal 15 is added to the reference image signal 17 used in step S501 to obtain the local decoded image signal 16 (step S506), which is stored in the frame memory 107 as the reference image signal (step S507).

[0033] Next, we will describe in more detail each component of the image encoding unit 100 described above. The image encoding unit 100 in Figure 1 is equipped with multiple prediction modes, each of which has a different method for generating the predicted image signal 11 and a different motion compensation block size. Specifically, the prediction unit 101 can generate the predicted image signal 11 in two main ways: intra-prediction (in-frame prediction), which generates a predicted image using a reference image signal 17 related to the frame (or field) to be encoded, and inter-prediction (inter-frame prediction), which generates a predicted image using a reference image signal 17 related to one or more encoded reference frames (reference fields). The prediction unit 101 selectively switches between intra-prediction and inter-prediction to generate the predicted image signal 11 for the block to be encoded.

[0034] Figure 6A shows an example of interpretation by the motion compensation unit 113. In interpretation, as shown in Figure 6A, a predicted image signal 11 is generated using a reference image signal 17 relating to a block 24 located at a spatially shifted position according to the motion vector 18a contained in the motion information 18, from a block 23 in the reference frame one frame prior, which has already been encoded and is at the same position as the block to be encoded (also called a predicted block), according to the motion vector 18a contained in the motion information 18. That is, in the generation of the predicted image signal 11, the reference image signal 17 relating to a block 24 in the reference frame, which is identified by the position (coordinates) of the block to be encoded and the motion vector 18a contained in the motion information 18, is used. Interpretation enables motion compensation with a small number of pixel precision (e.g., 1 / 2 pixel precision or 1 / 4 pixel precision), and the interpolated pixel values ​​are generated by performing a filtering process on the reference image signal 17. For example, in H.264, interpolation up to 1 / 4 pixel precision is possible for the luminance signal. When performing motion compensation with 1 / 4 pixel precision, the amount of information in the motion information 18 becomes four times that of integer pixel precision.

[0035] In interpretation, the method is not limited to using the reference frame from one frame prior, as shown in Figure 6A; any encoded reference frame may be used, as shown in Figure 6B. When reference image signals 17 for multiple reference frames at different time positions are stored, the information indicating which time position's reference image signal 17 was used to generate the predicted image signal 11 is represented by the reference frame number. The reference frame number is included in the motion information 18. The reference frame number can be changed on a region-by-region basis (picture, block, etc.). That is, a different reference frame can be used for each pixel block. For example, if an encoded reference frame from one frame prior is used for prediction, the reference frame number for this region is set to 0, and if an encoded reference frame from two frames prior is used for prediction, the reference frame number for this region is set to 1. As another example, if only one frame's worth of reference image signal 17 is stored in the frame memory 107 (the number of reference frames is 1), the reference frame number is always set to 0.

[0036] Furthermore, in interpretation, a block size suitable for the block to be encoded can be selected from among multiple motion compensation blocks. That is, the block to be encoded may be divided into multiple small pixel blocks, and motion compensation may be performed for each small pixel block. Figures 7A to 7C show the size of motion compensation blocks at the macroblock level, and Figure 7D shows the size of motion compensation blocks at the subblock level (pixel blocks of 8x8 pixels or less). As shown in Figure 7A, if the block to be encoded is 64x64 pixels, a 64x64 pixel block, a 32x64 pixel block, or a 32x32 pixel block can be selected as the motion compensation block. Also, as shown in Figure 7B, if the block to be encoded is 32x32 pixels, a 32x32 pixel block, a 32x16 pixel block, a 16x32 pixel block, or a 16x16 pixel block can be selected as the motion compensation block. Furthermore, as shown in Figure 7C, when the block to be encoded is 16x16 pixels, the motion compensation block can be set to a 16x16 pixel block, a 16x8 pixel block, an 8x16 pixel block, or an 8x8 pixel block, etc. Also, as shown in Figure 7D, when the block to be encoded is 8x8 pixels, the motion compensation block can be selected to be an 8x8 pixel block, an 8x4 pixel block, a 4x8 pixel block, or a 4x4 pixel block, etc.

[0037] As mentioned above, since the small pixel blocks (e.g., 4x4 pixel blocks) in the reference frame used for interpretation have motion information 18, the optimal shape of the motion compensation block and the motion vector can be used according to the local properties of the input image signal 10. In addition, the macroblocks and sub-macroblocks in Figures 7A to 7D can be combined arbitrarily. If the block to be encoded is a 64x64 pixel block as shown in Figure 7A, then by selecting the respective block sizes shown in Figure 7B for each of the four 32x32 pixel blocks obtained by dividing the 64x64 pixel block, blocks from 64x64 to 16x16 pixels can be used hierarchically. Similarly, if the block size shown in Figure 7D can be selected, block sizes from 64x64 to 4x4 can be used hierarchically.

[0038] Next, we will explain the motion reference block with reference to Figures 8A to 8F. Motion reference blocks are selected from the encoded regions (blocks) within the target frame and the reference frame according to a method determined by both the image encoding device in Figure 1 and the image decoding device described later. Figure 8A shows an example of the arrangement of motion reference blocks selected according to the position of the target block. In the example in Figure 8A, nine motion reference blocks A-D and TA-TE are selected from the encoded regions within the target frame and the reference frame. Specifically, from the target frame, four blocks A, B, C, and D adjacent to the left, above, upper right, and upper left of the target block are selected as motion reference blocks. From the reference frame, block TA, which is in the same position as the target block, and four pixel blocks TB, TC, TD, and TE adjacent to the right, below, left, and above block TA are selected as motion reference blocks. In this embodiment, motion reference blocks selected from the target frame are referred to as spatial motion reference blocks, and motion reference blocks selected from the reference frame are referred to as temporal motion reference blocks. The symbol p assigned to each motion reference block in Figure 8A indicates the index of the motion reference block. This index is numbered in the order of time-direction and spatial-direction motion reference blocks, but it is not limited to this order; as long as the indexes do not overlap, the order does not necessarily have to be this one. For example, time-direction and spatial-direction motion reference blocks can be numbered in any order.

[0039] Note that the spatial motion reference block is not limited to the example shown in Figure 8A, but may also be the block to which pixels a, b, c, and d adjacent to the block to be encoded belong (for example, a macroblock or submacroblock), as shown in Figure 8B. In this case, the relative positions (dx, dy) from the top-left pixel e in the block to be encoded to each pixel a, b, c, and d are set as shown in Figure 8C. Here, in the examples shown in Figures 8A and 8B, the macroblock is shown as an N×N pixel block.

[0040] Furthermore, as shown in Figure 8D, all blocks A1-A4, B1, B2, C, and D adjacent to the block to be encoded may also be selected as spatial motion reference blocks. In the example in Figure 8D, the number of spatial motion reference blocks is 8.

[0041] Furthermore, as shown in Figure 8E, the time-direction motion reference blocks may overlap in some parts, TA to TE, while as shown in Figure 8F, the blocks TA to TE may be spaced apart. In Figure 8E, the overlapping parts of time-direction motion reference blocks TA and TB are indicated by diagonal lines. Moreover, time-direction motion reference blocks are not necessarily limited to the block at the position corresponding to the block to be encoded (Collocate position) and the blocks located around it; they may be any block located at any position within the reference frame. For example, a block within the reference frame identified by the position of the reference block and the motion information 18 of any encoded block adjacent to the block to be encoded may be designated as the central block (e.g., block TA), and this central block and the blocks around it may be selected as the time-direction motion reference blocks. Furthermore, time-direction reference blocks do not need to be spaced equally apart from the central block.

[0042] In any of the above cases, the number and position of the spatial and temporal motion reference blocks can be set in any way, as long as the number and position of the spatial and temporal motion reference blocks are predetermined in the encoding and decoding devices. Furthermore, the size of the motion reference blocks does not necessarily have to be the same as the size of the blocks to be encoded. For example, as shown in Figure 8D, the size of the motion reference blocks may be larger or smaller than the blocks to be encoded. Moreover, the motion reference blocks are not limited to a square shape; they may be set to any shape, such as a rectangle. In addition, the motion reference blocks may be set to any size.

[0043] Furthermore, motion reference blocks and available blocks may be placed in either the temporal or spatial direction only. Also, motion reference blocks and available blocks may be placed in either the temporal direction or the spatial direction according to the type of slice, such as P-slice or B-slice.

[0044] Figure 9 shows how the available block acquisition unit 109 selects available blocks from motion reference blocks. Available blocks are blocks to which motion information can be applied to the blocks to be encoded, and each block has different motion information. The available block acquisition unit 109 refers to the reference motion information 19 and determines whether each motion reference block is an available block according to the method shown in Figure 9, and outputs the available block information 30.

[0045] As shown in Figure 9, first, motion reference blocks with index p of zero are selected (S800). In the explanation of Figure 9, it is assumed that motion reference blocks are processed sequentially from index p 0 to M-1 (where M is the number of motion reference blocks). Furthermore, the explanation assumes that the availability determination process for motion reference blocks with index p from 0 to p-1 has been completed, and the index of the motion reference block that is the target of the availability determination is p.

[0046] The available block acquisition unit 109 determines whether the motion reference block p has motion information 18, that is, whether at least one motion vector is assigned to it (S801). If the motion reference block p does not have a motion vector, that is, if the time-direction motion reference block p is a block in an I slice that does not have motion information, or if all the small pixel blocks in the time-direction motion reference block p are intra-predictively coded, the process proceeds to step S805. In step S805, the motion reference block p is determined to be an unavailable block.

[0047] If the motion reference block p has motion information in step S801, the process proceeds to step S802. The available block acquisition unit 109 selects the motion reference block q (available block q) that has already been selected as an available block. Here, q is a value smaller than p. Subsequently, the available block acquisition unit 109 compares the motion information 18 of the motion reference block p with the motion information 18 of the available block q and determines whether they have the same motion information (S803). If it is determined that the motion information 18 of the motion reference block p is the same as the motion information 18 of the motion reference block q selected as the available block, the process proceeds to step S805, and the motion reference block p is determined to be an unavailable block.

[0048] For all available blocks q that satisfy q < p, if it is determined in step S803 that the motion information 18 of the motion reference block p is not the same as the motion information 18 of the available block q, the process proceeds to step S804. In step S804, the available block acquisition unit 109 determines that the motion reference block p is an available block.

[0049] When it is determined that the motion reference block p is an available block or an unavailable block, the available block acquisition unit 109 determines whether the availability determination has been executed for all motion reference blocks (S806). If there is a motion reference block for which the availability determination has not been executed, for example, if p < M - 1, the process proceeds to step S807. Subsequently, the available block acquisition unit 109 increments the index p by 1 (step S807) and executes steps S801 to S806 again. When the availability determination is executed for all motion reference blocks in step S806, the availability determination process ends.

[0050] By performing the availability determination process described above, it is determined whether each motion reference block is an available block or an unavailable block. The available block acquisition unit 109 generates available block information 30, which includes information about available blocks. In this way, by selecting available blocks from among the motion reference blocks, the amount of information in the available block information 30 is reduced, and as a result, the amount of encoded data 14 can be reduced.

[0051] Figure 10 shows an example of the results of performing an availability determination process on the motion reference blocks shown in Figure 8A. In Figure 10, two spatial motion reference blocks (p=0,1) and two temporal motion reference blocks (p=5,8) are determined to be available blocks. Figure 11 shows an example of available block information 30 related to the example in Figure 10. As shown in Figure 11, the available block information 30 includes the index of the motion reference block, its availability, and the name of the motion reference block. In the example in Figure 11, the index p=0,1,5,8 are available blocks, and the number of available blocks is 4. The prediction unit 101 selects the optimal available block from these available blocks as the selected block and outputs information about the selected block (selected block information) 31. The selected block information 31 includes the number of available blocks and the index value of the selected available block. For example, if the number of available blocks is 4, the corresponding selected block information 31 is encoded by the variable-length coding unit 104 using a code table with a maximum of 4 entries.

[0052] In step S801 in Figure 9, if at least one of the blocks in the time-direction motion reference block p is an intra-predictive encoded block, the available block acquisition unit 109 may determine that the motion reference block p is an unavailable block. In other words, the process may proceed to step S802 only if all blocks in the time-direction motion reference block p are encoded using inter-prediction.

[0053] Figures 12A to 12E show examples in step S803 where the motion information 18 of motion reference block p and the motion information 18 of available block q are determined to be identical during the comparison of motion information 18. Figures 12A to 12E each show multiple blocks with diagonal lines and two white blocks. For the sake of simplicity, Figures 12A to 12E assume a case where the motion information 18 of these two white blocks is compared without considering the diagonal lines. One of the two white blocks is motion reference block p, and the other is motion reference block q (available block q) which has already been determined to be available. Unless otherwise specified, either of the two white blocks can be motion reference block p.

[0054] Figure 12A shows an example where both the motion reference block p and the available block q are spatial blocks. In the example in Figure 12A, if the motion information 18 of blocks A and B are identical, then it is determined that the motion information 18 are identical. In this case, the sizes of blocks A and B do not need to be the same.

[0055] Figure 12B shows an example where one of the motion reference block p and available block q is a spatial block A, and the other is a temporal block TB. In Figure 12B, there is one block containing motion information within the temporal block TB. If the motion information 18 of the temporal block TB and the motion information 18 of the spatial block A are identical, then it is determined that the motion information 18 are identical. In this case, the sizes of blocks A and TB do not need to be the same.

[0056] Figure 12C shows another example where one of the motion reference block p and available block q is a spatial block A, and the other is a temporal block TB. Figure 12C shows a case where the temporal block TB is divided into multiple subblocks, and there are multiple subblocks that have motion information 18. In the example of Figure 12C, if all blocks that have motion information 18 have the same motion information 18, and that motion information 18 is identical to the motion information 18 of spatial block A, then the motion information 18 is determined to be identical. In this case, the sizes of block A and TB do not need to be the same.

[0057] Figure 12D shows an example where both the motion reference block p and the available block q are time-direction blocks. In this case, if the motion information 18 of blocks TB and TE are identical, it is determined that the motion information 18 are identical.

[0058] Figure 12E shows another example where both the motion reference block p and the available block q are time-direction blocks. Figure 12E shows a case where time-direction blocks TB and TE are each divided into multiple subblocks, each containing multiple subblocks with motion information 18. In this case, the motion information 18 is compared for each subblock within the block, and if the motion information 18 is identical for all subblocks, it is determined that the motion information 18 of block TB and the motion information 18 of block TE are identical.

[0059] Figure 12F shows yet another example where both the motion reference block p and the available block q are time-direction blocks. Figure 12F shows the case where the time-direction block TE is divided into multiple subblocks, and block TE has multiple subblocks that have motion information 18. If all the motion information 18 in block TE is the same motion information 18 and is the same as the motion information 18 in block TD, then it is determined that the motion information 18 in block TD and TE are the same.

[0060] In this way, step S803 determines whether the motion information 18 of the motion reference block p and the motion information 18 of the available block q are identical. In the examples in Figures 12A to 12F, the number of available blocks q to be compared with the motion reference block p was set to 1. However, if the number of available blocks q is 2 or more, the motion information 18 of the motion reference block p may be compared with the motion information 18 of each available block q. Furthermore, when scaling, which will be described later, is applied, the scaled motion information 18 becomes the motion information 18 described above.

[0061] Furthermore, the determination that the motion information of motion reference block p and the motion information of available block q are identical is not limited to cases where each motion vector included in the motion information is perfectly identical. For example, if the norm of the difference between the two motion vectors is within a predetermined range, the motion information of motion reference block p and the motion information of available block q may be considered substantially identical.

[0062] Figure 13 shows a more detailed configuration of the prediction unit 101. As described above, the prediction unit 101 takes available block information 30, reference motion information 19, and reference image signal 17 as inputs and outputs a predicted image signal 11, motion information 18, and selected block information 31. The motion information selection unit 118, as shown in Figure 13, includes a spatial motion information acquisition unit 110, a temporal motion information acquisition unit 111, and a motion information switching switch 112.

[0063] The spatial direction motion information acquisition unit 110 receives available block information 30 and reference motion information 19 related to spatial direction motion reference blocks as input. The spatial direction motion information acquisition unit 110 outputs motion information 18A that includes motion information of each available block located in the spatial direction and the index value of the available block. When the information shown in Figure 11 is input as available block information 30, the spatial direction motion information acquisition unit 110 generates two motion information outputs 18A, each motion information output 18A including the available block and the motion information 19 that this available block possesses.

[0064] The time-direction motion information acquisition unit 111 receives available block information 30 and reference motion information 19 related to time-direction motion reference blocks as input. The time-direction motion information acquisition unit 111 outputs motion information 18B as motion information, which includes motion information 19 possessed by the available time-direction motion reference block identified by the available block information 30 and the index value of the available block. The time-direction motion reference block is divided into multiple small pixel blocks, and each small pixel block has motion information 19. The motion information 18B output by the time-direction motion information acquisition unit 111 includes a group of motion information 19 possessed by each small pixel block within the available block, as shown in Figure 14. When the motion information 18B includes a group of motion information 19, motion compensation prediction can be performed on the encoding target block in units of small pixel blocks obtained by dividing the encoding target block. When the information shown in Figure 11 is input as available block information 30, the time-direction motion information acquisition unit 111 generates two motion information outputs 18B, each motion information output including an available block and a group of motion information 19 possessed by this available block.

[0065] The time-direction motion information acquisition unit 111 may also calculate the average or representative value of the motion vectors included in the motion information 19 of each pixel subblock and output the average or representative value of the motion vectors as motion information 18B.

[0066] The motion information switching switch 112 in Figure 13 selects one appropriate available block as the selected block based on the motion information 18A and 18B output from the spatial motion information acquisition unit 110 and the temporal motion information acquisition unit 111, and outputs the motion information 18 (or a group of motion information 18) corresponding to the selected block to the motion compensation unit 113. The motion information switching switch 112 also outputs selected block information 31 related to the selected block. The selected block information 31 includes the name of the index p or motion reference block, and is also simply called selection information. The selected block information 31 is not limited to the index p and the name of the motion reference block, but can be any information that can identify the location of the selected block.

[0067] The motion information switching switch 112 selects, for example, the available block that minimizes the encoding cost derived by the cost formula shown in Equation 1 below as the selected block.

number

[0068] Here, J represents the coding cost, and D represents the coding distortion, which is the sum of squared errors between the input image signal 10 and the reference image signal 17. R represents the code amount estimated by the provisional coding, and λ represents the Lagrange coefficient determined by the quantization width, etc. Instead of Equation 1, the coding cost J may be calculated using only the code amount R or the coding distortion D, or the cost function of Equation 1 may be created using approximate values ​​of the code amount R or the coding distortion D. Furthermore, the coding distortion D is not limited to the sum of squared errors, but may also be the sum of absolute differences (SAD) of prediction errors. The code amount R may be limited to the code amount related to the motion information 18. Also, the selection of the block is not limited to the available block that minimizes the coding cost, but may also be one available block having a value within a certain range greater than or equal to the smallest coding cost.

[0069] The motion compensation unit 113 derives the position of the pixel block from which the reference image signal 17 is extracted as the predicted image signal 11, based on the motion information (or group of motion information) possessed by the selected block selected by the motion information selection unit 118. When a group of motion information is input to the motion compensation unit 113, the motion compensation unit 113 divides the pixel block from which the reference image signal 17 is extracted as the predicted image signal 11 into small pixel blocks (for example, 4x4 pixel blocks), and applies the corresponding motion information 18 to each of these small pixel blocks to obtain the predicted image signal 11 from the reference image signal 17. The position of the block from which the predicted image signal 11 is obtained is, for example, as shown in Figure 4A, a position shifted in the spatial direction from the small pixel block according to the motion vector 18a included in the motion information 18.

[0070] Motion compensation for the encoded blocks can be the same as that used in H.264. Here, as an example, a 1 / 4 pixel precision interpolation method is specifically described. In 1 / 4 pixel precision interpolation, if each component of the motion vector is a multiple of 4, the motion vector points to an integer pixel position. Otherwise, the motion vector points to a predicted position corresponding to the fractional precision interpolation position.

number

[0071] Here, x and y represent the vertical and horizontal indices indicating the starting position of the block to be predicted (e.g., the top-left vertex), and x_pos and y_pos represent the corresponding predicted positions of the reference image signal 17. (mv_x, mv_y) represents a motion vector with 1 / 4 pixel accuracy. Next, predicted pixels are generated by filling or interpolating the corresponding pixel positions of the reference image signal 17 with the determined pixel positions. Figure 15 shows an example of H.264 predicted pixel generation. In Figure 15, squares indicated by uppercase letters (squares with diagonal lines) indicate pixels at integer positions, and shaded squares indicate interpolated pixels at 1 / 2 pixel positions. Also, squares shown in white indicate interpolated pixels corresponding to 1 / 4 pixel positions. For example, in Figure 15, the interpolation of 1 / 2 pixels corresponding to the positions of letters b and h is calculated by the following formula 3.

number

[0072] Here, the letters shown in Equation 3 and Equation 4 below (e.g., b, h, C1, etc.) represent the pixel values ​​of the pixels assigned the same letters in Figure 16. Also, ">>" indicates a right shift operation, and ">> 5" is equivalent to division by 32. That is, the interpolated pixel at the 1 / 2 pixel position is calculated using a 6-tap FIR (Finite Impulse Response) filter (tap coefficients: (1,-5,20,20,-5,1) / 32).

[0073] Furthermore, the interpolation process for the quarter pixels corresponding to the positions of letters a and d in Figure 15 is calculated using the following formula 4.

number

[0074] Thus, the interpolated pixel at the 1 / 4 pixel position is calculated using a 2-tap average value filter (tap coefficient: (1 / 2, 1 / 2)). The interpolation of the 1 / 2 pixel corresponding to the letter j, which lies between the four integer pixel positions, is generated using both vertical 6 taps and horizontal 6 taps. Interpolated pixel values ​​are generated in a similar manner for pixel positions other than those described.

[0075] Furthermore, the interpolation process is not limited to the examples in Equations 3 and 4; other interpolation coefficients may be used to generate the results. Also, the interpolation coefficients may be fixed values ​​provided by the coding control unit 150, or they may be optimized for each frame based on the aforementioned coding cost, and the results may be generated using the optimized interpolation coefficients.

[0076] Furthermore, although this embodiment describes the process for predicting motion vector blocks in units of macroblocks (e.g., 16x16 pixel blocks), the prediction process may be performed not only in units of macroblocks, but also in units of 16x8 pixel blocks, 8x16 pixel blocks, 8x8 pixel blocks, 8x4 pixel blocks, 4x8 pixel blocks, or 4x4 pixel blocks. In this case, the information regarding the motion vector block is derived in units of pixel blocks. In addition, the above prediction process may be performed in units larger than 16x16 pixel blocks, such as 32x32 pixel blocks, 32x16 pixel blocks, or 64x64 pixel blocks.

[0077] When substituting a reference motion vector within a motion vector block as the motion vector for a small pixel block within a block to be encoded, (A) a negative value (inverted vector) of the reference motion vector may be substituted, or (B) a weighted average, median, maximum, or minimum value using the reference motion vector corresponding to the small block and the reference motion vectors adjacent to this reference motion vector may be substituted.

[0078] Figure 16 schematically shows the operation of the prediction unit 101. As shown in Figure 16, first, a reference frame (motion reference frame) containing a time-direction reference motion block is acquired (step S1501). The motion reference frame is typically the reference frame with the smallest temporal distance from the frame to be encoded, and is a reference frame that is temporally past. For example, the motion reference frame is the frame encoded immediately before the frame to be encoded. In other cases, any reference frame in which motion information 18 is stored in the motion information memory 108 may be acquired as the motion reference frame. Next, the spatial motion information acquisition unit 110 and the time-direction motion information acquisition unit 111 each acquire the available block information 30 output from the available block acquisition unit 109 (step S1502). Next, the motion information switching switch 112 selects one of the available blocks as a selected block according to, for example, formula 1 (step S1503). Subsequently, the motion compensation unit 113 copies the motion information of the selected block to the block to be encoded (step S1504). At this time, if the selected block is a spatial reference block, the motion information 18 contained in this selected block is copied to the encoding reference block, as shown in Figure 17. If the selected block is a temporal reference block, the group of motion information 18 contained in this selected block is copied to the encoding target block along with the position information. Next, the motion compensation unit 113 performs motion compensation using the motion information 18 or the group of motion information 18 copied, and outputs the predicted image signal 11 and the motion information 18 used for motion compensation prediction.

[0079] Figure 18 shows a more detailed configuration of the variable-length coding unit 104. As shown in Figure 18, the variable-length coding unit 104 comprises a parameter coding unit 114, a conversion coefficient coding unit 115, a selected block coding unit 116, and a multiplexing unit 117. The parameter coding unit 114 encodes parameters necessary for decoding, such as prediction mode information, block size information, and quantization parameter information, excluding conversion coefficient information 13 and selected block information 31, and generates encoded data 14A. The conversion coefficient coding unit 115 encodes the conversion coefficient information 13 to generate encoded data 14B. The selected block coding unit 116 encodes the selected block information 31 by referring to the available block information 30 to generate encoded data 14C.

[0080] If the available block information 30 includes an index and the availability of the corresponding motion reference block, as shown in Figure 19, then from the pre-configured set of motion reference blocks, unavailable motion reference blocks are excluded, and only the available motion reference blocks are converted into syntax (stds_idx). In Figure 19, five of the nine motion reference blocks are unavailable, so the syntax stds_idx is assigned sequentially from 0 to the remaining four motion reference blocks. In this example, the block information to be encoded is selected from four available blocks, rather than from nine, so the amount of code (number of bins) to be assigned is, on average, less.

[0081] Figure 20 shows an example of a code table representing the syntax stds_idx and its binary information (bin). As shown in Figure 18, the fewer the number of available motion reference blocks, the lower the average number of bins required to encode the syntax stds_idx. For example, if the number of available blocks is 4, the syntax stds_idx can be represented by 3 bits or less. The binary information (bin) of the syntax stds_idx may be binarized so that all stds_idx have the same number of bins for each number of available blocks, or it may be binarized according to a binarization method determined by pre-training. Alternatively, multiple binarization methods may be provided and switched adaptively for each block to be encoded.

[0082] Entropy coding (e.g., isometric coding, Huffman coding, or arithmetic coding) can be applied to these coding units 114, 115, and 116, and the generated coded data 14A, 14B, and 14C are multiplexed by the multiplexing unit 117 and output.

[0083] In this embodiment, we have described an example in which a frame encoded one frame before the frame to be encoded is referenced as a reference frame. However, the motion vector and reference frame number in the reference motion information 19 possessed by the selected block may be used to scale (or normalize) the motion vector, and the reference motion information 19 may be applied to the block to be encoded.

[0084] This scaling process will be explained in detail with reference to Figure 21. In Figure 21, tc represents the time distance (POC (display order number) distance) between the frame to be encoded and the motion reference frame, and is calculated by the following formula 5. In Figure 21, tr[i] represents the time distance between the motion reference frame and frame i referenced by the selected block, and is calculated by the following formula 6.

number

[0085] Here, curPOC represents the Picture Order Count (POC) of the frame to be encoded, colPOC represents the POC of the motion reference frame, and refPOC represents the POC of frame i referenced by the selected block. Clip(min,max,target) is a clipping function that outputs min if target is less than min, max if target is greater than max, and target otherwise. DiffPicOrderCnt(x,y) is a function that calculates the difference between two POCs.

[0086] If we let MVr=(MVr_x,MVr_y) be the motion vector of the selected block, and MV=(MV_x,MV_y) be the motion vector to be applied to the block to be encoded, then the motion vector MV can be calculated using the following formula 7.

number

[0087] Here, Abs(x) represents a function that extracts the absolute value of x. In this way, motion vector scaling converts the motion vector MVr assigned to the selected block into a motion vector MV between the frame to be encoded and the first motion reference frame.

[0088] Further examples of scaling motion vectors are described below. First, for each slice or frame, calculate the scaling factor (DistScaleFactor[i]) for all possible time distances tr that the motion reference frame can take, according to Equation 8 below. The number of scaling factors is equal to the number of frames referenced by the selected block, i.e., the number of reference frames.

number

[0089] The calculation of tx shown in equation 8 can be pre-programmed into a table.

[0090] When scaling each block to be encoded, the motion vector MV can be calculated using only multiplication, addition, and shift operations by using the following formula 9.

number

[0091] When such scaling is performed, the scaled motion information 18 is applied to the processing of the available block acquisition unit 109 together with the prediction unit 101. When scaling is performed, the reference frame referenced by the block to be encoded becomes a motion reference frame.

[0092] Figure 22 shows the syntax structure in the image encoding unit 100. As shown in Figure 22, the syntax mainly consists of three parts: high-level syntax 901, slice-level syntax 904, and macroblock-level syntax 907. High-level syntax 901 holds syntax information for layers above the slice level. Slice-level syntax 904 holds the information required for each slice, and macroblock-level syntax 907 holds the data required for each macroblock shown in Figures 7A to 7D.

[0093] Each part includes more detailed syntax. High-level syntax 901 includes sequence and picture-level syntax such as sequence parameter set syntax 902 and picture parameter set syntax 903. Slice-level syntax 904 includes slice header syntax 905 and slice data syntax 906, etc. Furthermore, macroblock-level syntax 907 includes macroblock layer syntax 908 and macroblock prediction syntax 909, etc.

[0094] Figures 23A and 23B show examples of macroblock layer syntax. The `available_block_num` shown in Figures 23A and 23B indicates the number of available blocks; if this value is greater than 1, encoding of the selected block information is required. Furthermore, `stds_idx` indicates the selected block information, and `stds_idx` is encoded using the code table corresponding to the number of available blocks mentioned above.

[0095] Figure 23A shows the syntax for encoding selected block information after mb_type. stds_idx is encoded when the mode indicated by mb_type is a defined size and mode (TARGET_MODE), and available_block_num is greater than 1. For example, stds_idx is encoded when the motion information of the selected block becomes available for block sizes of 64x64 pixels, 32x32 pixels, 16x16 pixels, or in direct mode.

[0096] Figure 23B shows the syntax when encoding the selected block information before mb_type. If available_block_num is greater than 1, stds_idx is encoded. If available_block_num is 0, conventional motion compensation, such as that used in H.264, is performed, and mb_type is encoded.

[0097] Furthermore, syntax elements not defined in this invention may be inserted between the rows of the tables shown in Figures 23A and 23B, and descriptions related to other conditional branching may also be included. Alternatively, the syntax table may be divided into multiple tables and merged. In addition, it is not necessary to use the same terminology, and it may be changed as appropriate depending on the form in which it is used. Moreover, each syntax element described in the macroblock layer syntax may be modified to be explicitly stated in the macroblock data syntax described later.

[0098] Furthermore, it is possible to reduce the amount of mb_type information by using the information from stds_idx. Figure 24A shows mb_type and the corresponding code table for B-slicing in H.264. In Figure 24A, N is a value representing the size of the block to be encoded, such as 16, 32, or 64, and M is half the value of N. Therefore, if mb_type is between 4 and 21, it indicates that the block to be encoded is a rectangular block. Also, L0, L1, and Bi in Figure 24A represent unidirectional prediction (List0 direction only), unidirectional prediction (List1 direction only), and bidirectional prediction, respectively. If the block to be encoded is a rectangular block, mb_type includes information indicating which of L0, L1, or Bi predictions was performed for each of the two rectangular blocks within the block to be encoded. Also, B_Sub means that the above processing is performed for each of the pixel blocks obtained by dividing the macroblock into four parts. For example, if the block to be encoded is a 64x64 pixel macroblock, the block to be encoded is further assigned an mb_type to each of the four 32x32 pixel blocks obtained by dividing this macroblock into four parts, and then encoded.

[0099] Here, if the selected block indicated by stds_idx is Spatial Left (a pixel block adjacent to the left of the block to be encoded), the motion information of the pixel block adjacent to the left of the block to be encoded is used as the motion information of the block to be encoded. Therefore, stds_idx is equivalent to performing prediction on the block to be encoded using the horizontally elongated rectangular blocks shown in Figure 24A where mb_type=4,6,8,10,12,14,16,18,20. Also, if the selected block indicated by stds_idx is Spatial Up, the motion information of the pixel block adjacent to the top of the block to be encoded is used as the motion information of the block to be encoded. Therefore, stds_idx is equivalent to performing prediction on the vertically elongated rectangular blocks shown in Figure 24A where mb_type=5,7,9,11,13,15,17,19,21. Consequently, by using stds_idx, it is possible to create a code table with the columns mb_type=4~21 in Figure 24A reduced, as shown in Figure 24B. Similarly, with respect to the mb_type and corresponding code table for P-slicing in H.264 shown in Figure 24C, it is possible to create a code table with a reduced number of mb_types, as shown in Figure 24D.

[0100] Alternatively, the information of stds_idx may be included in the mb_type information during encoding. Figure 25A shows a code table when the information of stds_idx is included in the mb_type information, and illustrates an example of mb_type and a code table corresponding to mb_type in a B slice. In Figure 25A, B_STDS_X (X=0,1,2) indicates a mode corresponding to stds_idx, and B_STDS_X is added for each available block (in Figure 25A, the number of available blocks is 3). Similarly, another example of mb_type for a P slice is shown in Figure 25B. The explanation for Figure 25B is the same as for the B slice and will therefore be omitted.

[0101] The order and binarization method (binning) of mb_type are not limited to the examples shown in Figures 25A and 25B, and mb_type may be encoded according to other orders and binarization methods. B_STDS_X and P_STDS_X do not need to be consecutive and may be placed between each mb_type. Furthermore, the binarization method (binning) may be designed based on pre-learned selection frequencies.

[0102] In this embodiment, the present invention is also applicable to extended macroblocks that perform motion compensation prediction by grouping multiple macroblocks together. Furthermore, in this embodiment, the encoding scan order can be any order. For example, the present invention is also applicable to line scans or Z scans.

[0103] As described above, the image encoding device according to this embodiment selects available blocks from a plurality of motion reference blocks, generates information to identify the motion reference blocks to be applied to the block to be encoded according to the number of available blocks selected, and encodes this information. Therefore, the image encoding device according to this embodiment can reduce the amount of code related to motion vector information, while performing motion compensation at a finer pixel block unit than the block to be encoded, thereby achieving high encoding efficiency.

[0104] (Second embodiment) Figure 26 shows an image encoding device according to a second embodiment of the present invention. The second embodiment mainly describes the differences and operation from the first embodiment. As shown in Figure 26, the image encoding unit 200 according to this embodiment has a different configuration of the prediction unit 201 and the variable-length encoding unit 204 from the first embodiment. As shown in Figure 27, the prediction unit 201 comprises a first prediction unit 101 and a second prediction unit 202, and generates a predicted image signal 11 by selectively switching between these first and second prediction units 101 and 202. The first prediction unit 101 has the same configuration as the prediction unit 101 according to the first embodiment (Figure 1), and generates a predicted image signal 11 according to a prediction method (first prediction method) that compensates for motion using motion information 18 possessed by the selected block. The second prediction unit 202 generates a predicted image signal 11 according to a prediction method (second prediction method) such as H.264 that compensates for motion using one motion vector for the block to be encoded. The second prediction unit 202 generates a predicted image signal 11B using the input image signal 10 and the reference image signal 17 from the frame memory.

[0105] Figure 28 schematically shows the configuration of the second prediction unit 202. As shown in Figure 28, the second prediction unit 202 includes a motion information acquisition unit 205 that generates motion information 21 using the input image signal 10 and the reference image signal 17, and a motion compensation unit 113 (Figure 1) that generates a predicted image signal 11A using the reference image signal 17 and the motion information 21. The motion information acquisition unit 205 determines the motion vector to be assigned to the block to be encoded, for example by block matching, based on the input image signal 10 and the reference image signal 17. As an evaluation criterion for matching, the value obtained by accumulating the difference between the input image signal 10 and the interpolated image after matching for each pixel is used.

[0106] The motion information acquisition unit 205 may determine the optimal motion vector using a value obtained by converting the difference between the predicted image signal 11 and the input image signal 10. Alternatively, the optimal motion vector may be determined by considering the magnitude of the motion vector, as well as the code amount of the motion vector and the reference frame number, or by using Equation 1. The matching method may be performed based on search range information provided from outside the image encoding device, or it may be performed hierarchically for each pixel precision. Alternatively, the motion information given by the encoding control unit 150 may be used as the output 21 of the motion information acquisition unit 205 without performing the search process.

[0107] The prediction unit 101 in Figure 27 further includes a prediction method switching switch 203 that selects and outputs either a prediction image signal 11A from the first prediction unit 101 or a prediction image signal 11B from the second prediction unit 202. For example, for each of the prediction image signals 11A and 11B, the prediction method switching switch 203 uses the input image signal 10 to calculate the encoding cost, for example, according to equation 1, and selects one of the prediction image signals 11A or 11B to output as the prediction image signal 11, such that the encoding cost is smaller. Furthermore, the prediction method switching switch 203 also outputs prediction switching information 32 indicating whether the outputted prediction image signal 11 was output from either the first prediction unit 101 or the second prediction unit 202, along with motion information 18 and selected block information 31. The output motion information 18 is encoded by the variable-length encoding unit 204 and then multiplexed into encoded data 14.

[0108] Figure 29 schematically shows the configuration of the variable-length coding unit 204. The variable-length coding unit 204 shown in Figure 29 includes a motion information coding unit 217 in addition to the configuration of the variable-length coding unit 104 shown in Figure 18. Furthermore, unlike the selection block coding unit 116 in Figure 18, the selection block coding unit 216 in Figure 29 codes the prediction switching information 32 to generate coded data 14D. When the prediction process is performed by the first prediction unit 101, the selection block coding unit 216 further codes the available block information 30 and the selected block information 31. The coded available block information 30 and selected block information 31 are included in the coded data 14D. When the prediction process is performed by the second prediction unit 202, the motion information coding unit 217 codes the motion information 18 to generate coded data 14E. The selected block coding unit 216 and the motion information coding unit 217 each determine whether the prediction process was performed by the first prediction unit 101 or the second prediction unit 202, based on the prediction switching information 32 which indicates whether the predicted image was generated by motion compensation prediction using the motion information of the selected block.

[0109] The multiplexing unit 117 receives encoded data 14A, B, D, E from the parameter encoding unit 114, the conversion coefficient encoding unit 115, the selected block encoding unit 216, and the motion information encoding unit, and multiplexes the received encoded data 14A, B, D, E.

[0110] Figures 30A and 30B show examples of macroblock layer syntax according to this embodiment. In Figure 30A, available_block_num indicates the number of available blocks. If this value is greater than 1, the selected block encoding unit 216 encodes the selected block information 31. stds_flag is a flag indicating whether the motion information of the selected block was used as the motion information of the target block in motion compensation prediction; that is, it indicates whether the prediction method switching switch 203 selected the first prediction unit 101 or the second prediction unit 202. If the number of available blocks is greater than 1 and stds_flag is 1, it indicates that the motion information of the selected block was used in motion compensation prediction. If stds_flag is 0, the motion information of the selected block is not used, and the difference value of the motion information 18 is encoded directly or predicted, similar to H.264. Furthermore, stds_idx indicates the selected block information, and the code table corresponding to the number of available blocks is as described above.

[0111] Figure 30A shows the syntax for encoding selected block information after mb_type. stds_flag and stds_idx are encoded only when the mode indicated by mb_type is a specified size or mode. For example, stds_flag and stds_idx are encoded when the selected block's motion information becomes available for block sizes of 64x64, 32x32, 16x16, or in direct mode.

[0112] Figure 30B shows the syntax when the selection block information is encoded before mb_type. For example, if stds_flag is 1, mb_type does not need to be encoded. If stds_flag is 0, mb_type is encoded.

[0113] As described above, the image coding device according to the second embodiment compresses and encodes the input image signal by selectively switching between the first prediction unit 101 according to the first embodiment and the second prediction unit 202 that utilizes a prediction method such as H.264, in order to reduce coding costs. Therefore, the image coding device according to the second embodiment has improved coding efficiency compared to the image coding device according to the first embodiment.

[0114] (Third embodiment) Figure 31 schematically shows an image decoding device according to the third embodiment. As shown in Figure 31, this image decoding device comprises an image decoding unit 300, a decoding control unit 350, and an output buffer 308. The image decoding unit 300 is controlled by the decoding control unit 350. The image decoding device according to the third embodiment corresponds to the image encoding device according to the first embodiment. That is, the decoding process by the image decoding device in Figure 31 has a complementary relationship with the encoding process by the image encoding process in Figure 1. The image decoding device in Figure 31 may be implemented by hardware such as an LSI chip, or it may be implemented by having a computer execute an image decoding program.

[0115] The image decoding device shown in Figure 31 comprises an encoded sequence decoding unit 301, an inverse quantization / inverse transform unit 302, an adder 303, a frame memory 304, a prediction unit 305, a motion information memory 306, and an available block acquisition unit 307. In the image decoding unit 300, encoded data 80 from a storage system or transmission system (not shown) is input to the encoded sequence decoding unit 301. This encoded data 80 corresponds, for example, to encoded data 14 transmitted in a multiplexed state from the image encoding device shown in Figure 1.

[0116] In this embodiment, the pixel block to be decoded (e.g., a macroblock) is simply referred to as the decoded block. Furthermore, the image frame containing the decoded block is referred to as the decoded frame.

[0117] In the coded sequence decoding unit 301, decoding is performed by syntactic analysis for each frame or field based on the syntax. Specifically, the coded sequence decoding unit 301 sequentially decodes the coded sequence of each syntax in a variable length and decodes coding parameters related to the block to be decoded, including conversion coefficient information 33, selected block information 61, and prediction information such as block size information and prediction mode information.

[0118] In this embodiment, the decoding parameters include the conversion coefficient 33, the selected block information 61, and the prediction information, and include all parameters necessary for decoding, such as information regarding the conversion coefficient and information regarding quantization. The prediction information, information regarding the conversion coefficient, and information regarding quantization are input to the decoding control unit 350 as control information 71. The decoding control unit 350 provides decoding control information 70, which includes the prediction information and parameters necessary for decoding, such as quantization parameters, to each part of the image decoding unit 300.

[0119] Furthermore, the coded sequence decoding unit 301 simultaneously decodes the coded data 80, as will be explained later, to obtain prediction information and selected block information 61. Motion information 38, including motion vectors and reference frame numbers, does not necessarily need to be decoded.

[0120] The transformation coefficients 33 decoded by the coded sequence decoding unit 301 are sent to the inverse quantization / inverse transform unit 302. Various information related to quantization decoded by the coded sequence decoding unit 301, namely the quantization parameters and the quantization matrix, are provided to the decoding control unit 350 and loaded into the inverse quantization / inverse transform unit 302 when inverse quantization is performed. The inverse quantization / inverse transform unit 302 inverse quantizes the transformation coefficients 33 according to the loaded quantization information, and then performs an inverse transform process (e.g., inverse discrete cosine transform) to obtain the prediction error signal 34. The inverse transform process by the inverse quantization / inverse transform unit 302 in Figure 31 is the inverse transform of the transformation process by the transformation / quantization unit in Figure 1. For example, when a wavelet transform is performed by the image encoding device (Figure 1), the inverse quantization / inverse transform unit 302 performs the corresponding inverse quantization and inverse wavelet transform.

[0121] The prediction error signal 34, restored by the inverse quantization / inverse transform unit 302, is input to the adder 303. The adder 303 adds the prediction error signal 34 to the prediction image signal 35 generated by the prediction unit 305 (described later) to generate the decoded image signal 36. The generated decoded image signal 36 is output from the image decoding unit 300, temporarily stored in the output buffer 308, and then output according to the output timing managed by the decoding control unit 350. This decoded image signal 36 is also stored in the frame memory 304 as a reference image signal 37. The reference image signal 37 is read sequentially from the frame memory 304, frame by frame or field by field, and input to the prediction unit 305.

[0122] The available block acquisition unit 307 receives reference motion information 39 from the motion information memory 306, which will be described later, and outputs available block information 60. The operation of the available block acquisition unit 307 is the same as that of the available block acquisition unit 109 (Figure 1) described in the first embodiment.

[0123] The motion information memory 306 receives motion information 38 from the prediction unit 305 and temporarily stores it as reference motion information 39. The motion information memory 306 temporarily stores the motion information 38 output from the prediction unit 305 as reference motion information 39. Figure 4 shows an example of the motion information memory 306. The motion information memory 306 holds multiple motion information frames 26 with different encoding times. Motion information 38 or a group of motion information 38 that has been decoded is stored as reference motion information 39 in a motion information frame 26 corresponding to the decoding time. In the motion information frame 26, the reference motion information 39 is stored, for example, in units of 4x4 pixel blocks. The reference motion information 39 held by the motion information memory 306 is read and referenced by the prediction unit 305 when generating motion information 38 of the block to be decoded.

[0124] Next, the motion reference blocks and available blocks according to this embodiment will be described. Motion reference blocks are candidate blocks selected from already decoded regions according to a predetermined method by the aforementioned image encoding device and image decoding device. Figure 8A shows an example of available blocks. In Figure 8A, a total of nine motion reference blocks are arranged: four motion reference blocks in the frame to be decoded and five motion reference blocks in the reference frame. Motion reference blocks A, B, C, and D in the frame to be decoded in Figure 8A are blocks adjacent to the block to be decoded to the left, above, upper right, and upper left, respectively. In this embodiment, motion reference blocks selected from the frame to be decoded, which includes the block to be decoded, are referred to as spatial motion reference blocks. Motion reference block TA in the reference frame is a pixel block in the reference frame at the same position as the block to be decoded, and pixel blocks TB, TC, TD, and TE adjacent to this motion reference block TA are selected as motion reference blocks. Motion reference blocks selected from among the pixel blocks in the reference frame are referred to as temporal motion reference blocks. Furthermore, the frame in which a time-direction motion reference block is located is called a motion reference frame.

[0125] The spatial motion reference block is not limited to the example shown in Figure 8A; as shown in Figure 8B, the pixel block to which pixels a, b, c, and d adjacent to the block to be decoded belong may also be selected as the spatial motion reference block. In this case, the relative positions (dx, dy) of pixels a, b, c, and d with respect to the top-left pixel in the block to be decoded are shown in Figure 8C.

[0126] Furthermore, as shown in Figure 8D, all pixel blocks A1-A4, B1, B2, C, and D adjacent to the block to be decoded may be selected as spatial motion reference blocks. In Figure 8D, the number of spatial motion reference blocks is 8.

[0127] Furthermore, the time-direction motion reference blocks TA~TE may partially overlap each other, as shown in Figure 8E, or they may be separated from each other, as shown in Figure 8F. Also, the time-direction motion reference blocks do not necessarily have to be located in or around the block at the collocate position; they can be any pixel block within the motion reference frame. For example, motion information from an already decoded block adjacent to the block to be decoded may be used to select the reference block pointed to by the motion vector contained in the motion information as the center of the motion reference block (e.g., block TA). Moreover, the time-direction reference blocks do not have to be arranged at equal intervals.

[0128] In the method of selecting motion reference blocks as described above, the number and position of motion reference blocks can be selected from any number and position, as long as both the image decoding device and the image decoding device share information regarding the number and position of spatial and temporal motion reference blocks. Furthermore, the size of the motion reference blocks does not necessarily have to be the same as the size of the block to be decoded. For example, as shown in Figure 8D, the size of the motion reference blocks may be larger, smaller, or any other size than the block to be decoded. Also, the shape of the motion reference blocks is not limited to a square; it may be rectangular or other shapes.

[0129] Next, we will explain the available blocks. Available blocks are pixel blocks selected from the motion reference blocks that can have motion information applied to the block to be decoded. Available blocks have different motion information from each other. Available blocks are selected by performing the available block determination process shown in Figure 9 on a total of nine motion reference blocks in the frame to be decoded and the reference frame, for example, as shown in Figure 8A. Figure 10 shows the result of performing the available block determination process shown in Figure 9. In Figure 10, pixel blocks with diagonal lines indicate unavailable blocks, and blocks that are filled in white indicate available blocks. That is, a total of four blocks are determined to be available blocks: two from the spatial motion reference blocks and two from the temporal motion reference blocks. The motion information selection unit 314 in the prediction unit 305 selects the optimal available block as the selected block from among these available blocks arranged in the temporal and spatial directions, according to the selected block information 61 received from the selected block decoding unit 323.

[0130] Next, the available block acquisition unit 307 will be described. The available block acquisition unit 307 has the same function as the available block acquisition unit 109 of the first embodiment, and acquires reference motion information 39 from motion information memory 306 and outputs available block information 60, which is information indicating whether each motion reference block is an available block or an unavailable block.

[0131] The operation of the available block acquisition unit 307 will be explained with reference to the flowchart in Figure 9. First, the available block acquisition unit 307 determines whether or not a motion reference block (index p) has motion information (step S801). That is, in step S801, it is determined whether or not at least one small pixel block in the motion reference block p has motion information. If it is determined that the motion reference block p does not have motion information, that is, if the time-direction motion reference block is a block in an I slice that does not have motion information, or if all small pixel blocks in the time-direction motion reference block have been intra-predictively decoded, the process proceeds to step S805. In step S805, this motion reference block p is determined to be an unavailable block.

[0132] If it is determined in step S801 that motion reference block p has motion information, the available block acquisition unit 307 selects a motion reference block q (referred to as available block q) that has already been determined to be an available block (step S802). Here, q is a value smaller than p. Next, the available block acquisition unit 307 compares the motion information of this motion reference block p with the motion information of the available block q for all q and determines whether motion reference block p has the same motion information as the available block q (S803). If motion reference block p has the same motion vector as the available block q, the process proceeds to step S805, in which step S805, this motion reference block p is determined to be an unavailable block by the available block acquisition unit 307. If motion reference block p has motion information different from all available blocks q, in step S804, this motion reference block p is determined to be an available block by the available block acquisition unit 307.

[0133] By performing the above-described available block determination process for all motion reference blocks, it is determined whether each motion reference block is available or unavailable, and available block information 60 is generated. An example of available block information 60 is shown in Figure 11. As shown in Figure 11, the available block information 60 includes the index p and availability of the motion reference block. In Figure 11, the available block information 60 indicates that motion reference blocks with index p of 0, 1, 5, and 8 were selected as available blocks, and the number of available blocks is 4.

[0134] Furthermore, in step S801 of Figure 9, if at least one of the blocks in the time-direction motion reference block p is an intra-predictive encoded block, the available block acquisition unit 307 may determine that the motion reference block p is an unavailable block. In other words, the process may proceed to step S802 only if all blocks in the time-direction motion reference block p are encoded using inter-prediction.

[0135] Figures 12A to 12E show examples in step S803 where the motion information 38 of motion reference block p and the motion information 38 of available block q are determined to be identical during the comparison of motion information 38. Figures 12A to 12E each show multiple blocks with diagonal lines and two white-filled blocks. For the sake of simplicity, Figures 12A to 12E assume a case where the motion information 38 of these two white-filled blocks is compared without considering the diagonal lines. One of the two white-filled blocks is motion reference block p, and the other is motion reference block q (available block q) that has already been determined to be available. Unless otherwise specified, either of the two white blocks can be motion reference block p.

[0136] Figure 12A shows an example where both the motion reference block p and the available block q are spatial blocks. In the example in Figure 12A, if the motion information 38 of blocks A and B are identical, then it is determined that the motion information 38 are identical. In this case, the sizes of blocks A and B do not need to be the same.

[0137] Figure 12B shows an example where one of the motion reference block p and available block q is a spatial block A, and the other is a temporal block TB. In Figure 12B, there is one block containing motion information within the temporal block TB. If the motion information 38 of the temporal block TB and the motion information 38 of the spatial block A are identical, then it is determined that the motion information 38 are identical. In this case, the sizes of blocks A and TB do not need to be the same.

[0138] Figure 12C shows another example where one of the motion reference block p and available block q is a spatial block A, and the other is a temporal block TB. Figure 12C shows a case where the temporal block TB is divided into multiple subblocks, and there are multiple subblocks that have motion information 38. In the example of Figure 12C, if all blocks that have motion information 38 have the same motion information 38, and that motion information 38 is identical to the motion information 38 of spatial block A, then the motion information 38 is determined to be identical. In this case, the sizes of block A and TB do not need to be the same.

[0139] Figure 12D shows an example where both the motion reference block p and the available block q are time-direction blocks. In this case, if the motion information 38 of blocks TB and TE are identical, it is determined that the motion information 38 are identical.

[0140] Figure 12E shows another example where both the motion reference block p and the available block q are time-direction blocks. Figure 12E shows a case where time-direction blocks TB and TE are each divided into multiple subblocks, each containing multiple subblocks with motion information 38. In this case, the motion information 38 is compared for each subblock within the block, and if the motion information 38 is identical for all subblocks, it is determined that the motion information 38 of block TB and the motion information 38 of block TE are identical.

[0141] Figure 12F shows yet another example where both the motion reference block p and the available block q are time-direction blocks. Figure 12F shows the case where the time-direction block TE is divided into multiple subblocks, and block TE has multiple subblocks that have motion information 38. If all the motion information 38 in block TE is the same motion information 38 and is the same as the motion information 38 in block TD, then it is determined that the motion information 38 in block TD and TE are the same.

[0142] In this way, step S803 determines whether the motion information 38 of the motion reference block p and the motion information 38 of the available block q are identical. In the examples in Figures 12A to 12F, the number of available blocks q to be compared with the motion reference block p was set to 1. However, if the number of available blocks q is 2 or more, the motion information 38 of the motion reference block p may be compared with the motion information 38 of each available block q. Furthermore, when scaling, which will be described later, is applied, the scaled motion information 38 becomes the motion information 38 described above.

[0143] Furthermore, the determination that the motion information of motion reference block p and the motion information of available block q are identical is not limited to cases where each motion vector included in the motion information is perfectly identical. For example, if the norm of the difference between the two motion vectors is within a predetermined range, the motion information of motion reference block p and the motion information of available block q may be considered substantially identical.

[0144] Figure 32 is a block diagram showing the coded sequence decoding unit 301 in more detail. As shown in Figure 32, the coded sequence decoding unit 301 includes a separation unit 320 that separates the coded data 80 into syntax units, a conversion coefficient decoding unit 322 that decodes the conversion coefficients, a selected block decoding unit 323 that decodes the selected block information, and a parameter decoding unit 321 that decodes the predicted block size and parameters related to quantization.

[0145] The parameter decoding unit 321 receives encoded data 80A from the separation unit, which includes parameters related to the predicted block size and quantization, and decodes the encoded data 80A to generate control information 71. The conversion coefficient decoding unit 322 receives encoded conversion coefficients 80B from the separation unit 320, decodes these encoded conversion coefficients 80B to obtain conversion coefficient information 33. The selected block decoding unit 323 takes encoded data 80C and available block information 60 related to the selected block as input and outputs selected block information 61. The input available block information 60 indicates the availability for each motion reference block, as shown in Figure 11.

[0146] Next, the prediction unit 305 will be described in detail with reference to Figure 33. As shown in Figure 33, the prediction unit 305 includes a motion information selection unit 314 and a motion compensation unit 313. The motion information selection unit 314 includes a spatial motion information acquisition unit 310, a temporal motion information acquisition unit 311, and a motion information switching switch 312. The prediction unit 305 basically has the same configuration and functions as the prediction unit 101 described in the first embodiment.

[0147] The prediction unit 305 takes available block information 60, selected block information 61, reference motion information 39, and reference image signal 37 as inputs and outputs a predicted image signal 35 and motion information 38. The spatial motion information acquisition unit 310 and the temporal motion information acquisition unit 311 have the same functions as the spatial motion information acquisition unit 110 and the temporal motion information acquisition unit 111 described in the first embodiment, respectively. The spatial motion information acquisition unit 310 uses the available block information 60 and reference motion information 39 to generate motion information 38A, which includes motion information and indexes for each available block located in the spatial direction. The temporal motion information acquisition unit 311 uses the available block information 60 and reference motion information 39 to generate motion information (or a group of motion information) 38B, which includes motion information and indexes for each available block located in the temporal direction.

[0148] The motion information switching switch 312 selects one motion information 38A from the spatial motion information acquisition unit 310 and one motion information (or group of motion information) 38B from the temporal motion information acquisition unit 311 according to the selection block information 61, and obtains motion information 38. The selected motion information 38 is sent to the motion compensation unit 313 and the motion information memory 306. The motion compensation unit 313 performs motion compensation prediction according to the selected motion information 38, similar to the motion compensation unit 113 described in the first embodiment, and generates a predicted image signal 35.

[0149] The motion vector scaling function of the motion compensation unit 313 is the same as that described in the first embodiment, so its explanation will be omitted.

[0150] Figure 22 shows the syntax structure in the image decoding unit 300. As shown in Figure 22, the syntax mainly consists of three parts: high-level syntax 901, slice-level syntax 904, and macroblock-level syntax 907. High-level syntax 901 holds syntax information for layers above the slice level. Slice-level syntax 904 holds the information required for each slice, and macroblock-level syntax 907 holds the data required for each macroblock shown in Figures 7A to 7D.

[0151] Each part includes more detailed syntax. High-level syntax 901 includes sequence and picture-level syntax such as sequence parameter set syntax 902 and picture parameter set syntax 903. Slice-level syntax 904 includes slice header syntax 905 and slice data syntax 906, etc. Furthermore, macroblock-level syntax 907 includes macroblock layer syntax 908 and macroblock prediction syntax 909, etc.

[0152] Figures 23A and 23B show examples of macroblock layer syntax. In Figures 23A and 23B, available_block_num indicates the number of available blocks; if this value is greater than 1, decoding of the selected block information is required. Furthermore, stds_idx indicates the selected block information, and stds_idx is encoded using the code table corresponding to the number of available blocks mentioned above.

[0153] Figure 23A shows the syntax for decoding selected block information after mb_type. stds_idx is decoded when the prediction mode indicated by mb_type is a defined size and mode (TARGET_MODE), and available_block_num is greater than 1. For example, stds_idx is encoded when the motion information of the selected block becomes available for block sizes of 64x64 pixels, 32x32 pixels, 16x16 pixels, or in direct mode.

[0154] Figure 23B shows the syntax for decoding the selected block information before mb_type. If available_block_num is greater than 1, stds_idx is decoded. If available_block_num is 0, conventional motion compensation, such as that used in H.264, is performed, and mb_type is encoded.

[0155] Syntax elements not defined in this invention may be inserted between the rows of the tables shown in Figures 23A and 23B, and descriptions relating to other conditional branching may also be included. Alternatively, the syntax table may be divided into multiple tables and merged. Furthermore, it is not necessary to use the same terminology, and it may be arbitrarily changed depending on the form in which it is used. In addition, each syntax element described in the macroblock layer syntax may be modified to be explicitly stated in the macroblock data syntax described later.

[0156] As described above, the image decoding device according to this embodiment decodes an image encoded by the image encoding device according to the first embodiment described above. Therefore, the image decoding according to this embodiment can reproduce a high-quality decoded image from relatively small encoded data.

[0157] (Fourth embodiment) Figure 34 schematically shows an image decoding device according to the fourth embodiment. As shown in Figure 34, the image decoding device comprises an image decoding unit 400, a decoding control unit 350, and an output buffer 308. The image decoding device according to the fourth embodiment corresponds to the image encoding device according to the second embodiment. In the fourth embodiment, the differences and operation from the third embodiment will be mainly described. As shown in Figure 34, the image decoding unit 400 according to this embodiment differs from the third embodiment in its coded sequence decoding unit 401 and prediction unit 405.

[0158] The prediction unit 405 of this embodiment generates a predicted image signal 35 by selectively switching between a prediction method that compensates for motion using motion information possessed by the selected block (first prediction method) and a prediction method that compensates for motion using a single motion vector for the block to be decoded, such as H.264 (second prediction method).

[0159] Figure 35 is a block diagram showing the coded sequence decoding unit 401 in more detail. The coded sequence decoding unit 401 shown in Figure 35 includes a motion information decoding unit 424 in addition to the configuration of the coded sequence decoding unit 301 shown in Figure 32. Furthermore, the selected block decoding unit 423 shown in Figure 35 differs from the selected block decoding unit 323 shown in Figure 32 in that it decodes coded data 80C related to the selected block to obtain prediction switching information 62. The prediction switching information 62 indicates whether the prediction unit 101 in the image encoding device in Figure 1 used the first prediction method or the second prediction method. If the prediction switching information 62 indicates that the prediction unit 101 used the first prediction method, that is, if the block to be decoded is coded using the first prediction method, the selected block decoding unit 423 decodes the selected block information in the coded data 80C to obtain selected block information 61. If the prediction switching information 62 indicates that the prediction unit 101 used the second prediction method, that is, if the block to be decoded is encoded using the second prediction method, the selected block decoding unit 423 does not decode the selected block information, and the motion information decoding unit 424 decodes the encoded motion information 80D to obtain the motion information 40.

[0160] Figure 36 is a block diagram showing the prediction unit 405 in more detail. The prediction unit 405 shown in Figure 34 comprises a first prediction unit 305, a second prediction unit 410, and a prediction method switching switch 411. The second prediction unit 410 uses the motion information 40 and reference image signal 37 decoded by the coded sequence decoding unit 401 to perform motion compensation prediction similar to the motion compensation unit 313 in Figure 33, and generates a predicted image signal 35B. The first prediction unit 305 is the same as the prediction unit 305 described in the third embodiment and generates a predicted image signal 35B. The prediction method switching switch 411 selects either the predicted image signal 35B from the second prediction unit 410 or the predicted image signal 35A from the first prediction unit 305 and outputs it as the predicted image signal 35 of the prediction unit 405 based on the prediction switching information 62. At the same time, the prediction method switching switch 411 sends the motion information used by the selected first prediction unit 305 or second prediction unit 410 as motion information 38 to the motion information memory 306.

[0161] Next, we will mainly explain the differences in the syntax structure of this embodiment compared to the third embodiment.

[0162] Figures 30A and 30B show examples of macroblock layer syntax according to this embodiment. In Figure 30A, available_block_num indicates the number of available blocks. If this value is greater than 1, the selected block decoding unit 423 decodes the selected block information in the encoded data 80C. stds_flag is a flag indicating whether the motion information of the selected block was used as the motion information of the block to be decoded in motion compensation prediction; that is, it indicates whether the prediction method switching switch 411 selected the first prediction unit 305 or the second prediction unit 410. If the number of available blocks is greater than 1 and stds_flag is 1, it indicates that the motion information of the selected block was used in motion compensation prediction. If stds_flag is 0, the motion information of the selected block is not used, and the difference value of the motion information is encoded directly or predicted, similar to H.264. Furthermore, stds_idx indicates the selected block information, and the code table corresponding to the number of available blocks is as described above.

[0163] Figure 30A shows the syntax for decoding the selected block information after mb_type. stds_flag and stds_idx are decoded only if the prediction mode indicated by mb_type matches the specified block size or mode. For example, stds_flag and stds_idx are decoded when the block size is 64x64, 32x32, or 16x16, or in direct mode.

[0164] Figure 30B shows the syntax for decoding the selected block information before mb_type. For example, if stds_flag is 1, mb_type does not need to be decoded. If stds_flag is 0, mb_type is decoded.

[0165] As described above, the image decoding device according to this embodiment decodes an image encoded by the image encoding device according to the second embodiment described above. Therefore, the image decoding according to this embodiment can reproduce a high-quality decoded image from relatively small encoded data.

[0166] It should be noted that the present invention is not limited to the embodiments described above, and the components can be modified and implemented in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the embodiments. For example, some components may be deleted from all the components shown in the embodiments. Moreover, components from different embodiments may be appropriately combined.

[0167] As an example, similar effects can be obtained by modifying the first to fourth embodiments described above as follows.

[0168] (1) In the first to fourth embodiments, the frame to be processed is divided into rectangular blocks such as 16x16 pixel blocks, and the encoding or decoding is described using the example of encoding or decoding in the order from the pixel block in the upper left of the screen to the pixel block in the lower right, as shown in Figure 4. However, the encoding or decoding order is not limited to this example. For example, the encoding or decoding order may be from the lower right to the upper left of the screen, or from the upper right to the lower left. Also, the encoding or decoding order may be in a spiral pattern from the center to the periphery of the screen, or from the periphery to the center of the screen.

[0169] (2) In the first to fourth embodiments, the example is described in which the luminance signal and the chrominance signal are not separated, and only one of the chrominance signal components is used. However, different prediction processes may be used for the luminance signal and the chrominance signal, or the same prediction process may be used. When different prediction processes are used, the prediction method selected for the chrominance signal is encoded / decoded in the same way as for the luminance signal.

[0170] It goes without saying that the invention can also be implemented in a similar manner by applying various prejudices, as long as they do not depart from the spirit of the present invention. [Industrial applicability]

[0171] The image coding / decoding method according to the present invention has industrial applicability because it can improve coding efficiency. [Explanation of Symbols]

[0172] 10...Input image signal, 11...Predicted image signal, 12...Prediction error image signal, 13...Quantization conversion coefficients, 14...Encoded data, 15...Decoded prediction error signal, 16...Locally decoded image signal, 17...Reference image signal, 18...Motion information, 20...Bitstream, 21...Motion information, 25,26...Information frames, 30...Available block information, 31...Selected block information, 32...Prediction switching information, 33...Conversion coefficient information, 34...Prediction error signal, 35...Predicted image signal, 36...Decoded image signal, 37...Reference image signal, 38...Motion information, 39...Reference motion information, 40...Motion information, 50...Encoding control information 51...Feedback information, 60...Available block information, 61...Selected block information, 62...Prediction switching information, 70...Decoding control information, 71...Control information, 80...Encoded data, 100...Image encoding unit, 101...Prediction unit, 102...Subtractor, 103...Conversion / quantization unit, 104...Variable length encoding unit, 105...Inverse quantization / inverse conversion unit, 106...Adder, 107...Frame memory, 108...Information memory, 109...Available block acquisition unit, 110...Spatial motion information acquisition unit, 111...Temporal motion information acquisition unit, 112...Information switching switch, 113...Motion compensation unit, 114...Pa Lameter coding unit, 115... Transformation coefficient coding unit, 116... Selected block coding unit, 117... Multiplexing unit, 118... Motion information selection unit, 120... Output buffer, 150... Coding control unit, 200... Image coding unit, 201... Prediction unit, 202... Second prediction unit, 203... Prediction method switching switch, 204... Variable length coding unit, 205... Motion information acquisition unit, 216... Selected block coding unit, 217... Motion information coding unit, 300... Image decoding unit, 301... Coding sequence decoding unit, 301... Code sequence decoding unit, 302... Inverse quantization / inverse transformation unit, 303... Adder, 304... Frame memory, 305... Prediction Measurement unit, 306... Information memory, 307... Available block acquisition unit, 308... Output buffer, 310... Spatial motion information acquisition unit, 311... Temporal motion information acquisition unit, 312... Motion information switching switch, 313... Motion compensation unit, 314... Information selection unit, 320... Separation unit, 321... Parameter decoding unit, 322... Transformation coefficient decoding unit, 323... Selected block decoding unit, 350... Decoding control unit, 400... Image decoding unit, 401... Encoded sequence decoding unit, 405... Prediction unit, 410... Second prediction unit, 411... Prediction method switching switch, 423... Selected block decoding unit, 424... Information decoding unit,901…High-level syntax, 902…Sequence parameter set syntax, 903…Picture parameter set syntax, 904…Slice-level syntax, 905…Slice header syntax, 906…Slice data syntax, 907…Macroblock-level syntax, 908…Macroblock layer syntax, 909…Macroblock prediction syntax.

Claims

[Claim 1] Equipped with a circuit for transmitting encoded data, The encoded data includes mode information relating to the prediction mode of the target block and identification information identifying the selected block. The identification information is generated by: determining whether a plurality of candidate blocks in a predetermined positional relationship with respect to the target block are available when the mode information indicates predetermined information, determining whether they are available in a predetermined order according to the positional relationship; assigning different indices to the available blocks, which are the candidate blocks determined to be available, in the predetermined order; selecting one of the available blocks as the selected block; and encoding the indices assigned to the selected block using a coding rule corresponding to the number of available blocks. The plurality of candidate blocks include a block adjacent to the upper left of the target block and a block adjacent to the target block above it. The determination described above includes determining the block adjacent to the upper left of the target block after determining the block adjacent to the target block, and determining that the candidate block is available if the candidate block has motion information that does not match the motion information corresponding to a candidate block that has already been determined to be available. Transmitter.

Images