Image coding method nd image decoding method

Abstract

To provide an image coding and image decoding method with high coding efficiency.SOLUTION: An image coding method includes: a first step of selecting one or more motion reference blocks from coded pixel blocks having motion information; a second step of selecting, from the motion reference blocks, one or more usable blocks that have different types of motion information from each other and are pixel blocks having candidates for motion information to be applied to a coding target block; a third step of selecting one selection block from the usable blocks; a fourth step of creating a prediction image of the coding target block by using the motion information of the selection block; a fifth step of coding a prediction error between the prediction image and an original image; and a sixth step of referring to a code table predetermined according to the number of usable blocks to code selection information for specifying the selection block.SELECTED DRAWING: Figure 16

Description

Technical Field

[0001] The present invention relates to methods for encoding and decoding moving images and still images.

Background Art

[0002] In recent years, a moving image encoding method with significantly improved encoding efficiency has been recommended jointly by ITU-T and ISO / IEC as ITU-T Rec. H.264 and ISO / IEC 14496-10 (hereinafter referred to as H.264). In H.264, prediction processing, conversion processing, and entropy encoding processing are performed in units of rectangular blocks (for example, 16×16 pixel block units, 8×8 pixel block units, etc.). In the prediction processing, for a rectangular block to be encoded (encoding target block), motion compensation for performing temporal prediction is performed by referring to an already encoded frame (reference frame). In such motion compensation, it is necessary to encode motion information including a motion vector as spatial shift information between the encoding target block and the block referred to in the reference frame and send it to the decoding side. Further, when performing motion compensation using a plurality of reference frames, it is necessary to encode the reference frame number together with the motion information. For this reason, the amount of code related to the motion information and the reference frame number may increase.

[0003] As an example of a method for obtaining a motion vector in motion compensation prediction, there is a direct mode in which a motion vector to be assigned to an encoding target block is derived from the motion vectors already assigned to the encoded blocks, and a prediction image is generated based on the derived motion vector (see Patent Document 1 and Patent Document 2). In the direct mode, since the motion vector is not encoded, the amount of code for the motion information can be reduced. The direct mode is adopted, for example, in H.264 / AVC.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

[0005] In direct mode, the motion vector of the block to be encoded is predicted and generated by a fixed method of calculating the motion vector from the median value of the motion vectors of the encoded blocks adjacent to the block to be encoded. Therefore, the degree of freedom in calculating the motion vector is low.

[0006] In order to increase the degree of freedom in calculating the motion vector, a method has been proposed in which one of a plurality of encoded blocks is selected and the motion vector is assigned to the block to be encoded. In this method, selection information for specifying the selected block must always be transmitted so that the decoding side can identify the selected encoded block. Therefore, when determining the motion vector to be assigned to the block to be encoded by selecting one of a plurality of encoded blocks, there is a problem that the amount of code related to the selection information increases.

[0007] The present invention has been made to solve the above problems, and an object thereof is to provide an image encoding and image decoding method with high encoding efficiency. [Means for Solving the Problems]

[0008] An image encoding method according to an embodiment of the present invention includes: a first step of selecting at least one motion reference block from among encoded pixel blocks having motion information; a pixel block having candidates for motion information to be applied to an encoding target block, and selecting at least one available block having different motion information from among the motion reference blocks as a second step; a third step of selecting one selected block from among the available blocks; a fourth step of generating a predicted image of the encoding target block using the motion information of the selected block; a fifth step of encoding a prediction error between the predicted image and the original image; and a sixth step of encoding selection information for specifying the selected block with reference to a pre-determined code table according to the number of available blocks.

[0009] An image decoding method according to another embodiment of the present invention includes: a first step of selecting at least one motion reference block from among decoded pixel blocks having motion information; a pixel block having candidates for motion information to be applied to a decoding target block, and selecting at least one available block having different motion information from among the motion reference blocks as a second step; a third step of obtaining selection information for specifying a selected block by decoding input encoded data with reference to a pre-determined code table according to the number of available blocks; a fourth step of selecting one selected block from among the available blocks according to the selection information; a fifth step of generating a predicted image of the decoding target block using the motion information of the selected block; a sixth step of decoding a prediction residual of the decoding target block from the encoded data; and a seventh step of obtaining a decoded image from the predicted image and the prediction residual.

Advantages of the Invention

[0010] According to the present invention, the encoding efficiency can be improved.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2A

Figure 2B

Figure 3

Figure 4

Figure 5

Figure 6A

Figure 6B

Figure 7A

Figure 7B

Figure 7C

Figure 7D

Figure 8A

Figure 8B

Figure 8C

Figure 8D

Figure 8E

Figure 8F

Figure 9

Figure 10

Figure 11

Figure 12A

Figure 12B

Figure 12C

Figure 12D

Figure 12E

Figure 12F

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23A

Figure 23B

Figure 24A

Figure 24B

Figure 24C

Figure 24D

Figure 25A

Figure 25B

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30A

Figure 30B

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Embodiments for Carrying Out the Invention

[0012] Hereinafter, a method and apparatus for image encoding and image decoding according to embodiments of the present invention will be described with reference to the drawings as necessary. In the following embodiments, parts with the same number are assumed to perform the same operations, and repeated explanations are omitted.

[0013] (First Embodiment) FIG. 1 schematically shows the configuration of an image encoding apparatus according to a first embodiment of the present invention. As shown in FIG. 1, this image encoding apparatus includes an image encoding unit 100, an encoding control unit 150, and an output buffer 120. This image encoding apparatus may be realized by hardware such as an LSI chip, or may be realized by causing a computer to execute an image encoding program.

[0014] An original image (input image signal) 10, which is a moving image or a still image, is input to the image encoding unit 100 in units of pixel blocks obtained by dividing the original image, for example. As will be described in detail later, the image encoding unit 100 compresses and encodes the input image signal 10 to generate encoded data 14. The generated encoded data 14 is temporarily stored in the output buffer 120 and is sent to a storage system (storage medium) or a transmission system (communication line) (not shown) at an output timing managed by the encoding control unit 150.

[0015] The encoding control unit 150 controls the entire encoding process of the image encoding unit 100, such as feedback control of the generated code amount, quantization control, prediction mode control, and entropy encoding control. Specifically, the encoding control unit 150 gives encoding control information 50 to the image encoding unit 100 and appropriately receives feedback information 51 from the image encoding unit 100. The encoding control information 50 includes prediction information, motion information 18, quantization parameter information, and the like. The prediction information includes prediction mode information and block size information. The motion information 18 includes a motion vector, a reference frame number, and a prediction direction (unidirectional prediction, bidirectional prediction). The quantization parameter information includes quantization parameters such as a quantization width (quantization step size) and a quantization matrix. The feedback information 51 includes the generated code amount by the image encoding unit 100 and is used, for example, to determine the quantization parameter.

[0016] The image encoding unit 100 encodes the input image signal 10 in units of pixel blocks (e.g., macro blocks, sub-blocks, one pixel, etc.) obtained by dividing the original image. For this purpose, the input image signal 10 is sequentially input to the image encoding unit 100 in units of pixel blocks obtained by dividing the original image. In the present embodiment, the processing unit for encoding is a macro block, and the pixel block (macro block) to be encoded corresponding to the input image signal 10 is simply referred to as an encoding target block. Further, an image frame including the encoding target block, that is, an image frame to be encoded is referred to as an encoding target frame.

[0017] Such an encoding target block may be, for example, a 16×16 pixel block as shown in FIG. 2A, or a 64×64 pixel block as shown in FIG. 2B. Further, the encoding target block may be a 32×32 pixel block, an 8×8 pixel block, or the like. Further, the shape of the macro block is not limited to the square shape examples shown in FIGS. 2A and 2B, and may be set to an arbitrary shape such as a rectangular shape. Furthermore, the above processing unit is not limited to a pixel block such as a macro block, and may be a frame or a field.

[0018] Note that the encoding process for each pixel block in the encoding target frame may be executed in any order. In the present embodiment, for simplicity of explanation, as shown in FIG. 3, the encoding process is executed for the pixel blocks row by row from the upper left pixel block to the lower right pixel block of the encoding target frame, that is, in raster scan order.

[0019] The image encoding unit 100 shown in FIG. 1 includes a prediction unit 101, a subtractor 102, a transform / quantization unit 103, a variable length encoding unit 104, an inverse quantization / inverse transform unit 105, an adder 106, a frame memory 107, a motion information memory 108, and an available block acquisition unit 109.

[0020] In the image encoding unit 100, the input image signal 10 is input to the prediction unit 101 and the subtractor 102. The subtractor 102 receives the input image signal 10 and also receives the predicted image signal 11 from the prediction unit 101, which will be described later. The subtractor 102 calculates the difference between the input image signal 10 and the predicted image signal 11, and generates a prediction error image signal 12.

[0021] The conversion and quantization unit 103 receives the prediction error image signal 12 from the subtractor 102, performs a conversion process on the received prediction error image signal 12, and generates conversion coefficients. The conversion process is, for example, an orthogonal transformation such as a discrete cosine transform (DCT: Discrete Cosine Transform). In other embodiments, the conversion and quantization unit 103 may generate conversion coefficients by using techniques such as wavelet transform and independent component analysis instead of the discrete cosine transform. Further, the conversion and quantization unit 103 quantizes the generated conversion coefficients based on the quantization parameters given by the encoding control unit 150. The quantized conversion coefficients (conversion coefficient information) 13 are output to the variable length encoding unit 104 and the inverse quantization and inverse conversion unit 105.

[0022] The inverse quantization and inverse conversion unit 105 inverse quantizes the quantized conversion coefficients 13 according to the quantization parameters given by the encoding control unit 150, that is, the same quantization parameters as those of the conversion and quantization unit 103. Subsequently, the inverse quantization and inverse conversion unit 105 performs an inverse conversion on the inverse quantized conversion coefficients to generate a decoded prediction error signal 15. The inverse conversion process by the inverse quantization and inverse conversion unit 105 coincides with the inverse conversion process of the conversion process by the conversion and quantization unit 103. For example, the inverse conversion process is an inverse discrete cosine transform (IDCT: Inverse Discrete Cosine Transform) or an inverse wavelet transform.

[0023] The adder 106 receives the decoded prediction error signal 15 from the inverse quantization and inverse transformation unit 105, and further receives the predicted image signal 11 from the prediction unit 101. The adder 106 adds the decoded prediction error signal 15 and the predicted image signal 11 to generate a local decoded image signal 16. The generated local decoded image signal 16 is stored in the frame memory 107 as a reference image signal 17. The reference image signal 17 stored in the frame memory 107 is read out and referred to by the prediction unit 101 when encoding a subsequent block to be encoded.

[0024] The prediction unit 101 receives the reference image signal 17 from the frame memory 107, and also receives available block information 30 from the available block acquisition unit 109 described later. Further, the prediction unit 101 receives the reference motion information 19 from the motion information memory 108 described later. The prediction unit 101 generates a predicted image signal 11, motion information 18, and selected block information 31 for the block to be encoded based on the reference image signal 17, the reference motion information 19, and the available block information 30. Specifically, the prediction unit 101 includes a motion information selection unit 118 that generates the motion information 18 and the selected block information 31 based on the available block information 30 and the reference motion information 19, and a motion compensation unit 113 that generates the predicted image signal 11 based on the motion information 18. The predicted image signal 11 is sent to the subtractor 102 and the adder 106. The motion information 18 is stored in the motion information memory 108 for subsequent prediction processing of the block to be encoded. Also, the selected block information 31 is sent to the variable length encoding unit 104. The prediction unit 101 will be described in detail later.

[0025] The motion information 18 is temporarily stored in the motion information memory 108 as the reference motion information 19. FIG. 4 shows an example of the configuration of the motion information memory 108. As shown in FIG. 4, the reference motion information 19 is held in the motion information memory 108 in units of frames, and the reference motion information 19 forms a motion information frame 25. The motion information 18 regarding the encoded blocks is sequentially input to the motion information memory 108, and as a result, the motion information memory 108 holds a plurality of motion information frames 25 with different encoding times.

[0026] The reference motion information 19 is stored in the motion information frame 25 in units of predetermined blocks (for example, 4×4 pixel block units). The motion vector block 28 shown in FIG. 4 indicates a pixel block of the same size as the block to be coded, the available block, the selected block, etc., and is, for example, a 16×16 pixel block. In the motion vector block 28, motion vectors are assigned, for example, for each 4×4 pixel block. The inter prediction process using the motion vector block is referred to as the motion vector block prediction process. The reference motion information 19 held by the motion information memory 108 is read out by the prediction unit 101 when generating the motion information 18. The motion information 18 of the available block as described later refers to the reference motion information 19 stored in the area where the available block in the motion information memory 108 is located.

[0027] Note that the motion information memory 108 is not limited to the example of storing the reference motion information 19 in units of 4×4 pixel blocks, and may store the reference motion information 19 in other pixel block units. For example, the pixel block unit for the reference motion information 19 may be 1 pixel or a 2×2 pixel block. Further, the shape of the pixel block for the reference motion information 19 is not limited to the example of a square shape, and can be an arbitrary shape.

[0028] The available block acquisition unit 109 in FIG. 1 acquires the reference motion information 19 from the motion information memory 108, and based on the acquired reference motion information 19, selects an available block that can be used for the prediction process of the prediction unit 101 from a plurality of blocks for which encoding has already been completed. The selected available block is sent to the prediction unit 101 and the variable length encoding unit 104 as the available block information 30. The encoded blocks that are candidates for selecting the available block are referred to as motion reference blocks. The method for selecting the motion reference blocks and the available blocks will be described in detail later.

[0029] The variable-length encoding unit 104 receives, in addition to the transform coefficient information 13, the selection block information 31 from the prediction unit 101, the encoding parameters such as the prediction information and quantization parameter from the encoding control unit 150, and the available block information 30 from the available block acquisition unit 109. The variable-length encoding unit 104 performs entropy encoding (e.g., fixed-length encoding, Huffman encoding, or arithmetic encoding, etc.) on the quantized transform coefficients 13, the selection block information 31, the available block information 30, and the encoding parameters, to generate the encoded data 14. The encoding parameters include all the parameters necessary for decoding, such as information regarding the transform coefficients, information regarding quantization, etc., together with the selection block information 31 and the prediction information. The generated encoded data 14 is temporarily stored in the output buffer 120, and is sent to an accumulation system or a transmission system (not shown).

[0030] FIG. 5 shows the processing procedure of the input image signal 10. As shown in FIG. 5, first, the predicted image signal 11 is generated by the prediction unit 101 (step S501). In the generation of the predicted image signal 11 in step S501, one of the available blocks described later is selected as the selection block, and the predicted image signal 11 is created using the selection block information 31, the motion information of the selection block, and the reference image signal 17. The difference between the predicted image signal 11 and the input image signal 10 is calculated by the subtracter 102, and the prediction error image signal 12 is generated (step S502).

[0031] Subsequently, the prediction error image signal 12 is subjected to orthogonal transformation and quantization by the transformation / quantization unit 103, to generate the transform coefficient information 13 (step S503). The transform coefficient information 13 and the selection block information 31 are sent to the variable-length encoding unit 104, where variable-length encoding is performed to generate the encoded data (step S504). Also, in step S504, according to the selection block information 31, the code table is switched so as to have the same number of entries in the code table as the number of available blocks, and the selection block information 31 is variable-length encoded. The bit stream 20 of the encoded data is sent to an accumulation system or a transmission path (not shown).

[0032] The conversion coefficient information 13 generated in step S503 is inverse quantized by the inverse quantization and inverse transformation unit 105 and subjected to an inverse transformation process to become a decoded prediction error signal 15 (step S505). The decoded prediction error signal 15 is added to the reference image signal 17 used in step S501 to become a local decoded image signal 16 (step S506) and is stored as a reference image signal in the frame memory 107 (step S507).

[0033] Next, each component of the above-described image encoding unit 100 will be described in more detail. The image encoding unit 100 in FIG. 1 is provided with a plurality of prediction modes, and in each prediction mode, the method for generating the prediction image signal 11 and the motion compensation block size are different from each other. Specifically, as a method for the prediction unit 101 to generate the prediction image signal 11, there are roughly two types: intra prediction (intra-frame prediction) in which a prediction image is generated using the reference image signal 17 for the encoding target frame (or field), and inter prediction (inter-frame prediction) in which a prediction image is generated using the reference image signal 17 for one or more encoded reference frames (reference fields). The prediction unit 101 selectively switches between intra prediction and inter prediction to generate the prediction image signal 11 for the encoding target block.

[0034] FIG. 6A shows an example of inter prediction by the motion compensation unit 113. In inter prediction, as shown in FIG. 6A, a block in the reference frame one frame before in which encoding has already been completed, which is a block at the same position as the block to be encoded (also referred to as a prediction block) 23, is spatially shifted according to the motion vector 18a included in the motion information 18. The prediction image signal 11 is generated using the reference image signal 17 of the block 24 at the shifted position. That is, in the generation of the prediction image signal 11, the reference image signal 17 regarding the block 24 in the reference frame, which is specified by the position (coordinates) of the block to be encoded and the motion vector 18a included in the motion information 18, is used. In inter prediction, motion compensation with sub-pixel accuracy (for example, 1 / 2 pixel accuracy or 1 / 4 pixel accuracy) is possible, and by performing filtering processing on the reference image signal 17, the values of interpolated pixels are generated. For example, in H.264, interpolation processing up to 1 / 4 pixel accuracy is possible for the luminance signal. When performing motion compensation with 1 / 4 pixel accuracy, the amount of information in the motion information 18 becomes four times that of integer pixel accuracy.

[0035] Note that in inter prediction, it is not limited to the example of using the reference frame one frame before as shown in FIG. 6A, and any encoded reference frame may be used as shown in FIG. 6B. When the reference image signals 17 regarding a plurality of reference frames with different temporal positions are held, the information indicating from which reference image signal 17 at which temporal position the prediction image signal 11 is generated is represented by the reference frame number. The reference frame number is included in the motion information 18. The reference frame number can be changed in units of regions (picture, block unit, etc.). That is, different reference frames can be used for each pixel block. As an example, when the reference frame one frame before that has been encoded is used for prediction, the reference frame number of this region is set to 0, and when the reference frame two frames before that has been encoded is used for prediction, the reference frame number of this region is set to 1. As another example, when only the reference image signal 17 for one frame is held in the frame memory 107 (the number of reference frames is 1), the reference frame number is always set to 0.

[0036] Furthermore, in inter prediction, a block size suitable for the block to be coded can be selected from among a plurality of motion compensation blocks. That is, the block to be coded may be divided into a plurality of small pixel blocks, and motion compensation may be performed for each small pixel block. FIGS. 7A to 7C show the sizes of motion compensation blocks in units of macroblocks, and FIG. 7D shows the sizes of motion compensation blocks in units of subblocks (pixel blocks of 8×8 pixels or less). As shown in FIG. 7A, when the block to be coded is 64×64 pixels, as the motion compensation block, a 64×64 pixel block, a 64×32 pixel block, a 32×64 pixel block, a 32×32 pixel block, etc. can be selected. Also, as shown in FIG. 7B, when the block to be coded is 32×32 pixels, as the motion compensation block, a 32×32 pixel block, a 32×16 pixel block, a 16×32 pixel block, a 16×16 pixel block, etc. can be selected. Furthermore, as shown in FIG. 7C, when the block to be coded is 16×16 pixels, the motion compensation block can be set to a 16×16 pixel block, a 16×8 pixel block, an 8×16 pixel block, an 8×8 pixel block, etc. Still further, as shown in FIG. 7D, when the block to be coded is 8×8 pixels, as the motion compensation block, an 8×8 pixel block, an 8×4 pixel block, a 4×8 pixel block, a 4×4 pixel block, etc. can be selected.

[0037] As described above, since the small pixel blocks (e.g., 4×4 pixel blocks) in the reference frame used for inter prediction have motion information 18, the optimal motion compensation block shape and motion vectors can be utilized according to the local properties of the input image signal 10. Also, the macroblocks and sub-macroblocks in FIGS. 7A to 7D can be arbitrarily combined. When the block to be coded is a 64×64 pixel block as shown in FIG. 7A, for each of the four 32×32 pixel blocks obtained by dividing the 64×64 pixel block, by selecting each block size shown in FIG. 7B, blocks of 64×64 to 16×16 pixels can be hierarchically utilized. Similarly, when it is possible to select up to the block size shown in FIG. 7D, blocks of 64×64 to 4×4 block sizes can be hierarchically utilized.

[0038] Next, with reference to FIGS. 8A to 8F, the motion reference blocks will be described. The motion reference blocks are selected from among the encoded regions (blocks) in the frame to be encoded and the reference frames according to the method determined by both the image encoding device of FIG. 1 and the image decoding device described later. FIG. 8A shows an example of the arrangement of the motion reference blocks selected according to the position of the block to be encoded. In the example of FIG. 8A, nine motion reference blocks A to D and TA to TE are selected from the encoded regions in the frame to be encoded and the reference frame. Specifically, from the frame to be encoded, four blocks A, B, C, and D adjacent to the left, upper, upper right, and upper left of the block to be encoded are selected as motion reference blocks, and from the reference frame, the block TA at the same position as the block to be encoded, and four pixel blocks TB, TC, TD, and TE adjacent to the right, lower, left, and upper sides of this block TA are selected as motion reference blocks. In the present embodiment, the motion reference blocks selected from the frame to be encoded are referred to as spatial-direction motion reference blocks, and the motion reference blocks selected from the reference frame are referred to as temporal-direction motion reference blocks. The symbol p assigned to each motion reference block in FIG. 8A indicates the index of the motion reference block. This index is numbered in the order of the temporal-direction and spatial-direction motion reference blocks, but is not limited to this. As long as the indices do not overlap, they do not necessarily have to be in this order. For example, the temporal-direction and spatial-direction motion reference blocks may be numbered in a scattered order.

[0039] Note that the spatial-direction motion reference blocks are not limited to the example shown in FIG. 8A. As shown in FIG. 8B, they may be blocks (for example, macroblocks or submacroblocks, etc.) to which the pixels a, b, c, and d adjacent to the block to be encoded belong. In this case, the relative positions (dx, dy) from the upper left pixel e in the block to be encoded to each of the pixels a, b, c, and d are set as shown in FIG. 8C. Here, in the examples shown in FIGS. 8A and 8B, the macroblock is shown as an N×N pixel block.

[0040] Also, as shown in FIG. 8D, all the blocks A1 to A4, B1, B2, C, and D adjacent to the block to be encoded may be selected as spatial-direction motion reference blocks. In the example of FIG. 8D, the number of spatial-direction motion reference blocks is eight.

[0041] Furthermore, as shown in FIG. 8E, the time-direction motion reference blocks may partially overlap, and as shown in FIG. 8F, the blocks TA to TE may be arranged separately. In FIG. 8E, the overlapping portions of the time-direction motion reference blocks TA and TB are shown by hatching. Moreover, the time-direction motion reference blocks are not limited to the blocks at the position (Collocate position) corresponding to the block to be encoded and the blocks located around it, and may be blocks arranged at any position within the reference frame. For example, using the block within the reference frame specified by the position of the reference block and the motion information 18 of any one of the encoded blocks adjacent to the block to be encoded as the central block (for example, block TA), this central block and the blocks around it may be selected as time-direction motion reference blocks. Furthermore, the time-direction reference blocks do not necessarily have to be arranged at equal intervals from the central block.

[0042] In any of the above cases, if the number and position of the spatial-direction and time-direction motion reference blocks are determined in advance by the encoding device and the decoding device, the number and position of the motion reference blocks may be set in any way. Also, the size of the motion reference block does not necessarily have to be the same as the size of the block to be encoded. For example, as shown in FIG. 8D, the size of the motion reference block may be larger or smaller than the block to be encoded. Furthermore, the motion reference block is not limited to a square shape and may be set to an arbitrary shape such as a rectangular shape. Also, the motion reference block may be set to any size.

[0043] Also, the motion reference block and the available block may be arranged in only one of the time direction and the space direction. Further, according to the type of slice such as a P slice or a B slice, the motion reference block and the available block in the time direction may be arranged, or the motion reference block and the available block in the space direction may be arranged.

[0044] FIG. 9 shows a method in which the available block acquisition unit 109 selects an available block from among the motion reference blocks. The available block is a block to which motion information can be applied to the block to be encoded and has different motion information from each other. The available block acquisition unit 109 refers to the reference motion information 19 and determines whether each motion reference block is an available block according to the method shown in FIG. 9, and outputs the available block information 30.

[0045] As shown in FIG. 9, first, a motion reference block with an index p of zero is selected (S800). In the description of FIG. 9, it is assumed that the motion reference blocks are processed in order with the index p from 0 to M−1 (M indicates the number of motion reference blocks). Also, it will be described assuming that the available determination process for the motion reference blocks with the index p from 0 to p−1 has been completed and the index of the motion reference block to be determined whether it is available is p.

[0046] The available block acquisition unit 109 determines whether the motion reference block p has the motion information 18, that is, whether at least one motion vector is assigned (S801). If the motion reference block p does not have a motion vector, that is, if the time-direction motion reference block p is a block in an I slice that does not have motion information, or if all the small pixel blocks in the time-direction motion reference block p are intra-predicted and encoded, the process proceeds to step S805. In step S805, the motion reference block p is determined to be an unavailable block.

[0047] If the motion reference block p has motion information in step S801, the process proceeds to step S802. The available block acquisition unit 109 selects the motion reference block q (available block q) that has already been selected as an available block. Here, q is a value smaller than p. Subsequently, the available block acquisition unit 109 compares the motion information 18 of the motion reference block p with the motion information 18 of the available block q and determines whether they have the same motion information (S803). If it is determined that the motion information 18 of the motion reference block p is the same as the motion information 18 of the motion reference block q selected as the available block, the process proceeds to step S805, and the motion reference block p is determined to be an unavailable block.

[0048] For all available blocks q that satisfy q < p, if it is determined in step S803 that the motion information 18 of the motion reference block p is not the same as the motion information 18 of the available block q, the process proceeds to step S804. In step S804, the available block acquisition unit 109 determines that the motion reference block p is an available block.

[0049] When it is determined that the motion reference block p is an available block or an unavailable block, the available block acquisition unit 109 determines whether an availability determination has been performed for all motion reference blocks (S806). If there is a motion reference block for which the availability determination has not been performed, for example, if p < M - 1, the process proceeds to step S807. Subsequently, the available block acquisition unit 109 increments the index p by 1 (step S807) and executes steps S801 to S806 again. When the availability determination is performed for all motion reference blocks in step S806, the availability determination process ends.

[0050] By executing the above-described availability determination process, it is determined whether each motion reference block is an available block or an unavailable block. The available block acquisition unit 109 generates available block information 30 including information regarding the available blocks. In this way, by selecting available blocks from among the motion reference blocks, the amount of information regarding the available block information 30 can be reduced, and as a result, the amount of the encoded data 14 can be reduced.

[0051] An example of the result of executing the availability determination process for the motion reference blocks shown in FIG. 8A is shown in FIG. 10. In FIG. 10, two spatial-direction motion reference blocks (p = 0, 1) and two temporal-direction motion reference blocks (p = 5, 8) are determined to be available blocks. An example of the available block information 30 regarding the example of FIG. 10 is shown in FIG. 11. As shown in FIG. 11, the available block information 30 includes the index of the motion reference block, the availability, and the motion reference block name. In the example of FIG. 11, the indexes p = 0, 1, 5, 8 are available blocks, and the number of available blocks is 4. The prediction unit 101 selects one optimal available block from among these available blocks as a selected block, and outputs information (selected block information) 31 regarding the selected block. The selected block information 31 includes the number of available blocks and the index value of the selected available block. For example, when the number of available blocks is 4, the corresponding selected block information 31 is encoded by the variable-length encoding unit 104 using a code table in which the maximum entry is 4.

[0052] Note that in step S801 of FIG. 9, when at least one of the blocks in the temporal-direction motion reference block p is an intra-prediction-encoded block, the available block acquisition unit 109 may determine that the motion reference block p is an unavailable block. That is, it may be configured to proceed to step S802 only when all the blocks in the temporal-direction motion reference block p are encoded by inter prediction.

[0053] Figures 12A to 12E show an example where, in the comparison of the motion information 18 in step S803, it is determined that the motion information 18 of the motion reference block p and the motion information 18 of the available block q are the same. Figures 12A to 12E each show a plurality of hatched blocks and two white blocks. In Figures 12A to 12E, for simplicity of explanation, it is assumed that the motion information 18 of these two white blocks is compared without considering the hatched blocks. Let one of the two white blocks be the motion reference block p, and the other be the motion reference block q (available block q) that has already been determined to be available. Unless otherwise specified, either of the two white blocks may be the motion reference block p.

[0054] Figure 12A shows an example where both the motion reference block p and the available block q are blocks in the spatial direction. In the example of Figure 12A, if the motion information 18 of blocks A and B is the same, it is determined that the motion information 18 is the same. At this time, it is not necessary for the sizes of blocks A and B to be the same.

[0055] Figure 12B shows an example where one of the motion reference block p and the available block q is the block A in the spatial direction and the other is the block TB in the temporal direction. In Figure 12B, there is one block having motion information within the temporal block TB. If the motion information 18 of the temporal block TB and the motion information 18 of the spatial block A are the same, it is determined that the motion information 18 is the same. At this time, it is not necessary for the sizes of blocks A and TB to be the same.

[0056] FIG. 12C shows another example where one of the motion reference block p and the available block q is the spatial block A and the other is the temporal block TB. FIG. 12C shows a case where the temporal block TB is divided into a plurality of small blocks and there are a plurality of small blocks having the motion information 18. In the example of FIG. 12C, if all the blocks having the motion information 18 have the same motion information 18 and the motion information 18 is the same as the motion information 18 of the spatial block A, it is determined that the motion information 18 is the same. At this time, it is not necessary for the sizes of the blocks A and TB to be the same.

[0057] FIG. 12D shows an example where both the motion reference block p and the available block q are temporal blocks. In this case, if the motion information 18 of the blocks TB and TE is the same, it is determined that the motion information 18 is the same.

[0058] FIG. 12E shows another example where both the motion reference block p and the available block q are temporal blocks. FIG. 12E shows a case where the temporal blocks TB and TE are each divided into a plurality of small blocks and there are a plurality of small blocks having the motion information 18 respectively. In this case, the motion information 18 is compared for each small block within the block, and if the motion information 18 is the same for all the small blocks, it is determined that the motion information 18 of the block TB and the motion information 18 of the block TE are the same.

[0059] FIG. 12F shows still another example where both the motion reference block p and the available block q are temporal blocks. FIG. 12F shows a case where the temporal block TE is divided into a plurality of small blocks and there are a plurality of small blocks having the motion information 18 in the block TE. When all the motion information 18 of the block TE is the same motion information 18 and is the same as the motion information 18 of the block TD, it is determined that the motion information 18 of the blocks TD and TE is the same.

[0060] In this way, in step S803, it is determined whether the motion information 18 of the motion reference block p is the same as the motion information 18 of the available block q. In the examples of FIGS. 12A to 12F, the number of available blocks q compared with the motion reference block p is described as 1. However, when the number of available blocks q is two or more, the motion information 18 of the motion reference block p may be compared with the motion information 18 of each available block q. Further, when applying scaling described later, the motion information 18 after scaling becomes the motion information 18 in the above description.

[0061] Note that the determination that the motion information of the motion reference block p is the same as the motion information of the available block q is not limited to the case where each motion vector included in the motion information completely matches. For example, if the norm of the difference between two motion vectors is within a predetermined range, the motion information of the motion reference block p and the motion information of the available block q may be regarded as substantially the same.

[0062] FIG. 13 shows a more detailed configuration of the prediction unit 101. As described above, this prediction unit 101 takes the available block information 30, the reference motion information 19, and the reference image signal 17 as inputs and outputs a predicted image signal 11, motion information 18, and selected block information 31. As shown in FIG. 13, the motion information selection unit 118 includes a spatial direction motion information acquisition unit 110, a temporal direction motion information acquisition unit 111, and a motion information switching switch 112.

[0063] The spatial direction motion information acquisition unit 110 is input with the available block information 30 and the reference motion information 19 regarding the spatial direction motion reference block. The spatial direction motion information acquisition unit 110 outputs motion information 18A including the motion information of each available block located in the spatial direction and the index value of the available block. When the information shown in FIG. 11 is input as the available block information 30, the spatial direction motion information acquisition unit 110 generates two motion information outputs 18A, and each motion information output 18A includes the available block and the motion information 19 of this available block.

[0064] The time-direction motion information acquisition unit 111 receives the available block information 30 and the reference motion information 19 regarding the time-direction motion reference block. The time-direction motion information acquisition unit 111 outputs, as motion information 18B, the motion information 19 and the index value of the available block that the available time-direction motion reference block specified by the available block information 30 has. The time-direction motion reference block is divided into a plurality of small pixel blocks, and each small pixel block has motion information 19. As shown in FIG. 14, the motion information 18B output by the time-direction motion information acquisition unit 111 includes a group of motion information 19 that each small pixel block within the available block has. When the motion information 18B includes the group of motion information 19, motion compensation prediction can be performed on the encoding target block in units of small pixel blocks obtained by dividing the encoding target block. When the information shown in FIG. 11 is input as the available block information 30, the time-direction motion information acquisition unit 111 generates two motion information outputs 18B, and each motion information output includes the available block and the group of motion information 19 that this available block has.

[0065] Note that the time-direction motion information acquisition unit 111 may obtain the average value or representative value of the motion vectors included in the motion information 19 that each small pixel block has, and output the average value or representative value of the motion vectors as the motion information 18B.

[0066] The motion information switching switch 112 in FIG. 13 selects, based on the motion information 18A and 18B output from the spatial-direction motion information acquisition unit 110 and the time-direction motion information acquisition unit 111, an appropriate available block as the selected block, and outputs the motion information 18 (or the group of motion information 18) corresponding to the selected block to the motion compensation unit 113. Also, the motion information switching switch 112 outputs the selected block information 31 regarding the selected block. The selected block information 31 includes the index p or the name of the motion reference block, etc., and is also simply referred to as selection information. The selected block information 31 is not limited to the index p and the name of the motion reference block, and any information can be used as long as the position of the selected block can be specified.

[0067] The motion information switching switch 112 selects, as a selection block, an available block that minimizes the encoding cost derived by, for example, the cost formula shown in the following Equation 1. [Equation]

[0068] Here, J represents the encoding cost, and D represents the encoding distortion indicating the sum of squared errors between the input image signal 10 and the reference image signal 17. Also, R represents the amount of code estimated by pseudo-encoding, and λ represents the Lagrange undetermined coefficient determined by the quantization width or the like. Instead of Equation 1, the encoding cost J may be calculated using only the amount of code R or the encoding distortion D, or a cost function of Equation 1 may be created using an approximated value of the amount of code R or the encoding distortion D. Further, the encoding distortion D is not limited to the sum of squared errors, and may be the sum of absolute differences (SAD) of prediction errors. The amount of code R may use only the amount of code related to the motion information 18. Also, the example in which the available block that minimizes the encoding cost is selected as the selection block is not limited, and one available block having a value within a range equal to or greater than the smallest value of the encoding cost may be selected as the selection block.

[0069] The motion compensation unit 113 derives the position of the pixel block from which the reference image signal 17 is taken out as the predicted image signal 11 based on the motion information (or a group of motion information) included in the selection block selected by the motion information selection unit 118. When a group of motion information is input to the motion compensation unit 113, the motion compensation unit 113 divides the pixel block from which the reference image signal 17 is taken out as the predicted image signal 11 into small pixel blocks (for example, 4×4 pixel blocks), and applies the corresponding motion information 18 to each of these small pixel blocks to obtain the predicted image signal 11 from the reference image signal 17. The position of the block where the predicted image signal 11 is obtained is, for example, as shown in FIG. 4A, a position shifted in the spatial direction according to the motion vector 18a included in the motion information 18 from the small pixel block.

[0070] For the motion compensation process for the block to be symbolized, the same process as the motion compensation process of H.264 can be used. Here, as an example, the interpolation method with 1 / 4 pixel accuracy will be specifically described. In the interpolation with 1 / 4 pixel accuracy, when each component of the motion vector is a multiple of 4, the motion vector indicates an integer pixel position. Otherwise, the motion vector indicates a prediction position corresponding to an interpolation position with fractional accuracy.

Number

[0071] Here, x and y indicate the vertical and horizontal indices indicating the start position (for example, the upper left vertex) of the block to be predicted, x_pos and y_pos indicate the corresponding prediction positions of the reference image signal 17. (mv_x, mv_y) indicates a motion vector with 1 / 4 pixel accuracy. Next, for the calculated pixel position, a predicted pixel is generated by interpolation or interpolation processing of the corresponding pixel position of the reference image signal 17. In FIG. 15, an example of generating a predicted pixel in H.264 is shown. In FIG. 15, the squares (hatched squares) indicated by capital letters of the alphabet indicate pixels at integer positions, and the squares shown shaded indicate interpolated pixels at 1 / 2 pixel positions. Also, the white squares indicate interpolated pixels corresponding to 1 / 4 pixel positions. For example, in FIG. 15, the interpolation process for 1 / 2 pixels corresponding to the positions of the letters b and h is calculated by the following formula 3.

Number

[0072] Here, the alphabets (e.g., b, h, C1, etc.) shown in Equation 3 and the following Equation 4 indicate the pixel values of the pixels to which the same alphabets are assigned in FIG. 16. Also, ">>" indicates a right shift operation, and ">> 5" is equivalent to division by 32. That is, the interpolated pixel at the 1 / 2 pixel position is calculated using a 6-tap FIR (Finite Impulse Response) filter (tap coefficients: (1, -5, 20, 20, -5, 1) / 32).

[0073] Also, the interpolation process for the 1 / 4 pixels corresponding to the positions of alphabets a and d in FIG. 15 is calculated by the following Equation 4.

Equation

[0074] In this way, the interpolated pixel at the 1 / 4 pixel position is calculated using a 2-tap average value filter (tap coefficients: (1 / 2, 1 / 2)). The interpolation process for the 1 / 2 pixel corresponding to alphabet j existing in the middle of the four integer pixel positions is generated using both the vertical 6 taps and the horizontal 6 taps. For pixel positions other than those described, the interpolated pixel values are generated in the same way.

[0075] Note that the interpolation process is not limited to the examples of Equation 3 and Equation 4, and other interpolation coefficients may be used. Also, as the interpolation coefficients, fixed values given from the encoding control unit 150 may be used, or the interpolation coefficients may be optimized for each frame based on the aforementioned encoding cost, and the optimized interpolation coefficients may be used to generate them.

[0076] In addition, in this embodiment, although the motion reference block has been described in terms of the processing related to the motion vector block prediction processing in units of macro blocks (for example, 16×16 pixel blocks), the prediction processing may be executed not only in units of macro blocks but also in units of 16×8 pixel blocks, 8×16 pixel blocks, 8×8 pixel blocks, 8×4 pixel blocks, 4×8 pixel blocks, or 4×4 pixel blocks. In this case, the information regarding the motion vector block is derived in units of pixel blocks. Further, the above prediction processing may be performed in units larger than the 16×16 pixel block, such as units of 32×32 pixel blocks, 32×16 pixel blocks, 64×64 pixel blocks, etc.

[0077] When substituting the reference motion vector in the motion vector block as the motion vector of the small pixel block in the block to be coded, (A) a negative value (inverted vector) of the reference motion vector may be substituted, or (B) a weighted average value, median value, maximum value, or minimum value using the reference motion vector corresponding to the small block and the reference motion vectors adjacent to this reference motion vector may be substituted.

[0078] FIG. 16 schematically shows the operation of the prediction unit 101. As shown in FIG. 16, first, a reference frame (motion reference frame) including a time-direction reference motion block is acquired (step S1501). The motion reference frame is typically the reference frame with the smallest temporal distance from the frame to be encoded and is a reference frame in the past in terms of time. For example, the motion reference frame is the frame encoded immediately before the frame to be encoded. In other examples, any reference frame in which the motion information 18 is stored in the motion information memory 108 may be acquired as the motion reference frame. Next, the spatial-direction motion information acquisition unit 110 and the time-direction motion information acquisition unit 111 each acquire the available block information 30 output from the available block acquisition unit 109 (step S1502). Next, the motion information switching switch 112 selects one of the available blocks as a selected block, for example, according to Equation 1 (step S1503). Subsequently, the motion compensation unit 113 copies the motion information of the selected block to the block to be encoded (step S1504). At this time, when the selected block is a spatial-direction reference block, as shown in FIG. 17, the motion information 18 of this selected block is copied to the encoding reference block. Also, when the selected block is a time-direction reference block, the group of the motion information 18 of this selected block is copied to the block to be encoded together with the position information. Next, motion compensation is executed using the motion information 18 or the group of the motion information 18 copied by the motion compensation unit 113, and the predicted image signal 11 and the motion information 18 used for the motion compensation prediction are output.

[0079] FIG. 18 shows a more detailed configuration of the variable-length encoding unit 104. As shown in FIG. 18, the variable-length encoding unit 104 includes a parameter encoding unit 114, a transform coefficient encoding unit 115, a selected block encoding unit 116, and a multiplexing unit 117. The parameter encoding unit 114 encodes parameters necessary for decoding, such as prediction mode information, block size information, quantization parameter information, etc., excluding the transform coefficient information 13 and the selected block information 31, and generates encoded data 14A. The transform coefficient encoding unit 115 encodes the transform coefficient information 13 and generates encoded data 14B. Also, the selected block encoding unit 116 refers to the available block information 30, encodes the selected block information 31, and generates encoded data 14C.

[0080] When the available block information 30 includes, as shown in FIG. 19, an index and the availability of a motion reference block corresponding to the index, among a plurality of preset motion reference blocks, the unavailable motion reference blocks are excluded, and only the available motion reference blocks are converted into syntax (stds_idx). In FIG. 19, since 5 out of 9 motion reference blocks are unavailable, for the 4 motion reference blocks excluding these, the syntax stds_idx is assigned in order from 0. In this example, since the selected block information to be encoded is selected from 4 available blocks instead of 9, the amount of code to be assigned (number of bins) can be reduced on average.

[0081] FIG. 20 shows an example of a code table showing the syntax stds_idx and the binary information (bin) of the syntax stds_idx. As shown in FIG. 18, the smaller the number of available motion reference blocks, the smaller the average number of bins required for encoding the syntax stds_idx. For example, when the number of available blocks is 4, the syntax stds_idx can be represented in 3 bits or less. The binary information (bin) of the syntax stds_idx may be binarized so that all stds_idxs have the same number of bins for each number of available blocks, or may be binarized according to a binarization method determined by pre-learning. Also, a plurality of binarization methods may be prepared and selectively switched for each block to be encoded.

[0082] Entropy encoding (e.g., fixed-length encoding, Huffman encoding, or arithmetic encoding, etc.) can be applied to these encoding units 114, 115, 116, and the generated encoded data 14A, 14B, 14C are multiplexed and output by the multiplexing unit 117.

[0083] In this embodiment, an example is described assuming that a frame encoded one frame before the frame to be encoded is referred to as a reference frame. However, the motion vector in the reference motion information 19 of the selected block and the reference frame number are used to scale (or normalize) the motion vector, and the reference motion information 19 may be applied to the block to be encoded.

[0084] <, This scaling process will be specifically described with reference to FIG. 21. Tc shown in FIG. 21 indicates the time distance (POC (number indicating the display order) distance) between the frame to be encoded and the motion reference frame, and is calculated by the following Equation 5. Tr[i] shown in FIG. 21 indicates the time distance between the motion reference frame and the frame i referred to by the selected block, and is calculated by the following Equation 6.

Equation

[0085] Here, curPOC represents the POC (Picture Order Count) of the frame to be encoded, colPOC represents the POC of the motion reference frame, and refPOC represents the POC of frame i that the selected block references. Also, Clip(min, max, target) is a clip function that outputs min if target is less than min, outputs max if target is greater than max, and outputs target otherwise. Additionally, DiffPicOrderCnt(x, y) is a function that calculates the difference between two POCs.

[0086] Assuming the motion vector of the selected block is MVr = (MVr_x, MVr_y) and the motion vector applied to the block to be encoded is MV = (MV_x, MV_y), the motion vector MV is calculated by the following Equation 7.

Equation

[0087] Here, Abs(x) represents a function that extracts the absolute value of x. In this way, in the scaling of the motion vector, the motion vector MVr assigned to the selected block is converted into the motion vector MV between the frame to be encoded and the first motion reference frame.

[0088] Also, another example regarding the scaling of the motion vector will be described below. First, for each slice or frame, the scaling factor (DistScaleFactor[i]) is obtained for all possible temporal distances tr of the motion reference frame according to the following Equation 8. The number of scaling factors is equal to the number of frames that the selected block references, i.e., the number of reference frames.

Equation

[0089] Regarding the calculation of tx shown in Equation 8, it may be pre-tabulated.

[0090] When performing scaling for each block to be symbolized, the motion vector MV can be calculated using only multiplication, addition, and shift operations by using the following Equation 9.

Equation

[0091] When such scaling processing is performed, the scaled motion information 18 is applied to the processing of the available block acquisition unit 109 together with the prediction unit 101. When the scaling processing is performed, the reference frame referred to by the block to be coded becomes the motion reference frame.

[0092] FIG. 22 shows the syntax structure in the image coding unit 100. As shown in FIG. 22, the syntax mainly includes three parts, namely, the high-level syntax 901, the slice-level syntax 904, and the macroblock-level syntax 907. The high-level syntax 901 holds syntax information of upper layers above the slice. The slice-level syntax 904 holds information necessary for each slice, and the macroblock-level syntax 907 holds data required for each macroblock shown in FIGS. 7A to 7D.

[0093] Each part includes more detailed syntax. The high-level syntax 901 includes sequence and picture-level syntax such as sequence parameter set syntax 902 and picture parameter set syntax 903. The slice-level syntax 904 includes slice header syntax 905, slice data syntax 906, and the like. Further, the macroblock-level syntax 907 includes macroblock layer syntax 908, macroblock prediction syntax 909, and the like.

[0094] Figures 23A and 23B show examples of macroblock layer syntax. The available_block_num shown in Figures 23A and 23B indicates the number of available blocks. When this value is greater than 1, encoding of the selected block information is required. Further, stds_idx indicates the selected block information, and stds_idx is encoded using a code table corresponding to the number of available blocks described above.

[0095] Figure 23A shows the syntax when encoding the selected block information after mb_type. When the mode indicated by mb_type is a defined size or a defined mode (TARGET_MODE), and available_block_num is greater than 1, stds_idx is encoded. For example, when the motion information of the selected block is available, stds_idx is encoded when the block size is 64×64 pixels, 32×32 pixels, 16×16 pixels, or in the case of the direct mode.

[0096] Figure 23B shows the syntax when encoding the selected block information before mb_type. When available_block_num is greater than 1, stds_idx is encoded. Also, if available_block_num is 0, conventional motion compensation represented by H.264 is performed, so mb_type is encoded.

[0097] Note that syntax elements not defined in the present invention may be inserted between the rows of the tables shown in Figures 23A and 23B, or descriptions regarding other conditional branches may be included. Alternatively, the syntax table may be divided into multiple tables or integrated. Also, it is not necessarily required to use the same terms, and they may be arbitrarily changed depending on the form of use. Further, each syntax element described in the macroblock layer syntax may be changed as specified in the macroblock data syntax described later.

[0098] Also, it is possible to reduce the information of mb_type by using the information of stds_idx. FIG. 24A shows the mb_type and the code table corresponding to the mb_type at the time of B slice in H.264. N shown in FIG. 24A is a value representing the size of the block to be coded, such as 16, 32, 64, etc., and M is a value half of N. Therefore, when mb_type is 4 to 21, it indicates that the block to be coded is a rectangular block. Also, L0, L1, and Bi in FIG. 24A indicate unidirectional prediction (only in the List0 direction), unidirectional prediction (only in the List1 direction), and bidirectional prediction, respectively. When the block to be coded is a rectangular block, mb_type includes information indicating which of L0, L1, and Bi prediction has been performed for each of the two rectangular blocks within the block to be coded. Also, B_Sub means performing the above processing for each of the pixel blocks obtained by dividing the macroblock into four. For example, when the block to be coded is a 64×64 pixel macroblock, the block to be coded is further assigned and coded with mb_type for each of the four 32×32 pixel blocks obtained by dividing this macroblock into four.

[0099] Here, when the selected block indicated by stds_idx is Spatial Left (a pixel block adjacent to the left side of the block to be coded), since the motion information of the pixel block adjacent to the left side of the block to be coded is used as the motion information of the block to be coded, stds_idx has the same meaning as performing prediction on the block to be coded using the horizontally long rectangular blocks indicated by mb_type = 4, 6, 8, 10, 12, 14, 16, 18, 20 in FIG. 24A. Also, when the selected block indicated by stds_idx is Spatial Up, since the motion information adjacent to the upper side of the block to be coded is used as the motion information of the block to be coded, stds_idx has the same meaning as performing prediction using the vertically long rectangular blocks indicated by mb_type = 5, 7, 9, 11, 13, 15, 17, 19, 21 in FIG. 24A. Therefore, by using stds_idx, it is possible to create a code table with the columns of mb_type = 4 to 21 in FIG. 24A reduced as shown in FIG. 24B. Similarly, regarding the code table corresponding to mb_type and mb_type at the time of P slice in H.264 shown in FIG. 24C, it is also possible to create a code table with the number of mb_type reduced as shown in FIG. 24D.

[0100] Also, the information of stds_idx may be included in the information of mb_type and coded. FIG. 25A is a code table when the information of stds_idx is included in the information of mb_type, and shows an example of the code table corresponding to mb_type and mb_type in B slice. B_STDS_X (X = 0, 1, 2) in FIG. 25A indicates the mode corresponding to stds_idx, and B_STDS_X is added by the number of available blocks (in FIG. 25A, the number of available blocks is 3). Another example of mb_type regarding P slice is shown in FIG. 25B. The description of FIG. 25B is omitted because it is the same as that of B slice.

[0101] The order of mb_type and the binarization method (binning) are not limited to the examples shown in FIGS. 25A and 25B, and mb_type may be encoded according to other orders and binarization methods. B_STDS_X and P_STDS_X do not have to be continuous and may be arranged between each mb_type. Also, the binarization method (binning) may be designed based on a pre-learned selection frequency.

[0102] In this embodiment, the present invention is also applicable to an extended macroblock that performs motion compensation prediction on a plurality of macroblocks together. Also, in this embodiment, the encoding scan order may be in any order. For example, the present invention is applicable to line scan or Z scan as well.

[0103] As described above, the image encoding apparatus according to this embodiment selects available blocks from a plurality of motion reference blocks, generates information for specifying motion reference blocks to be applied to an encoding target block according to the number of selected available blocks, and encodes this information. Therefore, according to the image encoding apparatus according to this embodiment, while reducing the amount of code related to motion vector information, motion compensation can be performed in units of small pixel blocks finer than the encoding target block, so that high encoding efficiency can be realized.

[0104] (Second Embodiment) FIG. 26 shows an image encoding apparatus according to a second embodiment of the present invention. In the second embodiment, differences and operations different from those in the first embodiment will be mainly described. As shown in FIG. 26, the image encoding unit 200 according to the present embodiment has different configurations of the prediction unit 201 and the variable length encoding unit 204 from those in the first embodiment. As shown in FIG. 27, the prediction unit 201 includes a first prediction unit 101 and a second prediction unit 202, and selectively switches between these first and second prediction units 101 and 202 to generate a predicted image signal 11. The first prediction unit 101 has the same configuration as the prediction unit 101 (FIG. 1) according to the first embodiment, and generates a predicted image signal 11 according to a prediction method (first prediction method) of performing motion compensation using the motion information 18 included in the selection block. The second prediction unit 202 generates a predicted image signal 11 according to a prediction method (second prediction method) such as H.264 of performing motion compensation using one motion vector for the block to be encoded. The second prediction unit 202 uses the input image signal 10 and the reference image signal 17 from the frame memory to generate a predicted image signal 11B.

[0105] FIG. 28 schematically shows the configuration of the second prediction unit 202. As shown in FIG. 28, the second prediction unit 202 includes a motion information acquisition unit 205 that generates motion information 21 using the input image signal 10 and the reference image signal 17, and a motion compensation unit 113 (FIG. 1) that generates a predicted image signal 11A using the reference image signal 17 and the motion information 21. The motion information acquisition unit 205 obtains a motion vector to be assigned to the block to be encoded, for example, by block matching, based on the input image signal 10 and the reference image signal 17. As an evaluation criterion for matching, a value obtained by accumulating the differences between the input image signal 10 and the interpolated image after matching for each pixel is used.

[0106] Note that the motion information acquisition unit 205 may determine an optimal motion vector using the value obtained by converting the difference between the predicted image signal 11 and the input image signal 10. Also, an optimal motion vector may be determined in consideration of the magnitude of the motion vector, as well as the amount of signs of the motion vector and the reference frame number, or by using Equation 1. The matching method may be executed based on the search range information provided from outside the image encoding device, or may be executed hierarchically for each pixel accuracy. Also, without performing the search process, the motion information given by the encoding control unit 150 may be used as the output 21 of the motion information acquisition unit 205.

[0107] The prediction unit 101 in FIG. 27 further includes a prediction method switching switch 203 that selects and outputs either the predicted image signal 11A from the first prediction unit 101 or the predicted image signal 11B from the second prediction unit 202. For example, for each of the predicted image signals 11A and 11B, the prediction method switching switch 203 obtains an encoding cost using the input image signal 10, for example, according to Equation 1, and selects either the predicted image signal 11A or 11B so that the encoding cost becomes smaller and outputs it as the predicted image signal 11. Further, the prediction method switching switch 203 also outputs prediction switching information 32 indicating whether the output predicted image signal 11 is output from either the first prediction unit 101 or the second prediction unit 202, together with the motion information 18 and the selection block information 31. The output motion information 18 is encoded by the variable length encoding unit 204 and then multiplexed into the encoded data 14.

[0108] FIG. 29 schematically shows the configuration of the variable length encoding unit 204. The variable length encoding unit 204 shown in FIG. 29 includes a motion information encoding unit 217 in addition to the configuration of the variable length encoding unit 104 shown in FIG. 18. Further, the selection block encoding unit 216 in FIG. 29 encodes the prediction switching information 32 to generate encoded data 14D, unlike the selection block encoding unit 116 in FIG. 18. When the first prediction unit 101 executes the prediction process, the selection block encoding unit 216 further encodes the available block information 30 and the selection block information 31. The encoded available block information 30 and selection block information 31 are included in the encoded data 14D. When the second prediction unit 202 executes the prediction process, the motion information encoding unit 217 encodes the motion information 18 to generate encoded data 14E. The selection block encoding unit 216 and the motion information encoding unit 217 determine which of the first prediction unit 101 and the second prediction unit 202 has executed the prediction process based on the prediction switching information 32 indicating whether the predicted image is generated by motion compensation prediction using the motion information of the selection block, respectively.

[0109] The multiplexing unit 117 receives the encoded data 14A, B, D, and E from the parameter encoding unit 114, the transform coefficient encoding unit 115, the selection block encoding unit 216, and the motion information encoding unit, and multiplexes the received encoded data 14A, B, D, and E.

[0110] Figures 30A and 30B each show an example of the macroblock layer syntax according to this embodiment. The available_block_num shown in Figure 30A indicates the number of available blocks. When this value is greater than 1, the selected block encoding unit 216 encodes the selected block information 31. Also, the stds_flag is a flag indicating whether the motion information of the selected block is used as the motion information of the block to be encoded in motion compensation prediction, that is, a flag indicating which of the first prediction unit 101 and the second prediction unit 202 the prediction method switching switch 203 selects. When the number of available blocks is greater than 1 and the stds_flag is 1, it indicates that the motion information of the selected block is used in motion compensation prediction. When the stds_flag is 0, without using the motion information of the selected block, the information of the motion information 18 or the predicted difference value is encoded in the same manner as H.264. Furthermore, the stds_idx indicates the selected block information, and the code table according to the number of available blocks is as described above.

[0111] Figure 30A shows the syntax when encoding the selected block information after the mb_type. The stds_flag and stds_idx are encoded only when the mode indicated by the mb_type is a defined size or a defined mode. For example, the stds_flag and stds_idx are encoded when the motion information of the selected block is available in the case of a block size of 64×64, 32×32, 16×16, or in the case of the direct mode.

[0112] Figure 30B shows the syntax when encoding the selected block information before the mb_type. For example, when the stds_flag is 1, the mb_type does not need to be encoded. When the stds_flag is 0, the mb_type is encoded.

[0113] As described above, the image encoding apparatus according to the second embodiment selectively switches between the first prediction unit 101 according to the first embodiment and the second prediction unit 202 that uses a prediction method such as H.264 so as to reduce the encoding cost, and compresses and encodes the input image signal. Therefore, in the image encoding apparatus according to the second embodiment, the encoding efficiency is improved as compared with the image encoding apparatus of the first embodiment.

[0114] (Third Embodiment) FIG. 31 schematically shows an image decoding apparatus according to the third embodiment. As shown in FIG. 31, this image decoding apparatus includes an image decoding unit 300, a decoding control unit 350, and an output buffer 308. The image decoding unit 300 is controlled by the decoding control unit 350. The image decoding apparatus according to the third embodiment corresponds to the image encoding apparatus according to the first embodiment. That is, the decoding process by the image decoding apparatus in FIG. 31 has a complementary relationship with the encoding process by the image encoding process in FIG. 1. The image decoding apparatus in FIG. 31 may be realized by hardware such as an LSI chip, or may be realized by causing a computer to execute an image decoding program.

[0115] The image decoding apparatus in FIG. 31 includes a coded sequence decoding unit 301, an inverse quantization / inverse transformation unit 302, an adder 303, a frame memory 304, a prediction unit 305, a motion information memory 306, and an available block acquisition unit 307. In the image decoding unit 300, the coded data 80 from an accumulation system or a transmission system (not shown) is input to the coded sequence decoding unit 301. This coded data 80 corresponds to, for example, the coded data 14 transmitted in a multiplexed state from the image encoding apparatus in FIG. 1.

[0116] In the present embodiment, a pixel block to be decoded (for example, a macroblock) is simply referred to as a block to be decoded. Also, an image frame including the block to be decoded is referred to as a frame to be decoded.

[0117] In the symbol sequence decoding unit 301, for each frame or field, decoding by syntax analysis is performed based on the syntax. Specifically, the symbol sequence decoding unit 301 sequentially performs variable-length decoding on the symbol sequences of each syntax, and decodes encoding parameters related to the block to be decoded, including the transform coefficient information 33, the selected block information 61, and prediction information such as block size information and prediction mode information.

[0118] In this embodiment, the decoding parameters include the transform coefficient 33, the selected block information 61, and the prediction information, and include all parameters necessary for decoding, such as information related to the transform coefficient and information related to quantization. The prediction information, the information related to the transform coefficient, and the information related to quantization are input to the decoding control unit 350 as control information 71. The decoding control unit 350 provides decoding control information 70 including parameters necessary for decoding, such as prediction information and quantization parameters, to each part of the image decoding unit 300.

[0119] Furthermore, as will be described later, the symbol sequence decoding unit 301 simultaneously decodes the encoded data 80 to obtain the prediction information and the selected block information 61. The motion information 38 including the motion vector and the reference frame number does not have to be decoded.

[0120] The transform coefficient 33 decoded by the symbol sequence decoding unit 301 is sent to the inverse quantization - inverse transform unit 302. Various information related to quantization decoded by the symbol sequence decoding unit 301, that is, the quantization parameter and the quantization matrix, is given to the decoding control unit 350 and loaded into the inverse quantization - inverse transform unit 302 when performing inverse quantization. The inverse quantization - inverse transform unit 302 inverse quantizes the transform coefficient 33 according to the loaded information related to quantization, and then performs an inverse transform process (for example, inverse discrete cosine transform, etc.) to obtain the prediction error signal 34. The inverse transform process by the inverse quantization - inverse transform unit 302 in FIG. 31 is the inverse transform of the transform process by the transform - quantization unit in FIG. 1. For example, when wavelet transform is performed by the image encoding device (FIG. 1), the inverse quantization - inverse transform unit 302 performs the corresponding inverse quantization and inverse wavelet transform.

[0121] The prediction error signal 34 restored by the inverse quantization and inverse transformation unit 302 is input to the adder 303. The adder 303 adds the prediction error signal 34 to the prediction image signal 35 generated by the prediction unit 305 described later to generate a decoded image signal 36. The generated decoded image signal 36 is output from the image decoding unit 300, temporarily stored in the output buffer 308, and then output according to the output timing managed by the decoding control unit 350. Also, this decoded image signal 36 is stored in the frame memory 304 as a reference image signal 37. The reference image signal 37 is sequentially read from the frame memory 304 frame by frame or field by field and input to the prediction unit 305.

[0122] The available block acquisition unit 307 receives the reference motion information 39 from the motion information memory 306 described later and outputs the available block information 60. The operation of the available block acquisition unit 307 is the same as that of the available block acquisition unit 109 (FIG. 1) described in the first embodiment.

[0123] The motion information memory 306 receives the motion information 38 from the prediction unit 305 and temporarily stores it as the reference motion information 39. The motion information memory 306 temporarily stores the motion information 38 output from the prediction unit 305 as the reference motion information 39. FIG. 4 shows an example of the motion information memory 306. The motion information memory 306 holds a plurality of motion information frames 26 with different encoding times. The decoded motion information 38 or the group of motion information 38 is stored in the motion information frame 26 corresponding to the decoding time as the reference motion information 39. In the motion information frame 26, the reference motion information 39 is stored, for example, in units of 4×4 pixel blocks. The reference motion information 39 held by the motion information memory 306 is read and referred to by the prediction unit 305 when generating the motion information 38 of the block to be decoded.

[0124] Next, the motion reference block and available block according to this embodiment will be described. The motion reference block is a candidate block selected from the already decoded area according to a method predetermined by the aforementioned image encoding device and image decoding device. FIG. 8A shows an example regarding the available block. In FIG. 8A, a total of nine motion reference blocks, i.e., four motion reference blocks in the frame to be decoded and five motion reference blocks in the reference frame, are arranged. The motion reference blocks A, B, C, D in the frame to be decoded in FIG. 8A are blocks adjacent to the block to be decoded on the left, above, upper right, and upper left. In this embodiment, the motion reference block selected from the frame to be decoded including the block to be decoded is referred to as a spatial-direction motion reference block. Also, the motion reference block TA in the reference frame is a pixel block at the same position as the block to be decoded in the reference frame, and pixel blocks TB, TC, TD, TE adjacent to this motion reference block TA are selected as motion reference blocks. The motion reference block selected from the pixel blocks in the reference frame is referred to as a temporal-direction motion reference block. Also, the frame in which the temporal-direction motion reference block is located is referred to as a motion reference frame.

[0125] The spatial-direction motion reference block is not limited to the example shown in FIG. 8A. As shown in FIG. 8B, pixel blocks to which pixels a, b, c, d adjacent to the block to be decoded belong may be selected as the spatial-direction motion reference block. In this case, the relative positions (dx, dy) of the pixels a, b, c, d with respect to the upper left pixel in the block to be decoded are shown in FIG. 8C.

[0126] Also, as shown in FIG. 8D, all pixel blocks A1 to A4, B1, B2, C, D adjacent to the block to be decoded may be selected as the spatial-direction motion reference block. In FIG. 8D, the number of spatial-direction motion reference blocks is eight.

[0127] Also, as shown in FIG. 8E, the time-direction motion reference blocks TA to TE may partially overlap each other, or as shown in FIG. 8F, they may be separated from each other. Further, the time-direction motion reference blocks do not necessarily have to be located at the Collocate position block and its surroundings, and any pixel block within the motion reference frame may be used. For example, by using the motion information of already decoded blocks adjacent to the block to be decoded, the reference block indicated by the motion vector included in the motion information may be selected as the center of the motion reference block (for example, block TA). Furthermore, the time-direction reference blocks do not have to be arranged at equal intervals.

[0128] In the method of selecting the motion reference block as described above, as long as the information regarding the number and position of the spatial-direction and time-direction motion reference blocks is shared between both the image decoding apparatus and the image decoding apparatus, the motion reference block may be selected from any number and position. Also, the size of the motion reference block does not necessarily have to be the same as the size of the block to be decoded. For example, as shown in FIG. 8D, the size of the motion reference block may be larger than, smaller than, or of any size compared to the size of the block to be decoded. Further, the shape of the motion reference block is not limited to a square shape and may be a rectangular shape.

[0129] Next, available blocks will be described. An available block is a pixel block selected from among the motion reference blocks and is a pixel block to which motion information can be applied to the block to be decoded. Available blocks have different motion information from each other. The available blocks are selected by executing the available block determination process shown in FIG. 9 with respect to a total of nine motion reference blocks in the frame to be decoded and the reference frame as shown in, for example, FIG. 8A. FIG. 10 shows the result of executing the available block determination process shown in FIG. 9. In FIG. 10, the pixel blocks filled with diagonal lines indicate unavailable blocks, and the white blocks indicate available blocks. That is, a total of four blocks, two from among the spatial direction motion reference blocks and two from among the temporal direction motion reference blocks, are determined as available blocks. The motion information selection unit 314 in the prediction unit 305 selects one optimal available block from among these available blocks arranged in the temporal and spatial directions as the selected block according to the selected block information 61 received from the selected block decoding unit 323.

[0130] Next, the available block acquisition unit 307 will be described. The available block acquisition unit 307 has the same function as the available block acquisition unit 109 of the first embodiment, acquires the reference motion information 39 from the motion information memory 306, and outputs available block information 60 which is information indicating an available block or an unavailable block for each motion reference block.

[0131] The operation of the available block acquisition unit 307 will be described with reference to the flowchart of FIG. 9. First, the available block acquisition unit 307 determines whether the motion reference block (index p) has motion information (step S801). That is, in step S801, it is determined whether at least one small pixel block in the motion reference block p has motion information. If it is determined that the motion reference block p does not have motion information, that is, if the temporal motion reference block is a block in an I slice without motion information, or if all small pixel blocks in the temporal motion reference block have been intra-predicted and decoded, the process proceeds to step S805. In step S805, this motion reference block p is determined to be an unavailable block.

[0132] If it is determined in step S801 that the motion reference block p has motion information, the available block acquisition unit 307 selects a motion reference block q (referred to as the available block q) that has already been determined to be an available block (step S802). Here, q is a value smaller than p. Subsequently, for all q, the available block acquisition unit 307 compares the motion information of this motion reference block p with the motion information of the available block q to determine whether the motion reference block p has the same motion information as the available block q (S803). If the motion reference block p has the same motion vector as the available block q, the process proceeds to step S805. In step S805, this motion reference block p is determined to be an unavailable block by the available block acquisition unit 307. If the motion reference block p has different motion information from all available blocks q, in step S804, this motion reference block p is determined to be an available block by the available block acquisition unit 307.

[0133] By executing the above-described available block determination process for all motion reference blocks, it is determined whether each motion reference block is an available block or an unavailable block, and available block information 60 is generated. An example of the available block information 60 is shown in FIG. 11. As shown in FIG. 11, the available block information 60 includes the index p of the motion reference block and the availability. In FIG. 11, the available block information 60 indicates that the motion reference blocks with indices p of 0, 1, 5, and 8 are selected as available blocks, and the number of available blocks is 4.

[0134] Note that in step S801 of FIG. 9, if at least one of the blocks in the time-direction motion reference block p is an intra-predicted coded block, the available block acquisition unit 307 may determine the motion reference block p as an unavailable block. That is, it may be configured to proceed to step S802 only when all the blocks in the time-direction motion reference block p are encoded by inter prediction.

[0135] FIGS. 12A to 12E show an example in which it is determined that the motion information 38 of the motion reference block p and the motion information 38 of the available block q are the same in the comparison of the motion information 38 in step S803. FIGS. 12A to 12E each show a plurality of hatched blocks and two white blocks. In FIGS. 12A to 12E, for simplicity of explanation, it is assumed that the motion information 38 of these two white blocks is compared without considering the hatched blocks. Let one of the two white blocks be the motion reference block p, and the other be the motion reference block q (available block q) that has already been determined to be available. Unless otherwise specified, either of the two white blocks may be the motion reference block p.

[0136] FIG. 12A shows an example in which both the motion reference block p and the available block q are blocks in the spatial direction. In the example of FIG. 12A, if the motion information 38 of blocks A and B is the same, it is determined that the motion information 38 is the same. At this time, it is not necessary for blocks A and B to have the same size.

[0137] FIG. 12B shows an example in which one of the motion reference block p and the available block q is the spatial block A and the other is the temporal block TB. In FIG. 12B, there is one block having motion information within the temporal block TB. If the motion information 38 of the temporal block TB and the motion information 38 of the spatial block A are the same, it is determined that the motion information 38 is the same. At this time, it is not necessary for blocks A and TB to have the same size.

[0138] FIG. 12C shows another example in which one of the motion reference block p and the available block q is the spatial block A and the other is the temporal block TB. FIG. 12C shows a case where the temporal block TB is divided into a plurality of small blocks and there are a plurality of small blocks having the motion information 38. In the example of FIG. 12C, if all the blocks having the motion information 38 have the same motion information 38 and the motion information 38 is the same as the motion information 38 of the spatial block A, it is determined that the motion information 38 is the same. At this time, it is not necessary for blocks A and TB to have the same size.

[0139] FIG. 12D shows an example in which both the motion reference block p and the available block q are temporal blocks. In this case, if the motion information 38 of blocks TB and TE is the same, it is determined that the motion information 38 is the same.

[0140] FIG. 12E shows another example where both the motion reference block p and the available block q are blocks in the time direction. FIG. 12E shows a case where the time-direction blocks TB and TE are each divided into a plurality of small blocks, and there are a plurality of small blocks each having motion information 38. In this case, the motion information 38 is compared for each small block within the block, and if the motion information 38 is the same for all small blocks, it is determined that the motion information 38 of block TB and the motion information 38 of block TE are the same.

[0141] FIG. 12F shows yet another example where both the motion reference block p and the available block q are blocks in the time direction. FIG. 12F shows a case where the time-direction block TE is divided into a plurality of small blocks, and there are a plurality of small blocks having motion information 38 in block TE. When all the motion information 38 of block TE is the same motion information 38 and is the same as the motion information 38 that block TD has, it is determined that the motion information 38 of block TD and TE is the same.

[0142] In this way, in step S803, it is determined whether the motion information 38 of the motion reference block p and the motion information 38 of the available block q are the same. In the examples of FIGS. 12A to 12F, the number of available blocks q compared with the motion reference block p was described as 1, but when the number of available blocks q is 2 or more, the motion information 38 of the motion reference block p and the motion information 38 of each available block q may be compared. Also, when applying the scaling described later, the motion information 38 after scaling becomes the motion information 38 in the above description.

[0143] Note that the determination that the motion information of the motion reference block p and the motion information of the available block q are the same is not limited to the case where each motion vector included in the motion information completely matches. For example, if the norm of the difference between two motion vectors is within a predetermined range, the motion information of the motion reference block p and the motion information of the available block q may be regarded as being substantially the same.

[0144] FIG. 32 is a block diagram showing the encoded sequence decoding unit 301 in more detail. As shown in FIG. 32, the encoded sequence decoding unit 301 includes a separation unit 320 that separates the encoded data 80 into syntax units, a transform coefficient decoding unit 322 that decodes the transform coefficients, a selection block decoding unit 323 that decodes the selection block information, and a parameter decoding unit 321 that decodes parameters such as the prediction block size and quantization parameters.

[0145] The parameter decoding unit 321 receives the encoded data 80A including the prediction block size and quantization parameters from the separation unit, decodes the encoded data 80A, and generates the control information 71. The transform coefficient decoding unit 322 receives the encoded transform coefficients 80B from the separation unit 320, decodes the encoded transform coefficients 80B, and obtains the transform coefficient information 33. The selection block decoding unit 323 takes as inputs the encoded data 80C related to the selection block and the available block information 60, and outputs the selection block information 61. The input available block information 60 indicates the availability for each motion reference block, as shown in FIG. 11.

[0146] Next, referring to FIG. 33, the prediction unit 305 will be described in detail. As shown in FIG. 33, the prediction unit 305 includes a motion information selection unit 314 and a motion compensation unit 313. The motion information selection unit 314 includes a spatial direction motion information acquisition unit 310, a temporal direction motion information acquisition unit 311, and a motion information switching switch 312. The prediction unit 305 basically has the same configuration and function as the prediction unit 101 described in the first embodiment.

[0147] The prediction unit 305 takes the available block information 60, the selected block information 61, the reference motion information 39, and the reference image signal 37 as inputs, and outputs a predicted image signal 35 and motion information 38. The spatial direction motion information acquisition unit 310 and the temporal direction motion information acquisition unit 311 have the same functions as the spatial direction motion information acquisition unit 110 and the temporal direction motion information acquisition unit 111 described in the first embodiment, respectively. The spatial direction motion information acquisition unit 310 uses the available block information 60 and the reference motion information 39 to generate motion information 38A including the motion information and indexes of each available block located in the spatial direction. The temporal direction motion information acquisition unit 311 uses the available block information 60 and the reference motion information 39 to generate motion information (or a group of motion information) 38B including the motion information and indexes of each available block located in the temporal direction.

[0148] The motion information switching switch 312 selects one of the motion information 38A from the spatial direction motion information acquisition unit 310 and the motion information (or a group of motion information) 38B from the temporal direction motion information acquisition unit 311 according to the selected block information 61 to obtain the motion information 38. The selected motion information 38 is sent to the motion compensation unit 313 and the motion information memory 306. The motion compensation unit 313 performs motion compensation prediction in the same manner as the motion compensation unit 113 described in the first embodiment according to the selected motion information 38, and generates a predicted image signal 35.

[0149] In the scaling function of the motion vector of the motion compensation unit 313, since it is the same as that described in the first embodiment, the description is omitted.

[0150] Figure 22 shows the syntax structure in the image decoding unit 300. As shown in Figure 22, the syntax mainly includes three parts, namely, the high-level syntax 901, the slice-level syntax 904, and the macroblock-level syntax 907. The high-level syntax 901 holds the syntax information of the upper layer above the slice. The slice-level syntax 904 holds the information required for each slice, and the macroblock-level syntax 907 holds the data required for each macroblock shown in Figures 7A to 7D.

[0151] Each part includes more detailed syntax. The high-level syntax 901 includes sequence and picture-level syntax such as sequence parameter set syntax 902 and picture parameter set syntax 903. The slice-level syntax 904 includes slice header syntax 905 and slice data syntax 906, etc. Furthermore, the macroblock-level syntax 907 includes macroblock layer syntax 908 and macroblock prediction syntax 909, etc.

[0152] Figures 23A and 23B show an example of the macroblock layer syntax. The available_block_num shown in Figures 23A and 23B indicates the number of available blocks. When this value is greater than 1, decoding of the selected block information is required. Furthermore, stds_idx indicates the selected block information, and stds_idx is encoded using a code table corresponding to the above-mentioned number of available blocks.

[0153] FIG. 23A shows the syntax for decoding the selected block information after mb_type. When the prediction mode indicated by mb_type is in a defined size or a defined mode (TARGET_MODE), and when available_block_num is greater than 1, stds_idx is decoded. For example, when the motion information of the selected block is available, stds_idx is encoded when the block size is 64×64 pixels, 32×32 pixels, 16×16 pixels, or in the case of the direct mode.

[0154] FIG. 23B shows the syntax for decoding the selected block information before mb_type. When available_block_num is greater than 1, stds_idx is decoded. Also, if available_block_num is 0, conventional motion compensation represented by H.264 is performed, so mb_type is encoded.

[0155] Syntax elements not defined in the present invention may be inserted between the rows of the tables shown in FIGS. 23A and 23B, or descriptions regarding other conditional branches may be included. Alternatively, the syntax table may be divided into or integrated with a plurality of tables. Also, it is not necessarily required to use the same terms, and they may be arbitrarily changed depending on the form of use. Furthermore, each syntax element described in the macroblock layer syntax may be changed as specified in the macroblock data syntax described later.

[0156] As described above, the image decoding apparatus according to the present embodiment decodes the image encoded by the image encoding apparatus according to the aforementioned first embodiment. Therefore, the image decoding according to the present embodiment can reproduce a high-quality decoded image from relatively small encoded data.

[0157] (Fourth Embodiment) FIG. 34 schematically shows an image decoding apparatus according to a fourth embodiment. As shown in FIG. 34, the image decoding apparatus includes an image decoding unit 400, a decoding control unit 350, and an output buffer 308. The image decoding apparatus according to the fourth embodiment corresponds to the image encoding apparatus according to the second embodiment. In the fourth embodiment, differences and operations different from those in the third embodiment will be mainly described. As shown in FIG. 34, the image decoding unit 400 according to the present embodiment is different from that in the third embodiment in the encoded sequence decoding unit 401 and the prediction unit 405.

[0158] The prediction unit 405 in the present embodiment selectively switches between a prediction method (first prediction method) of performing motion compensation using the motion information of the selected block and a prediction method (second prediction method) of performing motion compensation using one motion vector for the block to be decoded, such as H.264, to generate a predicted image signal 35.

[0159] FIG. 35 is a block diagram showing the encoded sequence decoding unit 401 in more detail. The encoded sequence decoding unit 401 shown in FIG. 35 includes a motion information decoding unit 424 in addition to the configuration of the encoded sequence decoding unit 301 shown in FIG. 32. Further, the selected block decoding unit 423 shown in FIG. 35 is different from the selected block decoding unit 323 shown in FIG. 32, and decodes the encoded data 80C related to the selected block to obtain prediction switching information 62. The prediction switching information 62 indicates which of the first and second prediction methods the prediction unit 101 in the image encoding apparatus of FIG. 1 used. When the prediction switching information 62 indicates that the prediction unit 101 used the first prediction method, that is, when the block to be decoded is encoded by the first prediction method, the selected block decoding unit 423 decodes the selected block information in the encoded data 80C to obtain the selected block information 61. When the prediction switching information 62 indicates that the prediction unit 101 used the second prediction method, that is, when the block to be decoded is encoded by the second prediction method, the selected block decoding unit 423 does not decode the selected block information, and the motion information decoding unit 424 decodes the encoded motion information 80D to obtain the motion information 40.

[0160] FIG. 36 is a block diagram showing the prediction unit 405 in more detail. The prediction unit 405 shown in FIG. 34 includes a first prediction unit 305, a second prediction unit 410, and a prediction method switching switch 411. The second prediction unit 410 performs motion compensation prediction similar to that of the motion compensation unit 313 in FIG. 33 using the motion information 40 and the reference image signal 37 decoded by the coded sequence decoding unit 401, and generates a predicted image signal 35B. The first prediction unit 305 is the same as the prediction unit 305 described in the third embodiment and generates a predicted image signal 35A. Further, the prediction method switching switch 411 selects either the predicted image signal 35B from the second prediction unit 410 or the predicted image signal 35A from the first prediction unit 305 based on the prediction switching information 62, and outputs it as the predicted image signal 35 of the prediction unit 405. At the same time, the prediction method switching switch 411 sends the motion information used by the selected first prediction unit 305 or second prediction unit 410 to the motion information memory 306 as the motion information 38.

[0161] Next, the differences from the third embodiment in the syntax structure related to this embodiment will be mainly described.

[0162] Figures 30A and 30B each show an example of the macroblock layer syntax according to this embodiment. The available_block_num shown in Figure 30A indicates the number of available blocks. When this value is greater than 1, the selected block decoding unit 423 decodes the selected block information in the encoded data 80C. Also, the stds_flag is a flag indicating whether the motion information of the selected block has been used as the motion information of the block to be decoded in motion compensation prediction, that is, a flag indicating which of the first prediction unit 305 and the second prediction unit 410 the prediction method switching switch 411 has selected. When the number of available blocks is greater than 1 and the stds_flag is 1, it indicates that the motion information of the selected block has been used in motion compensation prediction. When the stds_flag is 0, without using the motion information of the selected block, the motion information or the difference value predicted directly is encoded in the same way as H.264. Furthermore, the stds_idx indicates the selected block information, and the code table according to the number of available blocks is as described above.

[0163] Figure 30A shows the syntax when decoding the selected block information after mb_type. The stds_flag and stds_idx are decoded only when the prediction mode indicated by mb_type is for a determined block size or a determined mode. For example, when the block size is 64×64, 32×32, or 16×16, or in the case of the direct mode, the stds_flag and stds_idx are decoded.

[0164] Figure 30B shows the syntax when decoding the selected block information before mb_type. For example, when the stds_flag is 1, mb_type does not need to be decoded. When the stds_flag is 0, mb_type is decoded.

[0165] As described above, the image decoding apparatus according to the present embodiment decodes an image encoded by the image encoding apparatus according to the second embodiment described above. Therefore, the image decoding according to the present embodiment can reproduce a high-quality decoded image from relatively small encoded data.

[0166] Note that the present invention is not limited to the above-described embodiments as they are, and at the implementation stage, the components can be modified and embodied without departing from the gist thereof. Also, various inventions can be formed by appropriately combining a plurality of components disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0167] As an example of this, the same effects can be obtained even if the first to fourth embodiments described above are modified as follows.

[0168] (1) In the first to fourth embodiments, the frame to be processed is divided into rectangular blocks such as 16×16 pixel blocks, and the case of encoding or decoding in the order from the pixel block in the upper left of the screen to the pixel block in the lower right as shown in FIG. 4 is taken as an example for explanation. However, the encoding or decoding order is not limited to this example. For example, the encoding or decoding order may be in the order from the lower right to the upper left of the screen, or in the order from the upper right to the lower left. Also, the encoding or decoding order may be in a spiral order from the central part to the peripheral part of the screen, or in the order from the peripheral part to the central part of the screen.

[0169] (2) In the first to fourth embodiments, the case where the luminance signal and the color difference signal are not divided and only one color signal component is limited is taken as an example for explanation. However, different prediction processes may be used for the luminance signal and the color difference signal, or the same prediction process may be used. When different prediction processes are used, the prediction method selected for the color difference signal is encoded / decoded in the same manner as the luminance signal.

[0170] Needless to say, the present invention can be similarly implemented with various modifications without departing from the gist of the invention.

Industrial Applicability

[0171] The image encoding / decoding method according to the present invention has industrial applicability because it can improve the encoding efficiency.

Explanation of Signs

[0172] 10…Input image signal, 11…Predicted image signal, 12…Prediction error image signal, 13…Quantization conversion coefficient, 14…Encoded data, 15…Decoded prediction error signal, 16…Local decoded image signal, 17…Reference image signal, 18…Motion information, 20…Bit stream, 21…Motion information, 25, 26…Information frame, 30…Available block information, 31…Selected block information, 32…Prediction switching information, 33…Conversion coefficient information, 34…Prediction error signal, 35…Predicted image signal, 36…Decoded image signal, 37…Reference image signal, 38…Motion information, 39…Reference motion information, 40…Motion information, 50…Encoding control information, 51…Feedback information, 60…Available block information, 61…Selected block information, 62…Prediction switching information, 70…Decoding control information, 71…Control information, 80…Encoded data, 100…Image encoding unit, 101…Prediction unit, 102…Subtractor, 103…Conversion / quantization unit, 104…Variable length encoding unit, 105…Inverse quantization / inverse conversion unit, 106…Adder, 107…Frame memory, 108…Information memory, 109…Available block acquisition unit, 110…Spatial direction motion information acquisition unit, 111…Temporal direction motion information acquisition unit, 112…Information switching switch, 113…Motion compensation unit, 114…Parameter encoding unit, 115…Conversion coefficient encoding unit, 116…Selected block encoding unit, 117…Multiplexing unit, 118…Motion information selection unit, 120…Output buffer, 150…Encoding control unit, 200…Image encoding unit, 201…Prediction unit, 202…Second prediction unit, 203…Prediction method switching switch, 204…Variable length encoding unit, 205…Motion information acquisition unit, 216…Selected block encoding unit, 217…Motion information encoding unit, 300…Image decoding unit, 301…Encoded sequence decoding unit, 301…Code sequence decoding unit, 302…Inverse quantization / inverse conversion unit, 303…Adder, 304…Frame memory, 305…Prediction unit, 306…Information memory, 307…Available block acquisition unit, 308…Output buffer, 310…Spatial direction motion information acquisition unit, 311…Temporal direction motion information acquisition unit, 312…Motion information switching switch, 313…Motion compensation unit, 314…Information selection unit, 320…Demultiplexing unit, 321…Parameter decoding unit, 322…Conversion coefficient decoding unit, 323…Selected block decoding unit, 350…Decoding control unit, 400…Image decoding unit, 401…Encoded sequence decoding unit, 405…Prediction unit, 410…Second prediction unit, 411…Prediction method switching switch, 423…Selected block decoding unit, 424…Information decoding unit901…High-level syntax, 902…Sequence parameter set syntax, 903…Picture parameter set syntax, 904…Slice level syntax, 905…Slice header syntax, 906…Slice data syntax, 907…Macroblock level syntax, 908…Macroblock layer syntax, 909…Macroblock prediction syntax.,

Claims

Claim 1 comprising a circuit for transmitting encoded data, wherein the encoded data includes identification information for identifying a selection block, the identification information is generated by determining, in a predetermined order according to the positional relationship, whether a plurality of candidate blocks in a predetermined positional relationship with respect to the target block are available, and selecting one of the candidate blocks determined to be available as the selection block; the plurality of candidate blocks includes a block adjacent to the upper left of the target block and a block adjacent to the upper of the target block, the determining includes determining the block adjacent to the upper left of the target block after determining the block adjacent to the upper of the target block, and determining that the candidate block is available when the candidate block has movement information that does not match the movement information corresponding to the candidate block that has already been determined to be available; a transmission device.

Images