Image coding method and image decoding method
The image encoding and decoding methods enhance encoding efficiency by selecting motion reference blocks and using a code table to encode selection information, addressing the limitations of existing video coding methods in motion vector calculation.
Patent Information
- Application Number
- JP2024137017
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2030-04-08
AI Technical Summary
The existing video coding methods, such as H.264, have limited freedom in calculating motion vectors, leading to increased code amounts due to the need to transmit selection information for selecting motion vectors, which affects encoding efficiency.
An image encoding method that selects at least one motion reference block, generates a predicted image using the motion information of the selected block, and encodes selection information using a predetermined code table, while an image decoding method decodes this information to generate the predicted image.
Improves encoding efficiency by reducing the amount of code related to motion vector selection information, allowing for more flexible motion vector calculation.
Smart Images

Figure 2024157009000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an encoding and decoding method for moving and still images. [Background technology]
[0002] In recent years, a video coding method with significantly improved coding efficiency has been recommended as ITU-T Rec. H.264 and ISO / IEC 14496-10 (hereinafter referred to as H.264) by the collaboration of ITU-T and ISO / IEC. In H.264, prediction processing, transformation processing, and entropy coding processing are performed in rectangular block units (for example, 16×16 pixel block units, 8×8 pixel block units, etc.). In the prediction processing, motion compensation is performed for a rectangular block to be coded (block to be coded) by referring to a frame (reference frame) that has already been coded, to perform a prediction in the time direction. In such motion compensation, it is necessary to code motion information including a motion vector as spatial shift information between the block to be coded and a block to be referenced in the reference frame, and send it to the decoding side. Furthermore, when motion compensation is performed using a plurality of reference frames, it is necessary to code the reference frame number together with the motion information. This may increase the amount of code related to the motion information and the reference frame number.
[0003] One example of a method for obtaining a motion vector in motion compensation prediction is a direct mode in which a motion vector to be assigned to a block to be coded is derived from a motion vector assigned to an already coded block, and a predicted image is generated based on the derived motion vector (see Patent Documents 1 and 2). In the direct mode, the amount of coding of motion information can be reduced because the motion vector is not coded. The direct mode is adopted in, for example, H.264 / AVC. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Patent No. 4020789 [Patent Document 2] U.S. Patent No. 7,233,621 Summary of the Invention [Problem to be solved by the invention]
[0005] In the direct mode, the motion vector of the block to be coded is predicted and generated by a fixed method of calculating the motion vector from the median value of the motion vectors of the blocks that are already coded and are adjacent to the block to be coded. Therefore, the degree of freedom in calculating the motion vector is low.
[0006] In order to increase the degree of freedom in calculating a motion vector, a method has been proposed in which one of a plurality of coded blocks is selected and the motion vector is assigned to the block to be coded. In this method, selection information for identifying the selected block must be constantly transmitted so that the decoding side can identify the selected coded block. Therefore, when determining the motion vector to be assigned to the block to be coded by selecting one of a plurality of coded blocks, there is a problem that the amount of code related to the selection information increases.
[0007] The present invention has been made to solve the above problems, and has an object to provide an image coding and decoding method with high coding efficiency. [Means for solving the problem]
[0008] An image encoding method according to one embodiment of the present invention includes a first step of selecting at least one motion reference block from among encoded pixel blocks having motion information; a second step of selecting at least one available block from the motion reference blocks, the available block having a candidate for motion information to be applied to a block to be encoded, the available block having different motion information; a third step of selecting one selected block from among the available blocks; a fourth step of generating a predicted image of the block to be encoded using the motion information of the selected block; a fifth step of encoding a prediction error between the predicted image and an original image; and a sixth step of encoding selection information that identifies the selected block by referring to a code table predetermined according to the number of available blocks.
[0009] An image decoding method according to another embodiment of the present invention includes a first step of selecting at least one motion reference block from among decoded pixel blocks having motion information; a second step of selecting at least one available block from the motion reference blocks, which is a pixel block having candidates for motion information to be applied to a block to be decoded, the available block having different motion information; a third step of obtaining selection information for identifying the selected block by decoding input encoded data with reference to a code table predetermined according to the number of the available blocks; a fourth step of selecting one selected block from the available blocks according to the selection information; a fifth step of generating a predicted image of the block to be decoded using the motion information of the selected block; a sixth step of decoding a prediction residual of the block to be decoded from the encoded data; and a seventh step of obtaining a decoded image from the predicted image and the prediction residual. Effect of the Invention
[0010] According to the present invention, it is possible to improve the coding efficiency. [Brief description of the drawings]
[0011] [Figure 1] 1 is a block diagram illustrating an outline of a configuration of an image encoding device according to a first embodiment. [Figure 2A] 2 is a diagram showing an example of the size of a macroblock, which is a processing unit for encoding in the image decoding section shown in FIG. 1; [Figure 2B] 1. FIG. 4 is a diagram showing another example of the size of a macroblock, which is a processing unit for encoding by the image decoding unit shown in FIG. [Diagram 3] 2 is a diagram showing an order in which the image encoding unit shown in FIG. 1 encodes pixel blocks in a frame to be encoded. [Figure 4] 2 is a diagram showing an example of a motion information frame held in a motion information memory shown in FIG. 1; [Diagram 5] 2 is a flowchart showing an example of a procedure for processing the input image signal of FIG. 1. [Figure 6A] 2 is a diagram showing an example of an inter prediction process executed by the motion compensation unit in FIG. 1. [Figure 6B] 1. FIG. 4 is a diagram showing another example of the inter prediction process executed by the motion compensation unit in FIG. [Figure 7A] FIG. 13 is a diagram showing an example of the size of a motion compensation block used in inter prediction processing. [Figure 7B] FIG. 11 is a diagram showing another example of the size of a motion compensation block used in inter prediction processing. [Figure 7C] FIG. 11 is a diagram showing yet another example of the size of a motion compensation block used in inter prediction processing. [Figure 7D] FIG. 11 is a diagram showing another example of the size of a motion compensation block used in inter prediction processing. [Figure 8A] FIG. 13 is a diagram showing an example of an arrangement of spatial and temporal motion reference blocks. [Figure 8B] A figure showing another example of the arrangement of spatial motion reference blocks. [Figure 8C] 8C is a diagram showing the relative position of a spatial motion reference block with respect to the encoding target block shown in FIG. 8B. [Figure 8D] A diagram showing another example of the arrangement of time-direction motion reference blocks. [Figure 8E] A figure showing yet another example of the arrangement of time-direction motion reference blocks. [Figure 8F] A figure showing yet another example of the arrangement of time-direction motion reference blocks. [Figure 9] 10 is a flowchart showing an example of a method in which an available block acquisition unit in FIG. 1 selects an available block from among motion reference blocks. [Figure 10] 10 is a diagram showing an example of available blocks selected from the motion reference blocks shown in FIG. 8 according to the method of FIG. 9. FIG. [Figure 11] 2 is a diagram showing an example of available block information output by an available block acquisition unit in FIG. 1; [Figure 12A] 2 is a diagram showing an example of a determination of identity of motion information between blocks by the available block acquisition unit in FIG. 1; [Figure 12B] 1. FIG. 4 is a diagram showing another example of the determination of identity of motion information between blocks by the available block acquisition unit in FIG. [Figure 12C] 1. FIG. 4 is a diagram showing yet another example of the determination of identity of motion information between blocks by the available block acquisition unit in FIG. [Figure 12D] 1. FIG. 4 is a diagram showing another example of the determination of identity of motion information between blocks by the available block acquisition unit in FIG. [Figure 12E] 1. FIG. 4 is a diagram showing yet another example of the determination of identity of motion information between blocks by the available block acquisition unit in FIG. [Figure 12F] 1. FIG. 4 is a diagram showing another example of the determination of identity of motion information between blocks by the available block acquisition unit in FIG. [Figure 13] 2 is a block diagram illustrating a schematic configuration of a prediction unit in FIG. 1. [Figure 14] 14 is a diagram showing a group of motion information output by the time direction motion information acquisition unit in FIG. 13. FIG. [Figure 15] 14 is an explanatory diagram illustrating an interpolation process with low pixel accuracy that can be used in the motion compensation process by the motion compensation unit in FIG. 13. [Figure 16]14 is a flowchart showing an example of the operation of the prediction unit in FIG. 13. [Figure 17] 14 is a diagram showing how the motion compensation unit in FIG. 13 copies motion information of a time-direction motion reference block to a current block to be coded. [Figure 18] 2 is a block diagram showing a schematic configuration of a variable-length coding unit in FIG. 1. [Figure 19] FIG. 11 is a diagram illustrating an example of generating syntax according to available block information. [Figure 20] FIG. 13 is a diagram showing an example of binarization of selected block information syntax corresponding to available block information. [Figure 21] FIG. 11 is an explanatory diagram illustrating scaling of motion information. [Figure 22] FIG. 2 is a diagram illustrating a syntax structure according to an embodiment. [Figure 23A] FIG. 2 is a diagram showing an example of a macroblock layer synctas according to the first embodiment. [Figure 23B] FIG. 11 is a diagram showing another example of a macroblock layer synctas according to the first embodiment. [Figure 24A] FIG. 13 is a diagram showing mb_type in the case of B slices in H.264 and a code table corresponding to the mb_type. [Figure 24B] FIG. 2 is a diagram illustrating an example of a code table according to the embodiment. [Figure 24C] FIG. 13 is a diagram showing mb_type in P slices in H.264 and a code table corresponding to the mb_type. [Figure 24D] FIG. 11 is a diagram showing another example of a code table according to the embodiment. [Figure 25A] FIG. 13 is a diagram showing an example of mb_type in a B slice and a code table corresponding to the mb_type according to an embodiment. [Figure 25B] FIG. 13 is a diagram showing another example of mb_type in a P slice and a code table corresponding to the mb_type according to an embodiment. [Figure 26] FIG. 11 is a block diagram illustrating an outline of the configuration of an image encoding device according to a second embodiment. [Figure 27]FIG. 27 is a block diagram illustrating a schematic configuration of a prediction unit in FIG. 26. [Figure 28] FIG. 28 is a block diagram illustrating a schematic configuration of a second prediction unit in FIG. 27. [Figure 29] 27 is a block diagram showing a schematic configuration of a variable-length coding unit in FIG. 26. [Figure 30A] FIG. 11 is a diagram illustrating an example of macroblock layer syntax according to the second embodiment. [Figure 30B] FIG. 11 is a diagram illustrating another example of macroblock layer syntax according to the second embodiment. [Diagram 31] FIG. 11 is a block diagram illustrating an image decoding device according to a third embodiment. [Diagram 32] 32 is a block diagram showing in more detail the coded sequence decoding unit shown in FIG. 31. FIG. [Diagram 33] FIG. 32 is a block diagram showing in more detail the prediction unit shown in FIG. 31. [Diagram 34] FIG. 13 is a block diagram illustrating an image decoding device according to a fourth embodiment. [Diagram 35] FIG. 34 is a block diagram showing in more detail the coded sequence decoding unit shown in FIG. 33. [Diagram 36] FIG. 34 is a block diagram showing in more detail the prediction unit shown in FIG. 33. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] Hereinafter, an image encoding and decoding method and device according to an embodiment of the present invention will be described with reference to the drawings as necessary. Note that in the following embodiments, parts with the same numbers perform similar operations, and redundant description will be omitted.
[0013] (First embodiment) Fig. 1 shows a schematic configuration of an image coding device according to a first embodiment of the present invention. As shown in Fig. 1, this image coding device includes an image coding unit 100, a coding control unit 150, and an output buffer 120. This image coding device may be realized by hardware such as an LSI chip, or may be realized by causing a computer to execute an image coding program.
[0014] An original image (input image signal) 10, which is a moving image or a still image, is input to the image encoding unit 100, for example, in units of pixel blocks obtained by dividing the original image. As will be described in detail later, the image encoding unit 100 compression-encodes the input image signal 10 to generate encoded data 14. The generated encoded data 14 is temporarily stored in an output buffer 120, and is sent to a storage system (storage media) or a transmission system (communication line) (not shown) at an output timing managed by an encoding control unit 150.
[0015] The coding control unit 150 controls the overall coding process of the image coding unit 100, such as feedback control of the amount of generated code, quantization control, prediction mode control, and entropy coding control. Specifically, the coding control unit 150 provides coding control information 50 to the image coding unit 100, and appropriately receives feedback information 51 from the image coding unit 100. The coding control information 50 includes prediction information, motion information 18, quantization parameter information, and the like. The prediction information includes prediction mode information and block size information. The motion information 18 includes a motion vector, a reference frame number, and a prediction direction (unidirectional prediction, bidirectional prediction). The quantization parameter information includes a quantization parameter such as a quantization width (quantization step size) and a quantization matrix. The feedback information 51 includes the amount of generated code by the image coding unit 100, and is used to determine, for example, a quantization parameter.
[0016] The image encoding unit 100 encodes the input image signal 10 in units of pixel blocks (e.g., macroblocks, sub-blocks, one pixel, etc.) obtained by dividing an original image. For this reason, the input image signal 10 is sequentially input to the image encoding unit 100 in units of pixel blocks obtained by dividing the original image. In this embodiment, the encoding process unit is a macroblock, and a pixel block (macroblock) to be encoded that corresponds to the input image signal 10 is simply referred to as a target block to be encoded. Also, an image frame including a target block to be encoded, i.e., an image frame to be encoded, is referred to as a target frame to be encoded.
[0017] Such a coding target block may be, for example, a 16×16 pixel block as shown in FIG. 2A, or a 64×64 pixel block as shown in FIG. 2B. The coding target block may be a 32×32 pixel block, an 8×8 pixel block, or the like. The shape of the macroblock is not limited to the square example shown in FIG. 2A and FIG. 2B, and may be set to any shape such as a rectangle. Furthermore, the processing unit is not limited to a pixel block such as a macroblock, and may be a frame or a field.
[0018] Note that the encoding process for each pixel block in the encoding target frame may be performed in any order. For ease of explanation, in this embodiment, it is assumed that the encoding process is performed on the pixel blocks row by row from the upper left pixel block to the lower right pixel block of the encoding target frame, that is, in raster scan order, as shown in Fig. 3.
[0019] The image coding unit 100 shown in FIG. 1 includes a prediction unit 101, a subtractor 102, a transformation and quantization unit 103, a variable length coding unit 104, an inverse quantization and inverse transformation unit 105, an adder 106, a frame memory 107, a motion information memory 108, and an available block acquisition unit 109.
[0020] In the image encoding unit 100, an input image signal 10 is input to a prediction unit 101 and a subtractor 102. The subtractor 102 receives the input image signal 10, and also receives a predicted image signal 11 from the prediction unit 101, which will be described later. The subtractor 102 calculates the difference between the input image signal 10 and the predicted image signal 11, and generates a prediction error image signal 12.
[0021] The transform / quantization unit 103 receives the prediction error image signal 12 from the subtractor 102, and performs a transform process on the received prediction error image signal 12 to generate transform coefficients. The transform process is, for example, an orthogonal transform such as a discrete cosine transform (DCT). In another embodiment, the transform / quantization unit 103 may generate transform coefficients using a method such as a wavelet transform and independent component analysis instead of the discrete cosine transform. Furthermore, the transform / quantization unit 103 quantizes the generated transform coefficients based on a quantization parameter provided by the encoding control unit 150. The quantized transform coefficients (transform coefficient information) 13 are output to the variable length coding unit 104 and the inverse quantization / inverse transform unit 105.
[0022] The inverse quantization and inverse transform unit 105 inverse quantizes the quantized transform coefficients 13 according to the quantization parameters provided by the encoding control unit 150, i.e., the same quantization parameters as those of the transform and quantization unit 103. Next, the inverse quantization and inverse transform unit 105 performs inverse transform on the inverse quantized transform coefficients to generate a decoded prediction error signal 15. The inverse transform process by the inverse quantization and inverse transform unit 105 corresponds to the inverse transform process of the transform process by the transform and quantization unit 103. For example, the inverse transform process is an inverse discrete cosine transform (IDCT) or an inverse wavelet transform.
[0023] The adder 106 receives the decoded prediction error signal 15 from the inverse quantization and inverse transform unit 105, and further receives the predicted image signal 11 from the prediction unit 101. The adder 106 adds the decoded prediction error signal 15 and the predicted image signal 11 to generate a locally decoded image signal 16. The generated locally decoded image signal 16 is stored in a frame memory 107 as a reference image signal 17. The reference image signal 17 stored in the frame memory 107 is read out and referred to by the prediction unit 101 when encoding a subsequent block to be encoded.
[0024] The prediction unit 101 receives a reference image signal 17 from the frame memory 107, and also receives available block information 30 from an available block acquisition unit 109, which will be described later. Furthermore, the prediction unit 101 receives reference motion information 19 from a motion information memory 108, which will be described later. The prediction unit 101 generates a predicted image signal 11, motion information 18, and selected block information 31 of a block to be coded, based on the reference image signal 17, the reference motion information 19, and the available block information 30. Specifically, the prediction unit 101 includes a motion information selection unit 118 that generates the motion information 18 and the selected block information 31 based on the available block information 30 and the reference motion information 19, and a motion compensation unit 113 that generates a predicted image signal 11 based on the motion information 18. The predicted image signal 11 is sent to the subtractor 102 and the adder 106. The motion information 18 is stored in the motion information memory 108 for the subsequent prediction process for the block to be coded. Furthermore, the selected block information 31 is sent to the variable length coding unit 104. The prediction unit 101 will be described in detail later.
[0025] In the motion information memory 108, the motion information 18 is temporarily stored as reference motion information 19. Fig. 4 shows an example of the configuration of the motion information memory 108. As shown in Fig. 4, the motion information memory 108 holds the reference motion information 19 on a frame-by-frame basis, and the reference motion information 19 forms a motion information frame 25. The motion information 18 relating to already-encoded blocks is sequentially input to the motion information memory 108, and as a result, the motion information memory 108 holds a plurality of motion information frames 25 with different encoding times.
[0026] The reference motion information 19 is held in the motion information frame 25 in predetermined block units (for example, 4×4 pixel block units). The motion vector block 28 shown in FIG. 4 indicates a pixel block of the same size as the coding target block, the available block, and the selected block, for example, a 16×16 pixel block. A motion vector is assigned to the motion vector block 28, for example, for each 4×4 pixel block. The inter prediction process using the motion vector block is called a motion vector block prediction process. The reference motion information 19 held by the motion information memory 108 is read out by the prediction unit 101 when generating the motion information 18. The motion information 18 held by the available block as described later refers to the reference motion information 19 held in the area in the motion information memory 108 where the available block is located.
[0027] Note that the motion information memory 108 is not limited to the example in which the reference motion information 19 is stored in units of 4×4 pixel blocks, and may store the reference motion information 19 in other pixel block units. For example, the pixel block unit related to the reference motion information 19 may be one pixel or a 2×2 pixel block. Furthermore, the shape of the pixel block related to the reference motion information 19 is not limited to the example of a square shape, and may be any shape.
[0028] The available block acquisition unit 109 in Fig. 1 acquires reference motion information 19 from the motion information memory 108, and selects an available block that can be used for prediction processing by the prediction unit 101 from a plurality of blocks that have already been coded, based on the acquired reference motion information 19. The selected available block is sent to the prediction unit 101 and the variable length coding unit 104 as available block information 30. A coded block that is a candidate for selecting an available block is called a motion reference block. Methods for selecting a motion reference block and an available block will be described in detail later.
[0029] The variable length coding unit 104 receives, in addition to the transform coefficient information 13, selected block information 31 from the prediction unit 101, coding parameters such as prediction information and quantization parameters from the coding control unit 150, and available block information 30 from the available block acquisition unit 109. The variable length coding unit 104 performs entropy coding (for example, equal length coding, Huffman coding, or arithmetic coding) on the quantized transform coefficients 13, selected block information 31, available block information 30, and coding parameters to generate coded data 14. The coding parameters include all parameters required for decoding, such as information on transform coefficients and information on quantization, in addition to the selected block information 31 and prediction information. The generated coded data 14 is temporarily stored in the output buffer 120 and sent to an accumulation system or a transmission system (not shown).
[0030] Fig. 5 shows a processing procedure for an input image signal 10. As shown in Fig. 5, first, a predicted image signal 11 is generated by a prediction unit 101 (step S501). In generating the predicted image signal 11 in step S501, one of available blocks described later is selected as a selected block, and the predicted image signal 11 is created using selected block information 31, motion information of the selected block, and a reference image signal 17. A subtractor 102 calculates the difference between the predicted image signal 11 and the input image signal 10, and a prediction error image signal 12 is generated (step S502).
[0031] Next, the prediction error image signal 12 is subjected to orthogonal transformation and quantization by the transformation and quantization unit 103, and transformation coefficient information 13 is generated (step S503). The transformation coefficient information 13 and the selected block information 31 are sent to the variable-length coding unit 104, and variable-length coding is performed, and coded data 14 is generated (step S504). Also, in step S504, the code table is switched in accordance with the selected block information 31 so that the code table has entries equal to the number of available blocks, and the selected block information 31 is variable-length coded. A bit stream 20 of coded data is sent to a storage system or a transmission path (not shown).
[0032] The transform coefficient information 13 generated in step S503 is inverse quantized and inverse transformed by the inverse quantization and inverse transform unit 105 to become a decoded prediction error signal 15 (step S505). The decoded prediction error signal 15 is added to the reference image signal 17 used in step S501 to become a locally decoded image signal 16 (step S506), which is stored in the frame memory 107 as a reference image signal (step S507).
[0033] Next, each component of the image encoding unit 100 described above will be described in more detail. The image encoding unit 100 of Fig. 1 is provided with a plurality of prediction modes, and each prediction mode has a different method of generating a predicted image signal 11 and a different motion compensation block size. Specifically, the method by which the prediction unit 101 generates the predicted image signal 11 can be broadly divided into intra-prediction (intra-frame prediction) for generating a predicted image using a reference image signal 17 related to a frame (or field) to be encoded, and inter-prediction (inter-frame prediction) for generating a predicted image using a reference image signal 17 related to one or more encoded reference frames (reference fields). The prediction unit 101 selectively switches between intra-prediction and inter-prediction to generate a predicted image signal 11 of a block to be encoded.
[0034] FIG. 6A shows an example of inter prediction by the motion compensation unit 113. In inter prediction, as shown in FIG. 6A, a predicted image signal 11 is generated using a reference image signal 17 related to a block 24, which is a block in a reference frame one frame before that has already been coded and is located at the same position as the block to be coded (also called a predicted block) 23, and is spatially shifted according to a motion vector 18a included in the motion information 18. That is, in generating the predicted image signal 11, a reference image signal 17 related to the block 24 in the reference frame, which is specified by the position (coordinates) of the block to be coded and the motion vector 18a included in the motion information 18, is used. In inter prediction, motion compensation with a small number of pixels (for example, 1 / 2 pixel accuracy or 1 / 4 pixel accuracy) is possible, and the value of an interpolated pixel is generated by performing a filtering process on the reference image signal 17. For example, in H.264, an interpolation process up to 1 / 4 pixel accuracy is possible for a luminance signal. When motion compensation with 1 / 4 pixel accuracy is performed, the amount of information of the motion information 18 is four times that of the integer pixel accuracy.
[0035] In addition, in the inter prediction, the example is not limited to the use of the reference frame of one frame before as shown in FIG. 6A, and any coded reference frame may be used as shown in FIG. 6B. When the reference image signal 17 related to a plurality of reference frames having different time positions is held, information indicating the time position of the reference image signal 17 from which the predicted image signal 11 is generated is represented by a reference frame number. The reference frame number is included in the motion information 18. The reference frame number can be changed in area units (pictures, blocks, etc.). That is, a different reference frame can be used for each pixel block. As an example, when a coded reference frame of one frame before is used for prediction, the reference frame number of this area is set to 0, and when a coded reference frame of two frames before is used for prediction, the reference frame number of this area is set to 1. As another example, when only one frame of the reference image signal 17 is held in the frame memory 107 (the number of reference frames is 1), the reference frame number is always set to 0.
[0036] Furthermore, in inter prediction, a block size suitable for a block to be coded can be selected from among a plurality of motion compensation blocks. That is, a block to be coded may be divided into a plurality of small pixel blocks, and motion compensation may be performed for each small pixel block. Figures 7A to 7C show the size of a motion compensation block in units of a macroblock, and Figure 7D shows the size of a motion compensation block in units of a subblock (a pixel block of 8x8 pixels or less). As shown in Figure 7A, when a block to be coded is 64x64 pixels, a 64x64 pixel block, a 64x32 pixel block, a 32x64 pixel block, or a 32x32 pixel block may be selected as a motion compensation block. Also, as shown in Figure 7B, when a block to be coded is 32x32 pixels, a 32x32 pixel block, a 32x16 pixel block, a 16x32 pixel block, or a 16x16 pixel block may be selected as a motion compensation block. Furthermore, as shown in Fig. 7C, when the current block to be coded is a 16x16 pixel block, the motion compensation block can be set to a 16x16 pixel block, a 16x8 pixel block, an 8x16 pixel block, an 8x8 pixel block, etc. Furthermore, as shown in Fig. 7D, when the current block to be coded is a 8x8 pixel block, an 8x4 pixel block, a 4x8 pixel block, a 4x4 pixel block, etc. can be selected as the motion compensation block.
[0037] As described above, since a small pixel block (e.g., a 4×4 pixel block) in a reference frame used for inter prediction has motion information 18, the shape and motion vector of an optimal motion compensation block can be used according to the local characteristics of the input image signal 10. Moreover, the macroblocks and sub-macroblocks in FIG. 7A to FIG. 7D can be arbitrarily combined. When the encoding target block is a 64×64 pixel block as shown in FIG. 7A, by selecting each block size shown in FIG. 7B for each of four 32×32 pixel blocks obtained by dividing the 64×64 pixel block, blocks of 64×64 to 16×16 pixels can be used hierarchically. Similarly, when block sizes up to those shown in FIG. 7D can be selected, block sizes of 64×64 to 4×4 can be used hierarchically.
[0038] Next, the motion reference block will be described with reference to FIGS. 8A to 8F. The motion reference block is selected from the coded area (block) in the coding target frame and the reference frame according to a method determined by both the image coding device of FIG. 1 and the image decoding device described later. FIG. 8A shows an example of the arrangement of the motion reference block selected according to the position of the coding target block. In the example of FIG. 8A, nine motion reference blocks A to D and TA to TE are selected from the coded area in the coding target frame and the reference frame. Specifically, from the coding target frame, four blocks A, B, C, and D adjacent to the left, top, top right, and top left of the coding target block are selected as the motion reference blocks, and from the reference frame, block TA at the same position as the coding target block and four pixel blocks TB, TC, TD, and TE adjacent to the right, bottom, left, and top of this block TA are selected as the motion reference blocks. In this embodiment, the motion reference block selected from the coding target frame is called a spatial motion reference block, and the motion reference block selected from the reference frame is called a temporal motion reference block. The symbol p given to each motion reference block in FIG. 8A indicates the index of the motion reference block. The indexes are numbered in the order of the motion reference blocks in the time direction and the space direction, but this is not limiting and the order does not necessarily have to be different as long as the indexes do not overlap. For example, the motion reference blocks in the time direction and the space direction may be numbered out of order.
[0039] The spatial motion reference block is not limited to the example shown in Fig. 8A, but may be a block (e.g., a macroblock or a sub-macroblock, etc.) to which pixels a, b, c, and d adjacent to the encoding target block belong, as shown in Fig. 8B. In this case, the relative positions (dx, dy) from the upper left pixel e in the encoding target block to each of the pixels a, b, c, and d are set as shown in Fig. 8C. Here, in the examples shown in Figs. 8A and 8B, the macroblock is shown as an NxN pixel block.
[0040] Also, as shown in Fig. 8D, all blocks A1 to A4, B1, B2, C, and D adjacent to the encoding target block may be selected as spatial motion reference blocks. In the example of Fig. 8D, the number of spatial motion reference blocks is 8.
[0041] Furthermore, the time direction motion reference blocks may be arranged such that the blocks TA to TE overlap each other as shown in FIG. 8E, or may be arranged apart from each other as shown in FIG. 8F. In FIG. 8E, the overlapping parts of the time direction motion reference blocks TA and TB are indicated by diagonal lines. Furthermore, the time direction motion reference block is not necessarily limited to a block at a position (Collocate position) corresponding to the encoding target block and a block located around it, but may be a block located at any position in the reference frame. For example, a block in the reference frame specified by the position of the reference block and the motion information 18 of any encoded block adjacent to the encoding target block may be set as a central block (e.g., block TA), and the central block and its surrounding blocks may be selected as time direction motion reference blocks. Furthermore, the time direction reference blocks may not be arranged at equal intervals from the central block.
[0042] In any of the above cases, as long as the number and positions of the spatial and temporal motion reference blocks are predefined in the encoding device and the decoding device, the number and positions of the motion reference blocks may be set in any way. In addition, the size of the motion reference block does not necessarily have to be the same as the size of the block to be encoded. For example, as shown in FIG. 8D, the size of the motion reference block may be larger or smaller than the block to be encoded. In addition, the motion reference block is not limited to a square shape, and may be set to any shape such as a rectangular shape. In addition, the motion reference block may be set to any size.
[0043] In addition, the motion reference block and the available block may be arranged in only one of the time direction and the space direction. In addition, the motion reference block and the available block in the time direction may be arranged according to the type of slice, such as P slice or B slice, or the motion reference block and the available block in the space direction may be arranged.
[0044] Fig. 9 shows a method in which the available block acquisition unit 109 selects an available block from among the motion reference blocks. The available blocks are blocks for which motion information can be applied to the block to be coded, and have different motion information from each other. The available block acquisition unit 109 refers to the reference motion information 19, and determines whether each of the motion reference blocks is an available block or not according to the method shown in Fig. 9, and outputs the available block information 30.
[0045] As shown in Fig. 9, first, a motion reference block with index p of zero is selected (S800). In the explanation of Fig. 9, it is assumed that motion reference blocks with index p of 0 to M-1 (M indicates the number of motion reference blocks) are processed in order. In addition, it is assumed that the usability determination process for motion reference blocks with index p of 0 to p-1 is completed, and the index of the motion reference block to be determined as to whether it is usable or not is p.
[0046] The available block acquisition unit 109 determines whether the motion reference block p has motion information 18, that is, whether at least one motion vector is assigned (S801). If the motion reference block p does not have a motion vector, that is, if the temporal motion reference block p is a block in an I-slice that does not have motion information, or if all the subpixel blocks in the temporal motion reference block p are intra-prediction coded, the process proceeds to step S805. In step S805, the motion reference block p is determined to be an unavailable block.
[0047] If the motion reference block p has motion information in step S801, the process proceeds to step S802. The available block acquisition unit 109 selects the motion reference block q (available block q) that has already been selected as an available block. Here, q is a value smaller than p. Subsequently, the available block acquisition unit 109 compares the motion information 18 of the motion reference block p with the motion information 18 of the available block q and determines whether they have the same motion information (S803). If it is determined that the motion information 18 of the motion reference block p is the same as the motion information 18 of the motion reference block q selected as the available block, the process proceeds to step S805, and the motion reference block p is determined to be an unavailable block.
[0048] For all available blocks q that satisfy q < p, if it is determined in step S803 that the motion information 18 of the motion reference block p is not the same as the motion information 18 of the available block q, the process proceeds to step S804. In step S804, the available block acquisition unit 109 determines that the motion reference block p is an available block.
[0049] When it is determined that the motion reference block p is an available block or an unavailable block, the available block acquisition unit 109 determines whether the availability determination has been executed for all motion reference blocks (S806). If there is a motion reference block for which the availability determination has not been executed, for example, if p < M - 1, the process proceeds to step S807. Subsequently, the available block acquisition unit 109 increments the index p by 1 (step S807) and executes steps S801 to S806 again. When the availability determination is executed for all motion reference blocks in step S806, the availability determination process ends.
[0050] By executing the above-mentioned availability determination process, it is determined whether each motion reference block is an available block or an unavailable block. The available block acquisition unit 109 generates available block information 30 including information on the available blocks. In this way, by selecting an available block from among the motion reference blocks, the amount of information on the available block information 30 is reduced, and as a result, the amount of encoded data 14 can be reduced.
[0051] FIG. 10 shows an example of a result of performing an availability determination process on the motion reference blocks shown in FIG. 8A. In FIG. 10, two spatial motion reference blocks (p=0,1) and two temporal motion reference blocks (p=5,8) are determined to be available blocks. FIG. 11 shows an example of available block information 30 related to the example of FIG. 10. As shown in FIG. 11, the available block information 30 includes the index, availability, and motion reference block name of the motion reference block. In the example of FIG. 11, the indexes p=0,1,5,8 are available blocks, and the number of available blocks is 4. The prediction unit 101 selects an optimal available block from these available blocks as a selected block, and outputs information (selected block information) 31 related to the selected block. The selected block information 31 includes the number of available blocks and the index value of the selected available block. For example, when the number of available blocks is 4, the corresponding selected block information 31 is coded by the variable-length coding unit 104 using a code table with a maximum entry of 4.
[0052] 9, if at least one of the blocks in the temporal motion reference block p is an intra-prediction coded block, the available block acquisition unit 109 may determine that the motion reference block p is an unavailable block. That is, the process may proceed to step S802 only when all the blocks in the temporal motion reference block p are coded by inter-prediction.
[0053] 12A to 12E show an example in which the motion information 18 of the motion reference block p and the motion information 18 of the available block q are determined to be the same in the comparison of the motion information 18 in step S803. Each of FIGS. 12A to 12E shows a plurality of blocks with diagonal lines and two white blocks. For ease of explanation, FIGS. 12A to 12E assume that the motion information 18 of these two white blocks is compared without considering the diagonal lines. One of the two white blocks is assumed to be the motion reference block p, and the other is assumed to be the motion reference block q (available block q) that has already been determined to be available. Unless otherwise specified, either of the two white blocks may be the motion reference block p.
[0054] Fig. 12A shows an example in which both the motion reference block p and the available block q are spatial blocks. In the example of Fig. 12A, if the motion information 18 of blocks A and B is the same, it is determined that the motion information 18 is the same. In this case, the sizes of blocks A and B do not need to be the same.
[0055] FIG. 12B shows an example in which one of the motion reference block p and the available block q is a spatial block A, and the other is a temporal block TB. In FIG. 12B, there is one block having motion information in the temporal block TB. If the motion information 18 of the temporal block TB is the same as the motion information 18 of the spatial block A, it is determined that the motion information 18 is the same. In this case, the sizes of the blocks A and TB do not need to be the same.
[0056] Fig. 12C shows another example where one of the motion reference block p and the available block q is a spatial block A, and the other is a temporal block TB. Fig. 12C shows a case where the temporal block TB is divided into a plurality of small blocks, and there are a plurality of small blocks having motion information 18. In the example of Fig. 12C, if all blocks having motion information 18 have the same motion information 18, and the motion information 18 is the same as the motion information 18 of the spatial block A, it is determined that the motion information 18 is the same. In this case, the sizes of the blocks A and TB do not need to be the same.
[0057] 12D shows an example in which the motion reference block p and the available block q are both blocks in the time direction. In this case, if the motion information 18 of blocks TB and TE is the same, it is determined that the motion information 18 is the same.
[0058] Fig. 12E shows another example in which the motion reference block p and the available block q are both time-direction blocks. Fig. 12E shows a case in which the time-direction blocks TB and TE are each divided into a plurality of small blocks, each of which has a plurality of small blocks having motion information 18. In this case, the motion information 18 is compared for each small block in the block, and if the motion information 18 is the same for all the small blocks, it is determined that the motion information 18 of the block TB and the motion information 18 of the block TE are the same.
[0059] Fig. 12F shows yet another example in which both the motion reference block p and the available block q are blocks in the time direction. Fig. 12F shows a case in which a block TE in the time direction is divided into a plurality of small blocks, and the block TE has a plurality of small blocks having motion information 18. If all the motion information 18 of the block TE is the same motion information 18 and is the same as the motion information 18 of the block TD, it is determined that the motion information 18 of the block TD and the TE is the same.
[0060] In this way, in step S803, it is determined whether the motion information 18 of the motion reference block p and the motion information 18 of the available block q are the same. In the examples of Figures 12A to 12F, the number of available blocks q to be compared with the motion reference block p is 1, but when the number of available blocks q is 2 or more, the motion information 18 of the motion reference block p may be compared with the motion information 18 of each available block q. In addition, when scaling, which will be described later, is applied, the motion information 18 after scaling becomes the motion information 18 described above.
[0061] In addition, the determination that the motion information of the motion reference block p and the motion information of the available block q are the same is not limited to the case where the motion vectors included in the motion information are completely the same. For example, if the norm of the difference between the two motion vectors is within a predetermined range, the motion information of the motion reference block p and the motion information of the available block q may be considered to be substantially the same.
[0062] Fig. 13 shows a more detailed configuration of the prediction unit 101. As described above, the prediction unit 101 receives the available block information 30, the reference motion information 19, and the reference image signal 17 as input, and outputs the predicted image signal 11, the motion information 18, and the selected block information 31. As shown in Fig. 13, the motion information selection unit 118 includes a spatial motion information acquisition unit 110, a temporal motion information acquisition unit 111, and a motion information changeover switch 112.
[0063] The spatial motion information acquisition unit 110 receives available block information 30 and reference motion information 19 related to spatial motion reference blocks. The spatial motion information acquisition unit 110 outputs motion information 18A including motion information of each available block located in the spatial direction and an index value of the available block. When the information shown in Fig. 11 is input as the available block information 30, the spatial motion information acquisition unit 110 generates two motion information outputs 18A, and each motion information output 18A includes an available block and motion information 19 of the available block.
[0064] The time-direction motion information acquisition unit 111 receives the available block information 30 and the reference motion information 19 related to the time-direction motion reference block. The time-direction motion information acquisition unit 111 outputs the motion information 19 of the available time-direction motion reference block specified by the available block information 30 and the index value of the available block as the motion information 18B. The time-direction motion reference block is divided into a plurality of small pixel blocks, and each small pixel block has the motion information 19. The motion information 18B output by the time-direction motion information acquisition unit 111 includes a group of the motion information 19 of each small pixel block in the available block, as shown in FIG. 14. When the motion information 18B includes a group of the motion information 19, it is possible to perform motion compensation prediction on the encoding target block in units of the small pixel blocks obtained by dividing the encoding target block. When the information shown in FIG. 11 is input as the available block information 30, the time-direction motion information acquisition unit 111 generates two motion information outputs 18B, and each motion information output includes an available block and a group of the motion information 19 of the available block.
[0065] The time direction motion information acquisition section 111 may obtain an average value or a representative value of the motion vectors included in the motion information 19 of each pixel sub-block, and output the average value or the representative value of the motion vectors as the motion information 18B.
[0066] The motion information changeover switch 112 in FIG. 13 selects one appropriate available block as a selected block based on the motion information 18A and 18B output from the spatial motion information acquisition unit 110 and the temporal motion information acquisition unit 111, and outputs the motion information 18 (or a group of motion information 18) corresponding to the selected block to the motion compensation unit 113. The motion information changeover switch 112 also outputs selected block information 31 regarding the selected block. The selected block information 31 includes the index p or the name of the motion reference block, and is also simply referred to as selection information. The selected block information 31 is not limited to the index p and the name of the motion reference block, and may be any information as long as it can identify the position of the selected block.
[0067] The motion information changeover switch 112 selects, as a selected block, an available block that has a minimum coding cost derived by, for example, the cost formula shown in the following formula 1.
number
[0068] Here, J indicates the coding cost, and D indicates the coding distortion representing the sum of squared errors between the input image signal 10 and the reference image signal 17. R indicates the amount of code estimated by provisional coding, and λ indicates the Lagrange undetermined coefficient determined by the quantization width or the like. Instead of Equation 1, the coding cost J may be calculated using only the amount of code R or the coding distortion D, or the cost function of Equation 1 may be created using an approximate value of the amount of code R or the coding distortion D. Furthermore, the coding distortion D is not limited to the sum of squared errors, and may be the sum of absolute differences (SAD) of prediction errors. The amount of code R may use only the amount of code related to the motion information 18. Furthermore, the present invention is not limited to an example in which an available block with the smallest coding cost is selected as the selected block, and one available block having a value within a certain range equal to or greater than the smallest coding cost may be selected as the selected block.
[0069] The motion compensation unit 113 derives the position of a pixel block from which the reference image signal 17 is extracted as the predicted image signal 11, based on the motion information (or a group of motion information) of the selected block selected by the motion information selection unit 118. When a group of motion information is input to the motion compensation unit 113, the motion compensation unit 113 divides the pixel block from which the reference image signal 17 is extracted as the predicted image signal 11 into small pixel blocks (e.g., 4×4 pixel blocks), and applies the corresponding motion information 18 to each of these small pixel blocks, thereby obtaining the predicted image signal 11 from the reference image signal 17. The position of the block from which the predicted image signal 11 is obtained is a position shifted in the spatial direction from the small pixel block according to the motion vector 18a included in the motion information 18, as shown in FIG. 4A, for example.
[0070] The motion compensation process for the block to be coded can be the same as that of H.264. Here, as an example, a 1 / 4 pixel precision interpolation method will be specifically described. In the 1 / 4 pixel precision interpolation, if each component of the motion vector is a multiple of 4, the motion vector points to an integer pixel position. Otherwise, the motion vector points to a predicted position corresponding to the fractional precision interpolation position.
number
[0071] Here, x and y indicate vertical and horizontal indexes indicating the start position (for example, the upper left vertex) of the block to be predicted, and x_pos and y_pos indicate the corresponding prediction position of the reference image signal 17. (mv_x, mv_y) indicate a motion vector with 1 / 4 pixel accuracy. Next, for the determined pixel position, a prediction pixel is generated by filling or interpolating the corresponding pixel position of the reference image signal 17. FIG. 15 shows an example of prediction pixel generation in H.264. In FIG. 15, squares (diagonally lined squares) indicated by capital alphabets indicate pixels at integer positions, and squares displayed in a shaded pattern indicate interpolated pixels at 1 / 2 pixel positions. Moreover, squares displayed in white indicate interpolated pixels corresponding to 1 / 4 pixel positions. For example, in FIG. 15, the 1 / 2 pixel interpolation process corresponding to the positions of the alphabets b and h is calculated by the following formula 3.
number
[0072] Here, the alphabets (e.g., b, h, C1, etc.) in Equation 3 and Equation 4 below indicate pixel values of pixels to which the same alphabets are assigned in FIG. 16. Also, ">>" indicates a right shift operation, and ">> 5" corresponds to division by 32. That is, the interpolated pixel at the 1 / 2 pixel position is calculated using a 6-tap FIR (Finite Impulse Response) filter (tap coefficients: (1, -5, 20, 20, -5, 1) / 32).
[0073] Furthermore, the 1 / 4 pixel interpolation process corresponding to the positions of the letters a and d in FIG. 15 is calculated using the following formula 4.
number
[0074] In this way, the interpolated pixel at the 1 / 4 pixel position is calculated using a 2-tap average filter (tap coefficients: (1 / 2, 1 / 2)). The interpolation process for the 1 / 2 pixel corresponding to the alphabet j located between the four integer pixel positions is generated using 6 taps in both the vertical direction and 6 taps in the horizontal direction. Interpolated pixel values are generated in a similar manner for pixel positions other than those described above.
[0075] The interpolation process may be performed using other interpolation coefficients, not limited to the examples of Formula 3 and Formula 4. The interpolation coefficients may be fixed values provided by the encoding control unit 150, or may be optimized for each frame based on the above-mentioned encoding cost, and may be generated using the optimized interpolation coefficients.
[0076] In addition, in this embodiment, the motion reference block is described as a macroblock (e.g., 16×16 pixel block) unit for the motion vector block prediction process, but the prediction process may be performed not only in macroblock units, but also in 16×8 pixel block units, 8×16 pixel block units, 8×8 pixel block units, 8×4 pixel block units, 4×8 pixel block units, or 4×4 pixel block units. In this case, information about the motion vector block is derived in pixel block units. In addition, the prediction process may be performed in units larger than 16×16 pixel blocks, such as 32×32 pixel block units, 32×16 pixel block units, or 64×64 pixel block units.
[0077] When substituting a reference motion vector in a motion vector block as a motion vector for a small pixel block in the block to be encoded, (A) the negative value (inverted vector) of the reference motion vector may be substituted, or (B) a weighted average value, median value, maximum value, or minimum value using the reference motion vector corresponding to the small block and the reference motion vectors adjacent to this reference motion vector may be substituted.
[0078] FIG. 16 shows an outline of the operation of the prediction unit 101. As shown in FIG. 16, first, a reference frame (motion reference frame) including a time-direction reference motion block is acquired (step S1501). The motion reference frame is typically a reference frame having the smallest temporal distance from the encoding target frame, and is a reference frame located in the past in time. For example, the motion reference frame is a frame encoded immediately before the encoding target frame. In another example, any reference frame whose motion information 18 is stored in the motion information memory 108 may be acquired as the motion reference frame. Next, the spatial motion information acquisition unit 110 and the time motion information acquisition unit 111 each acquire the available block information 30 output from the available block acquisition unit 109 (step S1502). Next, the motion information changeover switch 112 selects one of the available blocks as a selected block according to, for example, Equation 1 (step S1503). Next, the motion compensation unit 113 copies the motion information of the selected selected block to the encoding target block (step S1504). At this time, if the selected block is a spatial reference block, the motion information 18 of the selected block is copied to the coding reference block as shown in Fig. 17. Also, if the selected block is a temporal reference block, a group of motion information 18 of the selected block is copied to the coding target block together with position information. Next, motion compensation is performed using the motion information 18 or the group of motion information 18 copied by the motion compensation unit 113, and the predicted image signal 11 and the motion information 18 used for the motion compensation prediction are output.
[0079] FIG. 18 shows a more detailed configuration of the variable length coding unit 104. As shown in FIG. 18, the variable length coding unit 104 includes a parameter coding unit 114, a transform coefficient coding unit 115, a selection block coding unit 116, and a multiplexing unit 117. The parameter coding unit 114 codes parameters required for decoding, such as prediction mode information, block size information, and quantization parameter information, excluding the transform coefficient information 13 and the selection block information 31, to generate coded data 14A. The transform coefficient coding unit 115 codes the transform coefficient information 13 to generate coded data 14B. Moreover, the selection block coding unit 116 codes the selection block information 31 with reference to the available block information 30, to generate coded data 14C.
[0080] When the available block information 30 includes an index and the availability of a motion reference block corresponding to the index, as shown in Fig. 19, among a plurality of preset motion reference blocks, unavailable motion reference blocks are excluded, and only available motion reference blocks are converted into syntax (stds_idx). In Fig. 19, five of nine motion reference blocks are unavailable, so the syntax stds_idx is assigned to the four remaining motion reference blocks in order starting from 0. In this example, the selection block information to be coded is selected from four available blocks, not nine, so the amount of code to be assigned (the number of bins) can be reduced on average.
[0081] FIG. 20 shows an example of a code table showing the syntax stds_idx and binary information (bin) of the syntax stds_idx. As shown in FIG. 18, the fewer the number of available motion reference blocks, the lower the average number of bins required for encoding the syntax stds_idx. For example, when the number of available blocks is 4, the syntax stds_idx can be represented by 3 bits or less. The binary information (bin) of the syntax stds_idx may be binarized so that all stds_idx has the same number of bins for each number of available blocks, or may be binarized according to a binarization method determined by pre-learning. In addition, a plurality of binarization methods may be prepared and switched appropriately for each block to be encoded.
[0082] Entropy coding (e.g., equal length coding, Huffman coding, or arithmetic coding) can be applied to these coding units 114, 115, and 116, and the generated coded data 14A, 14B, and 14C are multiplexed by a multiplexing unit 117 and output.
[0083] In this embodiment, an example is described assuming that a frame coded one frame before the frame to be coded is referenced as a reference frame, but the motion vector may be scaled (or normalized) using the motion vector and reference frame number in the reference motion information 19 possessed by the selected block, and the reference motion information 19 may be applied to the block to be coded.
[0084] This scaling process will be specifically described with reference to Fig. 21. tc in Fig. 21 indicates the time distance (POC (number indicating display order) distance) between the encoding target frame and the motion reference frame, and is calculated by the following formula 5. tr[i] in Fig. 21 indicates the time distance between the motion reference frame and frame i referenced by the selected block, and is calculated by the following formula 6.
number
[0085] Here, curPOC indicates the POC (Picture Order Count) of the frame to be coded, colPOC indicates the POC of the motion reference frame, and refPOC indicates the POC of the frame i referenced by the selected block. Clip(min, max, target) is a clip function that outputs min when target is smaller than min, outputs max when target is larger than max, and outputs target otherwise. DiffPicOrderCnt(x, y) is a function that calculates the difference between two POCs.
[0086] If the motion vector of the selected block is MVr=(MVr_x, MVr_y) and the motion vector to be applied to the current block is MV=(MV_x, MV_y), the motion vector MV is calculated by the following Equation 7.
number
[0087] Here, Abs(x) is a function that extracts the absolute value of x. In this way, in the motion vector scaling, the motion vector MVr assigned to the selected block is converted into the motion vector MV between the encoding target frame and the motion first reference frame.
[0088] Other examples of motion vector scaling are also described below. First, for each slice or frame, a scaling factor (DistScaleFactor[i]) is calculated for all possible time distances tr of the motion reference frame according to the following formula 8. The number of scaling factors is equal to the number of frames referenced by the selected block, i.e., the number of reference frames.
number
[0089] The calculation of tx shown in Equation 8 may be made into a table in advance.
[0090] When scaling each block to be coded, the motion vector MV can be calculated using only multiplication, addition, and shift operations by using the following formula 9.
number
[0091] When such a scaling process is performed, the scaled motion information 18 is applied to the process of the prediction unit 101 and the available block acquisition unit 109. When the scaling process is performed, the reference frame referred to by the current block to be coded becomes a motion reference frame.
[0092] Fig. 22 shows a syntax structure in the image encoding unit 100. As shown in Fig. 22, the syntax mainly includes three parts, namely, a high level syntax 901, a slice level syntax 904, and a macroblock level syntax 907. The high level syntax 901 holds syntax information of layers higher than the slice level. The slice level syntax 904 holds information required for each slice, and the macroblock level syntax 907 holds data required for each macroblock shown in Figs. 7A to 7D.
[0093] Each part includes more detailed syntax. High level syntax 901 includes sequence and picture level syntax such as sequence parameter set syntax 902 and picture parameter set syntax 903. Slice level syntax 904 includes slice header syntax 905 and slice data syntax 906. Furthermore, macroblock level syntax 907 includes macroblock layer syntax 908 and macroblock prediction syntax 909.
[0094] Figures 23A and 23B show examples of macroblock layer syntax. available_block_num shown in Figures 23A and 23B indicates the number of available blocks, and if this value is greater than 1, it is necessary to encode the selected block information. Furthermore, stds_idx indicates the selected block information, and stds_idx is encoded using a code table according to the number of available blocks described above.
[0095] 23A shows the syntax for encoding selected block information after mb_type. When the mode indicated by mb_type is a specified size or a specified mode (TARGET_MODE) and available_block_num is a value greater than 1, stds_idx is encoded. For example, stds_idx is encoded when the motion information of the selected block is available for block sizes of 64x64 pixels, 32x32 pixels, or 16x16 pixels, or in direct mode.
[0096] Fig. 23B shows the syntax when the selection block information is coded before mb_type. When available_block_num is a value greater than 1, stds_idx is coded. Also, when available_block_num is 0, conventional motion compensation represented by H.264 is performed, so mb_type is coded.
[0097] In addition, between the rows of the tables shown in Figures 23A and 23B, syntax elements not specified in the present invention may be inserted, and descriptions related to other conditional branches may be included. Alternatively, the syntax table may be divided and integrated into multiple tables. Also, it is not necessary to use the same terminology, and it may be changed arbitrarily depending on the form of use. Furthermore, each syntax element described in the macroblock layer syntax may be changed as specified in the macroblock data syntax described later.
[0098] Also, by using information on stds_idx, it is possible to reduce information on mb_type. FIG. 24A shows mb_type in the case of B slice in H.264 and a code table corresponding to mb_type. N shown in FIG. 24A is a value indicating the size of the block to be coded, such as 16, 32, 64, and the like, and M is a value half of N. Therefore, when mb_type is 4 to 21, it indicates that the block to be coded is a rectangular block. Also, L0, L1, and Bi in FIG. 24A indicate unidirectional prediction (only List0 direction), unidirectional prediction (only List1 direction), and bidirectional prediction, respectively. When the block to be coded is a rectangular block, mb_type includes information indicating which of the predictions L0, L1, and Bi has been performed for each of the two rectangular blocks in the block to be coded. Also, B_Sub means that the above processing is performed for each of the pixel blocks obtained by dividing a macroblock into four. For example, if the current block to be coded is a 64×64 pixel macroblock, the macroblock is divided into four to obtain four 32×32 pixel blocks, each of which is further assigned an mb_type and coded.
[0099] Here, when the selected block indicated by stds_idx is Spatial Left (a pixel block adjacent to the left side of the encoding target block), the motion information of the pixel block adjacent to the left side of the encoding target block is set as the motion information of the encoding target block, so stds_idx has the same meaning as performing prediction on the encoding target block using the horizontally long rectangular blocks indicated by mb_type=4,6,8,10,12,14,16,18,20 in FIG. 24A. When the selected block indicated by stds_idx is Spatial Up, the motion information adjacent to the upper side of the encoding target block is set as the motion information of the encoding target block, so stds_idx has the same meaning as performing prediction using the vertically long rectangular blocks indicated by mb_type=5,7,9,11,13,15,17,19,21 in FIG. 24A. Therefore, by using stds_idx, it is possible to create a code table in which the columns of mb_type=4 to 21 in FIG. 24A are reduced as shown in FIG. 24B. Similarly, with regard to mb_type in P slices in H.264 shown in FIG. 24C and a code table corresponding to mb_type, it is possible to create a code table in which the number of mb_types is reduced as shown in FIG. 24D.
[0100] Also, the information on stds_idx may be included in the information on mb_type and coded. FIG. 25A shows an example of a code table when the information on stds_idx is included in the information on mb_type, and shows an example of a code table corresponding to mb_type and mb_type in a B slice. B_STDS_X (X=0, 1, 2) in FIG. 25A indicates a mode corresponding to stds_idx, and B_STDS_X is added by the number of available blocks (the number of available blocks is 3 in FIG. 25A). Similarly, another example of mb_type for a P slice is shown in FIG. 25B. The description of FIG. 25B is omitted because it is the same as that of a B slice.
[0101] The order of mb_type and the binarization method (binning) are not limited to the examples shown in Figures 25A and 25B, and mb_type may be encoded according to other orders and binarization methods. B_STDS_X and P_STDS_X do not need to be consecutive, and may be placed between each mb_type. In addition, the binarization method (binning) may be designed based on the selection frequency learned in advance.
[0102] In this embodiment, the present invention is also applicable to extended macroblocks in which a plurality of macroblocks are collectively subjected to motion compensation prediction. Also, in this embodiment, the encoding scan order may be any order. For example, the present invention is also applicable to line scan or Z scan.
[0103] As described above, the image coding device according to this embodiment selects available blocks from a plurality of motion reference blocks, generates information for identifying a motion reference block to be applied to a block to be coded according to the number of selected available blocks, and codes this information. Therefore, according to the image coding device according to this embodiment, while reducing the amount of code related to the motion vector information, motion compensation can be performed in units of small pixel blocks smaller than the block to be coded, thereby realizing high coding efficiency.
[0104] Second embodiment FIG. 26 shows an image encoding device according to the second embodiment of the present invention. In the second embodiment, the parts and operations different from those of the first embodiment will be mainly described. As shown in FIG. 26, the image encoding unit 200 according to this embodiment is different from the first embodiment in the configuration of the prediction unit 201 and the variable length encoding unit 204. As shown in FIG. 27, the prediction unit 201 includes a first prediction unit 101 and a second prediction unit 202, and selectively switches between the first and second prediction units 101 and 202 to generate a predicted image signal 11. The first prediction unit 101 has the same configuration as the prediction unit 101 (FIG. 1) according to the first embodiment, and generates the predicted image signal 11 according to a prediction method (first prediction method) that performs motion compensation using the motion information 18 of the selected block. The second prediction unit 202 generates the predicted image signal 11 according to a prediction method (second prediction method) such as H.264 that performs motion compensation using one motion vector for the block to be encoded. The second prediction unit 202 generates a predicted image signal 11B using the input image signal 10 and a reference image signal 17 from a frame memory.
[0105] FIG. 28 is a schematic diagram of the configuration of the second prediction unit 202. As shown in FIG. 28, the second prediction unit 202 includes a motion information acquisition unit 205 that generates motion information 21 using the input image signal 10 and the reference image signal 17, and a motion compensation unit 113 (FIG. 1) that generates a predicted image signal 11A using the reference image signal 17 and the motion information 21. The motion information acquisition unit 205 obtains a motion vector to be assigned to a block to be coded, for example, by block matching, based on the input image signal 10 and the reference image signal 17. As a criterion for evaluating matching, a value obtained by accumulating the difference between the input image signal 10 and the interpolated image after matching for each pixel is used.
[0106] The motion information acquisition unit 205 may determine an optimal motion vector using a value obtained by converting the difference between the predicted image signal 11 and the input image signal 10. The optimal motion vector may be determined by considering the magnitude of the motion vector and the amount of code of the motion vector and the reference frame number, or by using Equation 1. The matching method may be performed based on search range information provided from outside the image encoding device, or may be performed hierarchically for each pixel precision. Alternatively, the motion information provided by the encoding control unit 150 may be output 21 from the motion information acquisition unit 205 without performing a search process.
[0107] The prediction unit 101 in FIG. 27 further includes a prediction method changeover switch 203 that selects and outputs either the predicted image signal 11A from the first prediction unit 101 or the predicted image signal 11B from the second prediction unit 202. For example, the prediction method changeover switch 203 uses the input image signal 10 to calculate the coding cost for each of the predicted image signals 11A and 11B, for example, according to Equation 1, and selects either the predicted image signal 11A or 11B so that the coding cost is smaller, and outputs it as the predicted image signal 11. Furthermore, the prediction method changeover switch 203 outputs prediction changeover information 32 indicating whether the output predicted image signal 11 is output from either the first prediction unit 101 or the second prediction unit 202, together with the motion information 18 and the selection block information 31. The output motion information 18 is coded by the variable-length coding unit 204 and then multiplexed into the coded data 14.
[0108] FIG. 29 shows a schematic configuration of the variable-length coding unit 204. The variable-length coding unit 204 shown in FIG. 29 includes a motion information coding unit 217 in addition to the configuration of the variable-length coding unit 104 shown in FIG. 18. The selection block coding unit 216 shown in FIG. 29 differs from the selection block coding unit 116 shown in FIG. 18 in that it codes the prediction switching information 32 to generate coded data 14D. When the first prediction unit 101 executes the prediction process, the selection block coding unit 216 further codes the available block information 30 and the selected block information 31. The coded available block information 30 and the selected block information 31 are included in the coded data 14D. When the second prediction unit 202 executes the prediction process, the motion information coding unit 217 codes the motion information 18 to generate coded data 14E. The selected block encoding unit 216 and the motion information encoding unit 217 each determine whether the prediction process was performed by the first prediction unit 101 or the second prediction unit 202 based on prediction switching information 32 indicating whether the predicted image was generated by motion compensation prediction using the motion information of the selected block.
[0109] The multiplexing unit 117 receives the encoded data 14A, B, D, and E from the parameter encoding unit 114, the transform coefficient encoding unit 115, the selection block encoding unit 216, and the motion information encoding unit, and multiplexes the received encoded data 14A, B, D, and E.
[0110] FIG. 30A and FIG. 30B each show an example of macroblock layer syntax according to this embodiment. available_block_num shown in FIG. 30A indicates the number of available blocks, and if this is a value greater than 1, the selected block encoding unit 216 encodes the selected block information 31. Furthermore, stds_flag is a flag indicating whether or not the motion information of the selected block has been used as the motion information of the block to be encoded in the motion compensation prediction, that is, a flag indicating whether the prediction method changeover switch 203 has selected the first prediction unit 101 or the second prediction unit 202. If the number of available blocks is greater than 1 and stds_flag is 1, it indicates that the motion information of the selected block has been used for the motion compensation prediction. Furthermore, if stds_flag is 0, the motion information of the selected block is not used, and the information of the motion information 18 is encoded directly or as a predicted differential value in the same manner as H.264. Furthermore, stds_idx indicates the selected block information, and the code table according to the number of available blocks is as described above.
[0111] Fig. 30A shows the syntax for encoding selected block information after mb_type. stds_flag and stds_idx are encoded only when the mode indicated by mb_type is a specified size or mode. For example, stds_flag and stds_idx are encoded when the motion information of the selected block is available for block sizes of 64x64, 32x32, 16x16, or in direct mode.
[0112] 30B shows the syntax when the selection block information is coded before mb_type. For example, when stds_flag is 1, mb_type does not need to be coded. When stds_flag is 0, mb_type is coded.
[0113] As described above, the image coding device according to the second embodiment selectively switches between the first prediction unit 101 according to the first embodiment and the second prediction unit 202 that uses a prediction method such as H.264, so as to reduce coding costs, and compression codes an input image signal. Therefore, the image coding device according to the second embodiment has improved coding efficiency compared to the image coding device according to the first embodiment.
[0114] (Third embodiment) FIG. 31 shows an outline of an image decoding device according to the third embodiment. As shown in FIG. 31, this image decoding device includes an image decoding unit 300, a decoding control unit 350, and an output buffer 308. The image decoding unit 300 is controlled by the decoding control unit 350. The image decoding device according to the third embodiment corresponds to the image coding device according to the first embodiment. That is, the decoding process by the image decoding device of FIG. 31 has a complementary relationship with the coding process by the image coding process of FIG. 1. The image decoding device of FIG. 31 may be realized by hardware such as an LSI chip, or may be realized by causing a computer to execute an image decoding program.
[0115] The image decoding device of Fig. 31 includes a coded sequence decoding unit 301, an inverse quantization and inverse transform unit 302, an adder 303, a frame memory 304, a prediction unit 305, a motion information memory 306, and an available block acquisition unit 307. In the image decoding unit 300, coded data 80 from a storage system or a transmission system (not shown) is input to the coded sequence decoding unit 301. This coded data 80 corresponds to, for example, the coded data 14 sent in a multiplexed state from the image coding device of Fig. 1.
[0116] In this embodiment, a pixel block (for example, a macroblock) to be decoded is simply called a current block to be decoded, and an image frame including the current block to be decoded is called a current frame to be decoded.
[0117] The coded sequence decoding unit 301 performs decoding by syntax analysis for each frame or field based on the syntax. Specifically, the coded sequence decoding unit 301 sequentially performs variable length decoding on the code sequence of each syntax, and decodes coding parameters related to the block to be decoded, including the transform coefficient information 33, the selected block information 61, and prediction information such as block size information and prediction mode information.
[0118] In this embodiment, the decoding parameters include the transform coefficients 33, the selected block information 61, and prediction information, and include all parameters required for decoding, such as information on the transform coefficients and information on quantization. The prediction information, information on the transform coefficients, and information on quantization are input to the decoding control unit 350 as control information 71. The decoding control unit 350 provides each unit of the image decoding unit 300 with decoding control information 70, which includes parameters required for decoding, such as the prediction information and the quantization parameters.
[0119] Furthermore, as will be described later, the coded sequence decoding unit 301 simultaneously decodes the coded data 80 to obtain prediction information and selected block information 61. The motion information 38 including the motion vector and the reference frame number does not have to be decoded.
[0120] The transform coefficients 33 decoded by the coded sequence decoding unit 301 are sent to the inverse quantization and inverse transform unit 302. Various information related to quantization decoded by the coded sequence decoding unit 301, i.e., quantization parameters and quantization matrices, are provided to the decoding control unit 350 and loaded into the inverse quantization and inverse transform unit 302 when performing inverse quantization. The inverse quantization and inverse transform unit 302 inverse quantizes the transform coefficients 33 according to the loaded information related to quantization, and then performs inverse transform processing (e.g., inverse discrete cosine transform, etc.) to obtain a prediction error signal 34. The inverse transform processing by the inverse quantization and inverse transform unit 302 in FIG. 31 is the inverse transform of the transform processing by the transform and quantization unit in FIG. 1. For example, when a wavelet transform is performed by the image encoding device (FIG. 1), the inverse quantization and inverse transform unit 302 executes the corresponding inverse quantization and inverse wavelet transform.
[0121] The prediction error signal 34 restored by the inverse quantization and inverse transform unit 302 is input to the adder 303. The adder 303 adds the prediction error signal 34 to a predicted image signal 35 generated by a prediction unit 305 (described later) to generate a decoded image signal 36. The generated decoded image signal 36 is output from the image decoding unit 300 and temporarily stored in an output buffer 308, and then output according to an output timing managed by a decoding control unit 350. In addition, this decoded image signal 36 is stored in a frame memory 304 as a reference image signal 37. The reference image signal 37 is sequentially read out from the frame memory 304 for each frame or each field, and input to the prediction unit 305.
[0122] The available block acquisition unit 307 receives reference motion information 39 from a motion information memory 306 (described later), and outputs available block information 60. The operation of the available block acquisition unit 307 is similar to that of the available block acquisition unit 109 (FIG. 1) described in the first embodiment.
[0123] The motion information memory 306 receives the motion information 38 from the prediction unit 305 and temporarily stores it as reference motion information 39. The motion information memory 306 temporarily saves the motion information 38 output from the prediction unit 305 as reference motion information 39. FIG. 4 shows an example of the motion information memory 306. The motion information memory 306 holds a plurality of motion information frames 26 with different encoding times. The motion information 38 or a group of motion information 38 for which decoding has been completed is stored as reference motion information 39 in the motion information frame 26 corresponding to the decoding time. In the motion information frame 26, the reference motion information 39 is stored in units of, for example, 4×4 pixel blocks. The reference motion information 39 held in the motion information memory 306 is read and referred to by the prediction unit 305 when generating the motion information 38 of the block to be decoded.
[0124] Next, a motion reference block and an available block according to this embodiment will be described. A motion reference block is a candidate block selected from an already decoded area according to a method determined in advance by the image encoding device and the image decoding device. FIG. 8A shows an example of an available block. In FIG. 8A, a total of nine motion reference blocks are arranged, including four motion reference blocks in a frame to be decoded and five motion reference blocks in a reference frame. The motion reference blocks A, B, C, and D in the frame to be decoded in FIG. 8A are adjacent blocks to the left, top, top right, and top left of the block to be decoded. In this embodiment, a motion reference block selected from a frame to be decoded that includes a block to be decoded is called a spatial motion reference block. In addition, a motion reference block TA in a reference frame is a pixel block at the same position as the block to be decoded in the reference frame, and pixel blocks TB, TC, TD, and TE adjacent to this motion reference block TA are selected as motion reference blocks. A motion reference block selected from pixel blocks in a reference frame is called a temporal motion reference block. Moreover, the frame in which the time-direction motion reference block is located is called a motion reference frame.
[0125] The spatial motion reference block is not limited to the example shown in Fig. 8A, and a pixel block to which pixels a, b, c, and d adjacent to the block to be decoded belong may be selected as the spatial motion reference block as shown in Fig. 8B. In this case, the relative positions (dx, dy) of pixels a, b, c, and d with respect to the upper left pixel in the block to be decoded are shown in Fig. 8C.
[0126] Alternatively, as shown in Fig. 8D, all pixel blocks A1 to A4, B1, B2, C, and D adjacent to the block to be decoded may be selected as spatial motion reference blocks. In Fig. 8D, the number of spatial motion reference blocks is eight.
[0127] Furthermore, the time direction motion reference blocks TA to TE may partially overlap each other as shown in FIG. 8E, or may be separated from each other as shown in FIG. 8F. Furthermore, the time direction motion reference block does not necessarily need to be located in the block at the collocate position or its periphery, and may be a pixel block at any position within the motion reference frame. For example, by using motion information of an already decoded block adjacent to the block to be decoded, a reference block indicated by a motion vector included in the motion information may be selected as the center of the motion reference block (for example, block TA). Furthermore, the time direction reference blocks do not need to be arranged at equal intervals.
[0128] In the above-mentioned method of selecting a motion reference block, as long as information on the number and positions of spatial and temporal motion reference blocks is shared between both the image decoding device and the image decoding device, the number and positions of the motion reference blocks may be selected from any number and positions. In addition, the size of the motion reference block does not necessarily have to be the same as the size of the block to be decoded. For example, as shown in FIG. 8D, the size of the motion reference block may be larger or smaller than the size of the block to be decoded, and may be any size. In addition, the shape of the motion reference block is not limited to a square shape, and may be a rectangle.
[0129] Next, the available block will be described. The available block is a pixel block selected from the motion reference blocks, and is a pixel block for which motion information can be applied to the block to be decoded. The available blocks have different motion information from each other. The available block is selected by performing the available block determination process shown in FIG. 9 on a total of nine motion reference blocks in the frame to be decoded and the reference frame as shown in FIG. 8A. FIG. 10 shows the result of performing the available block determination process shown in FIG. 9. In FIG. 10, the pixel blocks with diagonal lines indicate unavailable blocks, and the blocks filled with white indicate available blocks. That is, two of the spatial motion reference blocks and two of the temporal motion reference blocks, a total of four, are determined as available blocks. The motion information selection unit 314 in the prediction unit 305 selects an optimal available block as a selected block from among these available blocks arranged in the time and space directions according to the selected block information 61 received from the selected block decoding unit 323.
[0130] Next, a description will be given of the available block acquisition unit 307. The available block acquisition unit 307 has the same function as the available block acquisition unit 109 in the first embodiment, and acquires reference motion information 39 from the motion information memory 306, and outputs available block information 60 which is information indicating an available block or an unavailable block for each motion reference block.
[0131] The operation of the available block acquisition unit 307 will be described with reference to the flowchart of FIG. 9. First, the available block acquisition unit 307 determines whether or not the motion reference block (index p) has motion information (step S801). That is, in step S801, it is determined whether or not at least one small pixel block in the motion reference block p has motion information. If it is determined that the motion reference block p does not have motion information, that is, if the temporal motion reference block is a block in an I-slice without motion information, or if all small pixel blocks in the temporal motion reference block have been intra-prediction decoded, the process proceeds to step S805. In step S805, this motion reference block p is determined to be an unavailable block.
[0132] If it is determined in step S801 that the motion reference block p has motion information, the available block acquisition unit 307 selects a motion reference block q (referred to as available block q) that has already been determined to be an available block (step S802). Here, q is a value smaller than p. Next, the available block acquisition unit 307 compares the motion information of the motion reference block p with the motion information of the available block q for all q to determine whether the motion reference block p has the same motion information as the available block q (S803). If the motion reference block p has the same motion vector as the available block q, the process proceeds to step S805, where the available block acquisition unit 307 determines the motion reference block p as an unavailable block in step S805. If the motion reference block p has motion information different from all the available blocks q, the available block acquisition unit 307 determines the motion reference block p as an available block in step S804.
[0133] By executing the above-mentioned available block determination process for all motion reference blocks, it is determined whether each motion reference block is available or unavailable, and available block information 60 is generated. An example of the available block information 60 is shown in Fig. 11. As shown in Fig. 11, the available block information 60 includes the index p and availability of the motion reference block. In Fig. 11, the available block information 60 indicates that the motion reference blocks with index p of 0, 1, 5, and 8 have been selected as available blocks, and the number of available blocks is 4.
[0134] 9, if at least one of the blocks in the temporal motion reference block p is an intra-prediction coded block, the available block acquisition unit 307 may determine that the motion reference block p is an unavailable block. That is, the process may proceed to step S802 only when all the blocks in the temporal motion reference block p are coded by inter-prediction.
[0135] 12A to 12E show an example in which the motion information 38 of the motion reference block p and the motion information 38 of the available block q are determined to be the same in the comparison of the motion information 38 in step S803. Each of FIGS. 12A to 12E shows a plurality of blocks with diagonal lines and two white blocks. For ease of explanation, FIGS. 12A to 12E assume that the motion information 38 of these two white blocks is compared without considering the diagonal lines. One of the two white blocks is assumed to be the motion reference block p, and the other is assumed to be the motion reference block q (available block q) that has already been determined to be available. Unless otherwise specified, either of the two white blocks may be the motion reference block p.
[0136] Fig. 12A shows an example in which both the motion reference block p and the available block q are spatial blocks. In the example of Fig. 12A, if the motion information 38 of blocks A and B is the same, it is determined that the motion information 38 is the same. In this case, the sizes of blocks A and B do not need to be the same.
[0137] Fig. 12B shows an example in which one of the motion reference block p and the available block q is a spatial block A, and the other is a temporal block TB. In Fig. 12B, there is one block having motion information in the temporal block TB. If the motion information 38 of the temporal block TB is the same as the motion information 38 of the spatial block A, it is determined that the motion information 38 is the same. In this case, the sizes of the blocks A and TB do not need to be the same.
[0138] Fig. 12C shows another example where one of the motion reference block p and the available block q is a spatial block A, and the other is a temporal block TB. Fig. 12C shows a case where the temporal block TB is divided into a plurality of small blocks, and there are a plurality of small blocks having motion information 38. In the example of Fig. 12C, if all blocks having motion information 38 have the same motion information 38, and the motion information 38 is the same as the motion information 38 of the spatial block A, it is determined that the motion information 38 is the same. In this case, the sizes of the blocks A and TB do not need to be the same.
[0139] 12D shows an example in which the motion reference block p and the available block q are both blocks in the time direction. In this case, if the motion information 38 of blocks TB and TE is the same, it is determined that the motion information 38 is the same.
[0140] Fig. 12E shows another example in which the motion reference block p and the available block q are both time-direction blocks. Fig. 12E shows a case in which the time-direction blocks TB and TE are each divided into a plurality of small blocks, each of which has a plurality of small blocks having motion information 38. In this case, the motion information 38 is compared for each small block in the block, and if the motion information 38 is the same for all the small blocks, it is determined that the motion information 38 of the block TB and the motion information 38 of the block TE are the same.
[0141] Fig. 12F shows yet another example in which both the motion reference block p and the available block q are blocks in the time direction. Fig. 12F shows a case in which a block TE in the time direction is divided into a plurality of small blocks, and the block TE has a plurality of small blocks having motion information 38. If all the motion information 38 of the block TE is the same motion information 38 and is the same as the motion information 38 of the block TD, it is determined that the motion information 38 of the block TD and the TE is the same.
[0142] In this manner, in step S803, it is determined whether the motion information 38 of the motion reference block p and the motion information 38 of the available block q are the same. In the examples of Figures 12A to 12F, the number of available blocks q to be compared with the motion reference block p is 1, but when the number of available blocks q is 2 or more, the motion information 38 of the motion reference block p may be compared with the motion information 38 of each available block q. In addition, when scaling, which will be described later, is applied, the motion information 38 after scaling becomes the motion information 38 described above.
[0143] In addition, the determination that the motion information of the motion reference block p and the motion information of the available block q are the same is not limited to the case where the motion vectors included in the motion information are completely the same. For example, if the norm of the difference between the two motion vectors is within a predetermined range, the motion information of the motion reference block p and the motion information of the available block q may be considered to be substantially the same.
[0144] Fig. 32 is a block diagram showing in more detail the coded sequence decoding unit 301. As shown in Fig. 32, the coded sequence decoding unit 301 includes a separation unit 320 that separates coded data 80 into syntax units, a transform coefficient decoding unit 322 that decodes transform coefficients, a selection block decoding unit 323 that decodes selection block information, and a parameter decoding unit 321 that decodes parameters related to prediction block size and quantization, etc.
[0145] The parameter decoding unit 321 receives coded data 80A including parameters related to prediction block size and quantization from the separation unit, and decodes the coded data 80A to generate control information 71. The transform coefficient decoding unit 322 receives coded transform coefficients 80B from the separation unit 320, and decodes the coded transform coefficients 80B to obtain transform coefficient information 33. The selected block decoding unit 323 receives coded data 80C related to a selected block and available block information 60 as input, and outputs selected block information 61. The input available block information 60 indicates the availability of each motion reference block, as shown in FIG. 11 .
[0146] Next, the prediction unit 305 will be described in detail with reference to FIG. 33, the prediction unit 305 includes a motion information selection unit 314 and a motion compensation unit 313, and the motion information selection unit 314 includes a spatial motion information acquisition unit 310, a temporal motion information acquisition unit 311, and a motion information changeover switch 312. The prediction unit 305 basically has the same configuration and function as the prediction unit 101 described in the first embodiment.
[0147] The prediction unit 305 receives the available block information 60, the selected block information 61, the reference motion information 39, and the reference image signal 37 as inputs, and outputs a predicted image signal 35 and motion information 38. The spatial motion information acquisition unit 310 and the temporal motion information acquisition unit 311 have the same functions as the spatial motion information acquisition unit 110 and the temporal motion information acquisition unit 111 described in the first embodiment, respectively. The spatial motion information acquisition unit 310 uses the available block information 60 and the reference motion information 39 to generate motion information 38A including motion information and indexes of each available block located in the spatial direction. The temporal motion information acquisition unit 311 uses the available block information 60 and the reference motion information 39 to generate motion information (or a group of motion information) 38B including motion information and indexes of each available block located in the temporal direction.
[0148] The motion information changeover switch 312 selects one of the motion information 38A from the spatial motion information acquisition unit 310 and the motion information (or a group of motion information) 38B from the time motion information acquisition unit 311 in accordance with the selected block information 61, and obtains motion information 38. The selected motion information 38 is sent to the motion compensation unit 313 and the motion information memory 306. The motion compensation unit 313 performs motion compensation prediction in accordance with the selected motion information 38 in the same manner as the motion compensation unit 113 described in the first embodiment, and generates a predicted image signal 35.
[0149] The motion vector scaling function of the motion compensation unit 313 is similar to that described in the first embodiment, and therefore a description thereof will be omitted.
[0150] Fig. 22 shows a syntax structure in the image decoding unit 300. As shown in Fig. 22, the syntax mainly includes three parts, namely, a high level syntax 901, a slice level syntax 904, and a macroblock level syntax 907. The high level syntax 901 holds syntax information of layers higher than the slice level. The slice level syntax 904 holds information required for each slice, and the macroblock level syntax 907 holds data required for each macroblock shown in Figs. 7A to 7D.
[0151] Each part includes more detailed syntax. High level syntax 901 includes sequence and picture level syntax such as sequence parameter set syntax 902 and picture parameter set syntax 903. Slice level syntax 904 includes slice header syntax 905 and slice data syntax 906. Furthermore, macroblock level syntax 907 includes macroblock layer syntax 908 and macroblock prediction syntax 909.
[0152] Figures 23A and 23B show examples of macroblock layer syntax. available_block_num shown in Figures 23A and 23B indicates the number of available blocks, and if this is a value greater than 1, it is necessary to decode the selected block information. Furthermore, stds_idx indicates the selected block information, and stds_idx is encoded using a code table according to the number of available blocks described above.
[0153] 23A shows the syntax for decoding selected block information after mb_type. When the prediction mode indicated by mb_type is a specified size or a specified mode (TARGET_MODE) and available_block_num is a value greater than 1, stds_idx is decoded. For example, stds_idx is encoded when the motion information of the selected block is available for a block size of 64×64 pixels, 32×32 pixels, or 16×16 pixels, or in direct mode.
[0154] Fig. 23B shows the syntax when the selection block information is decoded before mb_type. When available_block_num is a value greater than 1, stds_idx is decoded. Also, when available_block_num is 0, conventional motion compensation represented by H.264 is performed, so mb_type is coded.
[0155] Between the rows of the tables shown in Figures 23A and 23B, syntax elements not specified in the present invention may be inserted, and descriptions related to other conditional branches may be included. Alternatively, the syntax table may be divided and integrated into multiple tables. Also, it is not necessary to use the same terminology, and it may be changed arbitrarily depending on the form of use. Furthermore, each syntax element described in the macroblock layer syntax may be changed as specified in the macroblock data syntax described later.
[0156] As described above, the image decoding device according to this embodiment decodes an image coded by the image coding device according to the above-mentioned embodiment 1. Therefore, the image decoding according to this embodiment can reproduce a high-quality decoded image from relatively small coded data.
[0157] (Fourth embodiment) FIG. 34 is a schematic diagram of an image decoding device according to the fourth embodiment. As shown in FIG. 34, the image decoding device includes an image decoding unit 400, a decoding control unit 350, and an output buffer 308. The image decoding device according to the fourth embodiment corresponds to the image coding device according to the second embodiment. In the fourth embodiment, the portions and operations different from those of the third embodiment will be mainly described. As shown in FIG. 34, the image decoding unit 400 according to this embodiment differs from the third embodiment in an encoded sequence decoding unit 401 and a prediction unit 405.
[0158] The prediction unit 405 of this embodiment selectively switches between a prediction method (first prediction method) that performs motion compensation using motion information possessed by a selected block, and a prediction method (second prediction method) that performs motion compensation using one motion vector for a block to be decoded, such as H.264, to generate a predicted image signal 35.
[0159] FIG. 35 is a block diagram showing the coded sequence decoding unit 401 in more detail. The coded sequence decoding unit 401 shown in FIG. 35 includes a motion information decoding unit 424 in addition to the configuration of the coded sequence decoding unit 301 shown in FIG. 32. The selected block decoding unit 423 shown in FIG. 35 differs from the selected block decoding unit 323 shown in FIG. 32 in that it decodes the coded data 80C related to the selected block to obtain prediction switching information 62. The prediction switching information 62 indicates whether the prediction unit 101 in the image coding device of FIG. 1 has used the first prediction method or the second prediction method. When the prediction switching information 62 indicates that the prediction unit 101 has used the first prediction method, that is, when the block to be decoded has been coded by the first prediction method, the selected block decoding unit 423 decodes the selected block information in the coded data 80C to obtain the selected block information 61. When the prediction switching information 62 indicates that the prediction unit 101 has used the second prediction method, i.e., when the block to be decoded has been encoded using the second prediction method, the selected block decoding unit 423 does not decode the selected block information, and the motion information decoding unit 424 decodes the encoded motion information 80D to obtain motion information 40.
[0160] FIG. 36 is a block diagram showing the prediction unit 405 in more detail. The prediction unit 405 shown in FIG. 34 includes a first prediction unit 305, a second prediction unit 410, and a prediction method changeover switch 411. The second prediction unit 410 performs motion compensation prediction similar to that of the motion compensation unit 313 of FIG. 33 using the motion information 40 decoded by the coded sequence decoding unit 401 and the reference image signal 37, and generates a predicted image signal 35B. The first prediction unit 305 is the same as the prediction unit 305 described in the third embodiment, and generates a predicted image signal 35B. In addition, the prediction method changeover switch 411 selects either the predicted image signal 35B from the second prediction unit 410 or the predicted image signal 35A from the first prediction unit 305 based on the prediction changeover information 62, and outputs it as the predicted image signal 35 of the prediction unit 405. At the same time, the prediction method changeover switch 411 sends the motion information used in the selected first prediction unit 305 or second prediction unit 410 to the motion information memory 306 as motion information 38 .
[0161] Next, the syntax structure of this embodiment will be described, focusing mainly on the differences from the third embodiment.
[0162] FIG. 30A and FIG. 30B each show an example of the macroblock layer syntax according to this embodiment. available_block_num shown in FIG. 30A indicates the number of available blocks, and if this is a value greater than 1, the selected block decoding unit 423 decodes the selected block information in the encoded data 80C. Also, stds_flag is a flag indicating whether or not the motion information of the selected block has been used as the motion information of the block to be decoded in the motion compensation prediction, that is, a flag indicating whether the prediction method changeover switch 411 has selected the first prediction unit 305 or the second prediction unit 410. If the number of available blocks is greater than 1 and stds_flag is 1, it indicates that the motion information of the selected block has been used for the motion compensation prediction. Also, if stds_flag is 0, the motion information of the selected block is not used, and the difference value obtained by directly or predicting the information of the motion information is encoded as in H.264. Furthermore, stds_idx indicates the selected block information, and the code table according to the number of available blocks is as described above.
[0163] 30A shows the syntax for decoding selected block information after mb_type. Only when the prediction mode indicated by mb_type is a specified block size or mode, stds_flag and stds_idx are decoded. For example, when the block size is 64×64, 32×32, or 16×16, or in the direct mode, stds_flag and stds_idx are decoded.
[0164] Figure 30B shows the syntax when the selected block information is decoded before mb_type. For example, if stds_flag is 1, mb_type does not need to be decoded. If stds_flag is 0, mb_type is decoded.
[0165] As described above, the image decoding device according to this embodiment decodes an image coded by the image coding device according to the above-mentioned embodiment 2. Therefore, the image decoding according to this embodiment can reproduce a high-quality decoded image from a relatively small amount of coded data.
[0166] The present invention is not limited to the above-described embodiment as it is, and the components can be modified and embodied in the implementation stage without departing from the gist of the invention. In addition, various inventions can be formed by appropriately combining the multiple components disclosed in the above-described embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined.
[0167] As an example, the same effect can be obtained by modifying the above-described first to fourth embodiments as follows.
[0168] (1) In the first to fourth embodiments, a frame to be processed is divided into rectangular blocks such as 16×16 pixel blocks, and encoding or decoding is performed in an order from the pixel block in the upper left of the screen to the pixel block in the lower right as shown in Fig. 4, but the encoding or decoding order is not limited to this example. For example, the encoding or decoding order may be from the lower right to the upper left of the screen, or from the upper right to the lower left. Moreover, the encoding or decoding order may be a spiral order from the center to the periphery of the screen, or from the periphery to the center of the screen.
[0169] (2) In the first to fourth embodiments, the luminance signal and the chrominance signal are not divided, and only one of the chrominance signal components is used. However, different prediction processes may be used for the luminance signal and the chrominance signal, or the same prediction process may be used. When different prediction processes are used, the prediction method selected for the chrominance signal is encoded / decoded in the same manner as for the luminance signal.
[0170] It goes without saying that the present invention can be implemented in a similar manner even if various modifications are made without departing from the spirit and scope of the present invention. [Industrial Applicability]
[0171] INDUSTRIAL APPLICABILITY The image encoding / decoding method according to the present invention can improve encoding efficiency and has industrial applicability. [Explanation of symbols]
[0172] 10...input image signal, 11...prediction image signal, 12...prediction error image signal, 13...quantized transform coefficient, 14...encoded data, 15...decoded prediction error signal, 16...locally decoded image signal, 17...reference image signal, 18...motion information, 20...bit stream, 21...motion information, 25, 26...information frame, 30...available block information, 31...selected block information, 32...prediction switching information, 33...transform coefficient information, 34...prediction error signal, 35...prediction image signal, 36...decoded image signal, 37...reference image signal, 38...motion information, 39...reference motion information, 40...motion information, 50...encoding control information , 51...feedback information, 60...available block information, 61...selected block information, 62...prediction switching information, 70...decoding control information, 71...control information, 80...encoded data, 100...image encoding unit, 101...prediction unit, 102...subtractor, 103...transformation and quantization unit, 104...variable length encoding unit, 105...inverse quantization and inverse transformation unit, 106...adder, 107...frame memory, 108...information memory, 109...available block acquisition unit, 110...spatial direction motion information acquisition unit, 111...temporal direction motion information acquisition unit, 112...information changeover switch, 113...motion compensation unit, 114...parallel image encoding unit, parameter coding unit, 115...transform coefficient coding unit, 116...selected block coding unit, 117...multiplexing unit, 118...motion information selection unit, 120...output buffer, 150...coding control unit, 200...image coding unit, 201...prediction unit, 202...second prediction unit, 203...prediction method changeover switch, 204...variable length coding unit, 205...motion information acquisition unit, 216...selected block coding unit, 217...motion information coding unit, 300...image decoding unit, 301...code string decoding unit, 301...code string decoding unit, 302...inverse quantization and inverse transform unit, 303...adder, 304...frame memory, 305...prediction measurement unit, 306... information memory, 307... available block acquisition unit, 308... output buffer, 310... spatial motion information acquisition unit, 311... time direction motion information acquisition unit, 312... motion information changeover switch, 313... motion compensation unit, 314... information selection unit, 320... separation unit, 321... parameter decoding unit, 322... transform coefficient decoding unit, 323... selected block decoding unit, 350... decoding control unit, 400... image decoding unit, 401... coded sequence decoding unit, 405... prediction unit, 410... second prediction unit, 411... prediction method changeover switch, 423... selected block decoding unit, 424... information decoding unit,901: high level syntax, 902: sequence parameter set syntax, 903: picture parameter set syntax, 904: slice level syntax, 905: slice header syntax, 906: slice data syntax, 907: macroblock level syntax, 908: macroblock layer syntax, 909: macroblock prediction syntax.
Claims
[Claim 1] a circuit for transmitting the encoded data; the encoded data includes mode information regarding a prediction mode of a current block; the encoded data further includes identification information for identifying a selected block when a predetermined condition is satisfied, and does not include the identification information when the predetermined condition is not satisfied; the identification information is generated by, when the mode information indicates a predetermined information, determining whether a plurality of candidate blocks having a predetermined positional relationship with respect to the target block are available in a predetermined order according to the positional relationship, and selecting, as the selected block, one block whose motion information is used as motion information of the target block from among the candidate blocks determined to be available; determining includes determining a candidate block as available if the candidate block has motion information that does not match motion information corresponding to a candidate block already determined to be available; the predetermined condition is that the number of the candidate blocks determined to be available is greater than 1; Transmitting device.
Citation Information
Patent Citations
Time and space prediction of bidirectional prediction (b) picture, and moving vector prediction of plural picture reference moving compensations
JP2004040785A
Motion vector encoding / decoding method and apparatus
JP2004056823A
Variable length code generating apparatus, variable length decoding apparatus, variable length code generating method, and variable length decoding method
JP2004104159A
Moving picture coding method and decoding method
JP2004165703A
Motion vector calculating method
JP2004208259A
