Receiving Method and Receiver

Abstract

To provide a reception method capable of efficiently coding images.SOLUTION: The reception method includes: a receiving step of receiving a bit stream including a video encoded by hierarchically encoding a plurality of images included in the video into a hierarchical structure having one or more layers, and control information; and a decoding step (S401) of decoding multiple images from the bit stream. The lowest level of the hierarchical structure includes an I picture and a P picture, any level other than the lowest level in the hierarchical structure includes B picture, and the control information includes a piece of information of the frame rate of the video. The number of hierarchies is determined in advance based on the frame rate of the video.SELECTED DRAWING: Figure 16

Description

Technical Field

[0001] The present invention relates to an image encoding method for encoding an image or an image decoding method for decoding an image.

Background Art

[0002] As a technique related to an image encoding method for encoding an image (including a moving image) and an image decoding method for decoding an image, there is a technique described in Non-Patent Document 1. In addition, there are regulations described in Non-Patent Document 2 as operation regulations related to encoding and decoding.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the image encoding method or image decoding method according to the related art, inefficient processing may be used.

[0005] Therefore, an object of the present invention is to provide an image encoding method for efficiently encoding an image or an image decoding method for efficiently decoding an image.

Means for Solving the Problems

[0006] To achieve the above object, a transmission method according to an aspect of the present invention includes a generation step of generating a bitstream including the moving image encoded by hierarchically encoding a plurality of images included in the moving image into a hierarchical structure having one or more numbers of hierarchies, and control information, and a transmission step of transmitting the bitstream and the control information, wherein the lowest hierarchy of the hierarchical structure includes I pictures and P pictures, and hierarchies other than the lowest hierarchy of the hierarchical structure include B pictures, the control information includes information on the frame rate of the moving image, and the number of hierarchies is determined in advance based on the frame rate of the moving image.

[0007] Also, a reception method according to an aspect of the present invention includes a reception step of receiving a bitstream including the moving image encoded by hierarchically encoding a plurality of images included in the moving image into a hierarchical structure having one or more numbers of hierarchies, and control information, and a decoding step of decoding the plurality of images from the bitstream, wherein the lowest hierarchy of the hierarchical structure includes I pictures and P pictures, and hierarchies other than the lowest hierarchy of the hierarchical structure include B pictures, the control information includes information on the frame rate of the moving image, and the number of hierarchies is determined in advance based on the frame rate of the moving image.

[0008] Note that these general or specific aspects may be implemented in a system, method, integrated circuit, computer program, or a recording medium such as a computer-readable CD-ROM, or may be implemented by any combination of a system, method, integrated circuit, computer program, and recording medium.

Advantages of the Invention

[0009] The present invention can provide an image encoding method capable of efficiently encoding an image or an image decoding method capable of efficiently decoding an image.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9A

Figure 9B

Figure 9C

Figure 9D

Figure 10

Figure 11A

Figure 11B

Figure 11C

Figure 11D

Figure 12A

Figure 12B

Figure 12C

Figure 12D

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22A

Figure 22B

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35A

Figure 35B

Mode for Carrying Out the Invention

[0011] (Knowledge on which the present invention is based) The present inventor has found that the following problems occur with respect to an image encoding device that encodes an image or an image decoding device that decodes an image, which are described in the "Background Art" section.

[0012] In recent years, the technological progress of digital video equipment has been remarkable, and there has been an increasing opportunity for a video signal (a plurality of pictures arranged in time series order) output from a video camera or a television tuner to be compression-encoded and the obtained encoded signal to be recorded on a recording medium such as a DVD or a hard disk.

[0013] As an image encoding standard, there is H.264 / AVC (MPEG-4 AVC). Further, as a next-generation standard, the HEVC (High Efficiency Video Coding) standard (Non-Patent Document 1) is being studied. Also, regulations on how to operate the image encoding standard are being studied (Non-Patent Document 2).

[0014] In the current operation regulations (Non-Patent Document 2), as shown in FIG. 1, the encoding structure is limited to three layers, and thereby, as shown in FIG. 2, the maximum number of display delay pictures is limited to two. In FIG. 1, TemporalId is an identifier of the layer of the encoding structure. The larger the TemporalId, the deeper the layer.

[0015] One square block represents a picture, and I within the block x is an I picture (intra-predicted picture), P x is a P picture (forward reference predicted picture), B x is a B picture (bi-directional reference predicted picture). The x / P x / B x of x indicates the display order and represents the order in which the pictures are displayed.

[0016] The arrow between pictures indicates a reference relationship. For example, picture B 1 refers to picture I 0 , picture B 2 and picture P 4 are used as reference images to generate a predicted image. Also, it is prohibited to use a picture with a TemporalId larger than one's own TemporalId as a reference image. Therefore, the picture decoding order is in ascending order of TemporalId as shown in FIG. 2, and pictures are in the order of picture I 0 , picture P 4 , picture B 2 , picture B 1 , picture B 3 .

[0017] By defining a hierarchy, time scalability can be given to the code sequence.

[0018] For example, when it is desired to obtain a 30fps video from a 60fps (frame per second) code sequence, the image decoding device decodes only the pictures with TemporalId0 and TemporalId1 in FIG. 1. Thereby, the image decoding device can obtain a 30fps image. And since the decoded images need to be output in order without gaps, the image decoding device outputs the pictures in order starting from picture I 2 after decoding picture B 0 . Therefore, the number of display delay pictures is two. Converting this to time, when the original frame rate is 30fps, the display delay time is 2 / 30 seconds, and when the frame rate is 60fps, the display delay time is 2 / 60 seconds.

[0019] By using a structure with high time scalability, when the bandwidth is congested or when an image decoding device with low processing power performs decoding processing, the image decoding device can decode only the pictures in the lower hierarchy of TemporalId and display the obtained images. In this way, the versatility is improved. However, if a deep hierarchical structure is allowed, there is a problem that the display delay increases.

[0020] However, even if the number of display delay pictures is specified in advance as described above, the display delay time varies depending on the frame rate. In the case of a frame rate lower than the standard frame rate (e.g., 30 fps) (e.g., 24 fps), the display delay time is 2 / 24 seconds, which is longer than 2 / 30 seconds at 30 fps.

[0021] An image encoding method according to an aspect of the present invention is an image encoding method for hierarchically encoding an image, including a layer number determination step of determining the number of layers so that the number of layers in the hierarchical encoding is equal to or less than a maximum number of layers determined according to the frame rate, and an encoding step of generating a bitstream by hierarchically encoding the image with the determined number of layers.

[0022] According to this, the image encoding method can increase the number of layers while suppressing an increase in the display delay time. Therefore, the image encoding method can efficiently encode an image.

[0023] For example, when the frame rate is 60 fps or less, the maximum number of layers may be 4 or less.

[0024] For example, when the frame rate is 120 fps, the maximum number of layers may be 5.

[0025] For example, the image encoding method further includes a picture type determination step of determining the picture type of the image so that the number of display delay pictures, which is the number of pictures from when the image is decoded to when it is output in an image decoding device, is equal to or less than a maximum number of pictures determined according to the frame rate, and in the encoding step, the image may be encoded with the determined picture type.

[0026] For example, in the picture type determination step, the picture type of the image may be determined so that the number of consecutive B pictures, which is the number of consecutive B pictures, is equal to or less than a maximum number of consecutive pictures determined according to the frame rate.

[0027] For example, the maximum number of pictures, the encoder transmission delay which is the time from when the image is input to the image encoding device until the bitstream is output, and the frame rate satisfy the following relationship: Maximum number of pictures = int(log 2 (Encoder transmission delay [s] × Frame rate [fps])) The maximum consecutive number, the encoder transmission delay, and the frame rate satisfy the following relationship: Maximum consecutive number = int(Encoder transmission delay [s] × Frame rate [fps] - 1) The maximum number of layers, the encoder transmission delay, and the frame rate may satisfy the following relationship.

[0028] Maximum number of layers = int(log 2 (Encoder transmission delay [s] × Frame rate [fps])) + 1 For example, the maximum number of pictures [i] in each layer, the encoder transmission delay, and the frame rate satisfy the following relationship: Maximum number of pictures [i] = int(log 2 (Encoder transmission delay [s] × Frame rate [fps] / 2 (n-i) )) The maximum consecutive number [i] in each layer, the encoder transmission delay, and the frame rate satisfy the following relationship: Maximum consecutive number [i] = int(Encoder transmission delay [s] × Frame rate [fps] / 2 (n-i) - 1) i is an integer less than or equal to the maximum number of layers, indicating the layer, and n may indicate (the maximum number of layers - 1).

[0029] Also, an image decoding method according to an aspect of the present invention is an image decoding method for decoding a bitstream obtained by hierarchically encoding an image, including an image decoding step of decoding the image from the bitstream, an information decoding step of decoding first information indicating the number of hierarchies in the hierarchical encoding from the bitstream, and a rearrangement step of rearranging and outputting the decoded image using the number of hierarchies indicated by the first information, wherein the number of hierarchies is equal to or less than a maximum number of hierarchies determined in advance according to the frame rate of the bitstream.

[0030] According to this, the image decoding method can decode a bitstream obtained by being efficiently encoded.

[0031] For example, when the frame rate is 60 fps or less, the maximum number of hierarchies may be 4 or less.

[0032] For example, when the frame rate is 120 fps, the maximum number of hierarchies may be 5.

[0033] For example, in the information decoding step, further, second information indicating the number of display delay pictures, which is the number of pictures from when an image is decoded to when it is output in an image decoding apparatus, is decoded from the bitstream, and in the rearrangement step, the decoded image may be rearranged and output using the number of hierarchies indicated by the first information and the number of display delay pictures indicated by the second information.

[0034] For example, in the information decoding step, further, third information indicating the number of consecutive B pictures, which is the number of consecutive B pictures, is decoded from the bitstream, and in the rearrangement step, the decoded image may be rearranged and output using the number of hierarchies indicated by the first information, the number of display delay pictures indicated by the second information, and the number of consecutive B pictures indicated by the third information.

[0035] For example, the maximum number of pictures, the encoder transmission delay which is the time from when the image is input to the image encoding device until the bitstream is output, and the frame rate satisfy the following relationship: Maximum number of pictures = int(log 2 (Encoder transmission delay [s] × Frame rate [fps])) The maximum consecutive number, the encoder transmission delay, and the frame rate satisfy the following relationship: Maximum consecutive number = int(Encoder transmission delay [s] × Frame rate [fps] - 1) The maximum number of hierarchical levels, the encoder transmission delay, and the frame rate may satisfy the following relationship.

[0036] Maximum number of hierarchical levels = int(log 2 (Encoder transmission delay [s] × Frame rate [fps])) + 1 For example, the maximum number of pictures [i] in each hierarchical level, the encoder transmission delay, and the frame rate satisfy the following relationship: Maximum number of pictures [i] = int(log 2 (Encoder transmission delay [s] × Frame rate [fps] / 2 (n-i) )) The maximum consecutive number [i] in each hierarchical level, the encoder transmission delay, and the frame rate satisfy the following relationship: Maximum consecutive number [i] = int(Encoder transmission delay [s] × Frame rate [fps] / 2 (n-i) - 1) i is an integer less than or equal to the maximum number of hierarchical levels, indicating the hierarchical level, and n may indicate (the maximum number of hierarchical levels - 1).

[0037] Also, an image encoding device according to an aspect of the present invention is an image encoding device that encodes an image, and includes a processing circuit and a storage device accessible from the processing circuit, and the processing circuit executes the image encoding method using the storage device.

[0038] According to this, the image encoding apparatus can increase the number of layers while suppressing an increase in the display delay time. Thus, the image encoding apparatus can efficiently encode an image.

[0039] Also, an image decoding apparatus according to an aspect of the present invention is an image decoding apparatus that decodes a bit stream obtained by encoding an image, and includes a processing circuit and a storage device accessible from the processing circuit. The processing circuit executes the image decoding method using the storage device.

[0040] According to this, the image decoding apparatus can decode a bit stream obtained by being efficiently encoded.

[0041] Note that these general or specific aspects may be implemented by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be implemented by any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0042] Hereinafter, embodiments will be specifically described with reference to the drawings. Note that each of the embodiments described below shows a specific example of the present invention. Numerical values, shapes, materials, components, arrangement positions and connection forms of components, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the present invention. In addition, among the components in the following embodiments, components not described in the independent claims indicating the most general concept are described as optional components.

[0043] (Embodiment 1) The image encoding apparatus according to the present embodiment increases the number of layers when the frame rate is high. Thereby, the number of layers can be increased while suppressing an increase in the display delay time.

[0044] <Overall Configuration> FIG. 3 is a block diagram showing the configuration of an image encoding apparatus 100 according to the present embodiment.

[0045] The image encoding device 100 shown in FIG. 3 generates a code sequence 155 (bit stream) by encoding the input image 153. This image encoding device 100 includes a limit value setting unit 101 and an encoding unit 102.

[0046] <Operation (Overall)> Next, with reference to FIG. 4, the flow of the entire encoding process will be described. FIG. 4 is a flowchart of the image encoding method according to the present embodiment.

[0047] First, the limit value setting unit 101 sets an encoding structure limit value 154 related to the encoding structure in hierarchical encoding (S101). Specifically, the limit value setting unit 101 sets the encoding structure limit value 154 using the frame rate 151 and the transmission delay time limit value 152.

[0048] Next, the encoding unit 102 encodes the encoding structure limit value 154 and generates a code sequence 155 by encoding the input image 153 using the encoding structure limit value 154 (S102).

[0049] <Configuration of the Limit Value Setting Unit 101> FIG. 5 is a block diagram showing an example of the internal configuration of the limit value setting unit 101.

[0050] As shown in FIG. 5, the limit value setting unit 101 includes a layer number setting unit 111, a layer number parameter setting unit 112, a display delay picture number setting unit 113, a B picture consecutive number setting unit 114, and a consecutive number parameter setting unit 115.

[0051] <Operation (Encoding Structure Limit Value Setting)> Next, with reference to FIG. 6, an example of the limit value setting process (S101 in FIG. 4) will be described. FIG. 6 is a flowchart of the limit value setting process according to the present embodiment.

[0052] First, the number-of-layers setting unit 111 sets the number of layers 161 of the encoding structure using the frame rate 151 and the transmission delay time limit value 152 input from outside the image encoding apparatus 100. For example, the number of layers 161 is calculated by the following (Equation 1) (S111).

[0053] Number of layers = int(log 2 (Transmission delay time limit value [s] × Frame rate [fps])) + 1 ··· (Equation 1)

[0054] In the above (Equation 1), int(x) means a function that returns an integer obtained by truncating the fractional part of x, and log 2 (x) means a function that returns the logarithm of x to the base 2. The transmission delay time limit value 152 indicates the maximum time from when the input image 153 is input to the image encoding apparatus 100 until the code sequence 155 of the input image 153 is output.

[0055] Next, the number-of-layers parameter setting unit 112 sets sps_max_sub_layers_minus1, which is the number-of-layers parameter 163, by the following (Equation 2) using the number of layers 161 (S112).

[0056] sps_max_sub_layers_minus1 = Number of layers - 1 ··· (Equation 2)

[0057] Next, the limit value setting unit 101 sets 0 to TId (S113). TId is a variable for identifying a layer and is used to identify the layer to be processed in the processing for each subsequent layer.

[0058] Next, the display delay picture number setting unit 113 sets the display delay picture number 164 of the layer where TemporalId is TId using the frame rate 151, the transmission delay time limit value 152, and the number-of-layers parameter 163 (S114). The display delay picture number 164 is the number of pictures from when the decoding of a picture starts at the time of picture decoding until the display of the picture starts. The display delay picture number 164 is calculated by the following (Equation 3).

[0059] The number of display delay pictures in the hierarchy of TemporalId being TId = int(log 2 (transmission delay time limit value [s] × frame rate [fps] ÷ 2 (n-TId) )) ··· (Equation 3)

[0060] In the above (Equation 3), n represents the maximum TemporalId and is the value of sps_max_sub_layers_minus1 calculated in step S112. The display delay picture number setting unit 113 sets the calculated number of display delay pictures in the hierarchy of TemporalId being TId to sps_max_num_reorder_pics[TId].

[0061] Next, the B-picture consecutive number setting unit 114 sets the B-picture consecutive number 162 in the hierarchy of TemporalId being TId by using the frame rate 151, the transmission delay time limit value 152, and the layer number parameter 163 (S115). The B-picture consecutive number 162 is the number of consecutive B pictures and is calculated by the following (Equation 4).

[0062] The B-picture consecutive number in the hierarchy of TemporalId being TId = int(transmission delay time limit value [s] × frame rate [fps] ÷ 2 (n-TId) - 1) ··· (Equation 4)

[0063] Next, the consecutive number parameter setting unit 115 sets the consecutive number parameter 165 in the hierarchy of TemporalId being TId by using the B-picture consecutive number 162 and the display delay picture number 164 (sps_max_num_reorder_pics[TId]) in the hierarchy of TemporalId being TId (S116). The consecutive number parameter 165 in the hierarchy of TemporalId being TId is set by the following (Equation 5).

[0064] The consecutive number parameter in the hierarchy of TemporalId being TId = the B-picture consecutive number in the hierarchy of TemporalId being TId - sps_max_num_reorder_pics[TId] + 1 ··· (Equation 5)

[0065] The calculated consecutive number parameter 165 is set to sps_max_latency_increase_plus1[TId].

[0066] Next, the limit value setting unit 101 moves to the processing target layer by adding 1 to TId (S117). Steps S114 to S117 are repeated until TId reaches the number of layers 161, that is, until the processing of all layers is completed (S118).

[0067] Here, the limit value setting unit 101 sets the display delay picture number 164 and the B picture consecutive number 162 in each layer after setting the number of layers 161, but the setting order is not limited to this.

[0068] <Configuration of the Encoding Unit 102> FIG. 7 is a block diagram showing the internal configuration of the encoding unit 102. As shown in FIG. 7, the encoding unit 102 includes an image rearrangement unit 121, a code block division unit 122, a subtraction unit 123, a transform quantization unit 124, a variable length encoding unit 125, an inverse transform quantization unit 126, an addition unit 127, a frame memory 128, an intra prediction unit 129, an inter prediction unit 130, and a selection unit 131.

[0069] <Operation (Encoding)> Next, with reference to FIG. 8, the encoding process (S102 in FIG. 4) according to the present embodiment will be described. FIG. 8 is a flowchart of the encoding process according to the present embodiment.

[0070] First, the variable length encoding unit 125 performs variable length encoding on sps_max_sub_layers_minus1, sps_max_num_reorder_pics[], and sps_max_latency_increase_plus1[] set by the limit value setting unit 101 (S121). sps_max_num_reorder_pics[] and sps_max_latency_increase_plus1[] exist for each layer, but the variable length encoding unit 125 encodes all of them.

[0071] Next, the image rearrangement unit 121 rearranges the input image 153 according to sps_max_sub_layers_minus1, sps_max_num_reorder_pics[], and sps_max_latency_increase_plus1[], and determines the picture type of the input image 153 (S122).

[0072] The image rearrangement unit 121 performs this rearrangement using sps_max_num_reorder_pics[sps_max_sub_layers_minus1] and SpsMaxLatencyPictures. SpsMaxLatencyPictures is calculated by the following (Equation 6).

[0073] SpsMaxLatencyPictures = sps_max_num_reorder_pics[sps_max_sub_layers_minus1] + sps_max_latency_increase_plus1[sps_max_sub_layers_minus1] - 1 ···(Equation 6)

[0074] Figures 9A to 9D are diagrams showing this rearrangement. Since the rearrangement as shown in Figures 9A to 9D is performed, the encoding unit 102 cannot start encoding the input image 153 until a plurality of input images 153 are input. That is, a delay occurs from when the first input image 153 is input until the output of the code stream 155 starts. This delay is the transmission delay time, and the above-described transmission delay time limit value 152 is the limit value of this transmission delay time.

[0075] In addition, FIGS. 9A to 9D show the number of transmitted delayed pictures corresponding to the coding structure limit value 154. FIG. 9A shows the number of transmitted delayed pictures when sps_max_num_reorder_pics[sps_max_sub_layers_minus1] is 1 and SpsMaxLatencyPictures is 2. FIG. 9B shows the number of transmitted delayed pictures when sps_max_num_reorder_pics[sps_max_sub_layers_minus1] is 2 and SpsMaxLatencyPictures is 3. FIG. 9C shows the number of transmitted delayed pictures when sps_max_num_reorder_pics[sps_max_sub_layers_minus1] is 3 and SpsMaxLatencyPictures is 7. FIG. 9B shows the number of transmitted delayed pictures when sps_max_num_reorder_pics[sps_max_sub_layers_minus1] is 4 and SpsMaxLatencyPictures is 7.

[0076] For example, in the case of FIG. 9A, pictures 0, 1, 2, and 3 are input to the image coding device 100 in this order, and the image coding device 100 encodes these pictures in the order of pictures 0, 3, 1, and 2. Since the image coding device 100 needs to transmit the code sequence without gaps, the transmission of the code sequence is not started until picture 3 is input. Therefore, a transmission delay of three pictures occurs from when picture 0 is input until the transmission of the code sequence is started. In addition, the picture rearrangement unit 121 determines the picture type of each picture and outputs information indicating which picture each picture uses as a reference picture to the inter prediction unit 130. Here, the picture type is an I picture, a P picture, and a B picture.

[0077] Next, the code block splitting unit 122 splits the input image 153 into code blocks 171 (S123).

[0078] Next, the intra prediction unit 129 generates a prediction block for intra prediction and calculates the cost of the prediction block (S124). The inter prediction unit 130 generates a prediction block for inter prediction and calculates the cost of the prediction block (S125). The selection unit 131 determines the prediction mode and the prediction block 177 to be used by using the calculated costs and the like (S126).

[0079] Next, the subtraction unit 123 generates a difference block 172 by calculating the difference between the prediction block 177 and the sign block 171 (S127). Next, the transform quantization unit 124 generates transform coefficients 173 by performing frequency conversion and quantization on the difference block 172 (S128). Next, the inverse transform quantization unit 126 restores the difference block 174 by performing inverse quantization and inverse frequency conversion on the transform coefficients 173 (S129). Next, the addition unit 127 generates a decoded block 175 by adding the prediction block 177 and the difference block 174 (S130). This decoded block 175 is stored in the frame memory 128 and used for the prediction processing by the intra prediction unit 129 and the inter prediction unit 130.

[0080] Next, the variable length coding unit 125 encodes prediction information 178 indicating the used prediction mode and the like (S131) and encodes the transform coefficients 173 (S132).

[0081] Then, the process proceeds to the next sign block (S133), and the coding unit 102 repeats steps S124 to S133 until the processing of all the sign blocks in the picture is completed (S134).

[0082] Then, the coding unit 102 repeats steps S122 to S134 until the processing of all the pictures is completed (S135).

[0083] <Effect> As described above, the image encoding apparatus 100 according to the present embodiment determines the encoding structure based on the frame rate 151 and the transmission delay time limit value 152. As a result, when the frame rate 151 is high, the image encoding apparatus 100 can increase the depth of the hierarchy without extending the display delay time of the decoder and the transmission delay time of the encoder, so that the time scalability can be enhanced. In addition, the compression performance can be improved by increasing the number of B pictures.

[0084] Also, even in the case of various frame rates, it is possible not to exceed the specified display delay time of the decoder and the transmission delay time of the encoder.

[0085] This will be described more specifically. FIG. 10 is a diagram showing the number of hierarchical levels 161, the number of display delay pictures 164, and the number of consecutive B pictures 162 calculated based on the frame rate 151 and the transmission delay time limit value 152. In addition, FIG. 10 shows an example when the transmission delay time limit value is 4 / 30 seconds.

[0086] In addition, FIGS. 11A to 11D are diagrams showing the encoding structure based on the conditions of FIG. 10. FIG. 11A shows the structure when the frame rate is 24 fps. FIG. 11B shows the structure when the frame rate is 30 fps. FIG. 11C shows the structure when the frame rate is 60 fps. FIG. 11D shows the structure when the frame rate is 120 fps.

[0087] In addition, the number of transmission delay pictures in the code sequence is shown in FIGS. 9A to 9D when the frame rates are 24 fps, 30 fps, 60 fps, and 120 fps, respectively. In addition, FIGS. 12A to 12D are diagrams showing the number of display delay pictures when the frame rates are 24 fps, 30 fps, 60 fps, and 120 fps.

[0088] As shown in FIG. 10, at all frame rates, the transmission delay time does not exceed the limit value of 4 / 30 seconds. Also, in the current operation regulations (Non-Patent Document 2), as shown in FIG. 11B, the encoding structure is limited up to three layers. That is, as shown in FIG. 12B, the number of display delay pictures is limited up to 2, and as shown in FIG. 9B, the number of transmission delay pictures in the code stream is limited up to 4. Also, in the case of 30 fps, the display delay time is 2 / 30 seconds, and the transmission delay time is 4 / 30 seconds. In this embodiment, when the transmission delay time limit value is set to 4 / 30 seconds, even if the number of layers is increased or decreased according to the frame rate, the transmission delay time does not exceed 4 / 30 seconds, and the display delay time does not exceed 2 / 30 seconds.

[0089] Also, in this embodiment, the image encoding apparatus 100 determines the encoding structure using the limit value of the transmission delay time instead of the display delay time. By thus limiting the encoding structure by the transmission delay time, the encoding structure can be determined so as not to exceed both the display delay time of 2 / 30 seconds and the transmission delay time of 4 / 30 seconds in the current operation regulations (Non-Patent Document 2). More specifically, when determining the number of display delay pictures so as not to exceed the display delay time of 2 / 30 seconds in the current operation regulations (Non-Patent Document 2), at 120 fps, the number of display delay pictures is 8 (8 / 120 seconds), and an encoding structure up to nine layers is permitted. However, when using an encoding structure of nine layers, the number of transmission delay pictures becomes 256 (256 / 120 seconds), greatly exceeding the transmission delay time of 4 / 30 seconds in the current operation regulations (Non-Patent Document 2). On the other hand, paying attention to the transmission delay time and determining the number of transmission delay pictures so as not to exceed the transmission delay time of 4 / 30 seconds, at 120 fps, the number of transmission delay pictures is limited to 16 (16 / 120 seconds), and the encoding structure is limited up to five layers. In this case, the display delay time does not exceed 2 / 30 seconds. Thus, by limiting the transmission delay time, both the transmission delay time and the display delay time can be appropriately limited.

[0090] Further, the image encoding device 100 sets the limit values of the encoding structure for each layer. Thereby, even in an image decoding device that decodes only pictures of a layer with a small TemporalId, the display delay time can be prevented from exceeding the specified time.

[0091] In the above description, the image encoding device 100 calculates the encoding structure limit values such as the number of layers by an arithmetic expression. However, a table shown in FIG. 10 may be stored in the memory in advance, and the encoding structure limit values corresponding to the frame rate 151 and the transmission delay time limit value 152 may be set with reference to the table. Further, the image encoding device 100 may use both the table and the arithmetic expression. For example, when the frame rate is 24 fps or less, the image encoding device 100 may set the encoding structure limit value using the table, and when the frame rate exceeds 24 fps, the image encoding device 100 may set the encoding structure limit value using the arithmetic expression.

[0092] Also, in the above description, the image encoding device 100 uses the transmission delay time limit value 152 and the frame rate 151 input from the outside, but this is not the only case. For example, the image encoding device 100 may use a fixed value determined in advance as at least one of the transmission delay time limit value 152 and the frame rate 151. Further, the image encoding device 100 may determine at least one of the transmission delay time limit value 152 and the frame rate 151 according to an internal state such as a buffer memory.

[0093] Also, the encoding structures shown in FIGS. 11A to 11D are merely examples and are not limited thereto. For example, the arrows indicating the reference images are not limited to this, and each picture may use, as a reference image, a picture having a TemporalId larger than its own TemporalId. For example, image B shown in FIG. 11B 1 may use image P 4 as a reference image.

[0094] Also, the limit values (number of hierarchical levels, number of consecutive B pictures, and number of pictures for display delay) of the above-described encoding structure are merely maximum values, and values smaller than the limit values may be used depending on the situation. For example, in the case where the frame rate is 30 fps in FIG. 10, the number of hierarchical levels is 3, the number of consecutive B pictures is 3[2], and the number of pictures for display delay is 2[2], and an encoding structure as shown in FIG. 11B is shown. However, the number of hierarchical levels may be 3 or less, and the number of consecutive B pictures and the number of pictures for display delay may be values corresponding to the number of hierarchical levels of 3 or less. For example, the number of hierarchical levels may be 2, the number of consecutive B pictures may be 2[1], and the number of pictures for display delay may be 2[2]. In this case, for example, the encoding structure shown in FIG. 11A is used. In that case, in the encoding structure encoding of step S121 shown in FIG. 8, information indicating the used encoding structure is encoded.

[0095] Also, in the above description, the number of pictures for display delay is set to sps_max_num_reorder_pics, but sps_max_num_reorder_pics may be a variable representing the number of pictures whose order is changed. For example, in the example shown in FIG. 9C, the input image 8, the input image 4, and the input image 2 are rearranged and encoded so as to be before the positions in the input order (display order). In this case, the number of pictures whose order is changed is 3, and this value 3 may be set to sps_max_num_reorder_pics.

[0096] Also, in the above description, the consecutive number parameter is set to sps_max_latency_increase_plus1, and the value of sps_max_num_reorder_pics + sps_max_latency_increase_plus1 - 1 (SpsMaxLatencyPictures) is treated as the number of consecutive B pictures. However, SpsMaxLatencyPictures may indicate the maximum value of the number of picture decoding times, which is the number of pictures decoded after the picture is stored in the buffer until it can be displayed after the decoding of the picture is completed. For example, in the case of the image P in FIG. 12B 4 In the case of the image P 4 After the decoding of the image P is completed, the image B 2 The image B1 and Image B 3 After three images, i.e., Image P 4 becomes displayable. Also, Image B 2 Image B 3 and Image P 4 are displayed in sequence. 3, which is the maximum number of picture decoding times, may be set to SpsMaxLatencyPictures.

[0097] In this embodiment, sps_max_num_reorder_pics and sps_max_latency_increase_plus1 are set for each layer and encoded, but this is not restrictive. For example, in a system that does not use temporal scalability, only the values of sps_max_num_reorder_pics and sps_max_latency_increase_plus1 for the deepest layer (the layer with the largest TemporalId) may be set and encoded.

[0098] In the above description, the frame rates are four types: 24fps, 30fps, 60fps, and 120fps, but other frame rates may be used. Also, the frame rate may be a numerical value including a decimal, such as 29.97fps.

[0099] Furthermore, the processing in this embodiment may be realized by software. And this software may be distributed by downloading or the like. Also, this software may be recorded on a recording medium such as a CD-ROM and distributed. Note that this also applies to other embodiments in this specification.

[0100] (Embodiment 2) In this embodiment, an image decoding apparatus corresponding to the image encoding apparatus described in Embodiment 1 will be described.

[0101] <Overall Configuration> FIG. 13 is a block diagram showing the configuration of an image decoding apparatus 200 in this embodiment.

[0102] The image decoder 200 shown in FIG. 13 generates an output image 263 by decoding the code sequence 251. The code sequence 251 is, for example, the code sequence 155 generated by the image encoder 100 in Embodiment 1. This image decoder 200 includes a variable-length decoder 201, an inverse transform quantization unit 202, an addition unit 203, a frame memory 204, an intra prediction block generation unit 205, an inter prediction block generation unit 206, a constraint value decoder 208, an image rearrangement unit 209, and a coding structure confirmation unit 210.

[0103] <Operation (Overall)> Next, with reference to FIG. 14, the image decoding process according to this embodiment will be described.

[0104] First, the variable-length decoder 201 decodes the coding structure constraint value 257 from the code sequence 251. This coding structure constraint value 257 includes sps_max_sub_layers_minus1, sps_max_num_reorder_pics, and sps_max_latency_increase_plus1. Note that the meanings of these pieces of information are the same as those in Embodiment 1. Next, the constraint value decoder 208 obtains the number of layers by adding 1 to sps_max_sub_layers_minus1, obtains the number of consecutive B pictures by the formula of sps_max_num_reorder_pics + sps_max_latency_increase_plus1 - 1, and obtains the number of display delay pictures by sps_max_num_reorder_pics (S201). Further, the constraint value decoder 208 obtains the coding structure 262 (number of layers, number of display delay pictures, and number of consecutive B pictures) from sps_max_num_reorder_pics and sps_max_latency_increase_plus1 of the layer of the TemporalId corresponding to the value of HighestTId252 input from the outside, and outputs the obtained coding structure 262 to the image rearrangement unit 209 and the coding structure confirmation unit 210. Here, HighestTId252 indicates the TemporalId of the maximum layer to be decoded.

[0105] Next, the encoding structure confirmation unit 210 checks whether each value of the encoding structure 262 complies with the operation regulations (S202). Specifically, the encoding structure confirmation unit 210 uses the transmission delay time limit value 253 input from the outside and the frame rate 256 obtained by variable-length decoding of the code sequence 251 to calculate each limit value according to the following (Equation 7) to (Equation 9), and determines whether the encoding structure is less than or equal to the calculated limit value.

[0106] Number of hierarchical levels = int(log 2 (Transmission delay time limit value [s] × Frame rate [fps])) + 1 ··· (Equation 7)

[0107] Number of display delay pictures [TId] = int(log 2 (Transmission delay time limit value [s] × Frame rate [fps] ÷ 2 (n-TId) )) ··· (Equation 8)

[0108] Number of consecutive B pictures [TId] = int(Transmission delay time limit value [s] × Frame rate [fps] ÷ 2 (n-TId) - 1) ··· (Equation 9)

[0109] If the encoding structure is larger than the limit value (Yes in S203), the encoding structure confirmation unit 210 displays an error to that effect (S204) and ends the decoding process.

[0110] Next, the variable-length decoding unit 201 decodes prediction information 255 indicating the prediction mode from the code sequence 251 (S205). If the prediction mode is intra prediction (Yes in S206), the intra prediction block generation unit 205 generates a prediction block 261 by intra prediction (S207). On the other hand, if the prediction mode is inter prediction (No in S206), the inter prediction block generation unit 206 generates a prediction block 261 by inter prediction (S208).

[0111] Next, the variable-length decoding unit 201 decodes the transform coefficients 254 from the code sequence 251 (S209). Next, the inverse transform quantization unit 202 restores the differential block 258 by performing inverse quantization and inverse frequency conversion on the transform coefficients 254 (S210). Next, the addition unit 203 generates a decoded block 259 by adding the differential block 258 and the prediction block 261 (S211). This decoded block 259 is stored in the frame memory 204 and used for prediction block generation processing by the intra prediction block generation unit 205 and the inter prediction block generation unit 206.

[0112] Then, the image decoding apparatus 200 moves the process to the next code block (S212), and repeats steps S205 to S212 until the processing of all the code blocks in the picture is completed (S213).

[0113] Note that the processing in steps S205 to S212 is performed only on pictures having a TemporalId less than or equal to HighestTId252 input from the outside.

[0114] Next, the image rearrangement unit 209 rearranges the decoded picture according to the coding structure 262 of the layer of HighestTId252 input from the outside, and outputs the rearranged decoded picture as the output image 263 (S214).

[0115] Then, the image decoding apparatus 200 repeats steps S205 to S214 until the processing of all pictures is completed (S215).

[0116] <Effect> As described above, the image decoding apparatus 200 according to the present embodiment can decode a code sequence generated by efficient coding. Further, the image decoding apparatus 200 can check whether the coding structure conforms to the operation regulations, stop the decoding process if it does not conform, and perform an error display.

[0117] Note that in the above description, the image decoding apparatus 200 decodes only pictures of layers equal to or lower than HighestTId252 in accordance with HighestTId252 input from the outside, but this is not the limit. The image decoding apparatus 200 may always decode pictures of all layers. Also, the image decoding apparatus 200 may use a fixed value predetermined as HighestTId252 and always decode only pictures of a predetermined layer indicated by HighestTId252.

[0118] Also, in the above description, the image decoding apparatus 200 checks whether the encoding structure 262 conforms to the operation regulations, but this function is not essential and it is not necessary to check the encoding structure 262.

[0119] Also, in the above description, the image decoding apparatus 200 uses the transmission delay time limit value 253 input from the outside, but a fixed value predetermined as the transmission delay time limit value 253 may be used.

[0120] Others are the same as those in the first embodiment, and thus are omitted.

[0121] Note that the order of each flow is not limited to the above as in the encoding side.

[0122] As described above in the first and second embodiments, the image encoding apparatus 100 according to the first embodiment is an image encoding apparatus that generates a code sequence 155 (bit stream) by hierarchically encoding an input image 153, and performs the processing shown in FIG. 15.

[0123] First, the image encoding device 100 determines the number of layers 161 in hierarchical encoding so that the number of layers 161 is less than or equal to the maximum number of layers determined in advance according to the frame rate (S301). Here, the maximum number of layers is the number of layers shown in FIG. 10. For example, when the frame rate is 24 fps, it is 2; when the frame rate is 30 fps, it is 3; when the frame rate is 60 fps, it is 4; when the frame rate is 120 fps, it is 5. In other words, when the frame rate is 60 fps or higher, the maximum number of layers is 4 or more. Also, when the frame rate is 60 fps or lower, the maximum number of layers is 4 or less. Further, when the frame rate is greater than 30 fps, the maximum number of layers is greater than 3.

[0124] Note that the image encoding device 100 further determines the picture type of the input image 153 so that the number of display delay pictures 164 is less than or equal to the maximum number of pictures determined in advance according to the frame rate. Here, the number of display delay pictures 164 is the number of pictures from when the image decoding device starts decoding the code sequence 155 generated by the image encoding device 100 until it outputs (displays) the decoded image. Also, the picture type is an I picture, a P picture, or a B picture. Here, the maximum number of pictures is the number of display delay pictures shown in FIG. 10. For example, when the frame rate is 24 fps, it is 1; when the frame rate is 30 fps, it is 2; when the frame rate is 60 fps, it is 3; when the frame rate is 120 fps, it is 4. In other words, when the frame rate is 60 fps or higher, the maximum number of pictures is 3 or more. Also, when the frame rate is 60 fps or lower, the maximum number of pictures is 3 or less. Further, when the frame rate is greater than 30 fps, the maximum number of pictures is greater than 2.

[0125] Further, the image encoding device 100 further determines the picture type of the input image 153 such that the B-picture consecutive number 162, which is the number of consecutive B pictures, is equal to or less than the maximum consecutive number determined in advance according to the frame rate. Here, the maximum consecutive number is the number of display delay pictures shown in FIG. 10. For example, when the frame rate is 24 fps, it is 2; when the frame rate is 30 fps, it is 3; when the frame rate is 60 fps, it is 7; and when the frame rate is 120 fps, it is 15. In other words, when the frame rate is 60 fps or higher, the maximum consecutive number is 7 or more. When the frame rate is 60 fps or lower, the maximum consecutive number is 7 or less. Also, when the frame rate is greater than 30 fps, the maximum consecutive number is greater than 3.

[0126] Also, as shown in FIG. 10, the image encoding device 100 may determine the maximum number of layers, the maximum number of pictures, and the B-picture consecutive number according to the frame rate. That is, the higher the frame rate, the more the image encoding device 100 may set the maximum number of layers, the maximum number of pictures, and the B-picture consecutive number.

[0127] Also, as described above, the number of layers 161, the display delay picture number 164, and the B-picture consecutive number 162 are calculated by the above (Equation 1), (Equation 3), and (Equation 4) using the frame rate 151 and the transmission delay time limit value 152. That is, the following relationship is satisfied among the maximum number of pictures, the encoder transmission delay (transmission delay time) which is the time from when the input image 153 is input to the image encoding device 100 until the code sequence 155 is output, and the frame rate.

[0128] Maximum number of pictures = int(log 2 (Encoder transmission delay [s] × Frame rate [fps]))

[0129] Also, the following relationship is satisfied among the maximum consecutive number, the encoder transmission delay, and the frame rate.

[0130] Maximum consecutive number = int(Encoder transmission delay [s] × Frame rate [fps] - 1)

[0131] The maximum number of layers, the encoder transmission delay, and the frame rate satisfy the following relationship.

[0132] Maximum number of layers = int(log 2 (Encoder transmission delay [s] × Frame rate [fps])) + 1

[0133] Also, the maximum number of pictures [i] in each layer, the encoder transmission delay, and the frame rate satisfy the following relationship.

[0134] Maximum number of pictures [i] = int(log 2 (Encoder transmission delay [s] × Frame rate [fps] / 2 (n-i) ))

[0135] The maximum consecutive number [i] in each layer, the encoder transmission delay, and the frame rate satisfy the following relationship.

[0136] Maximum consecutive number [i] = int(Encoder transmission delay [s] × Frame rate [fps] / 2 (n-i) - 1)

[0137] Here, i is an integer less than or equal to the maximum number of layers and indicates the layer. n indicates (the maximum number of layers - 1).

[0138] Next, the image encoding device 100 generates a code sequence 155 by hierarchically encoding the input image 153 with the determined number of layers 161 and picture type (S302). Also, the image encoding device 100 encodes the first information (sps_max_sub_layers_minus1), the second information (sps_max_num_reorder_pics), and the third information (sps_max_latency_increase_plus1) indicating the determined number of layers 161, the display delay picture number 164, and the B picture consecutive number 162.

[0139] Also, the image decoding device 200 according to Embodiment 2 is an image decoding device that generates an output image 263 by decoding a code sequence 251 (bitstream) obtained by hierarchically encoding an image, and performs the processing shown in FIG. 16.

[0140] First, the image decoding device 200 decodes an image from the code sequence 251 (S401).

[0141] Next, the image decoding device 200 decodes first information (sps_max_sub_layers_minus1) indicating the number of layers in the hierarchical encoding from the code sequence 251 (S402). For example, this number of layers is equal to or less than the maximum number of layers determined in advance according to the frame rate of the code sequence 251.

[0142] Furthermore, the image decoding device 200 decodes second information (sps_max_num_reorder_pics) indicating the number of display delay pictures from the code sequence 251. Also, the image decoding device 200 further decodes third information (sps_max_latency_increase_plus1) indicating the number of consecutive B pictures from the code sequence 251.

[0143] Next, the image decoding device 200 rearranges and outputs the decoded image by using the number of layers indicated by the first information, the number of display delay pictures indicated by the second information, and the number of consecutive B pictures indicated by the third information (S403).

[0144] Note that specific examples and limitations of the maximum number of layers, which is the maximum value of the number of layers, the maximum number of pictures, which is the maximum value of the number of display delay pictures, and the maximum number of consecutive pictures, which is the maximum value of the number of consecutive B pictures, are the same as those in the case of the image encoding device 100. Also, the relationships among the maximum number of layers, the maximum number of pictures, the maximum number of consecutive pictures, the frame rate, and the encoder transmission delay are the same as those in the case of the image encoding device 100.

[0145] As described above, the image decoding device and the image encoding device according to the embodiment have been described, but the present invention is not limited to this embodiment.

[0146] Also, each processing unit included in the image decoding device or the image encoding device according to the above embodiment is typically realized as an LSI which is an integrated circuit. These may be individually integrated into one chip, or may be integrated into one chip so as to include some or all of them.

[0147] Also, the integration into an integrated circuit is not limited to an LSI, and it may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.

[0148] In each of the above embodiments, each component may be configured by dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0149] In other words, the image decoding device and the image encoding device include a processing circuitry and a storage electrically connected to (accessible from) the processing circuitry. The processing circuitry includes at least one of dedicated hardware and a program execution unit. Further, when the processing circuitry includes a program execution unit, the storage stores a software program executed by the program execution unit. The processing circuitry uses the storage to execute the image decoding method or the image encoding method according to the above embodiment.

[0150] Furthermore, the present invention may be the above software program, or may be a non-transitory computer-readable recording medium on which the above program is recorded. Needless to say, the above program can be distributed via a transmission medium such as the Internet.

[0151] In addition, all the numbers used above are for illustrative purposes to specifically describe the present invention, and the present invention is not limited to the illustrated numbers.

[0152] In addition, the division of functional blocks in the block diagram is an example, and a plurality of functional blocks may be realized as one functional block, one functional block may be divided into a plurality, or part of the functions may be transferred to other functional blocks. Also, the functions of a plurality of functional blocks having similar functions may be processed by a single piece of hardware or software in parallel or time-division.

[0153] In addition, the order in which the steps included in the above image decoding method or image encoding method are executed is for illustrative purposes to specifically describe the present invention, and other orders may also be possible. Also, some of the above steps may be executed simultaneously (in parallel) with other steps.

[0154] In addition, the processing described in the above embodiments may be realized by centralized processing using a single device (system), or may be realized by distributed processing using a plurality of devices. Also, the computer that executes the above program may be singular or plural. That is, centralized processing or distributed processing may be performed.

[0155] In addition, the present invention is particularly effective in cases such as broadcasts for many end-users where the functions of the receiving terminals are diverse. For example, the signal of the data structure described above is broadcast. A terminal such as a 4K2K TV can decode full-layer data. On the other hand, a smartphone can decode up to the second layer. Also, depending on the congestion situation of the band, the transmitting device can send only the upper layer instead of the full layer. This enables flexible broadcasting and communication.

[0156] As described above, the image decoding device and the image encoding device according to one or more aspects of the present invention have been described based on the embodiments. However, the present invention is not limited to these embodiments. Without departing from the spirit of the present invention, various modifications conceived by those skilled in the art applied to these embodiments, or forms constructed by combining components in different embodiments, may also be included within the scope of one or more aspects of the present invention.

[0157] (Embodiment 3) By recording a program for realizing the configuration of the moving image encoding method (image encoding method) or the moving image decoding method (image decoding method) shown in each of the above embodiments on a storage medium, the processing shown in each of the above embodiments can be easily implemented in an independent computer system. The storage medium may be any medium capable of recording a program, such as a magnetic disk, an optical disk, a magneto-optical disk, an IC card, a semiconductor memory, or the like.

[0158] Furthermore, here, application examples of the moving image encoding method (image encoding method) and the moving image decoding method (image decoding method) shown in each of the above embodiments and a system using the same will be described. The system is characterized by having an image encoding / decoding device including an image encoding device using the image encoding method and an image decoding device using the image decoding method. Other configurations in the system can be appropriately changed as the case may be.

[0159] FIG. 17 is a diagram showing the overall configuration of a content supply system ex100 for realizing a content distribution service. The communication service providing area is divided into a desired size, and base stations ex106, ex107, ex108, ex109, and ex110, which are fixed radio stations, are installed in each cell.

[0160] This content supply system ex100 has various devices such as a computer ex111, a PDA (Personal Digital Assistant) ex112, a camera ex113, a mobile phone ex114, and a game console ex115 connected thereto via an Internet service provider ex102, a telephone network ex104, and a base station ex106 from the Internet ex101 through ex110.

[0161] However, the content supply system ex100 is not limited to the configuration as shown in FIG. 17, and any elements may be combined and connected. Also, each device may be directly connected to the telephone network ex104 without going through ex110 from the base station ex106 which is a fixed radio station. Further, each device may be directly connected to each other via short-range wireless or the like.

[0162] The camera ex113 is a device capable of shooting videos such as a digital video camera, and the camera ex116 is a device capable of shooting still images and videos such as a digital camera. Also, the mobile phone ex114 may be a mobile phone of the GSM (registered trademark) (Global System for Mobile Communications) system, CDMA (Code Division Multiple Access) system, W-CDMA (Wideband-Code Division Multiple Access) system, or LTE (Long Term Evolution) system, HSPA (High Speed Packet Access), or a PHS (Personal Handyphone System) or the like, and any of them is acceptable.

[0163] In the content supply system ex100, a camera ex113 or the like is connected to a streaming server ex103 through a base station ex109 and a telephone network ex104, enabling live distribution and the like. In live distribution, content captured by a user using a camera ex113 (for example, video of a music live performance, etc.) is subjected to encoding processing as described in each of the above embodiments (that is, functions as an image encoding device according to an aspect of the present invention) and transmitted to the streaming server ex103. On the other hand, the streaming server ex103 streams the transmitted content data to the requested client. Examples of the client include a computer ex111, a PDA ex112, a camera ex113, a mobile phone ex114, a game console ex115, etc., which are capable of decrypting the encoded data. Each device that receives the distributed data decrypts and plays back the received data (that is, functions as an image decoding device according to an aspect of the present invention).

[0164] Note that the encoding process of the captured data may be performed by the camera ex113, the streaming server ex103 that performs the data transmission process, or they may share the process. Similarly, the decoding process of the distributed data may be performed by the client, the streaming server ex103, or they may share the process. Also, not limited to the camera ex113, still image and / or moving image data captured by a camera ex116 may be transmitted to the streaming server ex103 via the computer ex111. In this case, the encoding process may be performed by any of the camera ex116, the computer ex111, the streaming server ex103, or they may share the process.

[0165] Also, these encoding / decoding processes are generally processed in a computer ex111 or an LSI ex500 possessed by each device. The LSI ex500 may be a one-chip configuration or a configuration consisting of multiple chips. Note that software for moving image encoding / decoding may be incorporated into some recording medium (such as a CD-ROM, a flexible disk, a hard disk, etc.) readable by a computer ex111 or the like, and the encoding / decoding process may be performed using the software. Further, when the mobile phone ex114 has a camera, the moving image data acquired by the camera may be transmitted. The moving image data at this time is data encoded by the LSI ex500 possessed by the mobile phone ex114.

[0166] Also, the streaming server ex103 may be a plurality of servers or a plurality of computers that distribute, record, and deliver data in a distributed manner.

[0167] As described above, in the content supply system ex100, the client can receive and reproduce the encoded data. In this way, in the content supply system ex100, the information transmitted by the user can be received, decoded, and reproduced by the client in real time, and even a user without special rights or facilities can realize personal broadcasting.

[0168] Note that, not limited to the example of the content supply system ex100, as shown in FIG. 18, at least any one of the moving image encoding device (image encoding device) or the moving image decoding device (image decoding device) of the above-described embodiments can also be incorporated into the digital broadcast system ex200. Specifically, in the broadcasting station ex201, multiplexed data in which music data and the like are multiplexed with video data is transmitted via radio waves to a communication or a satellite ex202. This video data is data encoded by the moving image encoding method described in the above-described embodiments (that is, data encoded by the image encoding device according to one aspect of the present invention). The receiving broadcast satellite ex202 transmits radio waves for broadcasting, and an antenna ex204 of a home capable of receiving satellite broadcasts receives this radio wave. The received multiplexed data is decoded and reproduced by a device such as a television (receiver) ex300 or a set-top box (STB) ex217 (that is, functions as an image decoding device according to one aspect of the present invention).

[0169] Also, it is possible to implement the moving image decoding device or the moving image encoding device shown in the above-described embodiments in a reader / recorder ex218 that reads and decodes multiplexed data recorded on a recording medium ex215 such as a DVD or a BD, or encodes a video signal on the recording medium ex215 and, in some cases, multiplexes and writes it with a music signal. In this case, the reproduced video signal is displayed on a monitor ex219, and the video signal can be reproduced by other devices and systems using the recording medium ex215 on which the multiplexed data is recorded. Further, a moving image decoding device may be implemented in a set-top box ex217 connected to a cable ex203 for cable television or an antenna ex204 for satellite / terrestrial wave broadcast, and this may be displayed on a monitor ex219 of a television. At this time, instead of the set-top box, a moving image decoding device may be incorporated in the television.

[0170] FIG. 19 is a diagram showing a television (receiver) ex300 using the moving image decoding method and the moving image encoding method described in each of the above embodiments. The television ex300 acquires or outputs multiplexed data in which audio data is multiplexed with video data via an antenna ex204 or a cable ex203 or the like that receives the above broadcast, and a tuner ex301, and demodulates the received multiplexed data or modulates it into multiplexed data to be transmitted externally, and a modulation / demodulation unit ex302, and a multiplexing / demultiplexing unit ex303 that separates the demodulated multiplexed data into video data and audio data or multiplexes the video data and audio data encoded by the signal processing unit ex306.

[0171] Also, the television ex300 has a signal processing unit ex306 having an audio signal processing unit ex304 and a video signal processing unit ex305 (functioning as an image encoding device or an image decoding device according to an aspect of the present invention) that decode audio data and video data respectively or encode respective information, and an output unit ex309 having a speaker ex307 that outputs the decoded audio signal and a display unit ex308 such as a display that displays the decoded video signal. Further, the television ex300 has an interface unit ex317 having an operation input unit ex312 or the like that receives input of user operations. Further, the television ex300 has a control unit ex310 that comprehensively controls each unit and a power supply circuit unit ex311 that supplies power to each unit. The interface unit ex317 may have, in addition to the operation input unit ex312, a bridge ex313 connected to an external device such as a reader / recorder ex218, a slot unit ex314 that enables mounting of a recording medium ex216 such as an SD card, a driver ex315 connected to an external recording medium such as a hard disk, a modem ex316 connected to a telephone network, and the like. Note that the recording medium ex216 enables electrical recording by a non-volatile / volatile semiconductor memory element for storing. Each unit of the television ex300 is connected to each other via a synchronization bus.

[0172] First, a configuration in which the TV ex300 decodes and plays multiplexed data acquired from the outside by the antenna ex204 or the like will be described. The TV ex300 receives a user operation from a remote controller ex220 or the like, and based on the control of a control unit ex310 having a CPU or the like, separates the multiplexed data demodulated by a modulation / demodulation unit ex302 in a multiplexing / demultiplexing unit ex303. Further, the TV ex300 decodes the separated audio data in an audio signal processing unit ex304, and decodes the separated video data in a video signal processing unit ex305 using the decoding method described in each of the above embodiments. The decoded audio signal and video signal are output from an output unit ex309 toward the outside. When outputting, these signals may be temporarily stored in buffers ex318, ex319, etc. so that the audio signal and the video signal are played back synchronously. Also, the TV ex300 may read multiplexed data from recording media ex215, ex216 such as a magnetic / optical disk or an SD card, instead of from a broadcast or the like. Next, a configuration in which the TV ex300 encodes an audio signal or a video signal and transmits it to the outside or writes it to a recording medium or the like will be described. The TV ex300 receives a user operation from a remote controller ex220 or the like, and based on the control of the control unit ex310, encodes the audio signal in the audio signal processing unit ex304, and encodes the video signal in the video signal processing unit ex305 using the encoding method described in each of the above embodiments. The encoded audio signal and video signal are multiplexed in the multiplexing / demultiplexing unit ex303 and output to the outside. When multiplexing, these signals may be temporarily stored in buffers ex320, ex321, etc. so that the audio signal and the video signal are synchronized. Note that a plurality of buffers ex318, ex319, ex320, ex321 may be provided as shown in the figure, or a configuration in which one or more buffers are shared may be used. Further, in addition to what is shown in the figure, for example, data may be stored in a buffer as a buffer material for avoiding system overflow and underflow between the modulation / demodulation unit ex302 and the multiplexing / demultiplexing unit ex303 or the like.

[0173] In addition to acquiring audio data and video data from broadcasts, recording media, etc., the TV ex300 is configured to accept AV inputs from microphones and cameras, and may perform encoding processing on the data acquired from them. Here, the TV ex300 has been described as being configured to perform the above encoding processing, multiplexing, and external output, but it may be configured such that these processes cannot be performed and only the above reception, decoding processing, and external output are possible.

[0174] Also, when reading or writing multiplexed data from a recording medium with the reader / writer ex218, the above decoding processing or encoding processing may be performed by either the TV ex300 or the reader / writer ex218, or the TV ex300 and the reader / writer ex218 may share the processing.

[0175] As an example, FIG. 20 shows the configuration of the information reproduction / recording unit ex400 when reading or writing data from / to an optical disk. The information reproduction / recording unit ex400 includes elements ex401, ex402, ex403, ex404, ex405, ex406, and ex407 described below. The optical head ex401 irradiates a laser spot on the recording surface of the recording medium ex215, which is an optical disk, to write information, and detects the reflected light from the recording surface of the recording medium ex215 to read information. The modulation recording unit ex402 electrically drives the semiconductor laser built in the optical head ex401 and modulates the laser light according to the recording data. The reproduction demodulation unit ex403 amplifies the reproduction signal obtained by electrically detecting the reflected light from the recording surface by the photodetector built in the optical head ex401, separates and demodulates the signal components recorded on the recording medium ex215, and reproduces the necessary information. The buffer ex404 temporarily holds the information to be recorded on the recording medium ex215 and the information reproduced from the recording medium ex215. The disk motor ex405 rotates the recording medium ex215. The servo control unit ex406 moves the optical head ex401 to a predetermined information track while controlling the rotational drive of the disk motor ex405, and performs tracking processing of the laser spot. The system control unit ex407 controls the entire information reproduction / recording unit ex400. The above reading and writing processes are realized by the system control unit ex407 using various information held in the buffer ex404, generating and adding new information as necessary, and causing the modulation recording unit ex402, the reproduction demodulation unit ex403, and the servo control unit ex406 to operate in cooperation, and performing information recording and reproduction through the optical head ex401. The system control unit ex407 is composed of, for example, a microprocessor, and executes those processes by executing a reading and writing program.

[0176] In the above, the optical head ex401 has been described as irradiating a laser spot, but a configuration using near-field light for higher-density recording may also be used.

[0177] Fig. 21 shows a schematic diagram of a recording medium ex215 which is an optical disc. On the recording surface of the recording medium ex215, guide grooves are formed in a spiral shape. On the information track ex230, address information indicating the absolute position on the disc is recorded in advance by changes in the shape of the grooves. This address information includes information for specifying the position of a recording block ex231 which is a unit for recording data, and in a device for recording or playing back, the recording block can be specified by playing back the information track ex230 and reading the address information. Further, the recording medium ex215 includes a data recording area ex233, an inner peripheral area ex232, and an outer peripheral area ex234. The area used for recording user data is the data recording area ex233, and the inner peripheral area ex232 and the outer peripheral area ex234 arranged inside or outside the data recording area ex233 are used for specific purposes other than recording user data. The information playback / recording unit ex400 reads and writes encoded audio data, video data, or multiplexed data obtained by multiplexing these data to the data recording area ex233 of such a recording medium ex215.

[0178] In the above, an optical disc such as a single-layer DVD or BD has been described as an example, but it is not limited to these, and an optical disc having a multi-layer structure and capable of recording not only on the surface may be used. Further, an optical disc having a structure for performing multi-dimensional recording / playback, such as recording information using lights of different colors with different wavelengths at the same location on the disc, or recording layers of different information from different angles, may be used.

[0179] Also, in the digital broadcast system ex200, it is also possible to receive data from a satellite ex202 or the like with a vehicle ex210 having an antenna ex205 and play back a video on a display device such as a car navigation ex211 included in the vehicle ex210. Note that the configuration of the car navigation ex211 may be a configuration in which a GPS receiving unit is added, for example, among the configurations shown in Fig. 19, and the same applies to a computer ex111, a mobile phone ex114, or the like.

[0180] FIG. 22A is a diagram showing a mobile phone ex114 using the moving image decoding method and the moving image encoding method described in the above embodiment. The mobile phone ex114 includes an antenna ex350 for transmitting and receiving radio waves to and from a base station ex110, a camera unit ex365 capable of capturing video and still images, and a display unit ex358 such as a liquid crystal display for displaying data obtained by decoding video captured by the camera unit ex365, video received by the antenna ex350, and the like. The mobile phone ex114 further includes a main body unit having an operation key unit ex366, an audio output unit ex357 such as a speaker for outputting audio, an audio input unit ex356 such as a microphone for inputting audio, a memory unit ex367 for storing encoded or decoded data such as captured video, still images, recorded audio, or received video, still images, mails, etc., or a slot unit ex364 which is an interface unit with a recording medium for storing data in the same manner.

[0181] Furthermore, a configuration example of the mobile phone ex114 will be described with reference to FIG. 22B. In the mobile phone ex114, a power supply circuit unit ex361, an operation input control unit ex362, a video signal processing unit ex355, a camera interface unit ex363, an LCD (Liquid Crystal Display) control unit ex359, a modulation / demodulation unit ex352, a multiplexing / demultiplexing unit ex353, an audio signal processing unit ex354, a slot unit ex364, and a memory unit ex367 are connected to each other via a bus ex370 with respect to a main control unit ex360 that comprehensively controls each part of the main body unit including the display unit ex358 and the operation key unit ex366.

[0182] When the end call and the power key are turned on by a user's operation, the power supply circuit unit ex361 starts the mobile phone ex114 in an operable state by supplying power from a battery pack to each part.

[0183] When the mobile phone ex114 is in the voice call mode, based on the control of the main control unit ex360 having a CPU, ROM, RAM, etc., the voice signal picked up by the voice input unit ex356 is converted into a digital voice signal by the voice signal processing unit ex354, and this is subjected to spread spectrum processing by the modulation / demodulation unit ex352. After performing digital-to-analog conversion processing and frequency conversion processing by the transmission / reception unit ex351, it is transmitted via the antenna ex350. Also, when the mobile phone ex114 is in the voice call mode, the received data received via the antenna ex350 is amplified and subjected to frequency conversion processing and analog-to-digital conversion processing, subjected to inverse spread spectrum processing by the modulation / demodulation unit ex352, converted into an analog voice signal by the voice signal processing unit ex354, and then output from the voice output unit ex357.

[0184] Furthermore, when sending an email in the data communication mode, the text data of the email input by operating the operation key unit ex366 of the main body or the like is sent to the main control unit ex360 via the operation input control unit ex362. The main control unit ex360 subjects the text data to spread spectrum processing by the modulation / demodulation unit ex352, performs digital-to-analog conversion processing and frequency conversion processing by the transmission / reception unit ex351, and then transmits it to the base station ex110 via the antenna ex350. When receiving an email, almost the reverse process is performed on the received data, and it is output to the display unit ex358.

[0185] When transmitting video, still images, or video and audio in the data communication mode, the video signal processing unit ex355 compresses and encodes the video signal supplied from the camera unit ex365 by the moving image encoding method shown in each of the above embodiments (i.e., functions as an image encoding device according to an aspect of the present invention), and sends the encoded video data to the multiplexing / demultiplexing unit ex353. Also, the voice signal processing unit ex354 encodes the voice signal picked up by the voice input unit ex356 while the camera unit ex365 is imaging video, still images, etc., and sends the encoded voice data to the multiplexing / demultiplexing unit ex353.

[0186] The multiplexing / demultiplexing unit ex353 multiplexes the encoded video data supplied from the video signal processing unit ex355 and the encoded audio data supplied from the audio signal processing unit ex354 in a predetermined manner, and the resulting multiplexed data is subjected to spread spectrum processing by the modulation / demodulation unit (modulation / demodulation circuit unit) ex352, and after digital-to-analog conversion processing and frequency conversion processing are performed by the transmission / reception unit ex351, it is transmitted via the antenna ex350.

[0187] When receiving the data of a moving image file linked to a homepage or the like in the data communication mode, or when receiving an e-mail with video and / or audio attached, in order to decode the multiplexed data received via the antenna ex350, the multiplexing / demultiplexing unit ex353 separates the multiplexed data into a bit stream of video data and a bit stream of audio data, supplies the encoded video data to the video signal processing unit ex355 via the synchronization bus ex370, and supplies the encoded audio data to the audio signal processing unit ex354. The video signal processing unit ex355 decodes the video signal by decoding it by a moving image decoding method corresponding to the moving image encoding method shown in each of the above embodiments (i.e., functions as an image decoding device according to an aspect of the present invention), and from the display unit ex358 via the LCD control unit ex359, for example, the video and still images included in the moving image file linked to the homepage are displayed. Also, the audio signal processing unit ex354 decodes the audio signal, and audio is output from the audio output unit ex357.

[0188] Also, terminals such as the mobile phone ex114 can have three implementation forms: a transmission / reception type terminal having both an encoder and a decoder, a transmission terminal having only an encoder, and a reception terminal having only a decoder, similar to the TV ex300. Furthermore, in the digital broadcast system ex200, although it has been described as receiving and transmitting multiplexed data in which music data and the like are multiplexed with video data, data in which character data related to video is multiplexed in addition to audio data may be used, or the video data itself instead of the multiplexed data may be used.

[0189] Thus, it is possible to use the moving image encoding method or the moving image decoding method shown in each of the above embodiments in any of the devices and systems described above, and by doing so, the effects described in each of the above embodiments can be obtained.

[0190] Further, the present invention is not limited to the above-described embodiments, and various modifications or corrections can be made without departing from the scope of the present invention.

[0191] (Embodiment 4) It is also possible to generate video data by appropriately switching, as necessary, between the moving image encoding method or apparatus shown in each of the above embodiments and a moving image encoding method or apparatus compliant with different standards such as MPEG-2, MPEG4-AVC, and VC-1.

[0192] Here, when generating a plurality of video data compliant with different standards respectively, it is necessary to select a decoding method corresponding to each standard when decoding. However, since it is impossible to identify which standard the video data to be decoded complies with, there arises a problem that an appropriate decoding method cannot be selected.

[0193] To solve this problem, the multiplexed data obtained by multiplexing audio data or the like with the video data has a configuration including identification information indicating which standard the video data complies with. A specific configuration of the multiplexed data including the video data generated by the moving image encoding method or apparatus shown in each of the above embodiments will be described below. The multiplexed data is a digital stream in the form of an MPEG-2 transport stream.

[0194] FIG. 23 is a diagram showing the configuration of multiplexed data. As shown in FIG. 23, the multiplexed data is obtained by multiplexing one or more of a video stream, an audio stream, a presentation graphics stream (PG), and an interactive graphics stream. The video stream shows the main video and sub-video of a movie, the audio stream (IG) shows the main audio part of the movie and the sub-audio mixed with the main audio, and the presentation graphics stream shows the subtitles of the movie. Here, the main video refers to the normal video displayed on the screen, and the sub-video refers to the video displayed in a small screen within the main video. Also, the interactive graphics stream shows an interactive screen created by arranging GUI components on the screen. The video stream is encoded by the moving image encoding method or apparatus shown in each of the above embodiments, or a moving image encoding method or apparatus compliant with conventional standards such as MPEG-2, MPEG4-AVC, and VC-1. The audio stream is encoded in a format such as Dolby AC-3, Dolby Digital Plus, MLP, DTS, DTS-HD, or linear PCM.

[0195] Each stream included in the multiplexed data is identified by a PID. For example, 0x1011 is assigned to the video stream used for the video of a movie, 0x1100 to 0x111F are assigned to the audio stream, 0x1200 to 0x121F are assigned to the presentation graphics, 0x1400 to 0x141F are assigned to the interactive graphics stream, 0x1B00 to 0x1B1F are assigned to the video stream used for the sub-video of the movie, and 0x1A00 to 0x1A1F are assigned to the audio stream used for the sub-audio mixed with the main audio, respectively.

[0196] FIG. 24 is a diagram schematically showing how multiplexed data is multiplexed. First, a video stream ex235 composed of a plurality of video frames and an audio stream ex238 composed of a plurality of audio frames are respectively converted into PES packet sequences ex236 and ex239, and then into TS packets ex237 and ex240. Similarly, the data of the presentation graphics stream ex241 and the interactive graphics ex244 are respectively converted into PES packet sequences ex242 and ex245, and further into TS packets ex243 and ex246. The multiplexed data ex247 is constituted by multiplexing these TS packets into one stream.

[0197] FIG. 25 shows in more detail how a video stream is stored in the PES packet sequence. The first row in FIG. 25 shows the video frame sequence of the video stream. The second row shows the PES packet sequence. As shown by the arrows yy1, yy2, yy3, yy4 in FIG. 25, the I picture, B picture, and P picture, which are a plurality of Video Presentation Units in the video stream, are divided for each picture and stored in the payload of the PES packet. Each PES packet has a PES header, and the PES header stores a PTS (Presentation Time-Stamp), which is the display time of the picture, and a DTS (Decoding Time-Stamp), which is the decoding time of the picture.

[0198] Figure 26 shows the format of the TS packet that is finally written to the multiplexed data. The TS packet is a fixed-length packet of 188 bytes composed of a 4-byte TS header that holds information such as the PID that identifies the stream and a 184-byte TS payload that stores the data. The above PES packet is split and stored in the TS payload. In the case of a BD-ROM, a 4-byte TP_Extra_Header is added to the TS packet to form a 192-byte source packet, which is written to the multiplexed data. Information such as ATS (Arrival_Time_Stamp) is described in the TP_Extra_Header. ATS indicates the transfer start time to the PID filter of the decoder for the TS packet. As shown in the lower part of Figure 26, source packets are arranged in the multiplexed data, and the number incremented from the head of the multiplexed data is called SPN (Source Packet Number).

[0199] In addition, the TS packets included in the multiplexed data include, in addition to each stream such as video, audio, and subtitles, PAT (Program Association Table), PMT (Program Map Table), PCR (Program Clock Reference), etc. PAT indicates what the PID of the PMT used in the multiplexed data is, and the PID of PAT itself is registered as 0. PMT has the PID of each stream such as video, audio, and subtitles included in the multiplexed data and the attribute information of the stream corresponding to each PID, and also has various descriptors regarding the multiplexed data. The descriptor includes copy control information that indicates whether to permit or not permit copying of the multiplexed data. PCR has the information of the STC time corresponding to the ATS at which the PCR packet is transferred to the decoder in order to synchronize the ATC (Arrival Time Clock), which is the time axis of ATS, and the STC (System Time Clock), which is the time axis of PTS and DTS.

[0200] FIG. 27 is a diagram for explaining in detail the data structure of a PMT. At the head of the PMT, a PMT header describing the length of the data contained in the PMT and the like is arranged. After that, a plurality of descriptors regarding the multiplexed data are arranged. The above copy control information and the like are described as descriptors. After the descriptors, a plurality of stream information regarding each stream contained in the multiplexed data are arranged. The stream information is composed of a stream descriptor in which a stream type, a PID of the stream, and stream attribute information (frame rate, aspect ratio, etc.) are described in order to identify the compression codec of the stream. The stream descriptors exist in the same number as the number of streams existing in the multiplexed data.

[0201] When recording on a recording medium or the like, the above multiplexed data is recorded together with a multiplexed data information file.

[0202] As shown in FIG. 28, the multiplexed data information file is management information of the multiplexed data, corresponds one-to-one with the multiplexed data, and is composed of multiplexed data information, stream attribute information, and an entry map.

[0203] As shown in FIG. 28, the multiplexed data information is composed of a system rate, a reproduction start time, and a reproduction end time. The system rate indicates the maximum transfer rate of the multiplexed data to the PID filter of the system target decoder described later. The interval of the ATSs contained in the multiplexed data is set to be equal to or less than the system rate. The reproduction start time is the PTS of the first video frame of the multiplexed data, and the reproduction end time is set to be the PTS of the last video frame of the multiplexed data plus the reproduction interval for one frame.

[0204] As shown in Fig. 29, for each stream included in the multiplexed data, the stream attribute information is registered for each PID. The attribute information has different information for each of the video stream, audio stream, presentation graphics stream, and interactive graphics stream. The video stream attribute information includes information such as what compression codec the video stream is compressed with, what the resolution of the individual picture data constituting the video stream is, what the aspect ratio is, and what the frame rate is. The audio stream attribute information includes information such as what compression codec the audio stream is compressed with, how many channels are included in the audio stream, what language it corresponds to, and what the sampling frequency is. These information are used for initialization of the decoder before playback by the player and the like.

[0205] In the present embodiment, among the above multiplexed data, the stream type included in the PMT is used. Also, when the multiplexed data is recorded on the recording medium, the video stream attribute information included in the multiplexed data information is used. Specifically, in the moving image encoding method or apparatus shown in each of the above embodiments, a step or means for setting unique information indicating that it is video data generated by the moving image encoding method or apparatus shown in each of the above embodiments for the stream type included in the PMT or the video stream attribute information is provided. With this configuration, it becomes possible to distinguish the video data generated by the moving image encoding method or apparatus shown in each of the above embodiments from the video data conforming to other standards.

[0206] Also, the steps of the moving image decoding method in this embodiment are shown in FIG. 30. In step exS100, the stream type included in the PMT from the multiplexed data or the video stream attribute information included in the multiplexed data information is acquired. Next, in step exS101, it is determined whether the stream type or the video stream attribute information indicates that the multiplexed data is generated by the moving image encoding method or apparatus shown in each of the above embodiments. And when it is determined that the stream type or the video stream attribute information is generated by the moving image encoding method or apparatus shown in each of the above embodiments, in step exS102, decoding is performed by the moving image decoding method shown in each of the above embodiments. Also, when the stream type or the video stream attribute information indicates that it conforms to conventional standards such as MPEG-2, MPEG4-AVC, VC-1, etc., in step exS103, decoding is performed by a moving image decoding method conforming to the conventional standard.

[0207] In this way, by setting a new unique value for the stream type or the video stream attribute information, it is possible to determine whether decoding can be performed by the moving image decoding method or apparatus shown in each of the above embodiments when decoding. Therefore, even when multiplexed data conforming to different standards is input, an appropriate decoding method or apparatus can be selected, so that decoding can be performed without causing an error. Also, the moving image encoding method or apparatus, or the moving image decoding method or apparatus shown in this embodiment can be used in any of the devices and systems described above.

[0208] (Embodiment 5) The moving image encoding method, apparatus, moving image decoding method, and apparatus shown in the above embodiments are typically realized by an LSI which is an integrated circuit. As an example, FIG. 31 shows the configuration of an LSIex500 integrated into one chip. LSIex500 includes elements ex501, ex502, ex503, ex504, ex505, ex506, ex507, ex508, ex509 to be described below, and each element is connected via a bus ex510. The power supply circuit section ex505 starts up to an operable state by supplying power to each section when the power is on.

[0209] For example, when performing encoding processing, based on the control of the control section ex501 which has a CPUex502, a memory controller ex503, a stream controller ex504, a drive frequency control section ex512, etc., LSIex500 inputs an AV signal from a microphone ex117, a camera ex113, etc. through an AV I / Oex509. The input AV signal is temporarily stored in an external memory ex511 such as an SDRAM. Based on the control of the control section ex501, the stored data is appropriately divided into multiple times according to the processing amount and processing speed and sent to a signal processing section ex507, where encoding of the audio signal and / or encoding of the video signal is performed. Here, the encoding process of the video signal is the encoding process described in the above embodiments. The signal processing section ex507 further performs processes such as multiplexing the encoded audio data and the encoded video data as the case may be, and outputs it to the outside through a stream I / Oex506. This output multiplexed data is transmitted toward a base station ex107 or written to a recording medium ex215. Note that when multiplexing, it is advisable to temporarily store the data in a buffer ex508 so as to be synchronized.

[0210] Note that in the above, the memory ex511 has been described as a configuration external to the LSIex500, but it may also be a configuration included inside the LSIex500. The buffer ex508 is not limited to one, and a plurality of buffers may be provided. Also, the LSIex500 may be integrated into one chip or may be made up of multiple chips.

[0211] Also, in the above description, the control unit ex501 is assumed to include the CPU ex502, memory controller ex503, stream controller ex504, drive frequency control unit ex512, etc. However, the configuration of the control unit ex501 is not limited to this configuration. For example, the signal processing unit ex507 may further include a CPU. By providing a CPU inside the signal processing unit ex507 as well, it becomes possible to further improve the processing speed. Also, as another example, the CPU ex502 may include the signal processing unit ex507, or a part of the signal processing unit ex507, for example, a voice signal processing unit. In such a case, the control unit ex501 has a configuration including the signal processing unit ex507 or the CPU ex502 having a part thereof.

[0212] Here, it is described as an LSI, but depending on the integration level, it may also be referred to as an IC, system LSI, super LSI, or ultra LSI.

[0213] Also, the method of integrating into an integrated circuit is not limited to LSI, and it may be realized by a dedicated circuit or a general-purpose processor. After manufacturing the LSI, an FPGA (Field Programmable Gate Array) that can be programmed, or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used. Such programmable logic devices can typically execute the moving image encoding method or moving image decoding method shown in each of the above embodiments by loading a program constituting software or firmware or reading it from a memory or the like.

[0214] Furthermore, if an integrated circuit technology that replaces the LSI appears due to the progress of semiconductor technology or other derived technologies, naturally, the integration of functional blocks may be performed using that technology. The application of biotechnology or the like is possible as an example.

[0215] (Embodiment 6) When decoding video data generated by the moving image encoding method or apparatus shown in each of the above embodiments, the processing amount is considered to increase compared to the case of decoding video data compliant with conventional standards such as MPEG-2, MPEG4-AVC, and VC-1. Therefore, in the LSIex500, it is necessary to set the driving frequency higher than the driving frequency of the CPUex502 when decoding video data compliant with conventional standards. However, when the driving frequency is increased, there arises a problem that the power consumption increases.

[0216] To solve this problem, moving image decoding apparatuses such as the TVex300 and the LSIex500 are configured to identify which standard the video data conforms to and switch the driving frequency according to the standard. FIG. 32 shows the configuration ex800 in the present embodiment. When the video data is generated by the moving image encoding method or apparatus shown in each of the above embodiments, the driving frequency switching unit ex803 sets the driving frequency high. Then, it instructs the decoding processing unit ex801 that executes the moving image decoding method shown in each of the above embodiments to decode the video data. On the other hand, when the video data is video data compliant with a conventional standard, the driving frequency is set lower than when the video data is generated by the moving image encoding method or apparatus shown in each of the above embodiments. Then, it instructs the decoding processing unit ex802 compliant with the conventional standard to decode the video data.

[0217] More specifically, the drive frequency switching unit ex803 is composed of the CPU ex502 and the drive frequency control unit ex512 in FIG. 31. Also, the decoding processing unit ex801 that executes the moving image decoding method shown in each of the above embodiments, and the decoding processing unit ex802 that conforms to the conventional standard correspond to the signal processing unit ex507 in FIG. 31. The CPU ex502 identifies which standard the video data conforms to. Then, based on the signal from the CPU ex502, the drive frequency control unit ex512 sets the drive frequency. Also, based on the signal from the CPU ex502, the signal processing unit ex507 decodes the video data. Here, for the identification of the video data, for example, it is conceivable to use the identification information described in Embodiment 4. Regarding the identification information, it is not limited to that described in Embodiment 4, and any information that can identify which standard the video data conforms to may be used. For example, when it is possible to identify which standard the video data conforms to based on an external signal that identifies whether the video data is for use on a television or for use on a disk, etc., it may be identified based on such an external signal. Also, the selection of the drive frequency in the CPU ex502 can be considered to be performed based on, for example, a look-up table that associates the standard of the video data as shown in FIG. 34 with the drive frequency. By storing the look-up table in the buffer ex508 or the internal memory of the LSI and having the CPU ex502 refer to this look-up table, it is possible to select the drive frequency.

[0218] Figure 33 shows the steps of implementing the method of this embodiment. First, in step exS200, the signal processing unit ex507 acquires identification information from the multiplexed data. Next, in step exS201, the CPU ex502 identifies whether the video data is generated by the encoding method or apparatus shown in each of the above embodiments based on the identification information. If the video data is generated by the encoding method or apparatus shown in each of the above embodiments, in step exS202, the CPU ex502 sends a signal for setting a high driving frequency to the driving frequency control unit ex512. Then, the driving frequency control unit ex512 sets a high driving frequency. On the other hand, if it is shown that the video data conforms to conventional standards such as MPEG-2, MPEG4-AVC, and VC-1, in step exS203, the CPU ex502 sends a signal for setting a low driving frequency to the driving frequency control unit ex512. Then, the driving frequency control unit ex512 sets a lower driving frequency compared to the case where the video data is generated by the encoding method or apparatus shown in each of the above embodiments.

[0219] Furthermore, in conjunction with the switching of the driving frequency, by changing the voltage applied to the LSI ex500 or the device including the LSI ex500, it is possible to further enhance the power saving effect. For example, when setting the driving frequency low, it is conceivable to set the voltage applied to the LSI ex500 or the device including the LSI ex500 lower compared to the case of setting the driving frequency high.

[0220] Also, the method of setting the driving frequency may be to set the driving frequency high when the processing amount during decoding is large, and set the driving frequency low when the processing amount during decoding is small, and is not limited to the above-described setting method. For example, if the processing amount for decoding video data conforming to the MPEG4-AVC standard is larger than the processing amount for decoding video data generated by the moving image encoding method or apparatus shown in each of the above embodiments, it is conceivable to reverse the setting of the driving frequency compared to the above-described case.

[0221] Furthermore, the method of setting the drive frequency is not limited to a configuration that lowers the drive frequency. For example, when the identification information indicates that the video data is generated by the moving image encoding method or apparatus shown in each of the above embodiments, the voltage applied to the LSIex500 or the apparatus including the LSIex500 is set high. When it indicates that the video data conforms to conventional standards such as MPEG-2, MPEG4-AVC, and VC-1, it is also conceivable to set the voltage applied to the LSIex500 or the apparatus including the LSIex500 low. As another example, when the identification information indicates that the video data is generated by the moving image encoding method or apparatus shown in each of the above embodiments, the driving of the CPUex502 is not stopped. When it indicates that the video data conforms to conventional standards such as MPEG-2, MPEG4-AVC, and VC-1, since there is a margin in processing, it is also conceivable to temporarily stop the driving of the CPUex502. Even when the identification information indicates that the video data is generated by the moving image encoding method or apparatus shown in each of the above embodiments, if there is a margin in processing, it is also conceivable to temporarily stop the driving of the CPUex502. In this case, it is conceivable to set the stop time shorter than when it indicates that the video data conforms to conventional standards such as MPEG-2, MPEG4-AVC, and VC-1.

[0222] In this way, by switching the drive frequency according to the standard to which the video data conforms, it becomes possible to achieve power saving. Also, when driving an apparatus including the LSIex500 or the LSIex500 using a battery, it is possible to extend the life of the battery with the power saving.

[0223] (Embodiment 7) In devices and systems such as televisions and mobile phones, multiple video data conforming to different standards may be input. In order to enable decoding even when multiple video data conforming to different standards are input, the signal processing unit ex507 of LSIex500 needs to support multiple standards. However, if the signal processing unit ex507 corresponding to each standard is used individually, there will be problems such as an increase in the circuit scale of LSIex500 and an increase in cost.

[0224] To solve this problem, a configuration is adopted in which a decoding processing unit for executing the moving image decoding method shown in each of the above embodiments and a decoding processing unit conforming to conventional standards such as MPEG-2, MPEG4-AVC, and VC-1 are partially shared. This configuration example is shown in ex900 of FIG. 35A. For example, the moving image decoding method shown in each of the above embodiments and the moving image decoding method conforming to the MPEG4-AVC standard have some common processing contents in processes such as entropy encoding, inverse quantization, deblocking filter, and motion compensation. For the common processing contents, a decoding processing unit ex902 corresponding to the MPEG4-AVC standard is shared, and for other processing contents specific to an aspect of the present invention that do not conform to the MPEG4-AVC standard, a dedicated decoding processing unit ex901 is used. In particular, since an aspect of the present invention is characterized by hierarchical encoding, for example, a dedicated decoding processing unit ex901 is used for inverse quantization, and for any one or all of the other processes of entropy decoding, inverse quantization, deblocking filter, and motion compensation, it is conceivable to share the decoding processing unit. Regarding the sharing of the decoding processing unit, for the common processing contents, the decoding processing unit for executing the moving image decoding method shown in each of the above embodiments is shared, and for the processing contents specific to the MPEG4-AVC standard, a configuration using a dedicated decoding processing unit may also be adopted.

[0225] Another example of sharing part of the processing is shown as ex1000 in FIG. 35B. In this example, a dedicated decoding processing unit ex1001 corresponding to the processing content specific to one aspect of the present invention, a dedicated decoding processing unit ex1002 corresponding to the processing content specific to other conventional standards, and a shared decoding processing unit ex1003 corresponding to the processing content common to the moving image decoding method according to one aspect of the present invention and the moving image decoding methods of other conventional standards are used. Here, the dedicated decoding processing units ex1001 and ex1002 are not necessarily specialized in the processing content specific to one aspect of the present invention or other conventional standards, and may be capable of executing other general-purpose processing. Also, the configuration of this embodiment can be implemented by LSI ex500.

[0226] As described above, by sharing the decoding processing unit for the processing content common to the moving image decoding method according to one aspect of the present invention and the moving image decoding methods of conventional standards, it is possible to reduce the circuit scale of the LSI and reduce the cost.

Industrial Applicability

[0227] The present invention can be applied to an image decoding method and apparatus, or an image encoding method and apparatus. Further, the present invention can be used in high-resolution information display devices or imaging devices such as televisions, digital video recorders, car navigation systems, mobile phones, digital cameras, and digital video cameras equipped with an image decoding device.

Explanation of Signs

[0228] 100 Image encoding device 101 Limiting value setting unit 102 Encoding unit 111 Number of layers setting unit 112 Number of layers parameter setting unit 113 Display delay picture number setting unit 114 B picture consecutive number setting unit 115 Consecutive number parameter setting unit 121, 209 Image rearrangement unit 122 Code block division unit 123 Subtraction unit 124 Transform quantization unit 125 Variable length coding unit 126, 202 Inverse transform quantization unit 127, 203 Addition unit 128, 204 Frame memory 129 Intra prediction unit 130 Inter prediction unit 131 Selection unit 151, 256 Frame rate 152, 253 Transmission delay time limit value 153 Input image 154, 257 Coding structure limit value 155, 251 Code sequence 161 Number of hierarchical levels 162 Number of consecutive B pictures 163 Hierarchical level parameter 164 Number of display delay pictures 165 Consecutive number parameter 171 Code block 172, 174, 258 Difference block 173, 254 Transform coefficient 175, 259 Decoded block 177, 261 Prediction block 178, 255 Prediction information 200 Image decoding device 201 Variable length decoding unit 205 Intra prediction block generation unit 206 Inter prediction block generation unit 208 Limit value decoding unit 210 Coding structure confirmation unit 252 HighestTId 262 Coding structure 263 Output image

Claims

1. a receiving step of receiving a bit stream including a video encoded by hierarchically encoding a plurality of images included in the video into a hierarchical structure having one or more layers, and control information; and decoding the plurality of images from the bitstream, the lowest layer of the hierarchical structure includes I pictures and P pictures; a hierarchy other than the lowest hierarchy of the hierarchical structure includes a B-picture; the control information includes information about a frame rate of the moving image, The number of layers is predetermined based on a frame rate of the video. Receiving method.

2. A processing circuit; a storage device accessible by the processing circuitry; The processing circuitry uses the storage device to: Executing the receiving method according to claim 1 Receiving device.

Images