Receiving method and receiving apparatus

Abstract

To provide a reception method capable of suppressing a processing load.SOLUTION: A reception method of receiving and decoding an encoded stream, the reception method including acquiring a decoding time of each of a plurality of pictures from the encoded stream, and decoding the plurality of pictures according to the decoding time, the decoding time being set such that the decoding time of each of the plurality of lower layer pictures is at equal intervals, and a timing at which each of the plurality of lower layer pictures is decoded is the same between a case where the plurality of pictures are decoded and a case where only the plurality of lower layer pictures are decoded, the plurality of pictures not belonging to the lowest layer include a bidirectional reference predictive picture, and when a frame rate at which all the pictures are decoded and displayed is f, the decoding time of each of all the pictures is set such that the decoding time of each of the plurality of lower layer pictures included in all the pictures is separated by a time indicated by n times a reciprocal of f.SELECTED DRAWING: Figure 15D

Description

[Technical Field]

[0001] The present invention relates to an image coding method for hierarchically coding an image, an image decoding method for decoding a hierarchically coded image, and the like. [Background technology]

[0002] BACKGROUND ART Conventionally, an image coding method for hierarchically coding an image and an image decoding method for decoding an image coded by the hierarchical coding have been proposed (see, for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11 12th Meeting: Geneva, CH, 14-23 Jan. 2013 JCTVC-L1003_v34.doc, High Efficiency Video Coding (HEVC) text specification draft 10 (for FDIS & Last Call) http: / / phenix.it-sudparis.eu / jct / doc_end_user / documents / 12_Geneva / wg11 / JCTVC-L1003-v34.zip Summary of the Invention [Problem to be solved by the invention]

[0004] However, the image encoding method and image decoding method of Non-Patent Document 1 have a problem in that they are unable to process images efficiently due to the high processing load.

[0005] Therefore, the present invention provides an image encoding method and an image decoding method that can reduce the processing load. [Means for solving the problem]

[0006] A receiving method according to one aspect of the present invention is a receiving method for receiving and decoding an encoded stream including a video in which, for each picture in a group of pictures following a randomly accessible I-picture in encoding order and belonging to one of a plurality of layers, the pictures are encoded without reference to other pictures belonging to a layer higher than the layer to which the picture belongs, the receiving method comprising the steps of: receiving the encoded stream including a plurality of encoded pictures and a decode time set for each of the plurality of pictures; obtaining the decode time of each of the plurality of pictures included in the encoded stream from the encoded stream; decoding each of the plurality of pictures or a plurality of lower layer pictures included in the encoded stream according to the obtained decode time; and decoding the decode time of each of the plurality of pictures included in the video by referring to the decode time of a previous plurality of pictures among the plurality of pictures included in the video that do not belong to the highest layer of the plurality of layers. The decoding times are set so that the decoding times of the plurality of lower layer pictures are equally spaced, and so that the timing at which each of the plurality of lower layer pictures is decoded is the same when the plurality of pictures included in the encoded video are decoded and when only the plurality of lower layer pictures among the plurality of pictures are decoded; the plurality of pictures that do not belong to the lowest layer include bidirectional reference predicted pictures; the decoding times of all pictures belonging to the group of pictures are earlier than the decoding times of all pictures that follow in display order, starting from a backward I-picture that is later than the group of pictures in display order; and the decoding times are set so that when the frame rate at which all pictures included in the encoded video are decoded and displayed is f, the decoding times of the plurality of lower layer pictures included in all of the pictures are spaced apart by a time indicated by n times the reciprocal of f (n is an integer greater than or equal to 2).

[0007] An image coding device according to one aspect of the present invention includes an acquisition unit that acquires a plurality of pictures including a plurality of first layer pictures and a plurality of second layer pictures arranged in display order; an association unit that associates the plurality of pictures with temporal IDs defined in a video coding standard so that each of the plurality of first layer pictures is associated with a smallest temporal ID and each of the plurality of second layer pictures is associated with a temporal ID greater than the smallest temporal ID; and a random access unit that (i) spaces a first decoding timing of each of the plurality of first layer pictures at regular intervals regardless of whether all coded pictures are decoded by a decoding device or whether only the coded picture associated with the smallest temporal ID is decoded by the decoding device, and (ii) all of the plurality of pictures included in each random access unit are decoded from other random access units. the first picture is arranged at the beginning of decoding order within the random access unit, and all of the plurality of pictures do not include a picture that precedes the first picture in the display order, and the first picture is arranged at the beginning of decoding order within the random access unit; an encoding unit that encodes the plurality of pictures arranged in an encoding order different from the display order corresponding to the decoding timing according to the decoding timing, so that a plurality of encoded pictures including a plurality of encoded first layer pictures and a plurality of encoded second layer pictures are generated; and a generation unit that generates an encoded stream including the plurality of encoded first layer pictures, the plurality of encoded second layer pictures, and the decoding timing.

[0008] A transmission method according to one aspect of the present invention is a transmission method for encoding and transmitting a moving image using an image encoding method for encoding a moving image, for each picture of a group of pictures following a randomly accessible I-picture in encoding order, the picture belonging to one of a plurality of layers, without referring to other pictures belonging to a layer higher than the layer to which the picture belongs. The transmission method includes setting the decode times of a plurality of pictures included in the moving image, the plurality of lower-layer pictures being some of the pictures included in the moving image and not belonging to the highest layer of the plurality of layers, so that the decode times of the plurality of lower-layer pictures are equally spaced and so that the timings at which the plurality of pictures included in the encoded moving image are decoded are the same between a case in which the plurality of pictures included in the encoded moving image are decoded and a case in which only the plurality of lower-layer pictures of the plurality of pictures are decoded. the decoding time is set so that, when a frame rate at which all pictures included in the coded video are decoded and displayed is f, the decoding times of the lower layer pictures included in all the pictures are spaced apart by a time indicated by n times the reciprocal of f (n is an integer equal to or greater than 2).

[0009] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium. [Effects of the Invention]

[0010] The image encoding method and image decoding method of the present invention can reduce the processing load. [Brief explanation of the drawings]

[0011] [Figure 1A] FIG. 1A is a diagram showing an example of how to achieve temporal scalability using HEVC. [Figure 1B] FIG. 1B is a diagram showing another example for realizing temporal scalability using HEVC. [Figure 2] FIG. 2 is a diagram for explaining the problem of not being able to decode a 120 fps coded stream. [Figure 3] FIG. 3 is a diagram illustrating the image coding device according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of encoding a moving image using three layers according to the first embodiment. [Figure 5] FIG. 5 is a diagram showing an example of encoding a moving image using four layers according to the first embodiment. [Figure 6] FIG. 6 is a diagram showing another example of encoding a video using three layers according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating the image decoding device according to the first embodiment. [Figure 8] FIG. 8 shows the decoding order and display order of each picture, as well as the DTS and PTS when a coded stream is played back at 120 fps according to the first modification of the first embodiment. [Figure 9] FIG. 9 is a block diagram showing a configuration of an image decoding device according to Variation 1 of Embodiment 1. In FIG. [Figure 10] FIG. 10 is a flowchart showing an example of the operation of the image decoding device according to Variation 1 of Embodiment 1 when decoding pictures in all layers. [Figure 11] FIG. 11 is a diagram showing an example of changing the DTS according to the second modification of the first embodiment. [Figure 12] FIG. 12 is a diagram illustrating pictures decoded in an open end random access unit according to the third modification of the first embodiment. [Figure 13] FIG. 13 is a flowchart showing an example of the operation of the image decoding device according to Variation 3 of Embodiment 1 to decode a video based on auxiliary information for playback control. [Figure 14] FIG. 14 shows an example of DTS or PTS conversion according to the fourth modification of the first embodiment. [Figure 15A] FIG. 15A is a block diagram of an image encoding device according to one aspect of the present invention. [Figure 15B] FIG. 15B is a flowchart showing an image coding method according to one embodiment of the present invention. [Figure 15C] FIG. 15C is a block diagram of an image decoding device according to one aspect of the present invention. [Figure 15D] FIG. 15D is a flowchart showing an image decoding method according to one embodiment of the present invention. [Figure 16] FIG. 16 is a diagram showing the overall configuration of a content supply system that realizes a content distribution service. [Figure 17] FIG. 17 is a diagram showing the overall configuration of a digital broadcasting system. [Figure 18] FIG. 18 is a block diagram showing an example of the configuration of a television. [Figure 19] FIG. 19 is a block diagram showing an example of the configuration of an information reproducing / recording unit that reads and writes information from and to a recording medium that is an optical disc. [Figure 20] FIG. 20 is a diagram showing an example of the structure of a recording medium that is an optical disc. [Figure 21A] FIG. 21A is a diagram showing an example of a mobile phone. [Figure 21B] FIG. 21B is a block diagram showing an example of the configuration of a mobile phone. [Figure 22] FIG. 22 is a diagram showing the structure of multiplexed data. [Figure 23]FIG. 23 is a diagram showing a schematic diagram of how each stream is multiplexed in multiplexed data. [Figure 24] FIG. 24 shows in more detail how a video stream is stored in a sequence of PES packets. [Figure 25] FIG. 25 is a diagram showing the structure of TS packets and source packets in multiplexed data. [Figure 26] FIG. 26 shows the data structure of a PMT. [Figure 27] FIG. 27 is a diagram showing the internal structure of the multiplexed data information. [Figure 28] FIG. 28 shows the internal structure of the stream attribute information. [Figure 29] FIG. 29 shows the steps for identifying video data. [Figure 30] FIG. 30 is a block diagram showing an example of the configuration of an integrated circuit that realizes the video encoding method and video decoding method according to each embodiment. [Figure 31] FIG. 31 is a diagram showing a configuration for switching the drive frequency. [Figure 32] FIG. 32 is a diagram showing steps for identifying video data and switching the drive frequency. [Figure 33] FIG. 33 is a diagram showing an example of a lookup table in which video data standards and drive frequencies are associated with each other. [Figure 34A] FIG. 34A is a diagram showing an example of a configuration in which modules of a signal processing unit are shared. [Figure 34B] FIG. 34B is a diagram showing another example of a configuration in which modules of a signal processing unit are shared. DETAILED DESCRIPTION OF THE INVENTION

[0012] (Findings that form the basis of the present invention) The present inventors have found that the following problems arise with respect to Non-Patent Document 1 described in the "Background Art" section.

[0013] In coding formats such as MPEG-4 AVC (Moving Picture Experts Group-4 Advanced Video Coding) or HEVC (High Efficiency Video Coding), temporal scalability (hereinafter referred to as temporal scalability) can be achieved by hierarchically encoding pictures. For example, decoding all pictures enables playback at 120 fps, while decoding only pictures belonging to a specific layer enables playback at 60 fps.

[0014] By utilizing this temporal scalability, double-speed playback becomes possible by decoding only pictures belonging to a specific layer at 120 fps intervals. Furthermore, playback devices that do not support decoding at 120 fps intervals (hereinafter referred to as 120 fps non-compatible playback devices) may be able to play back a 120 fps coded stream at 60 fps.

[0015] 1A and 1B are diagrams showing an example of how to achieve temporal scalability using HEVC. These diagrams show the reference relationships between layers and the decoding order (i.e., coding order) of each picture.

[0016] FIG. 1A shows an example in which each picture is coded using a three-layer structure. TId in FIG. 1A is a temporal ID, an identifier for identifying a layer. I indicates an I picture (an intra-predicted picture), P indicates a P picture (e.g., a forward reference predicted picture), and B indicates a B picture (e.g., a bidirectional reference predicted picture). The numbers to the right of I, P, and B indicate the display order of those pictures. Arrows indicate inter-picture reference relationships; for example, picture B2 references pictures I0 and P4. In other words, in this inter-layer reference relationship, a picture is coded by referencing pictures in the same layer as the picture or a layer lower than the layer to which the picture belongs, without referencing pictures in a layer higher than the layer to which the picture belongs. Here, decoding each picture in any of the layers with a temporal ID of 0 to 2 results in a frame rate of 120 fps. In this case, when each picture belonging to a layer with a temporal ID of 0 to 1 is decoded, the frame rate becomes 60 fps.

[0017] 1B shows an example in which each picture is coded using a four-layer structure. In this case, when each picture belonging to any of the layers with a temporal ID of 0 to 3 is decoded, the frame rate becomes 120 fps. In this case, too, when each picture belonging to any of the layers with a temporal ID of 0 to 2 is decoded, the frame rate becomes 60 fps.

[0018] As mentioned above, a playback device that does not support 120 fps may be able to achieve 60 fps playback by using temporal scalability, i.e., by decoding only pictures belonging to certain layers in a 120 fps encoded stream.

[0019] However, even in this case, a playback device that does not support 120 fps may have to decode each picture at intervals shorter than 1 / 60 seconds. Therefore, even if temporal scalability is used, a playback device that does not support 120 fps may be unable to decode a 120 fps coded stream due to the short interval for decoding each picture.

[0020] Fig. 2 is a diagram illustrating the problem that a non-120 fps compatible playback device cannot decode a 120 fps coded stream. The coded stream shown in Fig. 2 is the 120 fps coded stream shown in Fig. 1A. When a non-120 fps compatible playback device plays back the coded stream at 60 fps, it decodes only the pictures belonging to the layer with a Temporal ID of 0 and the pictures belonging to the layer with a Temporal ID of 1.

[0021] In Figure 2, T corresponds to the time corresponding to 120 fps, that is, 1 / 120 seconds. When broadcast content or stored content is displayed at a fixed frame rate, decoding is generally performed at the same frame rate. Therefore, during playback at 120 fps, the interval at which each picture is decoded (hereinafter referred to as the decoding interval) and the interval at which each picture is displayed (hereinafter referred to as the display interval) are both T.

[0022] Therefore, even when playing back at 60 fps, the decoding interval and display interval must both be the time corresponding to 60 fps, i.e., 2T. However, as shown in Figure 2, when playing back at 60 fps, the decoding interval between pictures I0 and P4, or the decoding interval between pictures P4 and B2, is T. A playback device that does not support 120 fps, which requires a decoding interval of 2T, faces the problem of not being able to decode pictures in time. In other words, a high processing load is a problem for playback devices that do not support 120 fps.

[0023] In order to solve such problems, one aspect of the present invention provides an image coding method for coding a moving image, for each picture belonging to one of a plurality of hierarchies, without referring to other pictures belonging to a hierarchical level higher than the hierarchical level to which the picture belongs; the image coding method determines the decode times of each of a plurality of pictures included in the moving image, which are some of the pictures included in the moving image and which do not belong to the highest hierarchical level of the plurality of hierarchies, so that the decode times of each of the plurality of lower-layer pictures are equally spaced and so that the timing at which each of the plurality of lower-layer pictures is decoded is the same between when the plurality of pictures included in the coded moving image are decoded and when only the plurality of lower-layer pictures of the plurality of pictures are decoded; the image coding method codes each of the plurality of pictures included in the moving image in accordance with the determined decode times; and generates an encoded stream including the coded pictures and the decode times determined for each of the plurality of pictures.

[0024] As a result, each of the multiple pictures included in the coded stream is coded without referring to other pictures belonging to a layer higher than the layer to which the picture belongs. Therefore, the image decoding device can decode only the multiple lower-layer pictures included in the coded stream. Furthermore, the decoding times of the multiple lower-layer pictures included in the coded stream are equally spaced. Therefore, when decoding only the multiple lower-layer pictures included in the coded stream, the image decoding device can sequentially decode the lower-layer pictures at equal time intervals. Therefore, by setting the equal intervals to an appropriate time, the processing load on the image decoding device can be reduced. In other words, the image decoding device can decode each picture at a frame rate appropriate to its own processing capability, without performing decoding at a high frame rate. Furthermore, the timing at which each of the multiple lower-layer pictures is decoded is the same when multiple pictures (e.g., all pictures) included in the coded stream are decoded and when only the multiple lower-layer pictures are decoded. Therefore, the image decoding device does not need to change the timing at which each of the multiple lower-layer pictures is decoded depending on whether all pictures in the coded stream are decoded or only the multiple lower-layer pictures are decoded. Therefore, the processing load on the image decoding device can be further reduced.

[0025] In addition, when determining the decoding time, the decoding times of each of a plurality of top-layer pictures, which are a portion of a plurality of pictures included in the video and belong to the highest layer, may be determined to be between the decoding times of each of the plurality of lower-layer pictures.

[0026] As a result, when the coded stream is decoded, the top layer picture and the lower layer picture are decoded alternately. Therefore, the time interval between the decoding of each of the multiple lower layer pictures in the coded stream is longer than the time interval between the decoding of each of all the pictures in the decoded stream. As a result, when decoding only the multiple lower layer pictures, the image decoding device can decode each picture at a frame rate that is reliably lower than when decoding each of all the pictures in the decoded stream. Therefore, the processing load of the image decoding device can be reliably reduced.

[0027] In addition, when determining the decoding times, the decoding times of each of the multiple pictures included in the video may be determined so that twice the interval between the decoding times of the multiple top-layer pictures and the multiple lower-layer pictures that are consecutive in decoding order is equal to the equally spaced time.

[0028] As a result, the interval between the decode times of each of the multiple lower-layer pictures is twice the interval between the decode times of the topmost and lower-layer pictures that are consecutive in decoding order, i.e., twice the interval between the times at which all of the pictures in the decoded stream are decoded. Therefore, if the frame rate when all of the pictures in the coded stream are decoded and displayed is 120 fps, the image decoding device can easily decode each of the multiple lower-layer pictures included in the coded stream at a time interval that is the reciprocal of the frame rate of 60 fps.

[0029] Furthermore, when the video has a plurality of random access units each consisting of a plurality of consecutive pictures in decoding order, the determination of the decode time may include determining, for each random access unit, the decode time of each picture in the random access unit so that all pictures in the random access unit, except for a picture that is displayed earlier in display order than the first picture in decoding order, can be decoded without referencing pictures included in other random access units. For example, the first picture is an I-picture in which pictures later than the first picture in decoding order are prohibited from referring to pictures earlier in decoding order than the first picture. Alternatively, the first picture is an I-picture in which pictures later than the first picture in decoding order and earlier in display order are permitted to refer to pictures earlier in decoding order than the first picture.

[0030] This allows the image decoding device to appropriately decode, for each random access unit, each of the multiple pictures included in that random access unit.

[0031] In addition, when determining the decoding times, if the frame rate at which all pictures included in the encoded video are decoded and displayed is f, the decoding times of each of the multiple lower-layer pictures included in all of the pictures may be determined so that the decoding times are separated by a time indicated by n times the reciprocal of f (n is an integer greater than or equal to 2).

[0032] This allows the image decoding device to decode each of the multiple lower layer pictures in order without burden, at time intervals that are n times the reciprocal of the frame rate.

[0033] In addition, the image encoding method may further include display delay information in the encoded stream indicating a display delay, which is the time between the decoding time of the first picture in decoding order included in the video and the display time of the first picture in display order included in the video.

[0034] This allows the image decoding device to acquire the display delay information from the encoded stream, and therefore, if the image decoding device starts decoding the encoded stream from a time that is earlier than the display start time by the display delay indicated by the display delay information, the image decoding device can display the moving image without delay from the display start time.

[0035] Furthermore, the image coding method may further include, in the coded stream, unequal interval information indicating that the decoding times determined for each of the plurality of pictures included in the video are not equally spaced.

[0036] This allows the image decoding device to acquire unequal interval information from the coded stream. Therefore, the image decoding device can determine that each of the multiple pictures included in the coded stream cannot be sequentially decoded at the display frame rate. As a result, the image decoding device can decode each of the multiple pictures included in the coded stream at an appropriate timing by referring to the decoding times determined for each of the multiple pictures.

[0037] In addition, an image decoding device according to one aspect of the present invention is an image decoding method for decoding an encoding stream containing moving images that have been encoded for each picture belonging to one of a plurality of hierarchies without referring to other pictures that belong to a hierarchical level higher than the hierarchical level to which the picture belongs, the image decoding method comprising: obtaining from the encoding stream the decoding times of each of a plurality of pictures included in the encoding stream, the decoding times being determined so that the decoding times of a plurality of lower-layer pictures that are some of the pictures included in the encoding stream and that do not belong to the highest hierarchical level among the plurality of hierarchies are equally spaced and so that the timing at which each of the plurality of lower-layer pictures is decoded is the same between when the plurality of pictures included in the encoding stream are decoded and when only the plurality of lower-layer pictures among the plurality of pictures are decoded; and decoding each of the plurality of pictures included in the encoding stream or the plurality of lower-layer pictures according to the obtained decoding times.

[0038] As a result, each of the multiple pictures included in the coded stream is coded without referring to other pictures belonging to a layer higher than the layer to which the picture belongs. Therefore, the image decoding device can decode only the multiple lower-layer pictures included in the coded stream. Furthermore, the decoding times of the multiple lower-layer pictures included in the coded stream are equally spaced. Therefore, when decoding only the multiple lower-layer pictures included in the coded stream, the image decoding device can sequentially decode the lower-layer pictures at equal time intervals. Therefore, if the equal intervals are appropriate, the processing load of the image decoding device can be reduced. In other words, the image decoding device can decode each picture at a frame rate appropriate to its own processing capability, without performing decoding at a high frame rate. Furthermore, the timing at which each of the multiple lower-layer pictures is decoded is the same when multiple pictures (e.g., all pictures) included in the coded stream are decoded and when only the multiple lower-layer pictures are decoded. Therefore, the image decoding device does not need to change the timing at which each of the multiple lower-layer pictures is decoded depending on whether all pictures in the coded stream are decoded or only the multiple lower-layer pictures are decoded. Therefore, the processing load on the image decoding device can be further reduced.

[0039] Furthermore, the image decoding method may further include, if the decoding times of each of the multiple pictures included in the encoded stream are not equally spaced, changing the decoding times of each of the multiple pictures to be equally spaced, and in decoding the encoded stream, decoding each of the multiple pictures included in the encoded stream or each of the multiple lower-layer pictures according to the changed decoding times.

[0040] This allows the decoding times of the pictures to be changed at regular intervals, allowing the image decoding device to decode each of the pictures included in the coded stream at regular time intervals, thereby further reducing the processing load on the image decoding device.

[0041] In addition, in the decoding of the coded stream, it may be determined, for each picture included in the coded stream, whether or not the decoding time acquired for that picture coincides with the generation timing of a processing signal that occurs at a predetermined cycle, and when it is determined that the decoding time coincides, the picture may be decoded. For example, the image decoding method may further determine, as the predetermined cycle, the reciprocal of a frame rate at which all pictures included in the coded stream are decoded and displayed.

[0042] This makes it possible to appropriately decode each of the multiple pictures at its corresponding decoding time, even if the decoding times of the multiple pictures are not equally spaced.

[0043] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or as any combination of a system, a method, an integrated circuit, a computer program, or a recording medium.

[0044] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0045] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present invention. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concepts are described as optional components.

[0046] (Embodiment 1) FIG. 3 is a diagram illustrating the image coding device according to this embodiment.

[0047] The image coding device 10 in this embodiment acquires video with a frame rate of 120 fps, encodes the video, and generates and outputs a coded stream. When all pictures included in this coded stream are decoded, the video is displayed at a frame rate of 120 fps. When only a portion of pictures included in this coded stream are decoded, the video is displayed at a frame rate of 60 fps. For example, the portion of pictures included in the coded stream are pictures that each belong to a layer other than the highest layer.

[0048] Figure 4 shows an example of encoding video using three layers in this embodiment. Image encoding device 10 in this embodiment generates a 120 fps encoded stream by encoding video based on a prediction structure similar to the picture prediction structure shown in Figure 1A. At this time, image encoding device 10 adjusts the decoding time stamps (DTS) of pictures that are decoded only when played back at 120 fps so that the decoding interval is 2T (1 / 60 seconds) when the encoded stream is played back at 60 fps. In other words, image encoding device 10 adjusts the DTS of multiple pictures belonging to the highest layer of multiple layers.

[0049] 5 is a diagram showing an example of encoding a video using four layers in this embodiment. Image encoding device 10 in this embodiment generates a 120 fps encoded stream by encoding the video based on a prediction structure similar to the picture prediction structure shown in FIG. 1B. In this case, image encoding device 10 adjusts the decoding times of pictures that are decoded only when the encoded stream is played back at 120 fps so that the decoding interval is 2T (1 / 60 seconds) when the encoded stream is played back at 60 fps.

[0050] When encoding a video, a random access unit called a GOP (Group Of Pictures) is configured so that decoding can start from a picture in the middle of the encoded stream. The first picture in a random access unit in decoding order is a random access point. For example, as shown in FIG. 4, pictures I0 to B9 form a first random access unit, and picture I16 is the first picture of a second random access unit. In the second random access unit, pictures such as picture B11 do not refer to picture I16 or pictures after picture I16 in decoding order, but are included in the second random access unit.

[0051] Furthermore, when recording a coded stream to be broadcast (i.e., a broadcast stream), recording may be performed up to the end of a random access unit such as the first random access unit. In this case, picture P12 can be decoded, but picture B11 cannot be decoded because it is included in the second random access unit. This makes the decoding process complicated.

[0052] Here, a picture that satisfies certain conditions is called an advance picture. The certain conditions for a picture are (1) that the picture refers to a picture included in the random access unit immediately preceding it in decoding order, and (2) that the display order of the picture precedes any picture included in the random access unit immediately preceding it in decoding order. Note that a random access unit is a unit defined by its coding order and decoding order, which are the same as the coding order. Furthermore, a random access unit immediately preceding a random access unit including an advance picture, such as the first random access unit, is called an open end random access unit. Furthermore, a picture included in an open end random access unit and that is later in display order than the advance picture is called an orphan picture. For example, picture B11 in FIG. 4 is an advance picture, and picture P12 is an orphan picture.

[0053] FIG. 6 is a diagram showing another example of encoding a moving image using three layers in this embodiment.

[0054] When realizing temporal scalability using three layers, the image coding device 10 in this embodiment may code images so that no open end random access units occur, as shown in FIG.

[0055] All pictures included in the second random access unit are later in display order than the last picture P12 of the first random access unit. Therefore, none of the pictures included in the second random access unit are advance pictures. As a result, the first random access unit is not an open end random access unit. However, picture B1 cannot refer to picture B2 because picture B2 is later in decoding order than picture B1. Similarly, picture B5 cannot refer to picture B6 because picture B6 is later in decoding order than picture B5. In this way, the prediction structure of pictures with a temporal ID of 2 is restricted.

[0056] In the examples of Figures 4 to 6, the cases where the number of layers is 3 and 4 are described using temporal scalability at 60p and 120p as examples. However, the combinations of frame rates and the number of layers that can be realized are not limited to these. When only pictures at lower layers are decoded and displayed, if the display interval between each picture at the display frame rate is T_pts, it is sufficient to ensure that the decoding interval between any consecutive pictures in decoding order is equal to or greater than T_pts. Image encoding device 10 performs video encoding of images so as to satisfy this condition.

[0057] FIG. 7 is a diagram illustrating the image decoding device 20 according to this embodiment.

[0058] The image decoding device 20 in this embodiment acquires the coded stream generated by the above-described image coding device 10. The image decoding device 20 displays moving images at a frame rate of 120 fps by decoding all pictures included in this coded stream. The image decoding device 20 also displays moving images at a frame rate of 60 fps by decoding a portion of a plurality of pictures included in this coded stream. For example, the portion of a plurality of pictures included in the coded stream are a plurality of pictures each belonging to a layer other than the highest layer.

[0059] As described above, in this embodiment, the decoding times of multiple pictures (hereinafter referred to as lower-layer pictures) included in a coded stream that do not belong to the highest layer are equally spaced. Therefore, when decoding only multiple lower-layer pictures in a coded stream, the image decoding device can sequentially decode these lower-layer pictures at equal time intervals. Therefore, by setting the equal intervals to an appropriate time (for example, the above-mentioned 2T=1 / 60 seconds), the processing load on the image decoding device can be reduced. In other words, the image decoding device can decode each picture at a frame rate that suits its own processing capability, without performing decoding at a high frame rate. For example, when decoding a 60 fps coded stream, it is sufficient to ensure that the time required to decode one picture is 1 / 60 seconds or less.

[0060] (Variation 1) Here, the DTS of the first picture in the random access unit will be explained.

[0061] FIG. 8 is a diagram showing the decoding order and display order of each picture, as well as DTS and PTS (Presentation Time Stamp) when the coded stream shown in FIG. 4 is played back at 120 fps.

[0062] Here, the subscripts (numbers) for the DTS and PTS indicate the display order. For example, in MPEG-2 TS (Transport Stream) for broadcasting, the DTS or PTS is indicated by the header of a PES packet. In MMT (MPEG Media Transport) or RTP (Real-time Transport Protocol), the DTS or PTS is indicated by the header of a transport layer packet or by header information of the coded stream included in the payload. In formats that transmit files without packetizing, such as MPEG-DASH (Dynamic Adaptive Streaming over HTTP), the DTS or PTS is included in the header of the file. Alternatively, in MPEG-4 AVC or HEVC, the DTS or PTS can be indicated in the coded stream by using SEI (Supplemental Enhancement Information), such as Picture Timing SEI or Buffering Period SEI.

[0063] In a conventional coded stream, pictures are displayed at a fixed frame rate, and if the interval between the PTSs of two consecutive pictures in display order is T, the interval between the DTSs of two consecutive pictures in decoding order is also always T. Therefore, a playback device (or an image decoding device) can start decoding a picture at the timing of the DTS of the first picture in decoding order, and then decode each of the subsequent pictures in order at intervals of T, without referring to the DTSs of the subsequent pictures.

[0064] On the other hand, in the coded stream in this embodiment, as shown in FIG. 8, the interval between the DTSs between picture I0 and picture P4 and the interval between the DTSs between picture P4 and picture B2 are 2T (T is, for example, 1 / 120 seconds). The interval between the DTSs between picture B2 and picture B1 and the interval between the DTSs between two consecutive pictures after picture B1 in decoding order are T. Therefore, in the coded stream in this embodiment, the interval between the DTSs between pictures is variable. Therefore, an image decoding device (or a playback device) needs to refer to the DTS for each picture and decode the picture at the timing of the DTS.

[0065] The image decoding device 20 according to this modification decodes or displays pictures based on the timing of a video processing signal generated at regular intervals. For example, if the display frame rate is 120 fps, the image decoding device 20 generates a video processing signal every T (e.g., 1 / 120 seconds) and decodes or displays pictures at the timing of the generation of the video processing signal. In the coded stream shown in Fig. 8, the interval between the DTSs between picture I0 and picture P4 and the interval between the DTSs between picture P4 and picture B2 are twice the period T of the video processing signal. Furthermore, the interval between the DTSs between two consecutive pictures after picture B2 in decoding order is equal to the period T of the video processing signal.

[0066] FIG. 9 is a block diagram showing the configuration of an image decoding device 20 in this modification.

[0067] The image decoding device 20 according to this modification has the same functions as the image decoding device 20 according to the above embodiment, and decodes an encoded picture at the timing indicated by the DTS of that picture. The image decoding device 20 includes a signal interval setting unit 21, a DTS acquisition unit 22, a determination unit 23, and a decoding unit 24.

[0068] FIG. 10 is a flowchart showing an example of the operation of the image decoding device 20 according to this modification when decoding pictures in all layers.

[0069] First, the signal interval setting unit 21 of the image decoding device 20 sets the interval or period (hereinafter referred to as V_period) for generating a video processing signal so that the V_period is the reciprocal of the frame rate when all layers are decoded and displayed (step S21).

[0070] Next, the DTS acquisition unit 22 acquires the DTS (hereinafter referred to as dts_i) of the picture to be decoded from the header of the PES packet storing the coded data of the picture (step S22). Next, the determination unit 23 monitors the timing of generation of the video processing signal and determines whether the timing matches dts_i (step S23). If the interval between the DTSs of the picture is N times the V_period, the timing of the video processing signal generated N times, counting from the decoding time of the picture decoded immediately before, matches dts_i. If it is determined that the timing matches ("YES" in step S23), the decoding unit 24 decodes the picture to be decoded (step S24). On the other hand, if it is determined that the timing does not match ("NO" in step S23), the determination unit 23 repeats the process of step S23.

[0071] Note that step S21 only needs to be performed once before the start of decoding of the first picture, and does not need to be performed for each picture being decoded. Furthermore, in the determination at step S23, if the difference between the timing of generation of the video processing signal and dts_i is smaller than a predetermined threshold, the two may be deemed to match. Furthermore, the operation shown in Fig. 10 can be applied not only to temporal scalability between decoding of all layers and decoding of a lower layer, but also to temporal scalability between two different lower layers.

[0072] In this way, the image decoding device 20 in this modification can decode pictures according to the variable DTS, even if the DTS interval is variable, as long as the DTS interval can be expressed as an integer multiple of the period of the video processing signal. As a result, the amount of processing required to determine the decoding timing can be reduced.

[0073] Consider a case where the frame rate when all layers are decoded (hereinafter referred to as FrameRateAll) is not an integer multiple of the frame rate when only lower layers are decoded (hereinafter referred to as FrameRatePart), such as in the case of temporal scalability between 50 fps and 120 fps. In this case, to ensure decoding in an image decoding device with a 50 fps decoding capability, the DTS interval when only lower layers are decoded needs to be 1 / 50 seconds. However, because FrameRateAll is not an integer multiple of FrameRatePart, not only is the picture decoding interval not constant (i.e., fixed interval) during playback at 120 fps, but the DTS interval (1 / 50 seconds) is not an integer multiple of the period of the video processing signal (1 / 120 seconds). As a result, decoding cannot be performed at the time indicated by the DTS, and overflow or underflow may occur in a coded picture buffer, for example. Therefore, when providing temporal scalability, the image coding device 10 according to this modification may determine a combination of layers that achieves temporal scalability so that the display frame rate when all layers are decoded is an integer multiple of the display frame rate when only lower layers are decoded. Furthermore, the image coding device 10 may store, in the coded stream, information indicating that the frame rate when all layers are decoded is an integer multiple of the frame rate when only some of the layers are decoded. Alternatively, the image coding device 10 may store this information in a descriptor or the like in a transport stream (TS) formed by multiplexing coded streams.

[0074] In addition, the difference between the DTS of the first picture in decoding order and the PTS of the first picture in display order in a random access unit is called the display delay. In the example shown in FIG. 8, the difference between DTS0 and PTS0 is the display delay. In a conventional coded stream with a fixed display frame rate, the DTS interval is also fixed and is the same as the PTS interval. Therefore, the display delay is expressed by multiplying the number of pictures decoded up to the PTS of the first picture in display order by the PTS interval. In the example shown in FIG. 8, two pictures, I0 and P4, are decoded up to the PTS of picture I0, so the conventional calculation method calculates the display delay as 2×T. However, in the example shown in FIG. 8, the PTS interval between pictures I0 and P4 and the PTS interval between pictures P4 and B2 are twice T, so the actual display delay is 4×T. As such, the conventional method cannot appropriately represent the display delay.

[0075] Therefore, the image coding device 10 according to this modification may include, in the coded stream, information indicating how many times the PTS interval the display delay is, in order to accurately represent the display delay even in a coded stream in which the DTS interval is not constant. In the example shown in FIG. 8 , the PTS interval is T, and the display delay is four times T, so the image coding device 10 expresses the display delay as 4. Alternatively, the image coding device 10 may indicate the display delay as an actual time length, such as "4×T." If the number of pictures to be decoded before the PTS of the first picture in display order is required, the image coding device 10 may indicate that number separately from the information indicating the display delay. Furthermore, the image coding device 10 may seamlessly connect multiple streams or multiple specific sections of a stream. In this case, the image coding device 10 encodes the streams or specific sections before and after the connection so that the display delay is equal, taking into account that the DTS intervals are variable in those streams or specific sections. In this case, the image coding device 10 may store information indicating that the display delay is equal before and after the seamless connection in the coded stream. Furthermore, when coding streams that are seamlessly connected to each other, the image coding device 10 may code the streams so that the display delay defined by this modification is equal before and after the seamless connection, rather than coding the streams so that the number of pictures that are decoded up to the PTS of the first picture in display order is equal before and after the seamless connection.

[0076] Furthermore, when the display frame rate is constant, i.e., when the PTS interval is constant, the image encoding device 10 according to this modification may signal auxiliary information indicating that the DTS interval is not constant. For example, the auxiliary information is a flag indicating whether the DTS interval is constant. In this case, when the flag indicating that the DTS interval is not constant is set, the image decoding device 20 performs the operation shown in FIG. 10 . When the flag is not set, the DTS interval is constant, so the image decoding device 20 may operate without referring to the DTS for each picture.

[0077] The image coding device 10 according to this modification may set the auxiliary information for each random access unit, each coded stream unit, or each unit referenced by a playlist in stored content. The auxiliary information may be stored in any of multiple layers in a content transmission / reception system, such as an area for storing private data in a TS packet (e.g., private_data_bytes), content management information in stored content, or SEI in a coded stream such as MPEG-4 AVC or HEVC. However, since it is desirable that the auxiliary information be accessible before decoding the coded stream, the image coding device 10 may store the auxiliary information in a higher layer in the content transmission / reception system, such as a multiplexing layer such as a TS or management information for multiplexed data. In this manner, in this modification, unequal interval information indicating that the decode times determined for each of multiple pictures included in a video are not equally spaced is included in the coded stream as the auxiliary information.

[0078] Furthermore, when decoding only the lower layers of a coded stream having temporal scalability, the image coding device 10 according to this modification may store, in the coded stream, information indicating that the DTS interval between any two consecutive pictures in decoding order is variable. This information indicates that, when decoding only the above-mentioned lower layers, assuming a constant display frame rate (hereinafter referred to as frame_rate), the DTS interval between any two consecutive pictures in decoding order is equal to or greater than 1 / frame_rate (seconds). For example, in MPEG2-TS, identification information of coded streams constituting a program is indicated in PMTs (Program Map Tables). The image coding device 10 according to this modification may indicate the above information by specifying a descriptor in the PMT. For example, an image decoding device 20 capable of decoding up to 60 fps will not decode or play back the coded stream if the descriptor in the PMT indicates that the DTS interval is less than 1 / 60 seconds. Alternatively, the image decoding device 20 can reset the DTS so that the decoding interval between each picture is 1 / 60 seconds or more, and then decode each picture. If resetting the DTS requires changing the PTS, the image decoding device 20 also changes the PTS.

[0079] (Variation 2) Next, the change of the decoding time of a picture will be described.

[0080] When decoding all layers, the image decoding device 20 in the above embodiment may change the DTS of a picture before starting decoding so that the DTS interval becomes 1 / frame_rate (seconds).

[0081] FIG. 11 is a diagram showing an example of changing the DTS.

[0082] As shown in FIG. 11 , the image decoding device 20 according to this modification changes the DTSs of pictures I0 and P4, thereby setting all DTS intervals to 1 / frame_rate (seconds). Pictures with variable DTS intervals are pictures that are decoded before the PTS of the first picture in display order. Changing the DTSs of these pictures ensures that the DTS intervals are fixed. Furthermore, if the DTS intervals are fixed, the decoding timing of pictures can be determined and decoded using the same method as in the past. Note that the DTS of pictures that are decoded before the PTS of the first picture in display order is simply changed to be delayed, and the DTS interval after the change is 1 / frame_rate (seconds). Therefore, no particular problems arise with decoder models such as the HRD (Hypothetical Reference Decoder) in MPEG-4 AVC or HEVC.

[0083] When multiplexing an encoded stream using a TS, the image encoding device 10 according to this modification may indicate the value of the changed DTS in, for example, a TREF (Timestamp Reference) field of the PES header. Even when using another multiplexing method, the image encoding device 10 may indicate the pre-change and post-change DTSs. Furthermore, information indicating the correspondence between the layer to be decoded and the DTS or PTS to be used, such as indicating that the changed DTS is to be used when decoding all layers, may be indicated in, for example, a TS descriptor, program information in the transport layer, or management information in the stored content. Furthermore, the image encoding device 10 may indicate, in program information in the transport layer or management information in the stored content, information indicating that the interval between the changed DTSs is fixed or that the interval is equal to the interval between the PTSs.

[0084] In this way, even if the image decoding device 20 has low processing power, it can decode only the lower layers. Also, an image decoding device 20 with high processing power that can decode all layers can determine that the changed DTS or PTS has been transmitted by analyzing the MPEG-2 TS descriptor or determining whether the TREF field in the PES packet header is present. Therefore, the image decoding device 20 can decode the coded stream using the changed DTS or PTS.

[0085] Furthermore, when recording an encoded stream on the assumption that all layers can be decoded, the image decoding device 20 may record the encoded stream after changing the DTS as described above. In this case, instead of using a field for storing the changed DTS or PTS, such as TREF, the DTS or PTS field in the PES header may be changed.

[0086] (Variation 3) Next, auxiliary information for playback control will be described.

[0087] FIG. 12 is a diagram illustrating pictures decoded in an open end random access unit.

[0088] For example, in the example shown in Figure 12, the end of the stored coded stream or the coded stream obtained via a communication network coincides with the end of the first random access unit, which is an open-end random access unit. In this case, pictures B11, B13, B14, and B15 included in the second random access unit cannot be decoded. However, picture P12 belonging to the first random access unit can be decoded. Here, picture B11 is an advance picture, and picture P12 is an orphan picture.

[0089] In a random access unit in a coded stream, all pictures except a predetermined picture among the pictures constituting the random access unit can be decoded without referring to other random access units. When the random access unit is configured as an open GOP (Group Of Pictures), the predetermined picture is a picture that appears earlier in the display order than the first picture in the random access unit in decoding order. Such a predetermined picture may refer to a picture included in a random access unit that immediately precedes the random access unit in decoding order. Therefore, when decoding starts from the beginning of the random access unit that is an open GOP, the predetermined picture cannot be decoded. Therefore, the image decoding device decodes and displays all pictures that appear later in the display order than the first picture in the random access unit as decodable pictures.

[0090] Here, the image decoding device 20 that has acquired the encoded data up to the open end random access unit of the encoded stream has not acquired the advance picture, and therefore cannot decode the advance picture. Therefore, the image encoding device 10 according to this modification includes auxiliary information for playback control in the encoded stream.

[0091] The auxiliary information for playback control is, for example, the information shown in (1) to (5) below.

[0092] (1) Information indicating whether the random access unit is an open-end random access unit. (2) Information indicating whether the random access unit is the last random access unit in a section that is played back continuously, such as the last random access unit in the coded stream or the last random access unit indicated by a playlist, etc. (3) Information indicating whether the picture is an isolated picture or whether it is the last isolated picture in display order in a random access unit. (4) Information indicating whether the picture is an advanced picture (5) Information indicating whether or not there is an isolated picture in the random access unit that is later in display order than the specified picture. In the above (2), if discontinuous sections in a coded stream or different coded streams are concatenated, the random access unit before the concatenated section cannot refer to pictures of the subsequent random access unit. Therefore, the random access unit before the concatenated section is treated the same as the last random access unit in the playback section.

[0093] The image coding device 10 according to this modification may set the auxiliary information for playback control as described above for each random access unit, for each coded stream, or for each unit referenced by a playlist in stored content. Furthermore, the auxiliary information may be stored in any of multiple layers in a content transmission / reception system, such as an area for storing private data in a TS packet (e.g., private_data_bytes), content management information in stored content, or SEI in a coded stream such as MPEG-4 AVC or HEVC. However, since it is desirable that the auxiliary information be referenceable before decoding the coded stream, the image coding device 10 may store the auxiliary information in a higher layer in a content transmission / reception system, such as a multiplexing layer such as a TS, or management information for multiplexed data.

[0094] The above information (1) and (2) is attribute information indicating the attributes of the random access unit. The image encoding device 10 stores the attribute information in an SEI that precedes the first picture of the random access unit in decoding order, or in a packet header or payload of a transport layer such as a TS packet or MMT packet that stores the first picture of the random access unit, or in a table that manages the attributes of the random access unit in content management information. In the transport layer, when information indicating a random access point is signaled, such as a random_access_indicator of a TS packet, the attribute information may be stored in a packet that indicates that it is a random access point.

[0095] Furthermore, the information in (3) and (4) above is attribute information for each picture constituting a random access unit. The image coding device 10 may store the attribute information collectively for each random access unit, or may store it for each picture. When storing it for each picture, the image coding device 10 adds an SEI to each random access unit in the coded stream, or stores the attribute information in the header or payload of a TS packet that stores the start data of the picture. Furthermore, the image coding device 10 may store the attribute information for a picture only when the random access unit is an open end random access unit.

[0096] Next, a method for decoding an image in an open end random access unit will be described.

[0097] FIG. 13 is a flowchart showing an example of the operation of the image decoding device 20 according to this modification to decode a moving image based on auxiliary information for playback control.

[0098] First, the image decoding device 20 determines whether auxiliary information for playback control is present in the content management information, the transport layer such as TS, or the coded stream (step S211). In a playback section that is played back continuously, auxiliary information for playback control is either provided for all random access units that make up the playback section, or is not provided. Therefore, the process of step S211 needs to be performed only for the first random access unit in the playback section.

[0099] Here, if it is determined that auxiliary information for playback control exists ("Yes" in step S211), the image decoding device 20 performs the processing of step S212, and if it is determined that auxiliary information does not exist ("No" in step S211), the image decoding device 20 performs the processing of step S215.

[0100] In step S215, the image decoding device 20 determines a picture to be decoded based on a predetermined method (step S215). In step S212, the image decoding device 20 determines whether the random access unit to be decoded satisfies the conditions that it is the last random access unit in a section to be continuously played back and is an open end random access unit (step S212).

[0101] Here, if it is determined that the condition is satisfied ("YES" in step S212), the image decoding device 20 determines the picture to be decoded by referring to the auxiliary information for playback control (step S213). On the other hand, if it is determined that the condition is not satisfied ("NO" in step S212), the image decoding device 20 determines to decode all pictures included in the random access unit (step S214). However, in this case, if the random access unit to be decoded is the first random access unit in the playback section, the image decoding device 20 does not decode a picture that refers to a picture included in a random access unit that immediately precedes the random access unit to be decoded in decoding order.

[0102] Then, the image decoding device 20 decodes the picture determined in any one of the processes in steps S213, S214, and S215 (step S216).

[0103] The process of step S212 may be performed for each random access unit. Furthermore, the picture determination in step S213 is performed at the start of decoding of a random access unit when auxiliary information for each of a plurality of pictures is stored collectively in the random access unit. Furthermore, the picture determination is performed for each picture when auxiliary information for each of a plurality of pictures is stored for each picture.

[0104] If the auxiliary information for playback control does not indicate information about each picture, the image decoding device 20 may determine whether or not a reference picture exists in step S213. This allows the image decoding device 20 to determine whether or not a picture can be decoded, and to determine which picture to decode.

[0105] Note that the image decoding device 20 may determine the picture in step S213 as follows.

[0106] For example, the image decoding device 20 determines only pictures that precede the advance picture in display order as pictures to be decoded, and determines isolated pictures as pictures not to be decoded.

[0107] Alternatively, the image decoding device 20 determines a picture that is earlier in display order than the advance picture and an orphan picture as pictures to be decoded. An advance picture that is earlier in display order than an orphan picture cannot be decoded. Therefore, the image decoding device 20 freezes and displays the decoding result of the picture that is immediately before the advance picture in display order among the decodable pictures at the timing indicated by the PTS of the advance picture. That is, the image decoding device 20 continues to display the decoding result of the immediately preceding picture even at the timing indicated by the PTS of the advance picture. Alternatively, the image decoding device 20 may display an image obtained by interpolating the decoding result of the advance picture and the decoding result of the decodable picture that is immediately before the advance picture in display order.

[0108] Here, an image decoding device capable of decoding at 120 fps can perform trick playback, such as quadruple-speed playback, by decoding only pictures belonging to a layer with a Temporal ID of 0 at a decoding interval of 120 fps. Therefore, a method for determining pictures to be decoded may be switched between normal playback, in which pictures in all layers are decoded and displayed, and trick playback. For example, in the example shown in FIG. 1A or 1B, the pictures with a Temporal ID of 0 are only I-pictures and P-pictures, and advance pictures are not included among these pictures. Therefore, the image decoding device 20 may not decode orphan pictures during normal playback, but may decode orphan pictures in trick playback, for example, when only pictures belonging to a layer with a Temporal ID of 0 are decoded and displayed. More generally, during trick playback, the image decoding device 20 may decode only pictures that precede advance pictures in display order among the pictures in the layer to be decoded.

[0109] Note that, when the random access unit is an open-end random access unit, the image encoding device 10 may store information for identifying the advance picture or the last isolated picture in display order in the encoded stream as attribute information of the random access unit. For example, if the decoding order of the advance picture is Nth in the random access unit, the image decoding device 20 may determine to decode only pictures having a PTS earlier than the PTS of the Nth picture. Alternatively, if the last isolated picture in display order is Nth in decoding order, the image decoding device 20 may determine not to decode pictures that are later in display order than the isolated picture.

[0110] (Variation 4) For example, when an image decoding device 20 according to this modification has an upper limit of decoding capability of 60 fps and acquires an encoded stream with a frame rate exceeding 60 fps, the image decoding device 20 may convert the DTS or PTS of each picture so that each picture included in the encoded stream can be decoded. For example, the image decoding device 20 may convert the DTS or PTS when acquiring and recording an encoded stream via broadcasting or a communication network. Alternatively, the image decoding device 20 may convert the DTS or PTS when transmitting an encoded stream recorded in a memory, a hard disk, or the like to an external device via a communication network, for example.

[0111] Fig. 14 shows an example of DTS or PTS conversion. The first row from the top in Fig. 14 shows the original DTS of all pictures in all layers included in the original coded stream. When all pictures are decoded and displayed, a video is displayed at a frame rate of 120 fps. The second row from the top in Fig. 14 shows each picture recorded when the original coded stream is recorded as a 60 fps coded stream, and the original DTS of those pictures. In a 60 fps coded stream recorded in this way, it is not guaranteed that the DTS interval between pictures will be 1 / 60 (seconds).

[0112] The third row from the top in FIG. 14 shows pictures to be recorded when the original coded stream is recorded as a 60 fps coded stream, and the DTSs of those pictures after the change. The image decoding device 20 in this modification changes the DTSs shown in the third row. As a result of changing the DTSs, it is guaranteed that the DTS interval between pictures is 1 / 60 (seconds). The fourth row from the top in FIG. 14 shows pictures to be recorded when the original coded stream is recorded as a 60 fps coded stream, and the PTSs of those pictures. There is no need to change the PTSs, and the same value as the original PTS can be used. Here, the DTSs are changed so as to be delayed from the original DTS, and the PTSs are not changed. Therefore, no overflow or underflow occurs in the upstream buffer (corresponding to the Coded Picture Buffer in MPEG-4 AVC or HEVC) provided in the image decoding device 20, or in the buffer for storing reference pictures (corresponding to the Decoded Picture Buffer in MPEG-4 AVC or HEVC).

[0113] Note that if it is necessary to change the PTS, the PTS may be changed to satisfy a buffer model (equivalent to a Hypothetical Reference Decoder in MPEG-4 AVC or HEVC). When an encoded stream is multiplexed using MPEG-2 TS, the PTS or DTS is indicated in the header of a PES packet. Therefore, the image decoding device 20 may change the PTS or DTS in the header of a PES packet. Alternatively, the image decoding device 20 may leave the value of the PTS or DTS unchanged and store the changed value in a TREF (Timestamp Reference) field in the PES packet header. Alternatively, the image decoding device 20 may change the PTS or DTS and then store the original value of the PTS or DTS in the TREF field.

[0114] In the above embodiment and its modifications, temporal scalability based on a combination of 60 fps and 120 fps has been described as an example, but the present invention may be applied to temporal scalability with other frame rate combinations. Also, in the above embodiment and its modifications, the layers to be decoded when realizing temporal scalability have been described as a combination of all layers and each layer except the top layer, but the present invention may be applied to other combinations of layers.

[0115] The image encoding device and image decoding device according to one or more aspects have been described above based on embodiments and their variations, but the present invention is not limited to these embodiments and their variations.

[0116] FIG. 15A is a block diagram of an image encoding device according to one aspect of the present invention.

[0117] An image coding device 100 according to one embodiment of the present invention is a device that codes moving images for each picture belonging to one of a plurality of layers without referring to other pictures belonging to a layer higher than the layer to which the picture belongs, and includes a determination unit 101, a coding unit 102, and a generation unit 103.

[0118] The determination unit 101 determines the decode times of each of a plurality of pictures included in the video, which are some of the pictures included in the video and which are lower-layer pictures that do not belong to the highest hierarchical layer of the plurality of hierarchical layers, so that the decode times of each of the lower-layer pictures are equally spaced. At this time, the determination unit 101 further determines the decode times of each of the plurality of pictures included in the video so that the timings at which each of the lower-layer pictures is decoded are the same between a case in which the plurality of pictures included in the coded video are decoded and a case in which only the plurality of lower-layer pictures of the plurality of pictures are decoded.

[0119] The encoding unit 102 encodes each of the multiple pictures included in the video in a coding order corresponding to the determined decoding times. The generation unit 103 generates a coded stream including the coded multiple pictures and the determined decoding times for each of the multiple pictures.

[0120] FIG. 15B is a flowchart showing an image coding method according to one embodiment of the present invention.

[0121] In an image coding method according to one aspect of the present invention, an image coding device 100 codes a moving image, for each picture belonging to one of a plurality of layers, without referring to other pictures belonging to a layer higher than the layer to which the picture belongs. This image coding method includes steps S101, S102, and S103. In step S101, the decode times of a plurality of pictures included in the moving image are determined so that the decode times of a plurality of lower-layer pictures, which are some of the pictures included in the moving image and do not belong to the highest layer of the plurality of layers, are equally spaced. At this time, the decode times of a plurality of pictures included in the coded moving image are determined so that the timings at which the plurality of lower-layer pictures are decoded are the same when all the pictures included in the coded moving image are decoded and when only the plurality of lower-layer pictures among the plurality of pictures are decoded.

[0122] In step S102, the multiple pictures included in the video are coded in a coding order according to the determined decoding times. In step S103, a coded stream is generated that includes the coded multiple pictures and the determined decoding times for each of the multiple pictures.

[0123] As a result, each of the multiple pictures included in the coded stream is coded without referring to other pictures belonging to a layer higher than the layer to which the picture belongs. Therefore, the image decoding device can decode only the multiple lower-layer pictures included in the coded stream. Furthermore, the decoding times of the multiple lower-layer pictures included in the coded stream are equally spaced. Therefore, when decoding only the multiple lower-layer pictures included in the coded stream, the image decoding device can sequentially decode the lower-layer pictures at equal time intervals. Therefore, by setting the equal intervals to an appropriate time, such as 1 / 60 seconds, the processing load of the image decoding device can be reduced. In other words, the image decoding device can decode each picture at a frame rate, such as 60 fps, that suits its own processing capability, rather than at a high frame rate, such as 120 fps. Furthermore, the timing at which each of the multiple lower-layer pictures is decoded is the same when multiple pictures (e.g., all pictures) included in the coded stream are decoded and when only the multiple lower-layer pictures are decoded. For example, as shown in Figure 4 or 6, pictures I0, P4, B2, I8, B6, etc., which are multiple lower-layer pictures, are decoded at the same timing when decoding is performed at 120 fps and when decoding is performed at 60 fps. Therefore, the image decoding device does not need to change the decoding timing of each of the multiple lower-layer pictures when decoding all pictures in the coded stream and when decoding only the multiple lower-layer pictures. This can further reduce the processing load of the image decoding device.

[0124] In determining the decoding times in step S101, the decoding times of the top layer pictures, which are some of the pictures included in the video and belong to the highest layer, are determined to be between the decoding times of the lower layer pictures. For example, in the example shown in Fig. 6, the decoding time of the top layer picture B1 is determined to be between the decoding times of the lower layer pictures P4 and B2, and the decoding time of the top layer picture B3 is determined to be between the decoding times of the lower layer pictures B2 and I8.

[0125] As a result, when the coded stream is decoded, the top layer picture and the lower layer picture are decoded alternately. Therefore, the time interval between the decoding of each of the multiple lower layer pictures in the coded stream is longer than the time interval between the decoding of each of all the pictures in the decoded stream. As a result, when decoding only the multiple lower layer pictures, the image decoding device can decode each picture at a frame rate that is reliably lower than when decoding each of all the pictures in the decoded stream. Therefore, the processing load of the image decoding device can be reliably reduced.

[0126] Furthermore, in determining the decode times in step S101, the decode times of the plurality of pictures included in the video are determined so that twice the interval between the decode times of the plurality of top-layer pictures and the plurality of lower-layer pictures that are consecutive in decoding order is equal to the above-mentioned evenly spaced time. For example, in the example shown in Fig. 6, the interval between the decode times of picture B1, which is the top-layer picture, and picture B2, which is the lower-layer picture that are consecutive in decoding order, is T = 1 / 120 seconds. Therefore, in step S101, the decode times of the plurality of pictures included in the video are determined so that 2 x T = 1 / 60 seconds is equal to the above-mentioned evenly spaced time.

[0127] As a result, the interval between the decode times of each of the multiple lower-layer pictures is twice the interval between the decode times of the topmost and lower-layer pictures that are consecutive in decoding order, i.e., twice the interval between the times at which all of the pictures in the decoded stream are decoded. Therefore, if the frame rate when all of the pictures in the coded stream are decoded and displayed is 120 fps, the image decoding device can easily decode each of the multiple lower-layer pictures included in the coded stream at a time interval that is the reciprocal of the frame rate of 60 fps.

[0128] Furthermore, when a video has a plurality of random access units each consisting of a plurality of consecutive pictures in decoding order, the decoding times may be determined in step S101 as follows. That is, in step S101, for each random access unit, the decoding times of each picture in the random access unit are determined so that all pictures in the random access unit, except for a picture that is displayed earlier in display order than the first picture in decoding order, can be decoded without referring to pictures included in other random access units. Here, the first picture is an I-picture (a so-called IDR picture: Instantaneous Decoding Refresh Picture) in which pictures later than the first picture in decoding order are prohibited from referring to pictures earlier in decoding order than the first picture. Alternatively, the first picture is an I-picture (a so-called CRA picture: Clean Random Access Picture) in which pictures later than the first picture in decoding order and earlier in display order are permitted to refer to pictures earlier in decoding order than the first picture. For example, in step S101, the image coding device 100 determines the decoding times of each of the pictures included in the video, as shown in Fig. 6. In the example shown in Fig. 6, picture I16 is a CRA picture because picture B13, which precedes picture I16 in display order, references picture P12 in the first random access unit.

[0129] This allows the image decoding device to appropriately decode, for each random access unit, each of the multiple pictures included in that random access unit.

[0130] Here, a detailed description will be given of the processing operation performed by the image coding device 100 in step S101 to determine the decoding times of multiple pictures included in a video, as in the example shown in FIG.

[0131] For example, if the frame rate at which all pictures included in the coded video are decoded and displayed is 120 fps, the image coding device 100, in step S101, determines the decode time for each of the pictures in a time unit (1 / 120 seconds) that is the reciprocal of the frame rate (120 fps). That is, the decode time determined for each picture is expressed as an integer multiple of the time unit plus an offset value. Alternatively, in step S101, the image coding device 100 may first determine the TId for each of the pictures included in the video, and then determine the decoding order for each of the pictures. Then, the image coding device 100 determines the DTS for each of the pictures in the time unit based on the determined decoding order.

[0132] For example, the image coding device 100 determines, for each picture arranged in display order, a TId, which is a value for identifying the layer of that picture, in the range from 0 to K (K is an integer equal to or greater than 1). A layer with a TId of K is the highest layer, and a layer with a Tid of 0 is the lowest layer. Specifically, the determining unit 101 determines the Tid of the first I-picture among multiple pictures arranged in display order in a moving image to be 0. Furthermore, for each of M consecutive pictures (M is an integer equal to or greater than 2) among multiple pictures arranged in display order after the first I-picture, the determining unit 101 determines the TId of the last I-picture or P-picture in the picture set to be 0. In a picture set, the last picture in display order is an I-picture or P-picture, and the TId of that I-picture or P-picture is determined to be 0. A picture whose TId is determined to be 0 is hereinafter referred to as a layer 0 picture. 1A, four pictures including picture B1, picture B2, picture B3, and picture P4 correspond to the above-mentioned picture set, and picture P4 is determined as the layer 0 picture.

[0133] Next, determining unit 101 identifies, as a candidate picture, at least one picture (e.g., a B picture) included in the picture set other than a layer 0 picture, and at least one picture that is a candidate for determining a TId. This candidate picture is a picture displayed between two default pictures whose TIds have already been determined. For example, in the example shown in FIG. 1A, if the TIds of pictures I0 and P4 have already been determined, pictures I0 and P4 are default pictures. In this case, in a picture set including pictures B1 to B3 and P4, picture B2, which is a picture displayed between pictures I0 and P4, is identified as a candidate picture.

[0134] Furthermore, if there are multiple candidate pictures identified in this way, determining unit 101 determines the TId of the first candidate picture in display order to be the value N obtained by adding 1 to the TId of the two default pictures corresponding to the first candidate picture, which is not the lower one. For example, in the example shown in FIG. 1B , if the TIds of pictures I0, P8, and B4 have already been determined, then pictures I0, P8, and B4 are default pictures. In this case, in a picture set including pictures B1 to B7 and P8, picture B2, which is displayed midway between pictures I0 and B4, and picture B6, which is displayed midway between pictures B4 and P8, are identified as candidate pictures. Therefore, the determination unit 101 determines the TId of the first candidate picture B2 in display order among these candidate pictures B2 and B6 to be the value (N=2) obtained by adding 1 to the TId=1 of the two default pictures I0 and B4 corresponding to the first candidate picture B2, which is not the lower one.

[0135] The determining unit 101 repeats this process of identifying candidate pictures and determining Tids until ITds have been determined for all pictures other than layer 0 pictures that are included in the picture set and for which N is equal to or less than K. As a result, TIds are determined for each picture, as shown in Figure 1A or 1B.

[0136] Next, the determining unit 101 determines that the decoding order of the first I-picture described above is 1. For example, as shown in Fig. 6, the determining unit 101 determines that the decoding order of the first I-picture, picture I0, is 1.

[0137] Then, for each of the above-mentioned picture sets, the determining unit 101 determines the decoding order of each of the multiple pictures in the picture set. Specifically, the determining unit 101 determines the decoding order of the layer 0 picture in the picture set to be first. Then, the determining unit 101 determines the decoding order of the multiple pictures included in the picture set other than the layer 0 picture, starting from the earliest picture in display order, so that the pictures immediately follow the previously determined decoding order. For example, in a picture set including pictures B1 to B7 and picture P8, the determining unit 101 determines the decoding order of picture P8, which is the layer 0 picture, to be first. Then, the determining unit 101 determines the decoding order of pictures B1 to B3 so that picture P8 is followed in order by picture B1, picture B2, and picture B3.

[0138] Furthermore, the determining unit 101 determines that multiple picture sets arranged in display order are to be decoded in the order in which they are arranged. That is, the determining unit 101 determines that the decoding order of the first picture in a picture set is the last decoding order of the immediately preceding picture set in display order, or the ordinal number obtained by adding 1 to the decoding order of the first I-picture (picture I0) described above.

[0139] Furthermore, after determining the decode time of the first I-picture (picture I0), the determining unit 101 determines the decode time of the picture (picture P4) immediately following in the decoding order of the I-picture to be the decode time of the I-picture plus the above-mentioned time unit (1 / 120 seconds) × 2. Furthermore, the determining unit 101 determines the decode time of each picture later in the decoding order than the above-mentioned immediately following picture (picture P4) to be the time obtained by adding the above-mentioned time unit to the decode time of the picture immediately preceding it in the decoding order.

[0140] As described above, the determining unit 101 determines the decode time of each picture, so that the decode times of the multiple lower layer pictures are spaced at equal intervals, that is, at intervals of twice the time unit (1 / 120 seconds) described above.

[0141] Furthermore, when determining the decoding times in step S101, if the frame rate at which all pictures included in the encoded video are decoded and displayed is f, the decoding times of each of the multiple lower-layer pictures included in all of the pictures may be determined so that the decoding times are separated by a time indicated by n times the reciprocal of f (n is an integer greater than or equal to 2).

[0142] This allows the image decoding device to decode each of the multiple lower layer pictures in order without burden, at time intervals that are n times the reciprocal of the frame rate.

[0143] In addition, an image encoding method according to one aspect of the present invention may further include, in the encoded stream, display delay information indicating a display delay, which is the time between the decoding time of the first picture in the decoding order included in the video and the display time of the first picture in the display order included in the video.

[0144] This allows the image decoding device to acquire the display delay information from the coded stream. Therefore, as shown in Fig. 8, if the image decoding device starts decoding the coded stream from a time earlier than the display start time by the display delay indicated by the display delay information, the image decoding device can display the moving image without delay from the display start time.

[0145] In addition, an image encoding method according to one aspect of the present invention may further include, in the encoded stream, unequal interval information indicating that the decoding times determined for each of the multiple pictures included in the moving image are not equally spaced.

[0146] This allows the image decoding device to obtain unequal interval information from the coded stream. Therefore, the image decoding device can determine that each of the multiple pictures included in the coded stream cannot be sequentially decoded at the display frame rate. As a result, the image decoding device can decode each of the multiple pictures included in the coded stream at an appropriate timing by referring to the decoding times determined for each of the multiple pictures included in the coded stream, according to the flowchart shown in FIG. 10 .

[0147] FIG. 15C is a block diagram of an image decoding device according to one aspect of the present invention.

[0148] An image decoding device 200 according to one aspect of the present invention is a device that decodes, for each picture belonging to one of a plurality of layers, a coded stream including video that has been coded without reference to other pictures belonging to a layer higher than the layer to which the picture belongs. The image decoding device 200 includes an acquisition unit 201 and a decoding unit 202.

[0149] The obtaining unit 201 obtains the decode times of each of the multiple pictures included in the coded stream from the coded stream. Here, the decode times of each of the multiple pictures are determined as follows. That is, these decode times are determined so that the decode times of multiple lower-layer pictures, which are some of the multiple pictures included in the coded stream and do not belong to the highest hierarchical layer among the multiple hierarchical layers, are equally spaced. Furthermore, these decode times are determined so that the timing at which each of the multiple lower-layer pictures is decoded is the same between when the multiple pictures included in the coded stream are decoded and when only multiple lower-layer pictures among the multiple pictures are decoded.

[0150] The decoding unit 202 decodes each of the multiple pictures or multiple lower layer pictures included in the coded stream according to the acquired decoding times.

[0151] FIG. 15D is a flowchart showing an image decoding method according to one embodiment of the present invention.

[0152] An image decoding method according to one aspect of the present invention is a method in which an image decoding device 200 decodes an encoded stream including video that has been encoded for each picture belonging to one of a plurality of layers without referring to other pictures belonging to a layer higher than the layer to which the picture belongs. This image decoding method includes steps S201 and S202.

[0153] In step S201, the decode time of each of the multiple pictures included in the coded stream is obtained from the coded stream. Here, the decode time of each of the multiple pictures is determined as follows. That is, these decode times are determined so that the decode times of multiple lower-layer pictures, which are some of the multiple pictures included in the coded stream and do not belong to the highest hierarchical layer among the multiple hierarchical layers, are equally spaced. Furthermore, these decode times are determined so that the timing at which each of the multiple lower-layer pictures is decoded is the same when the multiple pictures included in the coded stream are decoded and when only the multiple lower-layer pictures among the multiple pictures are decoded.

[0154] In step S202, the multiple pictures or multiple lower layer pictures included in the coded stream are decoded according to the acquired decoding times.

[0155] As a result, each of the multiple pictures included in the coded stream is coded without referring to other pictures belonging to a layer higher than the layer to which the picture belongs. Therefore, the image decoding device 200 can decode only the multiple lower-layer pictures included in the coded stream. Furthermore, the decoding times of the multiple lower-layer pictures included in the coded stream are equally spaced. Therefore, when decoding only the multiple lower-layer pictures included in the coded stream, the image decoding device 200 can sequentially decode the lower-layer pictures at equal time intervals. Therefore, if the equal intervals are appropriate, the processing load of the image decoding device 200 can be reduced. In other words, the image decoding device 200 can decode each picture at a frame rate appropriate to its own processing capability, without performing decoding at a high frame rate. Furthermore, the timing at which each of the multiple lower-layer pictures is decoded is the same when multiple pictures (e.g., all pictures) included in the coded stream are decoded and when only the multiple lower-layer pictures are decoded. Therefore, the image decoding device 200 does not need to change the timing of decoding each of the multiple lower-layer pictures depending on whether all pictures in the coded stream are being decoded or only the multiple lower-layer pictures are being decoded, thereby further reducing the processing load on the image decoding device 200.

[0156] In addition, an image decoding method according to one aspect of the present invention may further include, if the decoding times of each of multiple pictures included in an encoded stream are not equally spaced, changing the decoding times of each of the multiple pictures to be equally spaced, and, in decoding the encoded stream, decoding each of the multiple pictures or multiple lower-layer pictures included in the encoded stream according to the changed decoding times.

[0157] 11, the decoding times of the pictures are changed at regular intervals, so that the image decoding device 200 can decode the pictures included in the coded stream at regular time intervals, thereby further reducing the processing load on the image decoding device 200.

[0158] In addition, an image decoding method according to an aspect of the present invention may, in decoding an encoded stream, determine, for each picture included in the encoded stream, whether or not the decoding time acquired for that picture coincides with the generation timing of a processing signal (corresponding to the above-mentioned video processing signal) that occurs at a predetermined cycle, and decode the picture when it is determined that the decoding time coincides. For example, the image decoding method may further determine, as the above-mentioned predetermined cycle, the reciprocal of the frame rate at which all pictures included in the encoded stream are decoded and displayed.

[0159] As a result, as shown in the flowchart of FIG. 10, even if the decoding times of the plurality of pictures are not equally spaced, each of the plurality of pictures can be decoded appropriately at the decoding time of that picture.

[0160] In each of the above embodiments, each component may be configured with dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory. Here, software for realizing the image encoding device 10, 100 of the above embodiment or modification causes a computer to execute each step included in the flowchart shown in FIG. 15B. Furthermore, software for realizing the image decoding device 20, 200 of the above embodiment or modification causes a computer to execute each step included in the flowchart shown in FIG. 15D.

[0161] (Embodiment 2) By recording a program for implementing the video coding method (image coding method) or video decoding method (image decoding method) shown in each of the above embodiments on a storage medium, it becomes possible to easily perform the processes shown in each of the above embodiments on an independent computer system. The storage medium may be a magnetic disk, optical disk, magneto-optical disk, IC card, semiconductor memory, or any other medium capable of recording a program.

[0162] Furthermore, here, we will explain application examples of the video coding method (image coding method) and video decoding method (image decoding method) shown in each of the above embodiments, and a system using the same. The system is characterized by having an image coding / decoding device consisting of an image coding device using the image coding method and an image decoding device using the image decoding method. Other components of the system can be appropriately changed depending on the situation.

[0163] 16 is a diagram showing the overall configuration of a content supply system ex100 that provides a content distribution service. The area where communication services are provided is divided into cells of a desired size, and base stations ex106, ex107, ex108, ex109, and ex110, which are fixed wireless stations, are installed in each cell.

[0164] This content supply system ex100 is connected to the Internet ex101 via an Internet service provider ex102, a telephone network ex104, and base stations ex106 to ex110, and devices such as a computer ex111, a PDA (Personal Digital Assistant) ex112, a camera ex113, a mobile phone ex114, and a game console ex115.

[0165] However, the content supply system ex100 is not limited to the configuration shown in Fig. 16, and any combination of elements may be connected. Also, each device may be directly connected to the telephone network ex104 without going through base stations ex106 to ex110, which are fixed wireless stations. Also, each device may be directly connected to each other via short-range wireless or the like.

[0166] The camera ex113 is a device capable of shooting moving images, such as a digital video camera, and the camera ex116 is a device capable of shooting still images and moving images, such as a digital camera. The mobile phone ex114 may be any of a GSM (registered trademark) (Global System for Mobile Communications) system, a CDMA (Code Division Multiple Access) system, a W-CDMA (Wideband-Code Division Multiple Access) system, an LTE (Long Term Evolution) system, an HSPA (High Speed ​​Packet Access) mobile phone, or a PHS (Personal Handyphone System) system.

[0167] In the content supply system ex100, a camera ex113 and the like are connected to a streaming server ex103 via a base station ex109 and a telephone network ex104, thereby enabling live streaming and the like. In live streaming, a user shoots content (e.g., video of a live music concert) using the camera ex113, and encodes the content as described in the above embodiments (i.e., functions as an image encoding device according to an aspect of the present invention) and transmits the content to the streaming server ex103. Meanwhile, the streaming server ex103 streams the transmitted content data to a requesting client. Examples of clients include a computer ex111, a PDA ex112, a camera ex113, a mobile phone ex114, a game console ex115, and the like that are capable of decoding the encoded data. Each device that receives the distributed data decodes and plays back the received data (i.e., functions as an image decoding device according to an aspect of the present invention).

[0168] The encoding process of the captured data may be performed by the camera ex113, by the streaming server ex103 that processes the data transmission, or by a mutually shared responsibility. Similarly, the decoding process of the distributed data may be performed by the client, by the streaming server ex103, or by a mutually shared responsibility. Furthermore, still images and / or video data captured by camera ex116, not limited to camera ex113, may be transmitted to the streaming server ex103 via computer ex111. In this case, the encoding process may be performed by the camera ex116, the computer ex111, or the streaming server ex103, or by a mutually shared responsibility.

[0169] Furthermore, these encoding and decoding processes are generally performed by the computer ex111 or an LSIex500 possessed by each device. The LSIex500 may be a single chip or may be configured with multiple chips. It is also possible to embed video encoding and decoding software on some kind of recording medium (CD-ROM, flexible disk, hard disk, etc.) that can be read by the computer ex111, etc., and perform the encoding and decoding processes using that software. Furthermore, if the mobile phone ex114 is equipped with a camera, video data captured by the camera may be transmitted. This video data is data that has been encoded and processed by the LSIex500 possessed by the mobile phone ex114.

[0170] The streaming server ex103 may also be a plurality of servers or computers that process, record, and distribute data in a distributed manner.

[0171] In this way, the content delivery system ex100 allows a client to receive and play back encoded data. In this way, the content delivery system ex100 allows a client to receive, decode, and play back information sent by a user in real time, enabling even users without special rights or equipment to realize personal broadcasting.

[0172] In addition to the example of the content supply system ex100, as shown in FIG. 17, at least one of the video encoding device (image encoding device) or video decoding device (image decoding device) according to each of the above embodiments can also be incorporated into a digital broadcasting system ex200. Specifically, a broadcasting station ex201 communicates via radio waves multiplexed data in which music data and the like are multiplexed onto video data, or transmits the multiplexed data to a satellite ex202. This video data is data encoded using the video encoding method described in each of the above embodiments (i.e., data encoded by an image encoding device according to one aspect of the present invention). Receiving this, the broadcasting satellite ex202 transmits broadcast radio waves, which are received by a home antenna ex204 capable of receiving satellite broadcasts. The received multiplexed data is decoded and played back by a device such as a television (receiver) ex300 or a set-top box (STB) ex217 (i.e., functions as an image decoding device according to one aspect of the present invention).

[0173] The video decoding device or video encoding device described in each of the above embodiments can also be implemented in a reader / recorder ex218 that reads and decodes multiplexed data recorded on a recording medium ex215 such as a DVD or Blu-ray, or encodes a video signal onto the recording medium ex215 and, in some cases, multiplexes it with an audio signal before writing it. In this case, the reproduced video signal is displayed on a monitor ex219, and the video signal can be reproduced in another device or system using the recording medium ex215 on which the multiplexed data is recorded. Alternatively, a video decoding device may be implemented in a set-top box ex217 connected to a cable television cable ex203 or a satellite / terrestrial broadcast antenna ex204, and the video may be displayed on the television monitor ex219. In this case, the video decoding device may be incorporated into the television rather than the set-top box.

[0174] 18 is a diagram showing a television (receiver) ex300 that uses the video decoding method and video encoding method described in each of the above embodiments. The television ex300 includes a tuner ex301 that acquires or outputs multiplexed data in which audio data is multiplexed onto video data via an antenna ex204 that receives the broadcasts or a cable ex203, a modulation / demodulation unit ex302 that demodulates the received multiplexed data or modulates it into multiplexed data to be transmitted externally, and a multiplexing / demultiplexing unit ex303 that separates the demodulated multiplexed data into video data and audio data or multiplexes the video data and audio data encoded by a signal processing unit ex306.

[0175] The television ex300 also has a signal processing unit ex306 having an audio signal processing unit ex304 and a video signal processing unit ex305 (which function as an image encoding device or an image decoding device according to an embodiment of the present invention) that decode the audio data and the video data, respectively, or encode the respective information, and an output unit ex309 having a speaker ex307 that outputs the decoded audio signal and a display unit ex308 such as a display that displays the decoded video signal.The television ex300 also has an interface unit ex317 that has an operation input unit ex312 that accepts user operation input, etc.The television ex300 also has a control unit ex310 that controls each unit overall, and a power supply circuit unit ex311 that supplies power to each unit. In addition to the operation input unit ex312, the interface unit ex317 may have a bridge ex313 connected to an external device such as a reader / recorder ex218, a slot unit ex314 for allowing a recording medium ex216 such as an SD card to be attached, a driver ex315 for connecting to an external recording medium such as a hard disk, a modem ex316 for connecting to a telephone network, etc. The recording medium ex216 is a non-volatile / volatile semiconductor memory element that stores information and allows it to be electrically recorded. The various units of the television ex300 are connected to each other via a synchronous bus.

[0176] First, a configuration in which the television ex300 decodes and plays back multiplexed data acquired from an external source via an antenna ex204 or the like will be described. The television ex300 receives user operation via a remote controller ex220 or the like, and, under the control of a control unit ex310 having a CPU or the like, separates the multiplexed data demodulated by a modulation / demodulation unit ex302 in a multiplexing / separation unit ex303. The television ex300 then decodes the separated audio data in an audio signal processing unit ex304 and decodes the separated video data in a video signal processing unit ex305 using the decoding method described in each of the above embodiments. The decoded audio and video signals are output to the outside from an output unit ex309. When outputting, it is preferable to temporarily store these signals in buffers ex318, ex319, or the like so that the audio and video signals are played back in sync. The television ex300 may also read the multiplexed data from recording media ex215, ex216, such as magnetic / optical discs or SD cards, rather than from broadcasts or the like. Next, a configuration in which the television ex300 encodes audio and video signals and transmits them externally or writes them to a recording medium or the like will be described. The television ex300 receives user operation from a remote controller ex220 or the like, and, under the control of the control unit ex310, encodes the audio signal in the audio signal processing unit ex304 and encodes the video signal in the video signal processing unit ex305 using the encoding method described in each of the above embodiments. The encoded audio and video signals are multiplexed by the multiplexing / demultiplexing unit ex303 and output externally. When multiplexing, these signals may be temporarily stored in buffers ex320, ex321, etc., so that the audio and video signals are synchronized. Note that multiple buffers ex318, ex319, ex320, and ex321 may be provided as shown, or one or more buffers may be shared. Furthermore, data may be stored in buffers other than those shown in the figure, for example, between the modulation / demodulation unit ex302 and the multiplexing / demultiplexing unit ex303, as a buffer to prevent system overflow and underflow.

[0177] Furthermore, in addition to acquiring audio data and video data from broadcasts, recording media, etc., the television ex300 may also be configured to accept AV input from a microphone or camera and perform encoding processing on the data acquired from them. Note that while the television ex300 has been described here as being configured to be capable of the above encoding processing, multiplexing, and external output, it may also be configured not to be able to perform these processes and only be capable of the above reception, decoding processing, and external output.

[0178] Furthermore, when multiplexed data is read from or written to a recording medium using the reader / recorder ex218, the above-mentioned decoding or encoding process may be performed by either the television ex300 or the reader / recorder ex218, or the television ex300 and the reader / recorder ex218 may share the process.

[0179] As an example, Figure 19 shows the configuration of the information reproducing / recording unit ex400 when reading or writing data from an optical disc. The information reproducing / recording unit ex400 includes the following elements ex401, ex402, ex403, ex404, ex405, ex406, and ex407. The optical head ex401 writes information by irradiating a laser spot onto the recording surface of the recording medium ex215, which is an optical disc, and reads the information by detecting the light reflected from the recording surface of the recording medium ex215. The modulation / recording unit ex402 electrically drives the semiconductor laser built into the optical head ex401 and modulates the laser light according to the recorded data. The reproduction / demodulation unit ex403 amplifies the reproduction signal obtained by electrically detecting the light reflected from the recording surface using a photodetector built into the optical head ex401, and separates and demodulates the signal components recorded on the recording medium ex215 to reproduce the required information. The buffer ex404 temporarily stores information to be recorded on the recording medium ex215 and information reproduced from the recording medium ex215. The disk motor ex405 rotates the recording medium ex215. The servo control unit ex406 controls the rotation of the disk motor ex405, moves the optical head ex401 to a specified information track, and performs laser spot tracking. The system control unit ex407 controls the entire information reproduction / recording unit ex400. The system control unit ex407 performs the above read and write processes by using various information stored in the buffer ex404, generating and adding new information as needed, and recording and reproducing information through the optical head ex401 while coordinating the modulation recording unit ex402, reproduction demodulation unit ex403, and servo control unit ex406. The system control unit ex407 is composed of, for example, a microprocessor and performs these processes by executing read and write programs.

[0180] In the above description, the optical head ex401 is described as irradiating a laser spot, but it may be configured to perform higher density recording using near-field light.

[0181] FIG. 20 shows a schematic diagram of recording medium ex215, an optical disc. A spiral guide groove is formed on the recording surface of recording medium ex215, and address information indicating absolute positions on the disc is recorded in advance on information track ex230 by changing the shape of the groove. This address information includes information for identifying the position of recording block ex231, which is the unit of data recording. A recording or playback device can identify a recording block by reproducing information track ex230 and reading the address information. Recording medium ex215 also includes a data recording area ex233, an inner peripheral area ex232, and an outer peripheral area ex234. The data recording area ex233 is the area used for recording user data, while the inner peripheral area ex232 and outer peripheral area ex234, which are located either inner or outer than data recording area ex233, are used for specific purposes other than recording user data. The information reproducing / recording unit ex400 reads and writes encoded audio data, video data, or multiplexed data obtained by multiplexing these data, from the data recording area ex233 of such recording medium ex215.

[0182] The above explanation has been given using examples of optical discs such as single-layer DVDs and BDs, but the present invention is not limited to these and may be an optical disc with a multi-layer structure that allows recording on areas other than the surface. It may also be an optical disc with a structure that allows multidimensional recording / playback, such as recording information using light of various different wavelengths in the same location on the disc or recording different layers of information from various angles.

[0183] In addition, in the digital broadcasting system ex200, a car ex210 equipped with an antenna ex205 can receive data from a satellite ex202 or the like, and the video can be played on a display device such as a car navigation system ex211 installed in the car ex210. The car navigation system ex211 can be configured, for example, by adding a GPS receiving unit to the configuration shown in Fig. 18, and similar configurations can be considered for a computer ex111, a mobile phone ex114, and the like.

[0184] 21A is a diagram showing a mobile phone ex114 that uses the video decoding method and video encoding method described in the above embodiment. The mobile phone ex114 includes an antenna ex350 for transmitting and receiving radio waves to and from base station ex110, a camera unit ex365 capable of capturing video and still images, and a display unit ex358 such as an LCD display that displays decoded data of video captured by the camera unit ex365 and video received by the antenna ex350. The mobile phone ex114 further includes a main body unit having an operation key unit ex366, an audio output unit ex357 such as a speaker for outputting audio, an audio input unit ex356 such as a microphone for inputting audio, a memory unit ex367 for storing captured video, still images, recorded audio, or encoded or decoded data of received video, still images, email, etc., or a slot unit ex364 that serves as an interface with a recording medium for similarly storing data.

[0185] Furthermore, a configuration example of mobile phone ex114 will be described with reference to Fig. 21B. Mobile phone ex114 has a main control unit ex360 that comprehensively controls each unit of a main body unit including a display unit ex358 and an operation key unit ex366, and a power supply circuit unit ex361, an operation input control unit ex362, a video signal processing unit ex355, a camera interface unit ex363, an LCD (Liquid Crystal Display) control unit ex359, a modulation / demodulation unit ex352, a multiplexing / demultiplexing unit ex353, an audio signal processing unit ex354, a slot unit ex364, and a memory unit ex367, which are all connected to each other via a bus ex370.

[0186] When the end call and power key is turned on by the user, the power supply circuit unit ex361 starts up the mobile phone ex114 into an operable state by supplying power to each unit from the battery pack.

[0187] Based on the control of a main control unit ex360 having a CPU, ROM, RAM, etc., the mobile phone ex114 converts an audio signal collected by an audio input unit ex356 into a digital audio signal by an audio signal processing unit ex354 in a voice call mode, which undergoes spectrum spread processing by a modulation / demodulation unit ex352, digital-to-analog conversion processing and frequency conversion processing by a transmission / reception unit ex351, and then transmits the digital audio signal via an antenna ex350. Furthermore, the mobile phone ex114 amplifies received data received via the antenna ex350 in a voice call mode, performs frequency conversion processing and analog-to-digital conversion processing, performs spectrum despread processing by the modulation / demodulation unit ex352, converts the data into an analog audio signal by the audio signal processing unit ex354, and then outputs the data from an audio output unit ex357.

[0188] Furthermore, when sending an e-mail in data communication mode, the text data of the e-mail entered by operating the operation key unit ex366 or the like of the main unit is sent to the main control unit ex360 via the operation input control unit ex362. The main control unit ex360 performs spectrum spread processing on the text data in the modulation / demodulation unit ex352, performs digital-to-analog conversion processing and frequency conversion processing in the transmission / reception unit ex351, and then transmits the data to the base station ex110 via the antenna ex350. When receiving an e-mail, the received data is subjected to roughly the reverse processing and output to the display unit ex358.

[0189] When transmitting video, still images, or video and audio in the data communication mode, the video signal processing unit ex355 compresses and encodes the video signal supplied from the camera unit ex365 using the video encoding method described in each of the above embodiments (i.e., functions as an image encoding device according to one aspect of the present invention), and sends the encoded video data to the multiplexing / separating unit ex353. In addition, the audio signal processing unit ex354 encodes the audio signal collected by the audio input unit ex356 while the camera unit ex365 is capturing video, still images, etc., and sends the encoded audio data to the multiplexing / separating unit ex353.

[0190] The multiplexing / separation unit ex353 multiplexes the encoded video data supplied from the video signal processing unit ex355 and the encoded audio data supplied from the audio signal processing unit ex354 using a predetermined method, and the resulting multiplexed data is subjected to spectrum spreading processing in the modulation / demodulation unit (modulation / demodulation circuit unit) ex352, digital-to-analog conversion processing and frequency conversion processing in the transmission / reception unit ex351, and then transmitted via the antenna ex350.

[0191] When receiving video file data linked to a website or the like in data communication mode, or when receiving an email with video and / or audio attachments, the multiplexer / demultiplexer ex353 decodes the multiplexed data received via the antenna ex350 into a video data bitstream and an audio data bitstream. The multiplexer / demultiplexer ex353 then decodes the multiplexed data into a video data bitstream and an audio data bitstream via a synchronization bus ex370. The video signal processor ex355 decodes the video signal using a video decoding method corresponding to the video encoding method described in each of the above embodiments (i.e., functions as an image decoding device according to one aspect of the present invention). The display unit ex358 displays, via an LCD controller ex359, video and still images included in the video file linked to a website, for example. The audio signal processor ex354 decodes the audio signal, and audio is output from an audio output unit ex357.

[0192] Furthermore, like the television ex300, terminals such as the mobile phone ex114 can be implemented in three ways: a transmitting / receiving terminal with both an encoder and a decoder, a transmitting terminal with only an encoder, and a receiving terminal with only a decoder. Furthermore, in the digital broadcasting system ex200, it has been explained that multiplexed data in which music data and the like are multiplexed onto video data is received and transmitted, but the data may also be multiplexed with text data related to the video in addition to audio data, or it may be video data itself rather than multiplexed data.

[0193] In this way, it is possible to use the video encoding method or video decoding method shown in each of the above embodiments in any of the above-mentioned devices and systems, and by doing so, it is possible to obtain the effects described in each of the above embodiments.

[0194] Furthermore, the present invention is not limited to the above-described embodiment, and various modifications and alterations are possible without departing from the scope of the present invention.

[0195] (Embodiment 3) It is also possible to generate video data by switching between the video encoding method or device shown in each of the above embodiments and a video encoding method or device conforming to a different standard, such as MPEG-2, MPEG4-AVC, or VC-1, as needed.

[0196] When multiple pieces of video data conforming to different standards are generated, it is necessary to select a decoding method corresponding to each standard when decoding. However, since it is not possible to identify which standard the video data to be decoded conforms to, a problem arises in that it is not possible to select an appropriate decoding method.

[0197] To solve this problem, multiplexed data, which is video data multiplexed with audio data, etc., is configured to include identification information that indicates which standard the video data conforms to. A specific configuration of multiplexed data including video data generated by the video encoding methods or devices described in the above embodiments is described below. The multiplexed data is a digital stream in MPEG-2 transport stream format.

[0198] FIG. 22 shows the structure of multiplexed data. As shown in FIG. 22, the multiplexed data is obtained by multiplexing one or more of a video stream, an audio stream, a presentation graphics stream (PG), and an interactive graphics stream. The video stream represents the main video and secondary video of a movie, the audio stream (IG) represents the main audio portion of the movie and the secondary audio mixed with the main audio, and the presentation graphics stream represents the subtitles of the movie. Here, the main video refers to the normal video displayed on the screen, and the secondary video refers to the video displayed on a small screen within the main video. The interactive graphics stream represents an interactive screen created by arranging GUI components on the screen. The video stream is encoded using the video encoding method or device described in each of the above embodiments or a video encoding method or device conforming to conventional standards such as MPEG-2, MPEG4-AVC, or VC-1. The audio stream is encoded using a format such as Dolby AC-3, Dolby Digital Plus, MLP, DTS, DTS-HD, or Linear PCM.

[0199] Each stream included in the multiplexed data is identified by a PID. For example, 0x1011 is assigned to the video stream used for movie images, 0x1100 to 0x111F to the audio stream, 0x1200 to 0x121F to the presentation graphics, 0x1400 to 0x141F to the interactive graphics stream, 0x1B00 to 0x1B1F to the video stream used for movie secondary video, and 0x1A00 to 0x1A1F to the audio stream used for secondary audio to be mixed with the main audio.

[0200] 23 is a diagram showing how multiplexed data is multiplexed. First, a video stream ex235 consisting of multiple video frames and an audio stream ex238 consisting of multiple audio frames are converted into PES packet sequences ex236 and ex239, respectively, and then converted into TS packets ex237 and ex240. Similarly, presentation graphics stream ex241 and interactive graphics data ex244 are converted into PES packet sequences ex242 and ex245, respectively, and then converted into TS packets ex243 and ex246. Multiplexed data ex247 is constructed by multiplexing these TS packets into a single stream.

[0201] FIG. 24 shows in more detail how a video stream is stored in a PES packet sequence. The first row in FIG. 24 shows a video frame sequence of the video stream. The second row shows a PES packet sequence. As indicated by arrows yy1, yy2, yy3, and yy4 in FIG. 24, I-pictures, B-pictures, and P-pictures, which are multiple Video Presentation Units in the video stream, are divided into individual pictures and stored in the payload of a PES packet. Each PES packet has a PES header, which stores a Presentation Time-Stamp (PTS), which is the display time of the picture, and a Decoding Time-Stamp (DTS), which is the decoding time of the picture.

[0202] Figure 25 shows the format of the TS packet that is ultimately written to the multiplexed data. TS packets are 188-byte fixed-length packets consisting of a 4-byte TS header containing information such as a PID that identifies the stream, and a 184-byte TS payload that stores the data. The PES packets are divided and stored in the TS payload. In the case of BD-ROMs, a 4-byte TP_Extra_Header is added to the TS packet, forming a 192-byte source packet that is written to the multiplexed data. The TP_Extra_Header contains information such as an ATS (Arrival Time Stamp). The ATS indicates the start time of the transfer of the TS packet to the PID filter of the decoder. As shown in the lower part of Figure 25, source packets are lined up in the multiplexed data, and the number that increments from the beginning of the multiplexed data is called the SPN (Source Packet Number).

[0203] In addition to the individual streams (video, audio, subtitles, etc.), the TS packets contained in the multiplexed data also contain a Program Association Table (PAT), Program Map Table (PMT), and Program Clock Reference (PCR). The PAT indicates the PID of the PMT used in the multiplexed data, and the PAT's own PID is registered as 0. The PMT contains the PIDs of each stream (video, audio, subtitles, etc.) contained in the multiplexed data, as well as attribute information for the streams corresponding to each PID. It also contains various descriptors related to the multiplexed data. The descriptors include copy control information that indicates whether copying of the multiplexed data is permitted or prohibited. The PCR contains information about the Arrival Time Clock (ATC), which is the time axis of the ATS, and the System Time Clock (STC), which is the time axis of the PTS and DTS, and indicates the STC time corresponding to the ATS at which the PCR packet is transferred to the decoder.

[0204] Figure 26 is a diagram explaining the data structure of a PMT in detail. At the beginning of a PMT is a PMT header that describes the length of the data contained in the PMT, among other things. This is followed by multiple descriptors related to the multiplexed data. The above-mentioned copy control information and other information are written as descriptors. After the descriptors are multiple stream information items related to each stream included in the multiplexed data. The stream information consists of stream descriptors that describe the stream type to identify the stream compression codec, the stream PID, and stream attribute information (frame rate, aspect ratio, etc.). There are as many stream descriptors as there are streams in the multiplexed data.

[0205] When recording on a recording medium, the multiplexed data is recorded together with a multiplexed data information file.

[0206] As shown in FIG. 27, the multiplexed data information file is management information for multiplexed data, has one-to-one correspondence with the multiplexed data, and is made up of multiplexed data information, stream attribute information, and an entry map.

[0207] As shown in Fig. 27, the multiplexed data information consists of a system rate, a playback start time, and a playback end time. The system rate indicates the maximum transfer rate of the multiplexed data to the PID filter of the system target decoder, which will be described later. The interval between ATSs contained in the multiplexed data is set to be equal to or less than the system rate. The playback start time is set to the PTS of the first video frame of the multiplexed data, and the playback end time is set to the PTS of the last video frame of the multiplexed data plus the playback interval of one frame.

[0208] As shown in Figure 28, the stream attribute information for each stream included in the multiplexed data is registered for each PID. The attribute information has different information for each video stream, audio stream, presentation graphics stream, and interactive graphics stream. The video stream attribute information includes information such as the compression codec used to compress the video stream, the resolution of the individual picture data that make up the video stream, the aspect ratio, and the frame rate. The audio stream attribute information includes information such as the compression codec used to compress the audio stream, the number of channels included in the audio stream, the language it supports, and the sampling frequency. This information is used to initialize the decoder before playback on the player.

[0209] In this embodiment, the stream type included in the PMT of the multiplexed data is used. Furthermore, if multiplexed data is recorded on a recording medium, the video stream attribute information included in the multiplexed data information is used. Specifically, the video coding method or device shown in each of the above embodiments includes a step or means for setting, in the stream type included in the PMT or the video stream attribute information, unique information indicating that the video data is generated by the video coding method or device shown in each of the above embodiments. This configuration makes it possible to distinguish between video data generated by the video coding method or device shown in each of the above embodiments and video data that conforms to other standards.

[0210] FIG. 29 shows the steps of the video decoding method according to this embodiment. In step exS100, the stream type included in the PMT or the video stream attribute information included in the multiplexed data information is obtained from the multiplexed data. Next, in step exS101, it is determined whether the stream type or the video stream attribute information indicates that the multiplexed data was generated by the video coding method or device described in the above embodiments. If it is determined that the stream type or the video stream attribute information was generated by the video coding method or device described in the above embodiments, in step exS102, decoding is performed using the video decoding method described in the above embodiments. If the stream type or the video stream attribute information indicates that the data complies with a conventional standard such as MPEG-2, MPEG4-AVC, or VC-1, decoding is performed using the video decoding method according to the conventional standard in step exS103.

[0211] In this way, by setting a new unique value in the stream type or video stream attribute information, it is possible to determine whether the video decoding method or device shown in each of the above embodiments can decode the data when decoding. Therefore, even when multiplexed data conforming to a different standard is input, an appropriate decoding method or device can be selected, enabling decoding without errors. Furthermore, the video encoding method or device or video decoding method or device shown in this embodiment can be used in any of the above-mentioned devices and systems.

[0212] (Fourth embodiment) The video encoding method and device, and video decoding method and device described in each of the above embodiments are typically realized by an LSI, which is an integrated circuit. As an example, FIG. 30 shows the configuration of a single-chip LSI ex500. LSI ex500 includes elements ex501, ex502, ex503, ex504, ex505, ex506, ex507, ex508, and ex509, which are described below, and each element is connected via a bus ex510. When the power supply is on, a power supply circuit unit ex505 supplies power to each unit, thereby activating them into an operable state.

[0213] For example, when performing encoding processing, the LSI ex500 inputs AV signals from the microphone ex117, camera ex113, etc. via the AV I / O ex509 under the control of a control unit ex501 including a CPU ex502, a memory controller ex503, a stream controller ex504, a drive frequency control unit ex512, etc. The input AV signals are temporarily stored in an external memory ex511 such as an SDRAM. Under the control of the control unit ex501, the stored data is divided into multiple batches as appropriate depending on the processing volume and processing speed and sent to the signal processing unit ex507, where the audio signal and / or video signal is encoded. Here, the video signal encoding processing is the encoding processing described in each of the above embodiments. The signal processing unit ex507 may further perform processing such as multiplexing the encoded audio data and the encoded video data, and output the resulting data to the outside from the stream I / O ex506. This output multiplexed data is transmitted to the base station ex107 or written to a recording medium ex215. When multiplexing, it is advisable to temporarily store the data in a buffer ex508 to ensure synchronization.

[0214] Although the memory ex511 has been described above as being external to the LSIex500, it may be included within the LSIex500. The buffer ex508 is not limited to one, and multiple buffers may be provided. Furthermore, the LSIex500 may be formed as a single chip or multiple chips.

[0215] Furthermore, in the above description, the control unit ex501 is described as having a CPU ex502, a memory controller ex503, a stream controller ex504, a drive frequency control unit ex512, etc., but the configuration of the control unit ex501 is not limited to this configuration. For example, the signal processing unit ex507 may further include a CPU. By providing a CPU inside the signal processing unit ex507, it is possible to further improve processing speed. As another example, the CPU ex502 may include the signal processing unit ex507, or a part of the signal processing unit ex507, such as an audio signal processing unit. In such a case, the control unit ex501 is configured to include a CPU ex502 that includes the signal processing unit ex507, or a part of it.

[0216] Although we have referred to it as an LSI here, it may also be called an IC, system LSI, super LSI, or ultra LSI depending on the level of integration.

[0217] Furthermore, the method of integration is not limited to LSI, but may be realized by dedicated circuits or general-purpose processors. It is also possible to use FPGAs (Field Programmable Gate Arrays), which can be programmed after LSI manufacturing, or reconfigurable processors, which allow the connections and settings of circuit cells within LSIs to be reconfigured.

[0218] Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology or other derivative technologies, it is natural that such technology could be used to integrate functional blocks. The application of biotechnology is also a possibility.

[0219] (Embodiment 5) When decoding video data generated by the video encoding method or device described in each of the above embodiments, the amount of processing is likely to increase compared to when decoding video data conforming to conventional standards such as MPEG-2, MPEG4-AVC, or VC-1. Therefore, it is necessary to set the drive frequency of the LSIex500 to a higher frequency than the drive frequency of the CPUex502 when decoding video data conforming to conventional standards. However, increasing the drive frequency raises the problem of increased power consumption.

[0220] To solve this problem, video decoding devices such as televisions ex300 and LSIs ex500 are configured to identify the standard to which video data conforms and switch the drive frequency according to the standard. FIG. 31 shows a configuration ex800 in this embodiment. If the video data was generated using the video encoding method or device described in each of the above embodiments, a drive frequency switching unit ex803 sets a high drive frequency. The unit then instructs a decoding processing unit ex801, which executes the video decoding method described in each of the above embodiments, to decode the video data. On the other hand, if the video data conforms to a conventional standard, the unit sets a low drive frequency compared to when the video data was generated using the video encoding method or device described in each of the above embodiments. The unit then instructs a decoding processing unit ex802, which conforms to the conventional standard, to decode the video data.

[0221] More specifically, the drive frequency switching unit ex803 is composed of the CPU ex502 and drive frequency control unit ex512 in FIG. 30. The decoding processing unit ex801 that executes the video decoding method described in each of the above embodiments and the decoding processing unit ex802 that complies with the conventional standard correspond to the signal processing unit ex507 in FIG. 30. The CPU ex502 identifies the standard to which the video data conforms. The driving frequency control unit ex512 sets the drive frequency based on the signal from the CPU ex502. The signal processing unit ex507 decodes the video data based on the signal from the CPU ex502. Here, the video data can be identified using, for example, the identification information described in the third embodiment. The identification information is not limited to that described in the third embodiment, and may be any information that can identify the standard to which the video data conforms. For example, if it is possible to identify the standard to which the video data conforms based on an external signal that identifies whether the video data is for use on a television or a disc, then the identification may be based on such an external signal. Furthermore, the selection of the drive frequency in the CPUex502 can be performed based on a lookup table that associates the video data standard with the drive frequency, as shown in Fig. 33. The lookup table is stored in the buffer ex508 or the internal memory of the LSI, and the CPUex502 can select the drive frequency by referring to this lookup table.

[0222] FIG. 32 shows steps for implementing the method of this embodiment. First, in step exS200, the signal processing unit ex507 acquires identification information from the multiplexed data. Next, in step exS201, the CPU ex502 identifies, based on the identification information, whether the video data was generated by the encoding method or device described in any of the above embodiments. If the video data was generated by the encoding method or device described in any of the above embodiments, in step exS202, the CPU ex502 sends a signal to the driving frequency control unit ex512 to set the driving frequency to a high level. The driving frequency control unit ex512 then sets the driving frequency to a high level. On the other hand, if the video data indicates that it complies with a conventional standard such as MPEG-2, MPEG4-AVC, or VC-1, in step exS203, the CPU ex502 sends a signal to the driving frequency control unit ex512 to set the driving frequency to a low level. The driving frequency control unit ex512 then sets the driving frequency to a lower level than when the video data was generated by the encoding method or device described in any of the above embodiments.

[0223] Furthermore, by changing the voltage applied to the LSIex500 or a device including the LSIex500 in conjunction with switching the drive frequency, it is possible to further enhance the power saving effect. For example, when the drive frequency is set low, it is conceivable to set the voltage applied to the LSIex500 or a device including the LSIex500 lower in response to this change than when the drive frequency is set high.

[0224] Furthermore, the method of setting the drive frequency is not limited to the above-described setting method, and may be such that a high drive frequency is set when the decoding processing volume is large, and a low drive frequency is set when the decoding processing volume is small. For example, if the processing volume required to decode video data conforming to the MPEG4-AVC standard is larger than the processing volume required to decode video data generated by the video encoding method or device described in each of the above-described embodiments, the drive frequency may be set in the opposite way to the above-described setting method.

[0225] Furthermore, the method of setting the drive frequency is not limited to a configuration that lowers the drive frequency. For example, if the identification information indicates that the video data is generated by the video encoding method or device described in each of the above embodiments, the voltage applied to the LSIex500 or a device including the LSIex500 can be set high. If the identification information indicates that the video data complies with conventional standards such as MPEG-2, MPEG4-AVC, or VC-1, the voltage applied to the LSIex500 or a device including the LSIex500 can be set low. As another example, if the identification information indicates that the video data is generated by the video encoding method or device described in each of the above embodiments, the drive of the CPUex502 can be suspended without stopping. If the identification information indicates that the video data complies with conventional standards such as MPEG-2, MPEG4-AVC, or VC-1, the drive of the CPUex502 can be suspended temporarily because there is sufficient processing capacity. Even if the identification information indicates that the video data is generated by the video encoding method or device described in each of the above embodiments, the drive of the CPUex502 can be suspended temporarily if there is sufficient processing capacity. In this case, it is conceivable to set the stop time shorter than when the video data indicates that it is video data that conforms to conventional standards such as MPEG-2, MPEG4-AVC, and VC-1.

[0226] In this way, by switching the drive frequency depending on the standard to which the video data conforms, it is possible to achieve power savings. Furthermore, if the LSIex500 or a device including the LSIex500 is driven by a battery, the power savings can also extend the battery life.

[0227] (Embodiment 6) The above-mentioned devices and systems, such as televisions and mobile phones, may receive multiple inputs of video data conforming to different standards. To ensure that the signal processing unit ex507 of the LSIex500 can decode such inputs, the signal processing unit ex507 must support multiple standards. However, using separate signal processing units ex507 for each standard increases the circuit size of the LSIex500 and increases costs.

[0228] To solve this problem, a configuration is adopted in which a decoding processing unit for executing the video decoding method described in each of the above embodiments is partially shared with a decoding processing unit conforming to conventional standards such as MPEG-2, MPEG4-AVC, and VC-1. An example of this configuration is shown in ex900 of FIG. 34A . For example, the video decoding methods described in each of the above embodiments and video decoding methods conforming to the MPEG4-AVC standard share some of the processing content, such as entropy coding, inverse quantization, deblocking filtering, and motion compensation. A possible configuration is to share a decoding processing unit ex902 conforming to the MPEG4-AVC standard for the common processing content, and use a dedicated decoding processing unit ex901 for other processing content that is unique to one aspect of the present invention and does not conform to the MPEG4-AVC standard. Regarding the sharing of the decoding processing unit, a configuration may be adopted in which a decoding processing unit for executing the video decoding method described in each of the above embodiments is shared for the common processing content, and a dedicated decoding processing unit is used for processing content specific to the MPEG4-AVC standard.

[0229] Another example of partially sharing processing is shown in ex1000 in Figure 34B. In this example, a dedicated decoding processing unit ex1001 corresponding to processing content specific to one aspect of the present invention, a dedicated decoding processing unit ex1002 corresponding to processing content specific to another conventional standard, and a shared decoding processing unit ex1003 corresponding to processing content common to the video decoding method according to one aspect of the present invention and the video decoding method of another conventional standard are used. Here, the dedicated decoding processing units ex1001 and ex1002 are not necessarily specialized for processing content specific to one aspect of the present invention or another conventional standard, and may be capable of performing other general-purpose processing. The configuration of this embodiment can also be implemented using an LSI ex500.

[0230] In this way, by sharing a decoding processing unit for processing content that is common between a video decoding method according to one embodiment of the present invention and a video decoding method of a conventional standard, it is possible to reduce the circuit size of the LSI and reduce costs.

[0231] While the image encoding method and image decoding method according to one or more aspects have been described based on the embodiments, the present invention is not limited to these embodiments. As long as they do not deviate from the spirit of the present invention, various modifications conceivable by those skilled in the art to the present embodiments and forms constructed by combining components of different embodiments may also be included within the scope of one or more aspects. [Industrial Applicability]

[0232] The present invention can be applied to, for example, image encoding devices and image decoding devices, and more specifically, can be used in information display devices and imaging devices such as televisions, digital video recorders, car navigation systems, mobile phones, digital still cameras, and digital video cameras. [Explanation of symbols]

[0233] 10,100 Image encoding device 20,200 Image decoding device 21 Signal interval setting section 22 DTS Acquisition Department 23 Judgment section 24 Decoding unit 101 Decision Section 102 Encoding section 103 Generation part 201 Acquisition Department 202 Decoding unit

Claims

1. A receiving method for receiving and decoding an encoded stream including moving images, the moving images being pictures of a group of pictures following a randomly accessible I-picture in encoding order, the pictures belonging to any one of a plurality of layers being encoded without reference to other pictures belonging to a layer higher than the layer to which the picture belongs, the receiving method comprising: receiving the coded stream including a plurality of coded pictures and decoding times set for each of the plurality of pictures; acquiring, from the coded stream, decoding times of the respective pictures included in the coded stream, and decoding the respective pictures or the respective lower layer pictures included in the coded stream according to the acquired decoding times; The decoding times of the plurality of pictures included in the video are the decoding times of the plurality of lower-layer pictures, which are some of the plurality of pictures included in the video and do not belong to the highest hierarchical layer of the plurality of hierarchical layers, are set to be equally spaced, and the timings at which the plurality of lower-layer pictures are decoded are set to be the same between a case in which the plurality of pictures included in the coded video are decoded and a case in which only the plurality of lower-layer pictures of the plurality of pictures are decoded; the plurality of pictures not belonging to the lowest hierarchical layer include bidirectional reference predicted pictures; the decoding times of all pictures belonging to the group of pictures are earlier than the decoding times of all pictures following in display order from a subsequent I-picture that is later than the group of pictures in display order, In the setting of the decoding time, When a frame rate at which all pictures included in the coded video are decoded and displayed is f, the decode times of the plurality of lower layer pictures included in all the pictures are set so that the decode times are spaced apart by a time indicated by n times the reciprocal of f (n is an integer of 2 or more). Receiving method.

2. 1. A receiving device that receives and decodes an encoded stream including moving images, the moving images being pictures of a group of pictures following a randomly accessible I-picture in encoding order, the moving images being encoded for each picture belonging to one of a plurality of layers without reference to other pictures belonging to a layer higher than the layer to which the picture belongs, the receiving device comprising: a receiving unit that receives the coded stream including a plurality of coded pictures and a decoding time set for each of the plurality of pictures; a decoding unit that acquires, from the coded stream, a decoding time of each of the plurality of pictures included in the coded stream, and decodes each of the plurality of pictures or a plurality of lower layer pictures included in the coded stream according to the acquired decoding time; The decoding times of the plurality of pictures included in the video are the decoding times of the plurality of lower-layer pictures, which are some of the plurality of pictures included in the video and do not belong to the highest hierarchical layer of the plurality of hierarchical layers, are set to be equally spaced, and the timings at which the plurality of lower-layer pictures are decoded are set to be the same between a case in which the plurality of pictures included in the coded video are decoded and a case in which only the plurality of lower-layer pictures of the plurality of pictures are decoded; the plurality of pictures not belonging to the lowest hierarchical layer include bidirectional reference predicted pictures; the decoding times of all pictures belonging to the group of pictures are earlier than the decoding times of all pictures following in display order from a subsequent I-picture that is later than the group of pictures in display order, In the setting of the decoding time, When the frame rate at which all pictures included in the coded video are decoded and displayed is f, the decoding times of the plurality of lower layer pictures included in all the pictures are spaced apart by a time indicated by n times the reciprocal of f (n is an integer of 2 or more). the decoding times of all the pictures are set in Receiving device.

Images