Reception method

Abstract

To provide an encoded data receiving method capable of easily selecting hierarchically encoded data.SOLUTION: The receiving method includes a receiving step of receiving a packet including the basic layer level and a packet including the extended layer level in an encoded stream, and receiving a packet including information, and a decoding step of decoding encoded data using the received encoded stream and the received packet including the information. The packet including the information includes information indicating the asset ID of the basic layer level used for decoding the extended layer level and information for specifying the number of layers of the basic layer level.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a receiving method for receiving coded data that has been coded in a hierarchical manner. [Background technology]

[0002] Conventionally, a technique for transmitting coded data using a predetermined multiplexing method has been known. The coded data is generated by encoding content including video data and audio data based on a video coding standard such as High Efficiency Video Coding (HEVC).

[0003] The predetermined multiplexing method is, for example, MPEG-2 TS (Moving Picture Examples include MPEG Media Transport (MMT) and MPEG Experts Group-2 Transport Stream (MPEG Media Transport) (see Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Information technology - High efficiency coding and media delivery in heterogeneous environment - Part1:MPEG media transport(MMT), ISO / IEC DIS 23008-1 Summary of the Invention [Problem to be solved by the invention]

[0005] HEVC allows for hierarchical encoding, and the receiving side can select the frame rate of the video by selecting the hierarchically encoded data according to the layer level.

[0006] The present invention provides a receiving method for receiving coded data that allows easy selection of coded data that has been coded in a hierarchical manner. [Means for solving the problem]

[0007] A receiving method according to one embodiment of the present invention is a method for receiving encoded data in which video is coded in a temporal hierarchical manner and the coding order and display order differ, wherein the coded stream of the coded data includes packets in which the coded data is packetized into different packets depending on the hierarchical levels of the coded data, that is, a base layer level and an enhanced layer level, the packets being associated with different asset IDs depending at least on the base layer level and the enhanced layer level, and packets including information indicating the asset ID of the base layer level used for decoding the enhanced layer level and information for identifying the number of layers of the base layer level, wherein different packet IDs are assigned to the packets including the base layer level of the coded data, the packets including the enhanced layer level of the coded data, and the packets including the information, and the receiving method includes a receiving step of receiving the packets including the base layer level and the packets including the enhanced layer level in the coded stream and receiving the packets including the information, and a decoding step of decoding the coded data using the received coded stream and the received packets including the information.

[0008] These comprehensive or specific aspects may be realized as a system, an apparatus, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, etc. Furthermore, these comprehensive or specific aspects may be realized as any combination of a system, an apparatus, an integrated circuit, a computer program, and a recording medium. [Effects of the Invention]

[0009] According to the present invention, hierarchically encoded data can be easily selected. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram for explaining coded data that has been coded in a time scalable manner. [Figure 2] FIG. 2 is a first diagram illustrating the data structure of an encoded stream in MMT. [Figure 3] FIG. 3 is a second diagram for explaining the data structure of an encoded stream in MMT. [Figure 4] FIG. 4 is a diagram showing the correspondence between packet IDs and data (assets) in a coded stream according to the first embodiment. [Figure 5] FIG. 5 is a block diagram showing a configuration of a transmission device according to the first embodiment. [Figure 6] FIG. 6 is a flowchart of a transmission method according to the first embodiment. [Figure 7] FIG. 7 is a block diagram showing a configuration of a receiving device according to the first embodiment. [Figure 8] FIG. 8 is a flowchart of the receiving method according to the first embodiment. [Figure 9] FIG. 9 is a diagram conceptually showing a receiving method according to the first embodiment. [Figure 10] FIG. 10 is a block diagram showing a configuration of a receiving device according to the second embodiment. [Figure 11] FIG. 11 is a first diagram for explaining an outline of a transmission / reception method according to the second embodiment. [Figure 12] FIG. 12 is a second diagram for explaining the outline of the transmission and reception method according to the second embodiment. [Figure 13] FIG. 13 is a first diagram for explaining an example of packetizing encoded data in units of fragmented MFUs. [Figure 14] FIG. 14 is a second diagram for explaining an example of packetizing encoded data in units of fragmented MFUs. [Figure 15] FIG. 15 is a third diagram for explaining an example of packetizing encoded data in units of fragmented MFUs. [Figure 16] FIG. 16 is a diagram showing an example in which coded data is arranged in MP4 data in the same order. [Figure 17] FIG. 17 is a diagram showing a first example of arranging coded data in MP4 data for each hierarchical level. [Figure 18] FIG. 18 is a diagram illustrating a second example in which coded data is arranged in MP4 data for each hierarchical level. DETAILED DESCRIPTION OF THE INVENTION

[0011] (Findings that form the basis of the present invention) The video coding method HEVC (High Efficiency Video Coding) supports time scalability, and for example, it is possible to play back a 120 fps video as a 60 fps video. Fig. 1 is a diagram for explaining coded data coded in a time scalable manner.

[0012] Temporal IDs are assigned to each layer of temporally scalable encoded data. In Figure 1, for example, if pictures with a Temporal ID of 0 (I0, P4) and a picture with a Temporal ID of 1 (B2) are displayed, the video will be displayed at 60 fps, and if pictures with a Temporal ID of 2 (B1, B3) are additionally displayed, the video will be displayed at 120 fps.

[0013] In the example of FIG. 1, coded data with a Temporal ID of 0 or 1 is the base layer (base layer level), and data with a Temporal ID of 2 is the enhancement layer (enhancement layer level).

[0014] A picture in the base layer can be coded independently or can be decoded using other pictures in the base layer. In contrast, a picture in the enhancement layer cannot be decoded independently and can only be decoded after the reference picture at the start point of the arrow in Figure 1 is decoded. Therefore, a picture in the base layer that serves as a reference picture for a picture in the enhancement layer must be decoded before the picture in the enhancement layer.

[0015] Note that the decoding order is different from the image display order. In the example of Fig. 1, the image display order is (I0, B1, B2, B3, P4), while the decoding order is (I0, P4, B2, B1, B3). The image display order is determined based on a PTS (Presentation Time Stamp) assigned to each picture, and the decoding order is determined based on a DTS (Decode Time Stamp) assigned to each picture.

[0016] In not only temporal scalable coding but also spatial scalable coding and SNR scalable coding, when pictures are divided into a base layer and an enhancement layer, pictures belonging to the enhancement layer cannot be decoded independently and must be decoded together with pictures belonging to the base layer.

[0017] It is desirable that scalably coded (hierarchically coded) coded data can be easily selected on the receiving side (decoding side).

[0018] Therefore, a transmission method according to one aspect of the present invention is a method for transmitting coded data in which video is hierarchically coded, and includes a generation step of generating a coded stream in which the coded data is packetized into packets, the packets being assigned packet IDs that differ depending on at least the hierarchical level of the coded data, and information indicating the correspondence between the packet IDs and the hierarchical level, and a transmission step of transmitting the generated coded stream and the generated information indicating the correspondence.

[0019] This allows coded data to be sorted by layer level by filtering the packet ID, which means that coded data can be easily sorted on the receiving side.

[0020] In addition, the layer levels may include a base layer level and an enhanced layer level, and the encoded data of the base layer level may be decodable independently or by referring to the decoded data of other encoded data of the base layer level, and the encoded data of the enhanced layer level may be decodable by referring to the decoded data of the encoded data of the base layer level.

[0021] Furthermore, the generating step may generate a first encoded stream, which is an encoded stream that includes the packets into which the encoded data at the base layer level has been packetized, and does not include the packets into which the encoded data at the enhanced layer level has been packetized, and a second encoded stream, which is an encoded stream that includes the packets into which the encoded data at the enhanced layer level has been packetized, and does not include the packets into which the encoded data at the base layer level has been packetized, and the transmitting step may transmit the first encoded stream using a first transmission path, and transmit the second encoded stream using a second transmission path different from the first transmission path.

[0022] In the generating step, the first encoded stream and the second encoded stream may be generated in accordance with different multiplexing methods.

[0023] In addition, in the generating step, one of the first encoded stream and the second encoded stream may be generated in accordance with MPEG2-TS (Moving Picture Experts Group-2 Transport Stream), and the other of the first encoded stream and the second encoded stream may be generated in accordance with MMT (MPEG Media Transport).

[0024] Furthermore, one of the first transmission path and the second transmission path may be a transmission path for broadcasting, and the other of the first transmission path and the second transmission path may be a transmission path for communication.

[0025] In addition, the generating step may generate the coded stream including information indicating the correspondence relationship, and the transmitting step may transmit the coded stream including the information indicating the correspondence relationship.

[0026] Furthermore, the information indicating the correspondence may include either information indicating that the encoded stream can be decoded independently, or information indicating other encoded streams that are necessary for decoding the encoded stream.

[0027] These comprehensive or specific aspects may be realized as a system, an apparatus, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, etc. Furthermore, these comprehensive or specific aspects may be realized as any combination of a system, an apparatus, an integrated circuit, a computer program, and a recording medium.

[0028] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0029] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present invention. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concepts are described as optional components.

[0030] (Embodiment 1) [How to send] Hereinafter, a transmission method (transmission device) according to embodiment 1 will be described with reference to the drawings. In embodiment 1, a transmission method for transmitting encoded data in accordance with MMT will be described as an example.

[0031] First, the data structure of an encoded stream in MMT will be described. Figures 2 and 3 are diagrams for explaining the data structure of an encoded stream in MMT.

[0032] As shown in Fig. 2, the coded data is made up of a plurality of AUs (Access Units). The coded data is, for example, AV data coded based on a video coding standard such as HEVC. Specifically, the coded data includes video data, audio data, and associated metadata, still images, files, etc. When the coded data is video data, one AU is a unit equivalent to one picture (one frame).

[0033] In MMT, coded data is converted into MP4 data (with an MP4 header) in units of GOPs (Groups of Picture) according to the MP4 file format. The MP4 header included in the MP4 data describes the relative values ​​of the AU presentation time (the PTS mentioned above) and the decoding time (the DTS mentioned above). The MP4 header also describes the sequence number of the MP4 data. Note that MP4 data (MP4 file) is an example of an MPU (Media Processing Unit), which is a data unit defined in the MMT standard.

[0034] In the following, an example will be described in which MP4 data (files) are transmitted, but the transmitted data does not have to be MP4 data. For example, data in a file format other than an MP4 file may be used, and as long as the encoded data and information required for decoding the encoded data (for example, information included in the MP4 header) are transmitted, the receiving side can decode the encoded data.

[0035] 3, an encoded stream 10 in MMT includes program information 11, time offset information 12, and a plurality of MMT packets 13. In other words, the encoded stream 10 is a packet sequence of the MMT packets 13.

[0036] The coded stream 10 (MMT stream) is one of one or more streams that make up one MMT package. An MMT package corresponds to, for example, one broadcast program content.

[0037] The program information 11 includes information indicating that the coded stream 10 is a scalable coded stream (a stream including both a base layer and an enhancement layer), as well as information on the type of scalable coding and the number of layer levels (number of layers). Here, the type of scalable coding refers to temporal scalability, spatial scalability, SNR scalability, etc., and the number of layer levels refers to the number of layers such as the base layer and enhancement layers. Note that the program information 11 does not need to include all of the above information, and it is sufficient to include at least one of the information.

[0038] The program information 11 also includes, for example, information indicating the correspondence between a plurality of assets and packet IDs. An asset is a data entity including data with the same transport characteristics, such as video data or audio data. The program information 11 may also include a descriptor indicating the hierarchical relationship of each packet ID (or asset).

[0039] Specifically, the program information 11 is CI (Composition Information) and MPT (MMT Package Table) in MMT. Note that the program information 11 is PMT (Program Map Table) in MPEG2-TS, MPD (Media Presentation Description) in MPEG-DASH, etc.

[0040] The time offset information 12 is time information for determining the PTS or DTS of an AU. Specifically, the time offset information 12 is, for example, the absolute PTS or DTS of the first AU belonging to the base layer.

[0041] The MMT packet 13 is data in which MP4 data has been packetized. In the first embodiment, one MMT packet 13 includes one MP4 data (MPU). As shown in Fig. 3, the MMT packet 13 includes a header 13a (MTT packet header; in the case of MPEG2-TS, a TS packet header) and a payload 13b.

[0042] The payload 13b stores MP4 data. Note that the payload 13b may store divided MP4 data.

[0043] The header 13a is additional information related to the payload 13b, for example, the header 13a includes a packet ID.

[0044] The packet ID is an identification number that indicates the asset of the data included in the MMT packet 13 (payload 13b). The packet ID is an identification number that is unique to each asset that makes up the MMT package.

[0045] The coded stream 10 is characterized in that the video data of the base layer and the video data of the enhancement layer are treated as different assets. That is, different packet IDs are assigned to the MMT packets 13 of the coded stream 10 depending on the layer level of the coded data stored therein. Fig. 4 is a diagram showing the correspondence between packet IDs and data (assets) in the coded stream 10. Note that Fig. 4 is an example of the correspondence.

[0046] As shown in FIG. 4, in the first embodiment, a packet ID of "1" is assigned to MMT packets 13 obtained by packetizing video data of the base layer (encoded data at the base layer level). That is, the packet ID of "1" is described in header 13a. And a packet ID of "2" is assigned to MMT packets 13 obtained by packetizing video data of the enhancement layer (encoded data at the enhancement layer level). That is, the packet ID of "2" is described in header 13a.

[0047] Similarly, the MMT packet 13 into which audio data has been packetized is assigned a packet ID of "3," the MMT packet 13 into which time offset information 12 has been packetized is assigned a packet ID of "4," and the MMT packet 13 into which program information 11 has been packetized is assigned a packet ID of "5."

[0048] 4 is described in the program information 11 of the coded stream 10. Note that the correspondence includes information indicating that the MMT packet 13 assigned with the packet ID "1" and the MMT packet 13 assigned with the packet ID "2" are a pair and are used for scalability.

[0049] A description will now be given of a transmission method (transmission device) for transmitting the coded stream 10 according to the first embodiment as described above. Fig. 5 is a block diagram showing the configuration of the transmission device according to the first embodiment. Fig. 6 is a flowchart of the transmission method according to the first embodiment.

[0050] 5, the transmitting device 15 includes an encoding unit 16, a multiplexing unit 17, and a transmitting unit 18. The components of the transmitting device 15 are specifically realized by a microcomputer, a processor, a dedicated circuit, or the like.

[0051] In the method for transmitting the coded stream 10 according to the first embodiment, first, a coded stream 10 is generated that includes an MMT packet 13 to which a packet ID is assigned and information indicating the correspondence between the packet ID and the layer level (S11).

[0052] Specifically, when packetizing the coded data output from the coding unit 16, the multiplexing unit 17 determines (selects) a packet ID according to the layer level of the coded data. Subsequently, the multiplexing unit 17 generates an MMT packet 13 including the determined packet ID. Meanwhile, the multiplexing unit 17 generates information indicating the above-mentioned correspondence relationship. Then, the multiplexing unit 17 generates a coded stream 10 including the generated MMT packet 13 and the generated information indicating the correspondence relationship.

[0053] The generated coded stream 10 is transmitted by the transmitter 18 using a transmission path (S12).

[0054] In this way, if an encoded stream 10 including MMT packets 13 to which different packet IDs are assigned depending on the hierarchical level of the encoded data is transmitted, the encoded data can be easily selected on the receiving side using a conventional packet filter mechanism.

[0055] Note that the information indicating the correspondence between packet IDs and layer levels may not be included in the coded stream 10 and may be transmitted separately from the coded stream 10. Furthermore, if the correspondence between packet IDs and layer levels is already known on the receiving side, the information indicating the correspondence between packet IDs and layer levels does not need to be transmitted.

[0056] For example, the information indicating the correspondence may be included in program information that is repeatedly inserted into a continuous signal such as a broadcast signal, or may be obtained from a communication server before decoding begins.

[0057] [Receiving method] Hereinafter, a receiving method (receiving device) according to the first embodiment will be described with reference to the drawings. Fig. 7 is a block diagram showing the configuration of the receiving device according to the first embodiment. Fig. 8 is a flowchart of the receiving method according to the first embodiment.

[0058] In the following description, the base layer may be referred to as layer level A, and the enhancement layer may be referred to as layer level B.

[0059] 7, receiving device 20 includes packet filter 21, program information analysis unit 22, control unit 23, packet buffer 24, decoding unit 25, and presentation unit 26. Note that, among the components of receiving device 20, the components other than packet buffer 24 and presentation unit 26 are specifically realized by a microcomputer, a processor, a dedicated circuit, or the like. Packet buffer 24 is, for example, a storage device such as a semiconductor memory. Presentation unit 26 is, for example, a display device such as a liquid crystal panel.

[0060] 8, first, the packet filter 21 separates the MMT packets 13 included in the coded stream 10 (S21), and outputs the program information 11 to the program information analysis unit 22. Here, the packet filter 21 recognizes in advance the packet IDs of the MMT packets 13 including the program information 11 (or can acquire the packet IDs of the MMT packets 13 including the program information 11 from other control information), and therefore can separate the MMT packets 13 including the program information 11 from the coded stream 10.

[0061] Next, the program information analysis unit 22 analyzes (S22) the program information 11. The program information 11 includes the correspondence between packet IDs and assets, as described above.

[0062] Meanwhile, the control unit 23 determines at which layer level the encoded data (MMT packets 13) should be extracted (S23). This determination may be made based on a user input received by an input receiving unit (not shown in FIG. 7), or may be made according to the specifications of the presentation unit 26 (for example, the frame rate supported by the presentation unit 26, etc.).

[0063] Then, the packet filter 21 extracts (filters) the encoded data (MMT packets 13) of the determined layer level under the control of the control unit 23 (S24). The control unit 23 recognizes the packet ID for each layer level through the analysis by the program information analysis unit 22, and therefore can cause the packet filter 21 to extract the encoded data of the determined layer level.

[0064] Next, the packet buffer 24 buffers the coded data extracted by the packet filter 21 and outputs it to the decoding unit 25 at the timing of the DTS (S25). The timing of the DTS is calculated based on the program information 11, the time offset information 12, and, for example, the time information transmitted in the MP4 header. Note that, in cases where the same DTS is assigned to coded data of the base layer and coded data of the enhancement layer due to spatial scalability or the like, the decoding order may be rearranged so that the coded data of the base layer is decoded before the coded data of the enhancement layer.

[0065] The encoded data buffered by the packet buffer 24 is decoded by the decoding unit 25 and presented (displayed) by the presentation unit 26 at the timing of the PTS (S26). The timing of the PTS is calculated based on the program information 11, the time offset information 12, and the time information in the MP4 header.

[0066] Such a receiving method will be further explained with reference to Fig. 9. Fig. 9 is a diagram conceptually showing the receiving method according to the first embodiment.

[0067] 9, for example, when it is determined that the layer level is layer level A (only encoded data of the base layer is to be extracted), the packet filter 21 extracts all MMT packets 13 assigned with packet ID "1" and does not extract MMT packets 13 assigned with packet ID "2." As a result, video with a low frame rate (for example, 60 fps) is displayed by the presentation unit 26.

[0068] Furthermore, for example, when it is determined that the layer level is layer level A+B (the extraction target is both the coded data of the base layer and the coded data of the enhancement layer), the packet filter 21 extracts all MMT packets 13 assigned with packet ID "1" or "2." As a result, video with a high frame rate (for example, 120 fps) is displayed by the presentation unit 26.

[0069] In this way, the receiving device 20 can easily use the packet filter 21 to select coded data at the base layer level and coded data at the enhancement layer level.

[0070] (Embodiment 2) [Sending / receiving method] A transmission method and a reception method (reception device) according to the second embodiment will be described below with reference to the drawings. Fig. 10 is a block diagram showing the configuration of a reception device according to the second embodiment. Figs. 11 and 12 are diagrams for explaining an outline of the transmission and reception method according to the second embodiment. Note that the block diagram of the transmission device and the flowcharts of the reception method and transmission method are almost the same as those explained in the first embodiment except for the use of a layer level ID, and therefore explanations thereof will be omitted.

[0071] As shown in FIG. 10, a receiving device 20a according to the second embodiment differs from the receiving device 20 in that a hierarchical filter 27 is provided.

[0072] As shown in (1) of Figure 11, in the coded stream transmitted in the transmission method according to embodiment 2, the same packet ID (packet ID: 1 in Figure 10) is assigned to each of the MMT packets 13 of the base layer and the MMT packets 13 of the enhancement layer.

[0073] Then, a layer level ID, which is an identifier related to the layer level, is assigned separately from the packet ID to each MMT packet 13 that has been assigned the same packet ID. In the example of Fig. 10, for example, the layer level ID is assigned as A for the base layer and B for the enhancement layer. The packet ID and layer level ID are written, for example, in the header 13a (MTT packet header) corresponding to the MTT packet.

[0074] The hierarchical level ID may be defined as a new identifier or may be realized using private user data or other identifiers.

[0075] When the TS packet header is used, the layer level ID may be defined as a new identifier or may be realized using an existing identifier. For example, a function equivalent to the layer level ID can be realized using either or both of the transport priority identifier and the elementary stream priority identifier.

[0076] As shown in (2) of Fig. 11 and (2) of Fig. 12, the transmitted coded stream is packet-filtered by the packet filter 21 of the receiving device 20a. That is, the transmitted coded stream is filtered based on the packet ID attached to the packet header.

[0077] As shown in (3) of Figure 11 and (3) of Figure 12, the packet-filtered MMT packet 13 is further layer-level filtered by the layer filter 27 based on the layer level ID. The filtered encoded data is temporarily buffered in the packet buffer 24, and then decoded by the decoder 25 at the timing of the DTS. Then, as shown in (4) of Figure 11 and (4) of Figure 12, the decoded data is presented by the presenter 26 at the timing of the PTS.

[0078] Here, to obtain video in which only the base layer is decoded (for example, video at 60 fps), it is sufficient to decode only the MMT packets 13 (encoded data) of the lower layer with layer level ID "A." Therefore, in layer level filtering, only the MMT packets 13 with layer level ID "A" are extracted.

[0079] On the other hand, to obtain video in which the base layer and enhancement layer have been decoded (e.g., video at 120 fps), it is necessary to decode both the MMT packets 13 with layer level ID "A" in the lower layer and the MMT packets 13 with layer level ID "B" in the higher layer. Therefore, in layer level filtering, both the MMT packets 13 with layer level ID "A" and the MMT packets 13 with layer level ID "B" are extracted.

[0080] In this way, the receiving method (receiving device 20a) of embodiment 2 has two sequences: one that filters only MMT packets 13 with a layer level ID of "A" and decodes and presents video of only the base layer, and one that filters MMT packets 13 with a layer level ID of "A" or "B" and decodes and presents video of the base layer + enhancement layer.

[0081] In addition, which packet ID or layer level ID to filter in packet filtering and layer level filtering is determined taking into consideration the type of scalable encoding described in program information 11, information on the number of layers, and which layer of encoded data receiving device 20a will decode and display.

[0082] Such a determination is made by the receiving device 20a, for example, depending on the processing capabilities of the receiving device 20a. Note that the transmitting device may further transmit, as signaling information, information relating to the capabilities of the receiving device 20a required to decode and display the content. In such a case, the receiving device 20a makes the determination by comparing the signaling information with the capabilities of the receiving device 20a.

[0083] It is also possible to provide a filter unit that combines the packet filter 21 and the layer level filter 27, and the filter unit may perform filtering collectively based on the packet ID and layer level ID.

[0084] As described above, according to the transmission / reception method of the second embodiment, coded data can be sorted for each layer level by filtering the layer level ID. That is, coded data can be easily sorted on the receiving side. Furthermore, since packet IDs and layer level IDs are assigned separately, coded data of the base layer and coded data of the enhancement layer can be treated as the same stream in packet filtering.

[0085] Furthermore, by assigning a layer level ID to the packet, it becomes possible to extract the desired hierarchically encoded data by simply filtering, eliminating the need for reassembly.

[0086] Furthermore, since the desired hierarchically encoded data can be extracted using a hierarchical filter, a receiving device that supports decoding of only the base layer can reduce the memory required to buffer data packets of the enhancement layer.

[0087] [Example 1] In MMT, an MPU made up of MP4 data is fragmented into MFUs (Media Fragment Units), and a header 13a is added in units of MFUs to generate an MMT packet 13. Here, an MFU can be fragmented down to the minimum NAL unit.

[0088] An example of packetizing encoded data in fragmented MFU units will be described below as specific example 1 of embodiment 2. Figures 13, 14, and 15 are diagrams for explaining an example of packetizing encoded data in fragmented MFU units. Note that in Figures 13, 14, and 15, white AUs indicate AUs in the base layer, and hatched AUs indicate AUs in the enhancement layer (the same applies to Figures 16 to 18 below).

[0089] When fragmented MFUs are packetized, the same packet ID is assigned to the packet ID of the MMT packet header, and a layer level ID is also assigned to the MMT packet header. Furthermore, among 'ftyp', 'moov', and 'moof', an ID indicating that it is common is assigned to the MMT packet header of common data (common information) that is unrelated to the layer level. In Figure 13, layer level A is the base layer, layer level B is the enhancement layer, and layer level Z is the common information. However, the layer levels of the base layer and the common information may be the same.

[0090] In such a configuration, coded data of layer level B is treated as one asset. After filtering based on the packet ID is performed, the receiving device 20a is ready to perform filtering at the layer level.

[0091] When it is desired to decode both the coded data of the base layer and the coded data of the enhancement layer, the receiving device 20a performs filtering based on the packet ID, and then extracts all layer level IDs by filtering based on the layer level ID. That is, the layer level filtering extracts all of layer level A: base layer, layer level B: enhancement layer, and layer level Z: common information. The extracted data is as shown in FIG. 14.

[0092] When it is desired to decode only the coded data of the base layer, the receiving device 20a performs filtering based on the packet ID, and then filters and extracts the layer level A: base layer and the layer level Z: common information. The extracted data is as shown in (a) of FIG.

[0093] At this time, since the AUs of the enhancement layer are removed, the decoding unit 25 acquires the AUs of the base layer packed together as shown in (b) of Fig. 15. However, the time offset information and data size of the samples (AUs) described in 'moof' are information generated with the enhancement layer included. This causes a mismatch between the information described in the header and the actual data.

[0094] Therefore, it is necessary to store information necessary to reconstruct the MP4 data, such as separately storing the size and offset information of the removed AUs.

[0095] Therefore, when the decoding unit 25 acquires the AU of the base layer, or the DTS and PTS, the decoding process may be performed taking into account that the AU of the enhancement layer (removed by filtering) does not exist in 'mdat' in header information such as 'moof'.

[0096] For example, the offset information of the access unit (sample) in 'moof' is set assuming that an AU of the enhancement layer exists. Therefore, when the decoding unit 25 acquires only the base layer, it subtracts the size of the removed AU from the offset information. The data obtained by the subtraction is schematically shown in (c) of Figure 15.

[0097] Similarly, the DTS and PTS are calculated based on the sample_duration (the difference in DTS between consecutive access units) and sample_composition_time_offset (the difference between the DTS and PTS in an access unit) corresponding to the AU of the removed enhancement layer.

[0098] Instead of the subtraction described above, header information for decoding data from which only the base layer is extracted (header information for AUs containing only the base layer) may be described in advance in the MP4 header. Furthermore, the MP4 header may describe information for distinguishing between header information for decoding only the base layer and header information for decoding both the base layer and the enhancement layer.

[0099] [Example 2] As a second specific example of the second embodiment, an example in which MPUs are not fragmented but are packetized in units of MPUs will be described below.

[0100] First, an example in which coded data is arranged in MP4 data in the same order as it is will be described. Fig. 16 is a diagram showing an example in which coded data is arranged in MP4 data in the same order as it is (when AUs at different layer levels are simultaneously multiplexed).

[0101] When coded data is placed directly into MP4 data, base layer AUs and enhancement layer AUs are mixed in one track in the 'mdat' box. In this case, a layer level ID is assigned to each AU. The layer level ID for each AU (sample) is written in 'moov' or 'moof'. Note that it is desirable to place corresponding base layer AUs and enhancement layer AUs in the same 'mdat' box. Note that when a transport header is added to the MP4 data and packetized, the same packet ID is assigned.

[0102] In the above configuration, filtering on a packet basis is not possible because layer level IDs cannot be added to packet headers. Filtering is possible by analyzing the MP4 data.

[0103] As another method, the base layer and the enhancement layer of the coded data are divided into tracks, and the correspondence between them is described in the header.

[0104] The data packetized in this way is packet filtered in the receiving device 20a, and then the layer level of the AU is determined when the MP4 data is analyzed, and the AUs of the desired layer are extracted and decoded.

[0105] Next, a first example of arranging coded data in MP4 data for each hierarchical level will be described. Fig. 17 is a diagram showing a first example of arranging coded data in MP4 data for each hierarchical level.

[0106] When encoding data is placed into MP4 data for each hierarchical level, the encoded data is separated for each hierarchical level and placed in the 'mdat' box of the fragment for each hierarchical level. In this case, the hierarchical level ID is written in 'moof'. A hierarchical level ID is assigned to the common header to indicate that the information is common regardless of the hierarchical level.

[0107] In addition, the same packet ID is assigned to the packet header, so filtering on a packet-by-packet basis is also not possible in this case.

[0108] The data packetized in this way is packet filtered in the receiving device 20a, and then the layer level of the fragment is determined when the MP4 data is analyzed, and the fragment of the desired layer is extracted and decoded.

[0109] Finally, a second example in which coded data is arranged in MP4 data for each hierarchical level will be described. Fig. 18 is a diagram for explaining a second example in which coded data is arranged in MP4 data for each hierarchical level.

[0110] In this example, the coded data is separated for each hierarchical level and placed in an 'mdat' box for each hierarchical level.

[0111] MP4 data in which the AUs of the base layer are stored and MP4 data in which the AUs of the enhancement layer are stored are generated.

[0112] Layer level IDs are written in either or both of the MP4 data header and the transport packet header. Here, the layer level ID indicates the hierarchical relationship between MP4 data or transport packets. The same packet ID is assigned to the packet header.

[0113] The data thus packetized is packet filtered in the receiving device 20a, and then packets of the desired layer are extracted and decoded based on the layer level ID in the packet header.

[0114] [Variations] In the above-mentioned second embodiment, it has been explained that the packet ID and the layer level ID are assigned separately, but the layer level ID may be assigned using some bits of the packet ID, or new bits may be assigned as an extended packet ID. Note that assigning a layer level ID using some bits of the packet ID is equivalent to assigning a different packet ID for each layer level, based on the rule that the same ID is assigned except for the bit indicating the layer level ID.

[0115] In addition, in the above-described second embodiment, when both the coded data of the base layer and the coded data of the enhancement layer are decoded, the data of the base layer and the enhancement layer are extracted by filtering using packet filtering or layer level filtering. However, the coded data may be reconstructed after being separated into the base layer and the enhancement layer by layer level filtering.

[0116] (Other embodiments) The present invention is not limited to the above-described embodiment.

[0117] In the above first and second embodiments, the coded stream multiplexed according to MMT has been described. However, the coded stream may be multiplexed according to other multiplexing methods, such as MPEG2-TS or RTP (Real Transport Protocol). Also, the MMT packets may be transmitted according to MPEG-TS2. In either case, the coded data can be easily selected on the receiving side.

[0118] In the above-mentioned first and second embodiments, coded data of a base layer and coded data of an enhancement layer are mixed in one coded stream. However, a first coded stream consisting of coded data of the base layer and a second coded stream consisting of coded data of an enhancement layer may be generated separately. Here, more specifically, the first coded stream is a coded stream that includes packets of coded data of the base layer but does not include packets of coded data of the enhancement layer. The second coded stream is a coded stream that includes packets of coded data of the enhancement layer but does not include packets of coded data of the base layer.

[0119] In this case, the first encoded stream and the second encoded stream may be generated according to different multiplexing methods. For example, one of the first encoded stream and the second encoded stream may be generated according to MPEG2-TS, and the other of the first encoded stream and the second encoded stream may be generated according to MMT.

[0120] When two coded streams are generated according to different multiplexing methods, packet IDs or layer level IDs are assigned according to the respective multiplexing methods. In this case, the correspondence between the packet IDs and layer levels in each coded stream or the correspondence between the layer level IDs and layer levels in each coded stream is described in the common program information. Specifically, the common program information is included in only one of the two coded streams, or the common program information is included in both of the two coded streams.

[0121] The receiving device performs packet filtering and layer level filtering based on the correspondence described in the program information, extracts and decodes the coded data at the desired layer level, and displays one video from the two coded streams.

[0122] Furthermore, the first encoded stream and the second encoded stream may be transmitted using (physically) different transmission paths. Specifically, for example, one of the first encoded stream and the second encoded stream may be transmitted using a broadcast transmission path, and the other of the first encoded stream and the second encoded stream may be transmitted using a communication transmission path. Such transmission is expected, for example, in cases of hierarchical transmission or bulk transmission across channels. In this case, the correspondence between each packet ID and layer level ID is described in common program information. Note that the program information does not need to be common. It is sufficient that the receiving device can recognize the correspondence between the packet ID and the layer level, or the correspondence between the layer level ID and the layer level.

[0123] In the above-described first and second embodiments, the coded stream is composed of two layers, namely, one base layer and one enhancement layer, but the coded stream may be composed of three or more layer levels by configuring multiple enhancement layers. In this case, a different packet ID (or a different layer level ID) is assigned to each of the three layer levels.

[0124] In the above-described first and second embodiments, the transmitting device 15 includes the encoding unit 16, but the transmitting device may not have an encoding function. In this case, a coding device having an encoding function is provided separately from the transmitting device 15.

[0125] Similarly, in the above-mentioned first and second embodiments, the receiving devices 20 and 20a include the decoding unit 25, but the receiving devices 20 and 20a may not have the decoding function. In this case, a decoding device having the decoding function is provided separately from the receiving devices 20 and 20a.

[0126] In the first and second embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may also be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0127] Furthermore, in the first embodiment, the processing performed by a specific processing unit may be performed by another processing unit. Furthermore, the order of multiple processes may be changed, or multiple processes may be performed in parallel.

[0128] Note that a comprehensive or specific aspect of the present invention may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, etc. Also, a comprehensive or specific aspect of the present invention may be realized as any combination of a system, a method, an integrated circuit, a computer program, or a recording medium.

[0129] It should be noted that the present invention is not limited to these embodiments or their modifications. As long as they do not deviate from the spirit of the present invention, various modifications conceivable by those skilled in the art to the present embodiments or their modifications, or forms constructed by combining components of different embodiments or their modifications, are also included within the scope of the present invention.

[0130] A first transmitting device according to one aspect of the present invention transmits temporally scalable coded data. Here, the temporally scalable coded data includes base layer data that can be decoded using data included in the layers, and enhancement layer data that cannot be decoded independently and must be decoded together with the base layer data. Here, the base layer data is data for decoding, for example, 60p video, and the enhancement layer data is data for decoding, for example, 120p video when used together with the base layer data.

[0131] Base layer data is transmitted as a first asset assigned with a first packet ID, and enhancement layer data is transmitted as a second asset assigned with a second packet ID. The packet ID is written in the header of a packet storing the data. The transmitting device multiplexes and transmits the base layer data, enhancement layer data, and program information. Here, the program information may include, for example, an identifier indicating the hierarchical relationship of each packet ID (or the asset corresponding to each packet ID). Here, the information indicating the hierarchical relationship includes, for example, information indicating that the data of the first packet ID (first asset) can be independently decodable, and information indicating that the data of the second packet ID (second asset) cannot be independently decodable and must be decoded using the data of the first packet ID (first asset).

[0132] The program information may also include at least one of information indicating that the stream that constitutes the program is a scalable coded stream (a stream that includes both a base layer and an enhancement layer), information indicating the type of scalable coding, information indicating the number of layers, and information related to the layer level.

[0133] Also, according to one aspect of the present invention, a first receiving device receives temporally scalable coded data. Here, the temporally scalable coded data includes base layer data that can be decoded using data included in the layers, and enhancement layer data that cannot be decoded independently and must be decoded together with the base layer data. Here, the base layer data is data for decoding, for example, 60p video, and the enhancement layer data is data for decoding, for example, 120p video by using it together with the base layer data.

[0134] Base layer data is transmitted as a first asset assigned with a first packet ID, and enhancement layer data is transmitted as a second asset assigned with a second packet ID. The packet ID is written in the header of a packet storing the data. The transmitting device multiplexes and transmits the base layer data, enhancement layer data, and program information. Here, the program information may include, for example, an identifier indicating the hierarchical relationship of each packet ID (or the asset corresponding to each packet ID). Here, the information indicating the hierarchical relationship includes, for example, information indicating that the data of the first packet ID (first asset) can be independently decodable, and information indicating that the data of the second packet ID (second asset) cannot be independently decodable and must be decoded using the data of the first packet ID (first asset).

[0135] The program information may also include at least one of information indicating that the stream that constitutes the program is a scalable coded stream (a stream that includes both a base layer and an enhancement layer), information indicating the type of scalable coding, information indicating the number of layers, and information related to the layer level.

[0136] According to the first transmitting device and the second receiving device described above, the receiving side can obtain the packet ID (asset) required to decode the data of the selected packet ID (asset) from the program information and perform filtering based on the packet ID (asset). For example, when playing a first asset assigned with a first packet ID, the data of the first packet ID (first asset) can be decoded independently, so the data of the first packet ID (first asset) is obtained by filtering. On the other hand, when playing a second asset assigned with a second packet ID, the data of the second packet ID (second asset) cannot be decoded independently and must be decoded using the data of the first packet ID (first asset), so the data of the first packet ID (first asset) and the data of the second packet ID (second asset) are obtained by filtering.

[0137] In the above configuration, the program information describes information indicating data of packet IDs (asset) other than the packet ID (asset) that is necessary for decoding the data of each packet ID (asset). With this configuration, even if scalably encoded data includes three layers, namely, layer A, layer B that is decoded together with layer A, and layer C that is decoded together with layer A, once the layer to be played is selected, the receiving device can identify the packet ID (asset) that transmits the data necessary for decoding without performing complex determination.

[0138] In particular, considering that in the future there may be cases where the hierarchical depth is three or more, or where data encoded using multiple types of scalable encoding is multiplexed and transmitted, the above configuration is useful as it can identify the packet ID (asset) that transmits the data required for decoding without making complex decisions. [Industrial Applicability]

[0139] The present invention can be applied to television broadcasting, video distribution, and the like, as a method for transmitting coded data that allows the receiving side to easily select hierarchically coded coded data. [Explanation of symbols]

[0140] 10 Encoded Stream 11 Program Information 12 Time offset information 13 MMT packets 13a Header 13b payload 15 Transmitting device 16 Encoding section 17 Multiplexer 18 Transmitter 20, 20a Receiving device 21 Packet Filter 22 Program Information Analysis Unit 23 Control Unit 24 packet buffers 25 Decoding section 26 Presentation section

Claims

[Claim 1] A method for receiving coded data in which video is coded in a time-hierarchical manner and the coding order differs from the display order, comprising: The coded stream of the coded data includes packets in which the coded data is packetized into packets that differ depending on the layer levels of the coded data, that is, a base layer level and an enhancement layer level, and the packets are associated with different asset IDs depending on at least the base layer level and the enhancement layer level, and packets that include information indicating the asset ID of the base layer level used for decoding the enhancement layer level and information for identifying the number of layers of the base layer level, different packet IDs are assigned to the packet including the base layer level of the encoded data, the packet including the extension layer level of the encoded data, and the packet including the information; The receiving method includes: a receiving step of receiving the packet including the base layer level and the packet including the enhancement layer level in the coded stream, and receiving the packet including the information; and decoding the encoded data using the received encoded stream and the packets containing the received information. Receiving method.

Images