Method for transmission

Abstract

To provide a method for transmission which can reduce delays in the transmission of MP4-format files.SOLUTION: The method for transmission includes: a packeting step S11 of packeting data forming a MP4-format file, which are (1) sample data being data of coded picture signals or voice signals, (2) first meta data for decoding the sample data, and (3) second meta data for decoding the sample data, the second meta data including data generable only after generation of the sample data; and a transmission step S12 for sequentially sending the packeted first meta data, the packeted sample data, and the packeted second meta data.SELECTED DRAWING: Figure 24

Description

Technical Field

[0001] The present invention relates to a transmission method.

Background Art

[0002] With the advancement of broadcast and communication services, the introduction of ultra-high-definition moving image contents such as 8K (7680×4320 pixels: hereinafter also referred to as 8K4K) and 4K (3840×2160 pixels: hereinafter also referred to as 4K2K) is being considered. A receiving device needs to decode and display the encoded data of the received ultra-high-definition moving image in real time. However, in particular, moving images with a resolution such as 8K have a large processing load during decoding, and it is difficult to decode such a moving image in real time with a single decoder. Therefore, a method of reducing the processing load per decoder and achieving real-time processing by parallelizing the decoding process using a plurality of decoders has been considered.

[0003] Also, the encoded data is multiplexed based on a multiplexing method such as MPEG-2 TS (Transport Stream) or MMT (MPEG Media Transport) and then transmitted. For example, Non-Patent Document 1 discloses a technique of transmitting encoded media data for each packet according to MMT.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, conventionally, when encapsulating encoded data into an MP4 format file, header information such as moov and moof is created when the samples stored in the MP4 file are complete. When transmitting a file in such an MP4 format, since the transmitting device usually waits for the generation of the header information to transmit the data, there is a problem that the delay from the completion of data encoding to the start of transmission is large.

[0006] The present invention provides a transmission method and a reception method capable of reducing the End-to-End delay from the completion of data encoding to decoding and presentation when streaming an MP4 format file.

Means for Solving the Problems

[0007] To achieve the above object, a transmission method according to an aspect of the present invention includes a packetizing step of packetizing each of (1) sample data which is data obtained by encoding a video signal or an audio signal, (2) first metadata for decoding the sample data, and (3) second metadata for decoding the sample data, which can be generated only after the generation of the sample data, which constitute an MP4 format file, and a transmission step of transmitting the packetized first metadata, the packetized sample data, and the packetized second metadata in this order.

[0008] Further, a receiving method according to one aspect of the present invention includes a receiving step of receiving packetized first metadata, packetized sample data, and packetized second metadata in this order, and a reconstructing step of reconstructing a file in MP4 format including the received first metadata, the received second metadata, and the received sample data. And a decoding step of decoding the sample data included in the reconstructed MP4 format file using the first metadata and the second metadata, wherein the second metadata includes data that can be generated only after generation of the sample data on the transmission side.

[0009] In addition, these general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, and may be realized by any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

Effects of the Invention

[0010] The present invention can reduce the end-to-end delay in the transmission of files in MP4 format.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41

Figure 42

Figure 43

Figure 44

Figure 45

Figure 46

Figure 47

Figure 48

Figure 49

DETAILED DESCRIPTION OF THE INVENTION

[0012] The transmission method according to one aspect of the present invention includes a packetization step of packetizing each of (1) sample data which is data obtained by encoding a video signal or an audio signal, (2) first metadata for decoding the sample data, and (3) second metadata for decoding the sample data, which is data that can be generated only after the generation of the sample data, which constitute a file in the MP4 format, and a transmission step of transmitting the packetized first metadata, the packetized sample data, and the packetized second metadata in this order.

[0013] Thereby, it is possible to reduce the End-to-End delay in transmitting a file in the MP4 format.

[0014] Also, the data that can be generated only after the generation of the sample data may be at least a part of the data other than the sample data among the data stored in mdat in the MP4 format.

[0015] Also, the first metadata may be MPU (Media Processing Unit) metadata, and the second metadata may be movie fragment metadata.

[0016] Also, in the packetization step, packetization may be performed in the MMT (MPEG Media Transport) method.

[0017] Further, the receiving method according to one aspect of the present invention includes a receiving step of receiving packetized first metadata, packetized sample data, and packetized second metadata in this order, a reconstructing step of reconstructing a file in MP4 format including the received first metadata, the received second metadata, and the received sample data, and a decoding step of decoding the sample data included in the reconstructed file in MP4 format using the first metadata and the second metadata. The second metadata includes data that can be generated only after the generation of the sample data on the transmission side.

[0018] A transmission device according to one aspect of the present invention includes a multiplexing unit that packetizes each of (1) sample data that is data obtained by encoding a video signal or an audio signal, which constitutes a file in MP4 format, (2) first metadata for decoding the sample data, and (3) second metadata for decoding the sample data, which includes data that can be generated only after the generation of the sample data, and a transmission unit that transmits the packetized first metadata, the packetized sample data, and the packetized second metadata in this order.

[0019] A receiving device according to one aspect of the present invention includes a receiving unit that receives packetized first metadata, packetized sample data, and packetized second metadata in this order, a reconstructing unit that reconstructs a file in MP4 format including the received first metadata, the received second metadata, and the received sample data, and a decoding unit that decodes the sample data included in the reconstructed file in MP4 format using the first metadata and the second metadata. The second metadata includes data that can be generated only after the generation of the sample data on the transmission side.

[0020] A transmission method according to an aspect of the present invention encodes a video signal to generate encoded data including a plurality of access units, stores the plurality of access units in packets in units of access units or units obtained by dividing an access unit to generate a packet group, transmits the generated packet group as data, generates first information indicating the presentation time of the first presented access unit among the plurality of access units, and second information used for calculating the decoding time of the plurality of access units, and transmits the generated first information and the second information as control information.

[0021] In a reception method according to an aspect of the present invention, the second information is information used for calculating the decoding time of a part of the plurality of access units.

[0022] A transmission apparatus according to an aspect of the present invention receives a packet group in which encoded data including a plurality of access units is packetized in units of access units or units obtained by dividing an access unit, receives control information including first information indicating the presentation time of the first presented access unit among the plurality of access units, and second information used for calculating the decoding time of the plurality of access units, and decodes the access units included in the received packet group based on the first information and the second information.

[0023] A reception apparatus according to an aspect of the present invention includes an encoding unit that encodes a video signal to generate encoded data including a plurality of access units, a packet generation unit that stores the plurality of access units in packets in units of access units or units obtained by dividing an access unit to generate a packet group, a first transmission unit that transmits the generated packet group as data, an information generation unit that generates first information indicating the presentation time of the first presented access unit among the plurality of access units, and second information used for calculating the decoding time of the plurality of access units, and a second transmission unit that transmits the generated first information and the second information as control information.

[0024] Note that these general or specific aspects may be implemented in a system, method, integrated circuit, computer program, or a recording medium such as a computer-readable CD-ROM, or may be implemented by any combination of a system, method, integrated circuit, computer program, or recording medium.

[0025] Hereinafter, embodiments will be specifically described with reference to the drawings.

[0026] Note that all of the embodiments described below show general or specific examples. The numerical values, shapes, materials, components, arrangement positions and connection forms of the components, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the present invention. Also, among the components in the following embodiments, components not described in the independent claims indicating the most general concept are described as optional components.

[0027] (Findings underlying the present invention) In recent years, the resolution of displays such as TVs, smartphones, or tablet terminals has been increasing. In particular, in Japan's domestic broadcasting, 8K4K (resolution of 8K×4K) services are planned for 2020. In ultra-high-resolution moving images such as 8K4K, it is difficult to decode in real time with a single decoder, so a method of performing decoding processing in parallel using multiple decoders has been studied.

[0028] Since the encoded data is multiplexed and transmitted based on a multiplexing method such as MPEG-2 TS or MMT, the receiving device needs to separate the encoded data of the video from the multiplexed data prior to decoding. Hereinafter, the process of separating the encoded data from the multiplexed data is called demultiplexing.

[0029] When parallelizing the decoding process, it is necessary to distribute the encoded data to be decoded to each decoder. When distributing the encoded data, it is necessary to analyze the encoded data itself. Especially in the case of content such as 8K, since the bit rate is very high, the processing load involved in the analysis is large. Therefore, there has been a problem that the inverse multiplexing part becomes a bottleneck and real-time playback cannot be performed.

[0030] By the way, in video coding schemes such as H.264 and H.265 standardized by MPEG and ITU, a transmitting device can divide a picture into a plurality of regions called slices or slice segments, and encode each divided region so that it can be decoded independently. Therefore, for example, in the case of H.265, a receiving device that receives a broadcast can separate the data for each slice segment from the received data and output the data of each slice segment to a separate decoder, thereby realizing parallelization of the decoding process.

[0031] FIG. 1 is a diagram showing an example of dividing one picture into four slice segments in HEVC. For example, a receiving device includes four decoders, and each decoder decodes one of the four slice segments.

[0032] In conventional broadcasting, a transmitting device stores one picture (access unit in the MPEG system standard) in one PES packet and multiplexes the PES packets into a TS packet stream. For this reason, the receiving device needs to separate the payload of the PES packet and then analyze the data of the access unit stored in the payload to separate each slice segment and output the data of each separated slice segment to a decoder.

[0033] However, the inventor has found that there is a problem that it is difficult to perform this process in real time because the amount of processing for analyzing the data of the access unit and separating the slice segments is large.

[0034] FIG. 2 is a diagram showing an example in which data of a picture divided into slice segments is stored in the payload of a PES packet.

[0035] As shown in FIG. 2, for example, data of a plurality of slice segments (slice segments 1 to 4) is stored in the payload of one PES packet. Also, the PES packet is multiplexed into a TS packet stream.

[0036] (Embodiment 1) Hereinafter, a case where H.265 is used as a moving image encoding method will be described as an example, but this embodiment can also be applied when other encoding methods such as H.264 are used.

[0037] FIG. 3 is a diagram showing an example in which an access unit (picture) in this embodiment is divided into division units. The access unit is divided into a total of four tiles by being bisected in the horizontal and vertical directions respectively by a function called a tile introduced by H.265. Also, a slice segment and a tile are associated one-to-one.

[0038] The reason for bisecting in the horizontal and vertical directions in this way will be described. First, at the time of decoding, generally a line memory for storing data of one horizontal line is required, but when the resolution becomes ultra-high such as 8K4K, since the size in the horizontal direction becomes large, the size of the line memory increases. In the implementation of the receiving device, it is desirable to be able to reduce the size of the line memory. In order to reduce the size of the line memory, division in the vertical direction is required. A data structure called a tile is required for division in the vertical direction. For these reasons, tiles are used.

[0039] On the other hand, since an image generally has a high correlation in the horizontal direction, the encoding efficiency is improved if a wider range can be referred to in the horizontal direction. Therefore, from the viewpoint of encoding efficiency, it is desirable that the access unit be divided in the horizontal direction.

[0040] By bisecting the access unit both horizontally and vertically, these two characteristics can be achieved simultaneously, taking into account both the implementation surface and the coding efficiency. When a single decoder can decode 4K2K moving images in real time, the 8K4K image is divided into four equal parts, and each slice segment is divided into 4K2K, so that the receiving device can decode the 8K4K image in real time.

[0041] Next, the reason for associating the tiles and slice segments obtained by dividing the access unit both horizontally and vertically on a one-to-one basis will be explained. In H.265, an access unit is composed of units called a plurality of NAL (Network Adaptation Layer) units.

[0042] The payload of the NAL unit stores either an access unit delimiter indicating the start position of the access unit, an SPS (Sequence Parameter Set) which is initialization information commonly used at the sequence unit during decoding, a PPS (Picture Parameter Set) which is initialization information commonly used within a picture during decoding, an SEI (Supplemental Enhancement Information) which is not necessary for the decoding process itself but is required for processing and displaying the decoding result, and the encoded data of the slice segment. The header of the NAL unit contains type information for identifying the data stored in the payload.

[0043] Here, the transmitting device converts the encoded data into MPEG-2 TS, MMT (MPEG Media Transport), MPEG DASH (Dynamic Adaptive When multiplexing by a multiplexing format such as Streaming over HTTP or RTP (Real-time Transport Protocol), the basic unit can be set to a NAL unit. In order to store one slice segment in one NAL unit, it is desirable to divide the access unit into slice segment units when dividing the access unit into regions. For these reasons, the transmitting device associates tiles and slice segments one-to-one.

[0044] As shown in FIG. 4, the transmitting device can also set tiles 1 to 4 together in one slice segment. However, in this case, all tiles will be stored in one NAL unit, making it difficult for the receiving device to separate the tiles in the multiplexing layer.

[0045] Note that there are independent slice segments that can be decoded independently and reference slice segments that reference independent slice segments in a slice segment. Here, the case where independent slice segments are used will be described.

[0046] FIG. 5 is a diagram showing an example of the data of an access unit divided so that the boundaries between tiles and slice segments coincide as shown in FIG. 3. The data of the access unit includes a NAL unit storing an access unit delimiter arranged at the head, NAL units of SPS, PPS, and SEI arranged thereafter, and data of slice segments storing data from tile 1 to tile 4 arranged thereafter. Note that the data of the access unit may not include some or all of the NAL units of SPS, PPS, and SEI.

[0047] Next, the configuration of the transmitting device 100 according to the present embodiment will be described. FIG. 6 is a block diagram showing a configuration example of the transmitting device 100 according to the present embodiment. This transmitting device 100 includes an encoding unit 101, a multiplexing unit 102, a modulation unit 103, and a transmitting unit 104.

[0048] The encoding unit 101 generates encoded data by encoding the input image according to, for example, H.265. Further, as shown in FIG. 3 for example, the encoding unit 101 divides an access unit into four slice segments (tiles) and encodes each slice segment.

[0049] The multiplexing unit 102 multiplexes the encoded data generated by the encoding unit 101. The modulation unit 103 modulates the data obtained by multiplexing. The transmission unit 104 transmits the modulated data as a broadcast signal.

[0050] Next, the configuration of the receiving apparatus 200 according to the present embodiment will be described. FIG. 7 is a block diagram showing a configuration example of the receiving apparatus 200 according to the present embodiment. The receiving apparatus 200 includes a tuner 201, a demodulation unit 202, a demultiplexing unit 203, a plurality of decoding units 204A to 204D, and a display unit 205.

[0051] The tuner 201 receives a broadcast signal. The demodulation unit 202 demodulates the received broadcast signal. The demodulated data is input to the demultiplexing unit 203.

[0052] The demultiplexing unit 203 separates the demodulated data into division units and outputs the data for each division unit to the decoding units 204A to 204D. Here, the division unit is a divided area obtained by dividing an access unit, and is, for example, a slice segment in H.265. Also, here, an 8K4K image is divided into four 4K2K images. Therefore, there are four decoding units 204A to 204D.

[0053] The plurality of decoding units 204A to 204D operate synchronously with each other based on a predetermined reference clock. Each decoding unit decodes the encoded data of the division unit according to the DTS (Decoding Time Stamp) of the access unit and outputs the decoding result to the display unit 205.

[0054] The display unit 205 generates an 8K4K output image by integrating a plurality of decoding results output from the plurality of decoding units 204A to 204D. The display unit 205 displays the generated output image according to the PTS (Presentation Time Stamp) of the access unit obtained separately. Note that when integrating the decoding results, the display unit 205 may perform filter processing such as deblocking filtering in the boundary region of adjacent divided units such as the tile boundary so that the boundary is not visually prominent.

[0055] In the above description, the transmission device 100 and the reception device 200 that perform broadcast transmission or reception have been described as examples. However, the content may be transmitted and received via a communication network. When the reception device 200 receives the content via a communication network, the reception device 200 separates the multiplexed data from the IP packet received by a network such as Ethernet.

[0056] In broadcasting, the transmission delay from when the broadcast signal is transmitted until it reaches the reception device 200 is constant. On the other hand, in a communication network such as the Internet, due to the influence of congestion, the transmission delay from when the data transmitted from the server reaches the reception device 200 is not constant. Therefore, the reception device 200 often does not perform strict synchronous playback based on a reference clock such as the PCR in the MPEG-2 TS of the broadcast. Therefore, the reception device 200 may display the 8K4K output image at the display unit according to the PTS without strictly synchronizing each decoding unit.

[0057] Also, due to congestion in the communication network or the like, the decoding process for all divided units may not be completed at the time indicated by the PTS of the access unit. In this case, the reception device 200 skips the display of the access unit, or delays the display until at least four divided units have completed decoding and the generation of the 8K4K image is completed.

[0058] Note that content may be transmitted and received by using both broadcasting and communication. Also, this method can be applied when playing back multiplexed data stored in a recording medium such as a hard disk or memory.

[0059] Next, a multiplexing method for access units segmented into slice segments when MMT is used as the multiplexing method will be described.

[0060] FIG. 8 is a diagram showing an example when packetizing data of an access unit of HEVC into MMT. SPS, PPS, SEI, etc. are not necessarily included in the access unit, but here, cases where they exist will be exemplified.

[0061] NAL units arranged before the first slice segment in the access unit, such as the access unit delimiter, SPS, PPS, and SEI, are grouped together and stored in MMT packet #1. Subsequent slice segments are stored in separate MMT packets for each slice segment.

[0062] Note that, as shown in FIG. 9, NAL units arranged before the first slice segment in the access unit may be stored in the same MMT packet as the first slice segment.

[0063] Also, when NAL units such as End-of-Sequence or End-of-Bitstream indicating the end of a sequence or a stream are added after the final slice segment, they are stored in the same MMT packet as the final slice segment. However, since NAL units such as End-of-Sequence or End-of-Bitstream are inserted at the end point of decoding processing or the connection point of two streams, etc., it may be desirable for the receiving device 200 to easily obtain these NAL units in the multiplexing layer. In this case, these NAL units may be stored in an MMT packet different from the slice segment. Thereby, the receiving device 200 can easily separate these NAL units in the multiplexing layer.

[0064] Note that as the multiplexing method, TS, DASH, RTP, etc. may be used. Also in these methods, the transmitting device 100 stores different slice segments in different packets. Thereby, it can be guaranteed that the receiving device 200 can separate the slice segments in the multiplexing layer.

[0065] For example, when TS is used, the encoded data is packetized into PES packets in units of slice segments. When RTP is used, the encoded data is packetized into RTP packets in units of slice segments. Also in these cases, like the MMT packet #1 shown in FIG. 8, the NAL unit arranged before the slice segment and the slice segment may be packetized separately.

[0066] When TS is used, the transmitting device 100 indicates the unit of data stored in the PES packet, such as by using a data alignment descriptor. Also, since DASH is a method of downloading data units in the MP4 format called segments via HTTP or the like, the transmitting device 100 does not perform packetization of the encoded data at the time of transmission. For this reason, the transmitting device 100 may create subsamples in units of slice segments so that the receiving device 200 can detect slice segments in the multiplexing layer in MP4, and store information indicating the storage positions of the subsamples in the MP4 header.

[0067] Hereinafter, the MMT packetization of slice segments will be described in detail.

[0068] As shown in FIG. 8, by packetizing the encoded data, data that is commonly referred to during decoding of all slice segments in an access unit such as SPS and PPS is stored in MMT packet #1. In this case, the receiving device 200 concatenates the payload data of MMT packet #1 and the data of each slice segment, and outputs the obtained data to the decoding unit. In this way, the receiving device 200 can easily generate input data for the decoding unit by concatenating the payloads of a plurality of MMT packets.

[0069] FIG. 10 is a diagram showing an example in which input data to the decoding units 204A to 204D is generated from the MMT packet shown in FIG. 8. The demultiplexing unit 203 concatenates the payload data of MMT packet #1 and MMT packet #2, so that the decoding unit 204A generates the data necessary for decoding slice segment 1. The demultiplexing unit 203 similarly generates input data for the decoding units 204B to 204D. That is, the demultiplexing unit 203 concatenates the payload data of MMT packet #1 and MMT packet #3 to generate the input data for the decoding unit 204B. The demultiplexing unit 203 concatenates the payload data of MMT packet #1 and MMT packet #4 to generate the input data for the decoding unit 204C. The demultiplexing unit 203 concatenates the payload data of MMT packet #1 and MMT packet #5 to generate the input data for the decoding unit 204D.

[0070] Note that the demultiplexing unit 203 may remove NAL units that are not necessary for the decoding process, such as access unit delimiters and SEI, from the payload data of MMT packet #1, and separate only the NAL units of SPS and PPS that are necessary for the decoding process and add them to the data of the slice segment.

[0071] When the encoded data is packetized as shown in FIG. 9, the demultiplexing unit 203 outputs MMT packet #1 including the leading data of the access unit in the multiplexing layer to the first decoding unit 204A. Further, the demultiplexing unit 203 analyzes the MMT packet including the leading data of the access unit in the multiplexing layer, separates the NAL units of SPS and PPS, and adds the separated NAL units of SPS and PPS to each of the data of the second and subsequent slice segments to generate input data for each of the second and subsequent decoding units.

[0072] Furthermore, it is desirable that the receiving device 200 can identify the type of data stored in the MMT payload and the index number of the slice segment in the access unit when the slice segment is stored in the payload, using the information included in the header of the MMT packet. Here, the type of data refers to either the pre-slice segment data (collectively referred to as such for NAL units arranged before the first slice segment in the access unit) or the data of the slice segment. When storing a unit obtained by fragmenting an MPU such as a slice segment in the MMT packet, a mode for storing an MFU (Media Fragment Unit) is used. When using this mode, the transmitting device 100 can set, for example, a Data Unit, which is a basic unit of data in the MFU, to a sample (a data unit in MMT corresponding to an access unit) or a subsample (a unit obtained by dividing a sample).

[0073] At this time, the header of the MMT packet includes a field called Fragmentation indicator and a field called Fragment counter.

[0074] The Fragmentation indicator indicates whether the data stored in the payload of an MMT packet is a fragmentation of a Data unit, and if it is a fragmentation, whether the fragment is the first or last fragment in the Data unit, or a fragment that is neither the first nor the last. In other words, the Fragmentation indicator included in the header of a certain packet is identification information indicating which of the following cases it is: (1) the packet is the only one included in the Data unit which is the basic data unit, (2) the Data unit is divided and stored in multiple packets, and the packet is the first packet of the Data unit, (3) the Data unit is divided and stored in multiple packets, and the packet is a packet other than the first and the last packets of the Data unit, and (4) the Data unit is divided and stored in multiple packets, and the packet is the last packet of the Data unit.

[0075] The Fragment counter is an index number indicating which fragment in the Data unit the data stored in the MMT packet corresponds to.

[0076] Therefore, by the transmitting device 100 setting samples in the MMT to the Data unit and setting the pre-slice segment data and each slice segment in units of fragments of the Data unit respectively, the receiving device 200 can identify the type of data stored in the payload using the information included in the header of the MMT packet. That is, the demultiplexing unit 203 can generate input data to each decoding unit 204A to 204D by referring to the header of the MMT packet.

[0077] FIG. 11 is a diagram showing an example in the case where samples are set to the Data unit and the pre-slice segment data and the slice segments are packetized as fragments of the Data unit.

[0078] The pre-slice segment data and the slice segment are divided into five fragments from fragment #1 to fragment #5. Each fragment is stored in an individual MMT packet. At this time, the values of the Fragmentation indicator and the Fragment counter included in the header of the MMT packet are as shown in the figure.

[0079] For example, the Fragment indicator is a 2-bit binary value. The Fragment indicator of MMT packet #1, which is the start of the Data unit, the Fragment indicator of MMT packet #5, which is the last one, and the Fragment indicators of the packets in between, i.e., MMT packets #2 to #4, are set to different values respectively. Specifically, the Fragment indicator of MMT packet #1, which is the start of the Data unit, is set to 01, the Fragment indicator of MMT packet #5, which is the last one, is set to 11, and the Fragment indicators of the packets in between, i.e., MMT packets #2 to #4, are set to 10. In addition, when the Data unit contains only one MMT packet, the Fragment indicator is set to 00.

[0080] Also, the Fragment counter is 4, which is the value obtained by subtracting 1 from the total number of fragments, i.e., 5, in MMT packet #1, and it decreases by 1 in subsequent packets, and is 0 in the last MMT packet #5.

[0081] Therefore, the receiving device 200 can identify the MMT packet storing the pre-slice segment data by using either the Fragment indicator or the Fragment counter. Also, the receiving device 200 can identify the MMT packet storing the Nth slice segment by referring to the Fragment counter.

[0082] The header of the MMT packet separately includes the sequence number within the MPU of the Movie Fragment to which the Data unit belongs, the sequence number of the MPU itself, and the sequence number within the Movie Fragment of the sample to which the Data unit belongs. By referring to these, the demultiplexing unit 203 can uniquely determine the sample to which the Data unit belongs.

[0083] Furthermore, since the demultiplexing unit 203 can determine the index number of the fragment within the Data unit from the Fragment counter or the like, even when packet loss occurs, the slice segment stored in the fragment can be uniquely specified. For example, even if the fragment #4 shown in FIG. 11 cannot be obtained due to packet loss, the demultiplexing unit 203 knows that the next received fragment after fragment #3 is fragment #5, so the slice segment 4 stored in fragment #5 can be correctly output to the decoding unit 204D instead of the decoding unit 204C.

[0084] Note that when a transmission path that guarantees no packet loss is used, the demultiplexing unit 203 does not need to determine the type of data stored in the MMT packet or the index number of the slice segment by referring to the header of the MMT packet, and can simply process the arrived packets periodically. For example, when an access unit is transmitted by a total of 5 MMT packets including pre-slice data and 4 slice segments, after the receiving device 200 determines the pre-slice data of the access unit for which decoding is to be started, it can sequentially obtain the pre-slice data and the data of the 4 slice segments by processing the received MMT packets in order.

[0085] Hereinafter, a modification example of packetization will be described.

[0086] The slice segment does not necessarily have to be divided both horizontally and vertically within the plane of the access unit. As shown in FIG. 1, the access unit may be divided only horizontally or only vertically.

[0087] Also, when the access unit is divided only horizontally, it is not necessary to use tiles.

[0088] Also, the number of in-plane divisions in the access unit is arbitrary and is not limited to four. However, the area sizes of the slice segment and the tile need to be equal to or greater than the lower limit of the encoding standard such as H.265.

[0089] The transmitting device 100 may store identification information indicating the in-plane division method in the access unit in an MMT message, a descriptor of the TS, or the like. For example, information indicating the number of horizontal and vertical divisions in the plane may be stored. Alternatively, unique identification information may be assigned to the division method, such as being bisected horizontally and vertically as shown in FIG. 3, or being quartered horizontally as shown in FIG. 1. For example, when the access unit is divided as shown in FIG. 3, the identification information indicates mode 1, and when the access unit is divided as shown in FIG. 1, the identification information indicates mode 1.

[0090] Also, information indicating the constraints of the encoding conditions related to the in-plane division method may be included in the multiplexing layer. For example, information indicating that one slice segment is composed of one tile may be used. Alternatively, information indicating that the reference block for performing motion compensation during decoding of the slice segment or the tile is limited to the slice segment or tile at the same position within the screen, or is limited to blocks within a predetermined range in adjacent slice segments may be used.

[0091] Further, the transmitting device 100 may switch whether to divide an access unit into a plurality of slice segments according to the resolution of the moving image. For example, when the moving image to be processed has a resolution of 4K2K, the transmitting device 100 may not perform in-plane division, and when the moving image to be processed has a resolution of 8K4K, the access unit may be divided into four. By prescribing in advance the division method in the case of an 8K4K moving image, the receiving device 200 can determine the presence or absence of in-plane division and the division method by acquiring the resolution of the received moving image, and can switch the decoding operation.

[0092] Further, the receiving device 200 can detect the presence or absence of in-plane division by referring to the header of the MMT packet. For example, when the access unit is not divided, if the Data unit of MMT is set to sample, the fragmentation of the Data unit is not performed. Therefore, when the value of the Fragment counter included in the header of the MMT packet is always zero, the receiving device 200 can determine that the access unit is not divided. Alternatively, the receiving device 200 may detect whether the value of the Fragmentation indicator is always 01. When the value of the Fragmentation indicator is always 01, the receiving device 200 can also determine that the access unit is not divided.

[0093] Further, the receiving device 200 can also handle the case where the number of in-plane divisions in the access unit does not match the number of decoding units. For example, when the receiving device 200 includes two decoding units 204A and 204B that can decode 8K2K encoded data in real time, the de-multiplexing unit 203 outputs two of the four slice segments that make up the 8K4K encoded data to the decoding unit 204A.

[0094] FIG. 12 is a diagram showing an operation example when the data packetized by MMT as shown in FIG. 8 is input to two decoding units 204A and 204B. Here, it is desirable that the receiving apparatus 200 can integrate and output the decoding results in the decoding units 204A and 204B as they are. Therefore, the demultiplexing unit 203 selects slice segments to be output to each of the decoding units 204A and 204B so that the decoding results of each of the decoding units 204A and 204B are spatially continuous.

[0095] Further, the demultiplexing unit 203 may select a decoding unit to be used according to the resolution or frame rate of the encoded data of the moving image. For example, when the receiving apparatus 200 includes four 4K2K decoding units, if the resolution of the input image is 8K4K, the receiving apparatus 200 performs decoding processing using all four decoding units. Also, if the resolution of the input image is 4K2K, the receiving apparatus 200 performs decoding processing using only one decoding unit. Alternatively, even if the in-plane is divided into four, when the 8K4K can be decoded in real time by a single decoding unit, the demultiplexing unit 203 integrates all the divided units and outputs them to one decoding unit.

[0096] Furthermore, the receiving apparatus 200 may determine a decoding unit to be used in consideration of the frame rate. For example, when the receiving apparatus 200 includes two decoding units whose upper limit of the frame rate that can be decoded in real time is 60 fps when the resolution is 8K4K, there is a case where encoded data of 8K4K at 120 fps is input. At this time, assuming that the in-plane is composed of four divided units, similar to the example of FIG. 12, slice segment 1 and slice segment 2 are input to the decoding unit 204A, and slice segment 3 and slice segment 4 are input to the decoding unit 204B. Since each of the decoding units 204A and 204B can decode in real time up to 120 fps if it is 8K2K (half of the resolution of 8K4K), decoding processing is performed by these two decoding units 204A and 204B.

[0097] Also, even if the resolution and frame rate are the same, the processing amount will be different if the profile or level in the encoding method or the encoding method itself such as H.264 or H.265 is different. Therefore, the receiving device 200 may select a decoding unit to use based on this information. Note that when the receiving device 200 cannot decode all the encoded data received by broadcasting or communication, or when all the slice segments or tiles constituting the area selected by the user cannot be decoded, it may automatically determine slice segments or tiles that can be decoded within the processing range of the decoding unit. Or, the receiving device 200 may provide a user interface for the user to select an area to be decoded. At this time, the receiving device 200 may display a warning message indicating that not all areas can be decoded, or may display information indicating the number of areas, slice segments, or tiles that can be decoded.

[0098] Also, the above method can also be applied when MMT packets storing slice segments of the same encoded data are transmitted and received using a plurality of transmission paths such as broadcasting and communication.

[0099] Also, the transmitting device 100 may perform encoding so that the areas of each slice segment overlap in order to make the boundary of the division unit less noticeable. In the example shown in FIG. 13, an 8K4K picture is divided into four slice segments 1 to 4. Each of slice segments 1 to 3 is, for example, 8K×1.1K, and slice segment 4 is 8K×1K. Also, adjacent slice segments overlap each other. By doing so, at the boundary in the case of four-way division indicated by the dotted line, motion compensation during encoding can be efficiently executed, so the image quality at the boundary portion is improved. In this way, the image quality degradation at the boundary portion is reduced.

[0100] In this case, the display unit 205 cuts out an 8K×1K area from an 8K×1.1K area and integrates the obtained area. Note that the transmission device 100 may separately transmit information indicating whether the slice segments are encoded with overlap and the range of the overlap, included in the multiplexing layer or the encoded data.

[0101] Note that the same method can also be applied when tiles are used.

[0102] Hereinafter, the operation flow of the transmission device 100 will be described. FIG. 14 is a flowchart showing an operation example of the transmission device 100.

[0103] First, the encoding unit 101 divides a picture (access unit) into a plurality of slice segments (tiles) which are a plurality of regions (S101). Next, the encoding unit 101 generates encoded data corresponding to each of the plurality of slice segments by encoding each of the plurality of slice segments so that it can be independently decoded (S102). Note that the encoding unit 101 may encode the plurality of slice segments with a single encoding unit, or may perform parallel processing with a plurality of encoding units.

[0104] Next, the multiplexing unit 102 multiplexes the plurality of encoded data generated by the encoding unit 101 by storing the plurality of encoded data in a plurality of MMT packets (S103). Specifically, as shown in FIGS. 8 and 9, the multiplexing unit 102 stores the plurality of encoded data in a plurality of MMT packets so that the encoded data corresponding to different slice segments is not stored in one MMT packet. Further, as shown in FIG. 8, the multiplexing unit 102 stores control information commonly used for all decoding units in the picture in an MMT packet #1 different from the plurality of MMT packets #2 to #5 in which the plurality of encoded data are stored. Here, the control information includes at least one of an access unit delimiter, SPS, PPS, and SEI.

[0105] Note that the multiplexing unit 102 may store the control information in the same MMT packet as any one of the plurality of MMT packets in which the plurality of encoded data are stored. For example, as shown in FIG. 9, the multiplexing unit 102 may store the control information in the first MMT packet (MMT packet #1 in FIG. 9) among the plurality of MMT packets in which the plurality of encoded data are stored.

[0106] Finally, the transmitting device 100 transmits a plurality of MMT packets. Specifically, the modulation unit 103 modulates the data obtained by multiplexing, and the transmission unit 104 transmits the modulated data (S104).

[0107] FIG. 15 is a block diagram showing a configuration example of the receiving device 200, and is a diagram showing in detail the demultiplexing unit 203 shown in FIG. 7 and the subsequent configuration. As shown in FIG. 15, the receiving device 200 further includes a decoding command unit 206. The demultiplexing unit 203 includes a type determination unit 211, a control information acquisition unit 212, a slice information acquisition unit 213, and a decoded data generation unit 214.

[0108] Hereinafter, the operation flow of the receiving device 200 will be described. FIG. 16 is a flowchart showing an operation example of the receiving device 200. Here, the operation for one access unit is shown. When the decoding process for a plurality of access units is executed, the process of this flowchart is repeated.

[0109] First, the receiving device 200 receives, for example, a plurality of packets (MMT packets) generated by the transmitting device 100 (S201).

[0110] Next, the type determination unit 211 acquires the type of the encoded data stored in the received packet by analyzing the header of the received packet (S202).

[0111] Next, the type determination unit 211 determines whether the data stored in the received packet is pre-slice segment data or slice segment data based on the type of the acquired encoded data (S203).

[0112] When the data stored in the received packet is pre-slice segment data (Yes in S203), the control information acquisition unit 212 acquires the pre-slice segment data of the access unit to be processed from the payload of the received packet, and stores the pre-slice segment data in the memory (S204).

[0113] On the other hand, when the data stored in the received packet is slice segment data (No in S203), the receiving device 200 determines, using the header information of the received packet, which region among a plurality of regions the data stored in the received packet is encoded data of. Specifically, the slice information acquisition unit 213 acquires the index number Idx of the slice segment stored in the received packet by analyzing the header of the received packet (S205). Specifically, the index number Idx is the index number within the Movie Fragment of the access unit (sample in MMT).

[0114] Note that the process of this step S205 may be performed collectively in step S202.

[0115] Next, the decoded data generation unit 214 determines a decoding unit for decoding the slice segment (S206). Specifically, the index number Idx and a plurality of decoding units are associated in advance, and the decoded data generation unit 214 determines the decoding unit corresponding to the index number Idx acquired in step S205 as the decoding unit for decoding the slice segment.

[0116] Note that, as described in the example of FIG. 12, the decoded data generation unit 214 may determine a decoding unit for decoding the slice segment based on at least one of the resolution of the access unit (picture), the method of dividing the access unit into a plurality of slice segments (tiles), and the processing capabilities of the plurality of decoding units included in the receiving apparatus 200. For example, the decoded data generation unit 214 discriminates the method of dividing the access unit based on identification information in a descriptor such as an MMT message or a TS section.

[0117] Next, the decoded data generation unit 214 generates a plurality of input data (combined data) to be input to the plurality of decoding units by combining control information commonly used for all decoding units in the picture, which is included in any of the plurality of packets, and each of the plurality of encoded data of the plurality of slice segments. Specifically, the decoded data generation unit 214 acquires the data of the slice segment from the payload of the received packet. The decoded data generation unit 214 generates the input data to the decoding unit determined in step S206 by combining the pre-slice segment data stored in the memory in step S204 and the acquired data of the slice segment (S207).

[0118] After step S204 or S207, if the data of the received packet is not the final data of the access unit (No in S208), the processing after step S201 is performed again. That is, the above processing is repeated until the input data to the plurality of decoding units 204A to 204D corresponding to all the slice segments included in the access unit is generated.

[0119] Note that the timing at which the packet is received is not limited to the timing shown in FIG. 16, and a plurality of packets may be received in advance or sequentially and stored in a memory or the like.

[0120] On the other hand, when the data of the received packet is the final data of the access unit (Yes in S208), the decoding instruction unit 206 outputs the plurality of input data generated in step S207 to the corresponding decoding units 204A to 204D (S209).

[0121] Next, the plurality of decoding units 204A to 204D generate a plurality of decoded images by decoding the plurality of input data in parallel according to the DTS of the access unit (S210).

[0122] Finally, the display unit 205 generates a display image by combining the plurality of decoded images generated by the plurality of decoding units 204A to 204D, and displays the display image according to the PTS of the access unit (S211).

[0123] Note that the receiving device 200 acquires the DTS and PTS of the access unit by analyzing the header information of the MPU or the payload data of the MMT packet storing the header information of the Movie Fragment. Also, when TS is used as the multiplexing method, the receiving device 200 acquires the DTS and PTS of the access unit from the header of the PES packet. When RTP is used as the multiplexing method, the receiving device 200 acquires the DTS and PTS of the access unit from the header of the RTP packet.

[0124] In addition, when integrating the decoding results of multiple decoding units, the display unit 205 may perform filter processing such as deblocking filtering at the boundaries of adjacent divided units. Note that when displaying the decoding result of a single decoding unit, filter processing is unnecessary. Therefore, the display unit 205 may switch the processing according to whether to perform filter processing at the boundary of the decoding results of multiple decoding units. Whether filter processing is necessary may be predefined according to the presence or absence of division, etc. Alternatively, information indicating whether filter processing is necessary may be separately stored in the multiplexing layer. Also, information necessary for filter processing such as filter coefficients may be stored in the SPS, PPS, SEI, or within the slice segment. The decoding units 204A to 204D or the demultiplexing unit 203 acquire this information by analyzing the SEI and output the acquired information to the display unit 205. The display unit 205 performs filter processing using this information. Note that when this information is stored within the slice segment, it is desirable for the decoding units 204A to 204D to acquire this information.

[0125] Note that in the above description, an example in which the types of data stored in the fragment are two types, i.e., pre-slice segment data and slice segment, is shown. However, the number of data types may be three or more. In this case, branching according to the type is performed in step S203.

[0126] In addition, when the data size of the slice segment is large, the transmission device 100 may fragment the slice segment and store it in the MMT packet. That is, the transmission device 100 may fragment the pre-slice segment data and the slice segment. In this case, when the access unit and the data unit are set equal as in the packetization example shown in FIG. 11, the following problems occur.

[0127] For example, when slice segment 1 is divided into three fragments, slice segment 1 is divided into three packets with fragment counter values from 1 to 3 and transmitted. Also, for slice segments 2 and later, the fragment counter value becomes 4 or more, and the association between the fragment counter value and the data stored in the payload cannot be established. Therefore, the receiving device 200 cannot identify the packet storing the start data of the slice segment from the information in the header of the MMT packet.

[0128] In such a case, the receiving device 200 may analyze the data in the payload of the MMT packet to identify the start position of the slice segment. Here, as a format for storing NAL units in multiplexing layers in H.264 or H.265, there are two types: a byte stream format in which a start code consisting of a specific bit string is added immediately before the NAL unit header, and a NAL size format in which a field indicating the size of the NAL unit is added.

[0129] The byte stream format is used in MPEG-2 systems and RTP, etc. The NAL size format is used in MP4, and in DASH and MMT that use MP4, etc.

[0130] When the byte stream format is used, the receiving device 200 analyzes whether the start data of the packet matches the start code. If the start data of the packet matches the start code, the receiving device 200 can detect whether the data contained in the packet is the data of the slice segment by obtaining the type of the NAL unit from the subsequent NAL unit header.

[0131] On the other hand, in the case of the NAL size format, the receiving device 200 cannot detect the start position of the NAL unit based on the bit sequence. Therefore, in order for the receiving device 200 to obtain the start position of the NAL unit, it is necessary to shift the pointer by reading data by the size of the NAL unit in order from the first NAL unit of the access unit.

[0132] However, in the header of the MPU or Movie Fragment in MMT, when the size of the sub-sample unit is indicated and the sub-sample corresponds to pre-slice data or a slice segment, the receiving device 200 can specify the start position of each NAL unit based on the size information of the sub-sample. Therefore, the transmitting device 100 may include information indicating whether information in sub-sample units exists in the MPU or Movie Fragment in information such as MPT in MMT that the receiving device 200 obtains at the start of data reception.

[0133] Note that the data of the MPU is an extension based on the MP4 format. In MP4, there are a mode in which parameter sets such as SPS and PPS of H.264 or H.265 can be stored as sample data, and a mode in which they cannot be stored. Also, information for specifying this mode is shown as the entry name of SampleEntry. When the mode in which storage is possible is used and the parameter set is included in the sample, the receiving device 200 obtains the parameter set by the method described above.

[0134] On the one hand, when a non-storable mode is used, the parameter set is stored as Decoder Specific Information within the SampleEntry or is stored using a stream for the parameter set. Here, since the stream for the parameter set is not generally used, it is desirable for the transmitting device 100 to store the parameter set in the Decoder Specific Information. In this case, the receiving device 200 analyzes the SampleEntry transmitted as the metadata of the MPU or the metadata of the Movie Fragment in the MMT packet to obtain the parameter set referred to by the access unit.

[0135] When the parameter set is stored as sample data, the receiving device 200 can obtain the parameter set necessary for decoding by referring only to the sample data without referring to the SampleEntry. At this time, the transmitting device 100 does not necessarily have to store the parameter set in the SampleEntry. By doing so, the transmitting device 100 can use the same SampleEntry in different MPUs, thus reducing the processing load of the transmitting device 100 during MPU generation. Furthermore, there is an advantage that the receiving device 200 does not need to refer to the parameter set within the SampleEntry.

[0136] Alternatively, the transmitting device 100 may store one default parameter set in the SampleEntry and store the parameter set referred to by the access unit in the sample data. In conventional MP4s, since it was common to store the parameter set in the SampleEntry, there may be a receiving device that stops playback when there is no parameter set in the SampleEntry. By using the above method, this problem can be solved.

[0137] Alternatively, the transmitting device 100 may store the parameter set in the sample data only when a parameter set different from the default parameter set is used.

[0138] Note that since it is possible to store the parameter set in SampleEntry in both modes, the transmission device 100 may always store the parameter set in VisualSampleEntry, and the reception device 200 may always obtain the parameter set from VisualSampleEntry.

[0139] Note that in the MMT standard, header information of MP4 such as Moov and Moof is transmitted as MPU metadata or movie fragment metadata, but the transmission device 100 does not necessarily have to transmit MPU metadata and movie fragment metadata. Further, the reception device 200 can also determine whether SPS and PPS are stored in the sample data based on services of the ARIB (Association of Radio Industries and Businesses) standard, the type of asset, or the presence or absence of transmission of MPU meta.

[0140] FIG. 17 is a diagram showing an example in which pre-slice segment data and each slice segment are set in different Data units.

[0141] In the example shown in FIG. 17, the sizes of the pre-slice segment data and the data from slice segment 1 to slice segment 4 are Length#1 to Length#5, respectively. The field values of the Fragmentation indicator, Fragment counter, and Offset included in the header of the MMT packet are as shown in the figure.

[0142] Here, Offset is offset information indicating the bit length (offset) from the start of the encoded data of the sample (access unit or picture) to which the payload data belongs to the start byte of the payload data (encoded data) included in the MMT packet. Note that although the description will be given assuming that the value of the Fragment counter starts from a value obtained by subtracting 1 from the total number of fragments, it may start from other values.

[0143] FIG. 18 is a diagram showing an example when a Data unit is fragmented. In the example shown in FIG. 18, slice segment 1 is divided into three fragments and stored in MMT packets #2 to #4, respectively. Also in this case, if the data sizes of the respective fragments are Length#2_1 to Length#2_3, respectively, the values of the respective fields are as shown in the figure.

[0144] Thus, when a data unit such as a slice segment is set as the Data unit, the start of the access unit and the start of the slice segment can be determined as follows based on the field values of the MMT packet header.

[0145] The start of the payload in a packet where the value of Offset is 0 is the start of the access unit.

[0146] The start of the payload of a packet where the value of Offset is different from 0 and the value of the Fragmentation indcatorno is 00 or 01 is the start of the slice segment.

[0147] Also, when fragmentation of the Data unit does not occur and no packet loss occurs, the receiving apparatus 200 can specify the index number of the slice segment stored in the MMT packet based on the number of slice segments acquired after detecting the start of the access unit.

[0148] Also, even when the Data unit of the pre-slice segment data is fragmented, the receiving device 200 can similarly detect the access unit and the start of the slice segment.

[0149] Also, when packet loss occurs, or when the SPS, PPS, and SEI included in the pre-slice segment data are set in separate Data units, the receiving device 200 identifies the MMT packet storing the start data of the slice segment based on the analysis result of the MMT header, and then analyzes the header of the slice segment to identify the start position of the slice segment or tile within the picture (access unit). The processing amount related to the analysis of the slice header is small, and the processing load is not a problem.

[0150] Thus, each of the encoded data of the plurality of slice segments is associated one-to-one with a basic data unit (Data unit), which is a unit of data stored in one or more packets. Also, each of the plurality of encoded data is stored in one or more MMT packets.

[0151] The header information of each MMT packet includes a Fragmentation indicator (identification information) and an Offset (offset information).

[0152] The receiving device 200 determines that the start of the payload data included in the packet having header information including a Fragmentation indicator with a value of 00 or 01 is the start of the encoded data of each slice segment. Specifically, it determines that the start of the payload data included in the packet having header information including an Offset whose value is not 0 and a Fragmentation indicator whose value is 00 or 01 is the start of the encoded data of each slice segment.

[0153] Also, in the example of FIG. 17, the start of the Data unit is either the start of the access unit or the start of the slice segment, and the value of the Fragmentation indicator is 00 or 01. Further, the receiving device 200 can determine whether the start of the Data Unit is the access unit delimiter or the slice segment by referring to the type of the NAL unit, and can detect the start of the access unit or the slice segment without referring to the Offset.

[0154] In this way, by the transmitting device 100 performing packetization so that the start of the NAL unit always starts from the start of the payload of the MMT packet, including the case where the pre-slice segment data is divided into a plurality of Data units, the receiving device 200 can detect the start of the access unit or the slice segment by analyzing the Fragmentation indicator and the NAL unit header. The type of the NAL unit exists in the first byte of the NAL unit header. Therefore, when analyzing the header part of the MMT packet, the receiving device 200 can obtain the type of the NAL unit by additionally analyzing one byte of data. In the case of audio, the receiving device 200 only needs to be able to detect the start of the access unit, and can make a determination based on whether the value of the Fragmentation indicator is 00 or 01.

[0155] Also, as described above, when storing the encoded data encoded so as to enable split decoding into the PES packets of MPEG-2 TS, the transmitting device 100 can use the data alignment descriptor. Hereinafter, an example of a method for storing the encoded data into the PES packet will be described in detail.

[0156] For example, in HEVC, the transmitting device 100 can indicate whether the data stored in the PES packet is an access unit, a slice segment, or a tile by using a data alignment descriptor. The types of alignment in HEVC are defined as follows.

[0157] Alignment type = 8 indicates a slice segment of HEVC. Alignment type = 9 indicates a slice segment or an access unit of HEVC. Alignment type = 12 indicates a slice segment or a tile of HEVC.

[0158] Therefore, the transmitting device 100 can indicate that the data of the PES packet is either a slice segment or pre-slice-segment data, for example, by using type 9. Since a type for indicating a slice instead of a slice segment is also defined separately, the transmitting device 100 may use the type for indicating a slice instead of a slice segment.

[0159] Also, DTS and PTS included in the header of the PES packet are set only in the PES packet containing the leading data of the access unit. Therefore, the receiving device 200 can determine that the entire access unit or the leading division unit in the access unit is stored in the PES packet if the type is 9 and there is a DTS or PTS field in the PES packet.

[0160] Also, the transmitting device 100 may use fields such as transport_priority indicating the priority of the TS packet storing the PES packet including the head data of the access unit, so that the receiving device 200 can distinguish the data included in the packet. Also, the receiving device 200 may determine the data included in the packet by analyzing whether the payload of the PES packet is an access unit delimiter. Also, the data_alignment_indicator in the PES packet header indicates whether data is stored in the PES packet according to these types. If this flag (data_alignment_indicator) is set to 1, it is guaranteed that the data stored in the PES packet follows the type indicated by the data alignment descriptor.

[0161] Also, the transmitting device 100 may use the data alignment descriptor only when packetizing the PES packet in units such as slice segments that can be decoded separately. Thereby, the receiving device 200 can determine that the encoded data is packetized in units that can be decoded separately when the data alignment descriptor exists, and can determine that the encoded data is packetized in access unit units when the data alignment descriptor does not exist. Note that in the case where the data_alignment_indicator is set to 1 and the data alignment descriptor does not exist, it is defined in the MPEG-2 TS standard that the unit of packetization is the access unit.

[0162] If the data alignment descriptor is included in the PMT, the receiving device 200 determines that the PES packets are packetized in units that can be split and decoded, and can generate input data for each decoding unit based on the packetized units. Further, if the data alignment descriptor is not included in the PMT and it is determined that parallel decoding of the encoded data is necessary based on the program information or other descriptor information, the receiving device 200 generates input data for each decoding unit by analyzing the slice header of the slice segment or the like. Also, when the encoded data can be decoded by a single decoding unit, the receiving device 200 decodes the data of the entire access unit using the said decoding unit. Note that when information indicating whether the encoded data is composed of units that can be split and decoded, such as slice segments or tiles, is separately indicated by a descriptor in the PMT or the like, the receiving device 200 may determine whether the encoded data can be decoded in parallel based on the analysis result of the said descriptor.

[0163] Also, since the DTS and PTS included in the header of the PES packet are set only in the PES packet containing the leading data of the access unit, when the access unit is split and packetized into PES packets, the second and subsequent PES packets do not contain information indicating the DTS and PTS of the access unit. Therefore, when performing the decoding process in parallel, each decoding unit 204A to 204D and the display unit 205 use the DTS and PTS stored in the header of the PES packet containing the leading data of the access unit.

[0164] (Embodiment 2) In Embodiment 2, in MMT, a method of storing data in the NAL size format in an MPU based on the MP4 format will be described. Note that hereinafter, as an example, a method of storing in the MPU used in MMT will be described, but such a storage method is also applicable to DASH based on the same MP4 format.

[0165] [Method of Storing in MPU] In the MP4 format, multiple access units are grouped together and stored in a single MP4 file. The MPU used in MMT has data for each media stored in a single MP4 file, and the data can contain any number of access units. Since the MPU is a unit that can be decoded independently, for example, access units in GOP units are stored in the MPU.

[0166] Figure 19 is a diagram showing the configuration of the MPU. The beginning of the MPU is ftyp, mmpu, and moov, which are collectively defined as MPU metadata. The moov stores initialization information common to the file and the MMT hint track.

[0167] Also, the moof stores initialization information and sizes for each sample and subsample, information that can identify the presentation time (PTS) and decoding time (DTS) (sample_duration, sample_size, sample_composition_time_offset), and a data_offset indicating the position of the data, etc.

[0168] Also, multiple access units are each stored as samples in mdat (mdat box). The data excluding samples in moof and mdat is defined as movie fragment metadata (hereinafter referred to as MF metadata), and the sample data in mdat is defined as media data.

[0169] Figure 20 is a diagram showing the configuration of MF metadata. As shown in Figure 20, MF metadata consists more specifically of the type, length, and data of the moof box (moof), and the type and length of the mdat box (mdat).

[0170] When storing access units in MP4 data, there are modes where parameter sets such as the SPS and PPS of H.264 and H.265 can be stored as sample data, and modes where they cannot be stored.

[0171] Here, in the mode where it cannot be stored as described above, the parameter set is stored in the Decoder Specific Information of the SampleEntry in moov. Also, in the mode where it can be stored as described above, the parameter set is included in the sample.

[0172] The MPU metadata, MF metadata, and media data are each stored in the MMT payload, and as an identifier for identifying these data, a fragment type (FT) is stored in the header of the MMT payload. FT = 0 indicates MPU metadata, FT = 1 indicates MF metadata, and FT = 2 indicates media data.

[0173] Note that in FIG. 19, an example where the MPU metadata unit and the MF metadata unit are stored in the MMT payload as data units is illustrated. However, units such as ftyp, mmpu, moov, and moof may be stored in the MMT payload in data unit units as data units. Similarly, in FIG. 19, an example where the sample unit is stored in the MMT payload as a data unit is illustrated. However, data units may be composed of sample units or NAL units, and such data units may be stored in the MMT payload in data unit units. Such data units may be further stored in the MMT payload in fragmented units.

[0174] [Conventional Transmission Method and Problem] Conventionally, when encapsulating a plurality of access units in the MP4 format, moov and moof were created when all the samples to be stored in the MP4 were ready.

[0175] When transmitting the MP4 format in real time using broadcasting or the like, for example, assuming that the samples stored in one MP4 file are in GOP units, since moov and moof are created after the time samples in GOP units are accumulated, a delay due to encapsulation occurs. Due to such encapsulation on the transmitting side, the End-to-End delay always becomes longer by the GOP unit time. This makes it difficult to provide services in real time, and in particular, when live content is transmitted, it leads to deterioration of the service for viewers.

[0176] Figure 21 is a diagram for explaining the transmission order of data. When applying MMT to broadcasting, as shown in Fig. 21(a), when placing and transmitting in the order of MPU composition on the MMT packet (transmitting in the order of MMT packet #1, #2, #3, #4, #5, #6), a delay due to encapsulation occurs in the transmission of the MMT packet.

[0177] In order to prevent this delay due to encapsulation, as shown in Fig. 21(b), a method has been proposed in which MPU header information such as MPU metadata and MF metadata is not sent (packets #1 and #2 are not transmitted, and packets #3 - #6 are transmitted in this order). Also, as shown in Fig. 20(c), a method can be considered in which media data is transmitted first without waiting for the creation of MPU header information, and MPU header information is transmitted after the transmission of the media data (transmitting in the order of #3 - #6, #1, #2).

[0178] When the receiving device does not receive the MPU header information, it decodes without using the MPU header information. Also, when the MPU header information is sent after the media data in the receiving device, it waits to obtain the MPU header information and then decodes.

[0179] However, in a conventional MP4-compliant receiving device, it is not guaranteed that decoding can be performed without using MPU header information. Also, when the receiving device performs decoding without using the MPU header through special processing and uses the conventional transmission method, the decoding process becomes complicated, and real-time decoding is likely to be difficult. Further, when the receiving device performs decoding after waiting to acquire the MPU header information, it is necessary to buffer media data until the receiving device acquires the header information. However, the buffer model is not defined, and decoding is not guaranteed.

[0180] Therefore, as shown in FIG. 20(d), the transmitting device according to Embodiment 2 transmits the MPU metadata before the media data by storing only the information common to the MPU metadata. Then, the transmitting device according to Embodiment 2 transmits the MF metadata, which causes a delay in generation, after the media data. Thereby, a transmission method or a reception method that can guarantee decoding of media data is provided.

[0181] Hereinafter, the reception method when each of the transmission methods shown in FIGS. 21(a) to 21(d) is used will be described.

[0182] In each of the transmission methods shown in FIG. 21, first, the MPU data is configured in the order of MPU metadata, MFU metadata, and media data.

[0183] After configuring the MPU data, when the transmitting device transmits the data in the order of MPU metadata, MF metadata, and media data as shown in FIG. 21(a), the receiving device can perform decoding by any of the following methods (A-1) and (A-2).

[0184] (A-1) After acquiring the MPU header information (MPU metadata and MF metadata), the receiving device decodes the media data using the MPU header information.

[0185] (A-2) The receiving device decodes the media data without using the MPU header information.

[0186] All of these methods incur a delay due to encapsulation on the transmitting side, but have the advantage that the receiving device does not need to buffer the media data to obtain the MPU header. When not buffering, there is no need to mount memory for buffering, and furthermore, no buffering delay occurs. Also, the method of (A-1) can be applied to conventional receiving devices because decoding is performed using MPU header information.

[0187] When the transmitting device transmits only the media data as shown in Fig. 21(b), the receiving device can perform decoding by the following method (B-1).

[0188] (B-1) The receiving device decodes the media data without using the MPU header information.

[0189] Also, although not shown, when the MPU metadata is transmitted before the media data in Fig. 21(b) is transmitted, decoding can be performed by the following method (B-2).

[0190] (B-2) The receiving device decodes the media data using the MPU metadata.

[0191] Both of the above methods (B-1) and (B-2) have the advantage that no delay due to encapsulation occurs on the transmitting side and there is no need to buffer the media data to obtain the MPU header. However, both methods (B-1) and (B-2) do not perform decoding using the MPU header information, so special processing may be required for decoding.

[0192] When the transmitting device transmits data in the order of media data, MPU metadata, and MF metadata as shown in Fig. 21(c), the receiving device can perform decoding by either of the following methods (C-1) and (C-2).

[0193] After the receiving device acquires the MPU header information (MPU metadata and MF metadata), it decrypts the media data.

[0194] The receiving device decrypts the media data without using the MPU header information.

[0195] When the method in (C-1) above is used, it is necessary to buffer the media data for acquiring the MPU header information. In contrast, when the method in (C-2) above is used, it is not necessary to perform buffering for acquiring the MPU header information.

[0196] Also, in either of the methods (C-1) and (C-2) above, no delay due to encapsulation occurs on the transmission side. Also, since the method in (C-2) does not use the MPU header information, special processing may be required.

[0197] When the transmitting device transmits data in the order of MPU metadata, media data, and MF metadata as shown in (d) of FIG. 21, the receiving device can perform decryption by either of the following methods (D-1) and (D-2).

[0198] The receiving device acquires the MPU metadata, then further acquires the MF metadata, and then decrypts the media data.

[0199] The receiving device acquires the MPU metadata and decrypts the media data without using the MF metadata.

[0200] When the method in (D-1) above is used, it is necessary to buffer the media data for acquiring the MF metadata, but in the case of the method in (D-2), it is not necessary to perform buffering for acquiring the MF metadata.

[0201] Since the method in (D-2) above does not perform decryption using the MF metadata, special processing may be required.

[0202] As described above, when it is possible to decode using MPU metadata and MF metadata, there is an advantage that even a conventional MP4 receiving device can decode it.

[0203] In FIG. 21, the MPU data is configured in the order of MPU metadata, MFU metadata, and media data. In moof, the position information (offset) for each sample and subsample is determined based on this configuration. Also, the MF metadata includes data other than the media data in the mdat box (the size and type of the box).

[0204] Therefore, when the receiving device identifies the media data based on the MF metadata, the receiving device reconfigures the data in the order in which the MPU data was configured regardless of the order in which the data was transmitted, and then performs decoding using the moov of the MPU metadata or the moof of the MF metadata.

[0205] In FIG. 21, the MPU data is configured in the order of MPU metadata, MFU metadata, and media data, but the MPU data may be configured in an order different from that in FIG. 21, and the position information (offset) may be determined.

[0206] For example, the MPU data may be configured in the order of MPU metadata, media data, and MF metadata, and negative position information (offset) may be indicated in the MF metadata. Also in this case, regardless of the order in which the data is transmitted, the receiving device reconfigures the data in the order in which the MPU data was configured on the transmitting side, and then performs decoding using moov or moof.

[0207] Note that the transmitting device may signal information indicating the order in which the MPU data is configured, and the receiving device may reconfigure the data based on the signaled information.

[0208] As described above, as shown in Fig. 21(d), the receiving device receives the packetized MPU metadata, the packetized media data (sample data), and the packetized MF metadata in this order. Here, the MPU metadata is an example of the first metadata, and the MF metadata is an example of the second metadata.

[0209] Next, the receiving device reconstructs the MPU data (a file in the MP4 format) including the received MPU metadata, the received MF metadata, and the received sample data. Then, the sample data included in the reconstructed MPU data is decoded using the MPU metadata and the MF metadata. The MF metadata is metadata that includes data (for example, length stored in mbox) that can be generated only after the generation of the sample data on the transmission side.

[0210] Note that the operation of the receiving device is, in more detail, performed by each component that constitutes the receiving device. For example, the receiving device includes a receiving unit that receives the above data, a reconstructing unit that reconstructs the above MPU data, and a decoding unit that decodes the above MPU data. Note that each of the receiving unit, the generating unit, and the decoding unit is realized by a microcomputer, a processor, a dedicated circuit, or the like.

[0211] [Method of Decoding Without Using Header Information] Next, a method of decoding without using header information will be described. Here, a method of decoding without using header information in the receiving device will be described regardless of whether or not header information is sent on the transmission side. That is, this method is applicable regardless of which transmission method described with reference to Fig. 21 is used. However, some decoding methods are decoding methods applicable only to specific transmission methods.

[0212] FIG. 22 is a diagram showing an example of a method of performing decoding without using header information. In FIG. 22, only the MMT payload containing only media data and the MMT packet are illustrated, and the MMT payload and MMT packet containing MPU metadata and MF metadata are not illustrated. Also, in the following description of FIG. 22, it is assumed that media data belonging to the same MPU is transmitted continuously. Further, a case where samples are stored in the payload as media data will be described as an example, but in the following description of FIG. 22, naturally, NAL units may be stored, or fragmented NAL units may be stored.

[0213] In order to decode media data, the receiving device must first obtain initialization information necessary for decoding. Also, if the media is video, the receiving device must obtain initialization information for each sample, identify the start position of the MPU, which is a random access unit, and obtain the start positions of the sample and the NAL unit. Also, the receiving device needs to identify the decoding time (DTS) and presentation time (PTS) of each sample.

[0214] Therefore, the receiving device can perform decoding without using header information by using, for example, the following method. When NAL units or units obtained by fragmenting NAL units are stored in the payload, "sample" may be read as "NAL unit in the sample" in the following description.

[0215] <Random access (= identifying the first sample of the MPU)> When header information is not transmitted, there are the following Method 1 and Method 2 for the receiving device to identify the first sample of the MPU. When header information is transmitted, Method 3 can be used.

[0216] [Method 1] The receiving device obtains the samples included in the MMT packet in which 'RAP_flag = 1' in the MMT packet header.

[0217] [Method 2] The receiving device acquires a sample with'sample number = 0' in the MMT payload header.

[0218] [Method 3] When at least one of the MPU metadata and the MF metadata is transmitted before or after the media data, the receiving device acquires a sample included in the MMT payload in which the fragment type (FT) in the MMT payload header has switched to the media data.

[0219] In Methods 1 and 2, when multiple samples belonging to different MPUs are mixed in one payload, it is impossible to determine which NAL unit is a random access point (RAP_flag = 1 or sample number = 0). Therefore, restrictions such as not mixing samples of different MPUs in one payload, or when samples of different MPUs are mixed in one payload, setting the RAP_flag to 1 when the last (or first) sample is a random access point are necessary.

[0220] Also, in order for the receiving device to acquire the start position of the NAL unit, it is necessary to shift the data read pointer by the size of the NAL unit in order from the first NAL unit of the sample.

[0221] When the data is fragmented, the receiving device can identify the data unit by referring to the fragment_indicator and the fragment_number.

[0222] <Determination of the DTS of the Sample> There are the following Method 1 and Method 2 for determining the DTS of the sample.

[0223] [Method 1] The receiving device determines the DTS of the first sample based on the predicted structure. However, since this method requires analysis of the encoded data and may be difficult to decode in real time, the following Method 2 is desirable.

[0224] [Method 2] The receiving device separately transmits the DTS of the first sample and acquires the transmitted DTS of the first sample. Examples of the method for transmitting the DTS of the first sample include a method of transmitting the DTS of the MPU first sample using MMT-SI, a method of transmitting the DTS for each sample using the MMT packet header extension area, etc. Note that the DTS may be an absolute value or a relative value with respect to the PTS. Also, it may be signaled whether the DTS of the first sample is included on the transmitting side.

[0225] Note that in both Method 1 and Method 2, the DTS of the subsequent samples is calculated assuming a fixed frame rate.

[0226] As a method of storing the DTS for each sample in the packet header, in addition to using the extension area, there is a method of storing the DTS of the sample included in the MMT packet in the 32-bit NTP timestamp field in the MMT packet header. When the DTS cannot be represented by the number of bits (32 bits) of one packet header, the DTS may be represented using a plurality of packet headers. Also, the DTS may be represented by combining the NTP timestamp field of the packet header and the extension area. When the DTS information is not included, it is set to a known value (for example, ALL0).

[0227] <Determination of PTS of Sample> The receiving device acquires the PTS of the first sample from the MPU timestamp descriptor for each asset included in the MPU. For the subsequent sample PTS, the receiving device calculates it from parameters indicating the display order of the samples, such as POC, assuming a fixed frame rate. Thus, in order to calculate the DTS and PTS without using the header information, transmission at a fixed frame rate is essential.

[0228] Also, when MF metadata is being transmitted, the receiving device can calculate the absolute values of DTS and PTS from the relative time information of DTS and PTS from the start sample indicated in the MF metadata and the absolute value of the timestamp of the MPU start sample indicated in the MPU timestamp descriptor.

[0229] Note that when calculating DTS and PTS by analyzing the encoded data, the receiving device may calculate using the SEI information included in the access unit.

[0230] <Initialization information (parameter set)> [In the case of video] In the case of video, the parameter set is stored in the sample data. Also, when MPU metadata and MF metadata are not transmitted, it is guaranteed that the parameter set necessary for decoding can be obtained by referring only to the sample data.

[0231] Also, as shown in FIGS. 21(a) and (d), when MPU metadata is transmitted before the media data, it may be stipulated that the parameter set is not stored in the SampleEntry. In this case, the receiving device refers only to the parameter set within the sample without referring to the parameter set of the SampleEntry.

[0232] Also, when MPU metadata is transmitted before the media data, a parameter set common to the MPU or a default parameter set is stored in the SampleEntry, and the receiving device may refer to the parameter set of the SampleEntry and the parameter set within the sample. By storing the parameter set in the SampleEntry, it becomes possible to perform decoding even with a conventional receiving device that cannot play back if there is no parameter set in the SampleEntry.

[0233] [In the case of audio] In the case of audio, a LATM header is required for decoding, and in MP4, it is essential that the LATM header be included in the sample entry. However, if the header information is not transmitted, it is difficult for the receiving device to obtain the LATM header. Therefore, the LATM header is included in control information such as SI separately. Note that the LATM header may be included in a message, table, or descriptor. Note also that the LATM header may be included within a sample.

[0234] Before starting decoding, the receiving device obtains the LATM header from SI or the like and starts decoding the audio. Alternatively, as shown in FIGS. 21(a) and 21(d), when the MPU metadata is transmitted before the media data, the receiving device can receive the LATM header before the media data. Therefore, when the MPU metadata is transmitted before the media data, decoding can be performed even using a conventional receiving device.

[0235] <Others> The transmission order and the type of transmission order may be notified as control information such as an MMT packet header, a payload header, or an MPT or other table, message, descriptor, etc. Note that the type of transmission order here is, for example, the four types of transmission orders shown in FIGS. 21(a) to 21(d), and an identifier for identifying each type may be stored in a place where it can be obtained before the start of decoding.

[0236] Also, different types of transmission orders may be used for audio and video, or the same type of transmission order may be used for audio and video. Specifically, for example, audio may be transmitted in the order of MPU metadata, MF metadata, and media data as shown in FIG. 21(a), and video may be transmitted in the order of MPU metadata, media data, and MF metadata as shown in FIG. 21(d).

[0237] By the method described above, the receiving device can perform decoding without using header information. Also, when the MPU metadata is transmitted before the media data (in (a) and (d) of FIG. 21), even a conventional receiving device can perform decoding.

[0238] In particular, since the MF metadata is transmitted after the media data (in (d) of FIG. 21), it is possible to avoid the delay caused by encapsulation and to perform decoding even with a conventional receiving device.

[0239] [Configuration and Operation of Transmitting Device] Next, the configuration and operation of the transmitting device will be described. FIG. 23 is a block diagram of the transmitting device according to Embodiment 2, and FIG. 24 is a flowchart of the transmitting method according to Embodiment 2.

[0240] As shown in FIG. 23, the transmitting device 15 includes an encoding unit 16, a multiplexing unit 17, and a transmitting unit 18.

[0241] The encoding unit 16 generates encoded data by encoding video or audio to be encoded according to, for example, H.265 (S10).

[0242] The multiplexing unit 17 multiplexes (packetizes) the encoded data generated by the encoding unit 16 (S11). Specifically, the multiplexing unit 17 packetizes each of the sample data, MPU metadata, and MF metadata that make up a file in the MP4 format. The sample data is data obtained by encoding a video signal or an audio signal, the MPU metadata is an example of the first metadata, and the MF metadata is an example of the second metadata. Both the first metadata and the second metadata are metadata used for decoding the sample data, but the difference between them is that the second metadata includes data that can be generated only after the generation of the sample data.

[0243] Here, data that can be generated only after the generation of sample data is, for example, data other than the sample data stored in mdat in the MP4 format (data in the header of mdat. That is, type and length illustrated in FIG. 20). Here, the second metadata may include at least a part of length of this data.

[0244] The transmission unit 18 transmits the packetized MP4 format file (S12). The transmission unit 18 transmits the MP4 format file by, for example, the method shown in (d) of FIG. 21. That is, the packetized MPU metadata, the packetized sample data, and the packetized MF metadata are transmitted in this order.

[0245] Each of the encoding unit 16, the multiplexing unit 17, and the transmission unit 18 is realized by a microcomputer, a processor, a dedicated circuit, or the like.

[0246] [Configuration of receiving device] Next, the configuration and operation of the receiving device will be described. FIG. 25 is a block diagram of the receiving device according to Embodiment 2.

[0247] As shown in FIG. 25, the receiving device 20 includes a packet filtering unit 21, a transmission order type determination unit 22, a random access unit 23, a control information acquisition unit 24, a data acquisition unit 25, a PTS, DTS calculation unit 26, an initialization information acquisition unit 27, a decoding command unit 28, a decoding unit 29, and a presentation unit 30.

[0248] [Operation 1 of receiving device] First, when the medium is video, the operation of the receiving device 20 for specifying the MPU start position and the NAL unit position will be described. FIG. 26 is a flowchart of such an operation of the receiving device 20. Here, it is assumed that the transmission order type of the MPU data is stored in the SI information by the transmission device 15 (multiplexing unit 17).

[0249] First, the packet filtering unit 21 performs packet filtering on the received file. The transmission order type determination unit 22 analyzes the SI information obtained by the packet filtering to acquire the transmission order type of the MPU data (S21).

[0250] Next, the transmission order type determination unit 22 determines (discriminates) whether the data after packet filtering contains MPU header information (at least one of MPU metadata or MF metadata) (S22). If the MPU header information is included (Yes in S22), the random access unit 23 identifies the MPU start sample by detecting that the fragment type of the MMT payload header switches to media data (S23).

[0251] On the other hand, if the MPU header information is not included (No in S22), the random access unit 23 identifies the MPU start sample based on the RAP_flag of the MMT packet header or the sample number of the MMT payload header (S24).

[0252] Also, the transmission order type determination unit 22 determines whether the packet-filtered data contains MF metadata (S25). If it is determined that the MF metadata is included (Yes in S25), the data acquisition unit 25 acquires the NAL unit by reading the NAL unit based on the samples, sub-sample offsets, and size information included in the MF metadata (S26). On the other hand, if it is determined that the MF metadata is not included (No in S25), the data acquisition unit 25 acquires the NAL unit by reading the data of the NAL unit size in order from the start NAL unit of the sample (S27).

[0253] Note that even when it is determined in step S22 that the MPU header information is included, the receiving device 20 may specify the MPU start sample using the process of step S24 instead of step S23. Also, when it is determined that the MPU header information is included, the processes of step S23 and step S24 may be used in combination.

[0254] Further, even when it is determined in step S25 that the MF metadata is included, the receiving device 20 may acquire the NAL unit using the process of step S27 without using the process of step S26. Also, when it is determined that the MF metadata is included, the processes of step S23 and step S24 may be used in combination.

[0255] Also, it is assumed that in the case where it is determined in step S25 that the MF metadata is included and the MF data is transmitted after the media data. In this case, the receiving device 20 may buffer the media data and wait until the MF metadata is acquired and then perform the process of step S26, or the receiving device 20 may determine whether to perform the process of step S27 without waiting for the acquisition of the MF metadata.

[0256] For example, the receiving device 20 may determine whether to wait for the acquisition of the MF metadata based on whether it has a buffer with a buffer size capable of buffering the media data. Also, the receiving device 20 may determine whether to wait for the acquisition of the MF metadata based on whether the end-to-end delay becomes smaller. Also, the receiving device 20 may mainly perform the decoding process using the process of step S26 and may use the process of step S27 in the case of the processing mode when packet loss or the like occurs.

[0257] In addition, when the transmission order type is predetermined, steps S22 and S26 may be omitted. In this case, the receiving device 20 may determine the method for specifying the MPU start sample and the method for specifying the NAL unit in consideration of the buffer size and the End-to-End delay.

[0258] In addition, when the transmission order type is known in advance, the transmission type order determination unit 22 in the receiving device 20 is unnecessary.

[0259] Also, although not described in FIG. 26 above, the decoding command unit 28 outputs the data acquired by the data acquisition unit to the decoding unit 29 based on the PTS and DTS calculated by the PTS, DTS calculation unit 26 and the initialization information acquired by the initialization information acquisition unit 27. The decoding unit 29 decodes the data, and the presentation unit 30 presents the decoded data.

[0260] [Operation of Receiving Device 2] Next, the operation in which the receiving device 20 acquires initialization information based on the transmission order type and decodes the media data based on the initialization information will be described. FIG. 27 is a flowchart of such an operation.

[0261] First, the packet filtering unit 21 performs packet filtering on the received file. The transmission order type determination unit 22 analyzes the SI information obtained by the packet filtering and acquires the transmission order type (S301).

[0262] Next, the transmission type order determination unit 22 determines whether MPU metadata has been transmitted (S302). If it is determined that the MPU metadata has been transmitted (Yes in S302), the transmission type order determination unit 22 determines whether the MPU metadata has been transmitted before the media data as a result of the analysis in step S301 (S303). If the MPU metadata has been transmitted before the media data (Yes in S303), the initialization information acquisition unit 27 decrypts the media data based on the common initialization information included in the MPU metadata and the initialization information of the sample data (S304).

[0263] On the other hand, if it is determined that the MPU metadata has been transmitted after the media data (No in S303), the data acquisition unit 25 buffers the media data until the MPU metadata is acquired (S305), and performs the process of step S304 after the MPU metadata is acquired.

[0264] Also, in step S302, if it is determined that the MPU metadata has not been transmitted (No in S302), the initialization information acquisition unit 27 decrypts the media data based only on the initialization information of the sample data (S306).

[0265] Note that if decryption of the media data is guaranteed only when based on the initialization information of the sample data on the transmission side, the processes based on the determinations in step S302 and step S303 are not performed, and the process of step S306 is used.

[0266] Further, before step S305, the receiving device 20 may determine whether to buffer the media data. In this case, if the receiving device 20 determines to buffer the media data, it proceeds to the process of step S305, and if it determines not to buffer the media data, it proceeds to the process of step S306. The determination of whether to buffer the media data may be made based on the buffer size and occupancy of the receiving device 20, or for example, the determination may be made considering the End-to-End delay, such as selecting the one with a smaller End-to-End delay.

[0267] [Operation of Receiving Device 3] Here, the details of the transmission method and reception method when the MF metadata is transmitted after the media data (in (c) and (d) of FIG. 21) will be described. Hereinafter, the case of (d) in FIG. 21 will be described as an example. In transmission, it is assumed that only the method of (d) in FIG. 21 is used and no signaling of the transmission order type is performed.

[0268] As described above, when data is transmitted in the order of MPU metadata, media data, and MF metadata as shown in (d) of FIG. 21, (D-1) After the receiving device 20 acquires the MPU metadata and further acquires the MF metadata, it decrypts the media data.

[0269] (D-2) After the receiving device 20 acquires the MPU metadata, it decrypts the media data without using the MF metadata.

[0270] These two decryption methods are possible.

[0271] Here, although D-1 requires buffering of media data for MF metadata acquisition, it can be decoded using the MPU header information, so it can be decoded by a conventional MP4-compliant receiver. Also, although D-2 does not require buffering of media data for MF metadata acquisition, it cannot be decoded using MF metadata, so special processing is required for decoding.

[0272] Also, the method of (d) in FIG. 21 has the advantage that since the MF metadata is transmitted after the media data, no delay due to encapsulation occurs and the End-to-End delay can be reduced.

[0273] The receiving device 20 can select the above two decoding methods according to the capabilities of the receiving device 20 and the quality of service provided by the receiving device 20.

[0274] The transmitting device 15 must ensure that it can decode while reducing the occurrence of buffer overflow and underflow in the decoding operation of the receiving device 20. As elements for defining the decoder model when decoding using the method of D-1, for example, the following parameters can be used.

[0275] · Buffer size for reconstructing the MPU (MPU buffer) For example, the buffer size = maximum rate × maximum MPU time × α, where the maximum rate is the upper limit rate of the profile and level of the encoded data + the overhead of the MPU header. Also, the maximum MPU time is the maximum time length of the GOP when 1 MPU = 1 GOP (video).

[0276] Here, the audio may be in the same GOP unit as the above video or in another unit. α is a margin for not causing an overflow and may be multiplied or added to the maximum rate × maximum MPU time. When multiplied, α ≧ 1, and when added, α ≧ 0.

[0277] ·Upper limit of the decoding delay time from when data is input to the MPU buffer until it is decoded. (TSTD_delay in the STD of MPEG-TS) For example, at the time of transmission, considering the maximum MPU time and the upper limit value of the decoding delay time, the DTS is set so that the acquisition completion time of the MPU data in the receiver <= DTS.

[0278] Also, when the transmission device 15 decodes using the D-1 method, it may assign DTS and PTS according to the decoder model. Thereby, the transmission device 15 guarantees the decoding of the receiving device that decodes using the D-1 method, and at the same time, it may transmit the auxiliary information required when decoding is performed using the D-2 method.

[0279] For example, the transmission device 15 can guarantee the operation of the receiving device that decodes using the D-2 method by signaling the pre-buffering time in the decoder buffer when decoding using the D-2 method.

[0280] The pre-buffering time may be included in SI control information such as messages, tables, and descriptors, or may be included in the headers of MMT packets and MMT payloads. Also, the SEI in the encoded data may be overwritten. The DTS and PTS for decoding using the D-1 method are stored in the MPU time stamp descriptor and SamplleEntry, and the DTS and PTS for decoding using the D-2 method, or the pre-buffering time, may be described in the SEI.

[0281] When the receiving device 20 only supports the MP4-compliant decoding operation using the MPU header, the receiving device 20 selects the decoding method D-1, and when it supports both D-1 and D-2, it may select either one.

[0282] The transmission device 15 may assign DTS and PTS so as to guarantee the decoding operation of one (in this description, D-1), and may also transmit auxiliary information for assisting the decoding operation of the other.

[0283] Also, when the method of D-2 is used, compared with the case where the method of D-1 is used, the end-to-end delay is likely to increase due to the delay caused by the pre-buffering of the MF metadata. Therefore, when the receiving device 20 wants to reduce the end-to-end delay, it may select and decode using the method of D-2. For example, when the receiving device 20 always wants to reduce the end-to-end delay, it may always use the method of D-2. Also, the receiving device 20 may use the method of D-2 only when operating in a low-latency presentation mode where live content, channel selection, zapping operations, etc. need to be presented with low latency.

[0284] FIG. 28 is a flowchart of such a receiving method.

[0285] First, the receiving device 20 receives an MMT packet and acquires MPU data (S401). Then, the receiving device 20 (transmission order type determination unit 22) determines whether to present the program in the low-latency presentation mode (S402).

[0286] When the program is not presented in the low-latency presentation mode (No in S402), the receiving device 20 (random access unit 23 and initialization information acquisition unit 27) performs random access using the header information to acquire initialization information (S405). Also, the receiving device 20 (PTS, DTS calculation unit 26, decoding command unit 28, decoding unit 29, presentation unit 30) performs decoding and presentation processing based on the PTS and DTS given on the transmission side (S406).

[0287] On the other hand, when presenting the program in the low-latency presentation mode (Yes in S402), the receiving device 20 (random access unit 23 and initialization information acquisition unit 27) uses a decoding method without using header information to acquire random access and initialization information (S403). Further, the receiving device 20 performs decoding and presentation processing based on auxiliary information for decoding without using the PTS, DTS, and header information given on the transmission side (S404). Note that in steps S403 and S404, the processing may be performed using MPU metadata.

[0288] [Transmission and reception method using auxiliary data] As described above, the transmission and reception operations in the case where the MF metadata is transmitted after the media data (the cases of (c) and (d) in FIG. 21) have been described. Next, a method will be described in which the transmission device 15 transmits auxiliary data having some functions of the MF metadata, so that decoding can be started earlier and the End-to-End delay can be reduced. Here, an example in which auxiliary data is further transmitted based on the transmission method shown in (d) of FIG. 21 will be described, but the method using auxiliary data is also applicable to the transmission methods shown in (a) to (c) of FIG. 21.

[0289] FIG. 29(a) is a diagram showing an MMT packet transmitted using the method shown in (d) of FIG. 21. That is, the data is transmitted in the order of MPU metadata, media data, and MF metadata.

[0290] Here, sample #1, sample #2, sample #3, and sample #4 are samples included in the media data. Note that here, an example in which the media data is stored in MMT packets in sample units is described, but the media data may be stored in MMT packets in NAL unit units, or may be stored in units obtained by dividing NAL units. Note that there may be a case where a plurality of NAL units are aggregated and stored in an MMT packet.

[0291] As described in the above D-1, in the case of the method shown in (d) of FIG. 21, that is, when data is transmitted in the order of MPU metadata, media data, and MF metadata, after acquiring the MPU metadata, the MF metadata is further acquired, and then there is a method of decrypting the media data. In such a D-1 method, buffering of media data for acquiring MF metadata is required. However, since decryption is performed using the MPU header information, the D-1 method has the advantage of being applicable to conventional MP4-compliant receiving devices. On the other hand, the receiving device 20 has the disadvantage that it has to wait until the MF metadata is acquired before starting decryption.

[0292] In contrast, as shown in (b) of FIG. 29, in the method using auxiliary data, the auxiliary data is transmitted before the MF metadata.

[0293] The MF metadata includes information indicating the DTS, PTS, offset, and size of all samples included in the movie fragment. In contrast, the auxiliary data includes information indicating the DTS, PTS, offset, and size of some of the samples included in the movie fragment.

[0294] For example, the MF metadata includes information of all samples (sample #1 - sample #4), while the auxiliary data includes information of some samples (sample #1 - #2).

[0295] In the case shown in (b) of FIG. 29, since the auxiliary data is used, samples #1 and #2 can be decrypted, so the End-to-End delay is reduced compared to the D-1 transmission method. Note that the auxiliary data may include sample information combined in any way, and the auxiliary data may be transmitted repeatedly.

[0296] For example, in (c) of FIG. 29, when transmitting auxiliary information at timing A, the transmission device 15 includes the information of sample #1 in the auxiliary information. When transmitting auxiliary information at timing B, the transmission device 15 includes the information of sample #1 and sample #2 in the auxiliary information. When transmitting auxiliary information at timing C, the transmission device 15 includes the information of sample #1, sample #2, and sample #3 in the auxiliary information.

[0297] Note that the MF metadata includes the information of sample #1, sample #2, sample #3, and sample #4 (the information of all samples in the movie fragment).

[0298] The auxiliary data does not necessarily need to be transmitted immediately after generation.

[0299] Note that in the header of the MMT packet or MMT payload, a type indicating that auxiliary data is stored is specified.

[0300] For example, when the auxiliary data is stored in the MMT payload using the MPU mode, a data type indicating that it is auxiliary data is specified as the fragment_type field value (for example, FT = 3). The auxiliary data may be data based on the moof configuration or other configurations.

[0301] When the auxiliary data is stored in the MMT payload as a control signal (descriptor, table, message), a descriptor tag, table ID, message ID, etc. indicating that it is auxiliary data are specified.

[0302] Also, PTS or DTS may be stored in the header of the MMT packet or MMT payload.

[0303] [Example of generating auxiliary data] Hereinafter, an example in which the transmission device generates auxiliary data based on the moof configuration will be described. FIG. 30 is a diagram for explaining an example in which the transmission device generates auxiliary data based on the moof configuration.

[0304] In a normal MP4, as shown in FIG. 20, a moof is created for a movie fragment. The moof contains information indicating the DTS, PTS, offset, and size of the samples included in the movie fragment.

[0305] Here, the transmission device 15 constructs an MP4 (MP4 file) using only some of the sample data among the sample data that constitutes the MPU, and generates auxiliary data.

[0306] For example, as shown in FIG. 30(a), the transmission device 15 generates an MP4 using only sample #1 among samples #1 - #4 that constitute the MPU, and among them, uses the moof + mdat header as auxiliary data.

[0307] Next, as shown in FIG. 30(b), the transmission device 15 generates an MP4 using samples #1 and #2 among samples #1 - #4 that constitute the MPU, and among them, uses the moof + mdat header as the next auxiliary data.

[0308] Next, as shown in FIG. 30(c), the transmission device 15 generates an MP4 using samples #1, #2, and #3 among samples #1 - #4 that constitute the MPU, and among them, uses the moof + mdat header as the next auxiliary data.

[0309] Next, as shown in FIG. 30(d), the transmission device 15 generates all MP4s among samples #1 - #4 that constitute the MPU, and among them, the moof + mdat header becomes movie fragment metadata.

[0310] Note that here, the transmission device 15 generates auxiliary data for each sample, but it may also generate auxiliary data for every N samples. The value of N is an arbitrary number. For example, when transmitting auxiliary data M times when transmitting one MPU, N = total samples / M may also be used.

[0311] Note that the information indicating the offset of the sample in moof may be the offset value after the sample entry area of the subsequent number of samples is secured as a NULL area.

[0312] Note that auxiliary data may be generated so as to form a configuration for fragmenting MF metadata.

[0313] [Example of receiving operation using auxiliary data] The reception of the auxiliary data generated as described in FIG. 30 will be described. FIG. 31 is a diagram for explaining the reception of the auxiliary data. In FIG. 31(a), it is assumed that the number of samples constituting the MPU is 30, and the auxiliary data is generated and transmitted every 10 samples.

[0314] In FIG. 30(a), the auxiliary data #1 includes the sample information of samples #1 - #10, the auxiliary data #2 includes the sample information of samples #1 - #20, and the MF metadata includes the sample information of samples #1 - #30.

[0315] Note that samples #1 - #10, samples #11 - #20, and samples #21 - #30 are stored in one MMT payload, but they may be stored in units of samples or NALs, or may be stored in units of fragments or aggregations.

[0316] The receiving device 20 receives the packets of the MPU meta, samples, MF meta, and auxiliary data respectively.

[0317] The receiving device 20 concatenates the sample data in the order of reception (at the back), and updates the auxiliary data so far after receiving the latest auxiliary data. Further, the receiving device 20 can configure a complete MPU by finally replacing the auxiliary data with the MF metadata.

[0318] When the receiving device 20 receives the auxiliary data #1, it concatenates the data as shown in the upper part of (b) in FIG. 31 to form an MP4. As a result, the receiving device 20 can parse Samples #1 - #10 using the MPU metadata and the information of the auxiliary data #1, and can perform decoding based on the PTS, DTS, offset, and size information included in the auxiliary data.

[0319] Also, when the receiving device 20 receives the auxiliary data #2, it concatenates the data as shown in the middle part of (b) in FIG. 31 to form an MP4. As a result, the receiving device 20 can parse Samples #1 - #20 using the MPU metadata and the information of the auxiliary data #2, and can perform decoding based on the PTS, DTS, offset, and size information included in the auxiliary data.

[0320] Also, when the receiving device 20 receives the MF metadata, it concatenates the data as shown in the lower part of (b) in FIG. 31 to form an MP4. As a result, the receiving device 20 can parse Samples #1 - #30 using the MPU metadata and the MF metadata, and can perform decoding based on the PTS, DTS, offset, and size information included in the MF metadata.

[0321] In the case where there is no auxiliary data, since the receiving device 20 can obtain sample information for the first time after receiving the MF metadata, it was necessary to start decoding after receiving the MF metadata. However, by the transmitting device 15 generating and transmitting the auxiliary data, the receiving device 20 can obtain sample information using the auxiliary data without waiting for the reception of the MF metadata, so the decoding start time can be advanced. Furthermore, by the transmitting device 15 generating the auxiliary data based on the moof described with reference to FIG. 30, the receiving device 20 can use the conventional MP4 parser as it is and perform parsing.

[0322] In addition, the newly generated auxiliary data and MF metadata include information on samples that overlap with the auxiliary data transmitted in the past. Therefore, even if the past auxiliary data cannot be obtained due to packet loss or the like, it is possible to reconstruct the MP4 and obtain sample information (PTS, DTS, size, and offset) by using the newly obtained auxiliary data and MF metadata.

[0323] Note that the auxiliary data does not necessarily have to include information on past sample data. For example, auxiliary data #1 may correspond to sample data #1-#10, and auxiliary data #2 may correspond to sample data #11-#20. For example, as shown in (c) of FIG. 31, the transmission device 15 may sequentially transmit the complete MF metadata as a data unit and the fragmented unit of the data unit as auxiliary data.

[0324] In addition, the transmission device 15 may repeatedly transmit the auxiliary data or repeatedly transmit the MF metadata for packet loss countermeasures.

[0325] Note that the MMT packet and MMT payload in which the auxiliary data is stored include an MPU sequence number and an asset ID, similar to the MPU metadata, MF metadata, and sample data.

[0326] The reception operation using the auxiliary data as described above will be described with reference to the flowchart of FIG. 32. FIG. 32 is a flowchart of the reception operation using the auxiliary data.

[0327] First, the receiving device 20 receives the MMT packet and analyzes the packet header and payload header (S501). Next, the receiving device 20 analyzes whether the fragment type is auxiliary data or MF metadata (S502). If the fragment type is auxiliary data, the receiving device 20 overwrites and updates the past auxiliary data (S503). At this time, if there is no past auxiliary data of the same MPU, the receiving device 20 uses the received auxiliary data as new auxiliary data as it is. Then, the receiving device 20 acquires samples based on the MPU metadata, auxiliary data, and sample data, and performs decoding (S507).

[0328] On the other hand, if the fragment type is MF metadata, the receiving device 20 overwrites the past auxiliary data with the MF metadata in step S505 (S505). Then, the receiving device 20 acquires samples in the form of a complete MPU based on the MPU metadata, MF metadata, and sample data, and performs decoding (S506).

[0329] Although not shown in FIG. 32, in step S502, if the fragment type is MPU metadata, the receiving device 20 stores the data in the buffer, and if it is sample data, it stores the data concatenated at the back for each sample in the buffer.

[0330] If the auxiliary data cannot be acquired due to packet loss, the receiving device 20 can overwrite with the latest auxiliary data or decode the samples using the past auxiliary data.

[0331] Note that the transmission period and number of transmissions of the auxiliary data may be predetermined values. Information on the transmission period and number of transmissions (count, countdown) may be transmitted together with the data. For example, a time stamp such as the transmission period, number of transmissions, and initial_cpb_removal_delay may be stored in the data unit header.

[0332] By transmitting the auxiliary data including the information of the first sample of the MPU one or more times before the initial_cpb_removal_delay, it becomes possible to comply with the CPB buffer model. At this time, a value based on the picture timing SEI is stored in the MPU timestamp descriptor.

[0333] Note that the transmission method in the reception operation where such auxiliary data is used is not limited to the MMT method, and is applicable to cases such as streaming transmission of packets configured in the ISOBMFF file format such as MPEG-DASH.

[0334] [Transmission method when one MPU is composed of multiple movie fragments] In the description after FIG. 19 above, one MPU was composed of one movie fragment, but here, the case where one MPU is composed of multiple movie fragments will be described. FIG. 33 is a diagram showing the configuration of an MPU composed of multiple movie fragments.

[0335] In FIG. 33, the samples (#1-#6) stored in one MPU are divided and stored in two movie fragments. The first movie fragment is generated based on samples #1-#3, and the corresponding moof box is generated. The second movie fragment is generated based on samples #4-#6, and the corresponding moof box is generated.

[0336] The headers of the moof box and mdat box in the first movie fragment are stored in the MMT payload and MMT packet as movie fragment metadata #1. On the other hand, the headers of the moof box and mdat box in the second movie fragment are stored in the MMT payload and MMT packet as movie fragment metadata #2. In FIG. 33, the MMT payload in which the movie fragment metadata is stored is hatched.

[0337] Note that the number of samples constituting the MPU and the number of samples constituting the movie fragment are arbitrary. For example, the number of samples constituting the MPU may be the number of samples in GOP units, and two movie fragments may be constituted with the number of samples that is half of the number of samples in GOP units as the movie fragment.

[0338] Note that, here, an example in which one MPU includes two movie fragments (moof box and mdat box) is shown. However, the number of movie fragments included in one MPU may not be two but three or more. Also, the samples stored in the movie fragment may be divided into arbitrary numbers of samples instead of equal numbers of samples.

[0339] Note that, in FIG. 33, the MPU metadata unit and the MF metadata unit are each stored in the MMT payload as data units. However, the transmission device 15 may store units such as ftyp, mmpu, moov, and moof in the MMT payload in data unit units, or may store the data units in the MMT payload in units obtained by fragmenting the data units. Also, the transmission device 15 may store the data units in the MMT payload in units obtained by aggregating the data units.

[0340] Also, in FIG. 33, the samples are stored in the MMT payload in sample units. However, the transmission device 15 may constitute data units in NAL unit units or units obtained by grouping a plurality of NAL units instead of sample units, and store the data units in the MMT payload in data unit units. Also, the transmission device 15 may store the data units in the MMT payload in units obtained by fragmenting the data units, or may store the data units in the MMT payload in units obtained by aggregating the data units.

[0341] In addition, in FIG. 33, the MPU is configured in the order of moof#1, mdat#1, moof#2, mdat#2, and an offset is given to moof#1 assuming that the corresponding mdat#1 is attached at the rear. However, the offset may be given assuming that mdat#1 is attached before moof#1. However, in this case, movie fragment metadata cannot be generated in the form of moof+mdat, and the headers of moof and mdat are transmitted separately.

[0342] Next, the transmission order of the MMT packet when the MPU having the configuration described with reference to FIG. 33 is transmitted will be described. FIG. 34 is a diagram for explaining the transmission order of the MMT packet.

[0343] FIG. 34(a) shows the transmission order when the MMT packet is transmitted in the configuration order of the MPU shown in FIG. 33. Specifically, FIG. 34(a) shows an example of transmitting in the order of MPU meta, MF meta#1, media data#1 (samples #1-#3), MF meta#2, and media data#2 (samples #4-#6).

[0344] FIG. 34(b) shows an example of transmitting in the order of MPU meta, media data#1 (samples #1-#3), MF meta#1, media data#2 (samples #4-#6), and MF meta#2.

[0345] FIG. 34(c) shows an example of transmitting in the order of media data#1 (samples #1-#3), MPU meta, MF meta#1, media data#2 (samples #4-#6), and MF meta#2.

[0346] MF meta#1 is generated using samples #1-#3, and MF meta#2 is generated using samples #4-#6. Therefore, when the transmission method of FIG. 34(a) is used, a delay due to encapsulation occurs in the transmission of the sample data.

[0347] On the other hand, when the transmission methods shown in FIGS. 34(b) and 34(c) are used, samples can be transmitted without waiting to generate MF meta, so there is no delay due to encapsulation and the End-to-End delay can be reduced.

[0348] Also, in the transmission order shown in FIG. 34(a), since one MPU is divided into a plurality of movie fragments and the number of samples stored in MF meta is smaller than that in the case of FIG. 19, the amount of delay due to encapsulation can be made smaller than in the case of FIG. 19.

[0349] In addition to the methods shown here, for example, the transmission device 15 may concatenate MF meta #1 and MF meta #2 and transmit them together at the end of the MPU. In this case, MF meta of different movie fragments may be aggregated and stored in one MMT payload. Also, MF meta of different MPUs may be aggregated together and stored in the MMT payload.

[0350] [Receiving Method in the Case where One MPU is Composed of a Plurality of Movie Fragments] Here, an operation example of the receiving device 20 that receives and decodes an MMT packet transmitted in the transmission order described in FIG. 34(b) will be described. FIGS. 35 and 36 are diagrams for explaining such an operation example.

[0351] The receiving device 20 receives MMT packets including MPU meta, samples, and MF meta transmitted in the transmission order shown in FIG. 35, respectively. The sample data is concatenated in the order of reception.

[0352] At time T1 when the receiving device 20 receives MF meta #1, the receiving device 20 concatenates the data as shown in FIG. 36(1) to form an MP4. Thereby, the receiving device 20 can acquire samples #1-#3 based on the MPU meta data and the information of MF meta #1, and can perform decoding based on the PTS, DTS, offset, and size information included in the MF meta.

[0353] Also, at time T2 which is the time when the receiving device 20 receives MF meta #2, the receiving device 20 concatenates data as shown in (2) of FIG. 36 to form MP4. Thereby, the receiving device 20 can acquire samples #4-#6 based on the MPU metadata and the information of MF meta #2, and can perform decoding based on the PTS, DTS, offset, and size information of the MF meta. Further, the receiving device 20 may concatenate data as shown in (3) of FIG. 36 to form MP4, and thereby acquire samples #1-#6 based on the information of MF meta #1 and MF meta #2.

[0354] By dividing one MPU into a plurality of movie fragments, the time until the first MF meta in the MPU is acquired is shortened, so that the decoding start time can be advanced. Also, the buffer size for accumulating samples before decoding can be reduced.

[0355] Note that the transmission device 15 may set the division unit of the movie fragment so that the time from when the first sample in the movie fragment is transmitted (or received) to when the MF meta corresponding to the movie fragment is transmitted (or received) is shorter than the initial_cpb_removal_delay specified by the encoder. By setting in this way, the reception buffer can follow the cpb buffer, and low-latency decoding can be realized. In this case, absolute times based on the initial_cpb_removal_delay can be used for PTS and DTS.

[0356] Also, the transmission device 15 may divide the movie fragment at equal intervals, or divide subsequent movie fragments at an interval shorter than that of the previous movie fragment. Thereby, the receiving device 20 can surely receive the MF meta including the information of the sample before decoding the sample, and continuous decoding becomes possible.

[0357] The following two methods can be used as the calculation methods for the absolute times of PTS and DTS.

[0358] (1) The absolute timestamps of PTS and DTS are determined based on the reception times (T1 or T2) of MF meta #1 and MF meta #2, and the relative timestamps of PTS and DTS included in the MF meta.

[0359] (2) The absolute timestamps of PTS and DTS are determined based on the absolute timestamps signaled from the transmission side, such as the MPU timestamp descriptor, and the relative timestamps of PTS and DTS included in the MF meta.

[0360] Also, (2 - A) the absolute timestamp signaled by the transmission device 15 may be an absolute timestamp calculated based on the initial_cpb_removal_delay specified by the encoder.

[0361] Also, (2 - B) the absolute timestamp signaled by the transmission device 15 may be an absolute timestamp calculated based on the predicted value of the reception time of the MF meta.

[0362] Note that MF meta #1 and MF meta #2 may be repeatedly transmitted. By repeatedly transmitting MF meta #1 and MF meta #2, the receiving device 20 can acquire them again even if it fails to acquire the MF meta due to packet loss or the like.

[0363] The payload header of the MFU including the samples constituting the movie fragment can store an identifier indicating the order of the movie fragments. On the other hand, the identifier indicating the order of the MF meta constituting the movie fragment is not included in the MMT payload. For this reason, the receiving device 20 identifies the order of the MF meta by the packet_sequence_number. Alternatively, the transmission device 15 may store and signal an identifier indicating to which movie fragment the MF meta belongs in the control information (message, table, descriptor), MMT header, MMT payload header, or data unit header.

[0364] Note that the transmission device 15 may transmit the MPU meta, MF meta, and samples in a predetermined transmission order, and the reception device 20 may perform reception processing based on the predetermined transmission order. Further, the transmission device 15 may signal the transmission order, and the reception device 20 may select (determine) the reception processing based on the signaling information.

[0365] The above-described reception method will be described with reference to FIG. 37. FIG. 37 is a flowchart of the operation of the reception method described in FIGS. 35 and 36.

[0366] First, the reception device 20 discriminates (identifies) whether the data included in the payload is MPU metadata, MF metadata, or sample data (MFU) based on the fragment type indicated in the MMT payload (S601, S602). If the data is sample data, the reception device 20 buffers the sample and waits for the reception and decoding start of the MF metadata corresponding to the sample (S603).

[0367] On the other hand, in step S602, if the data is MF metadata, the reception device 20 acquires sample information (PTS, DTS, position information, and size) from the MF metadata, acquires a sample based on the acquired sample information, and decodes and presents the sample based on PTS and DTS (S604).

[0368] Although not shown, when the data is MPU metadata, the MPU metadata includes initialization information necessary for decoding. Therefore, the reception device 20 accumulates this and uses it for decoding the sample data in step S604.

[0369] Note that when the reception device 20 accumulates the received MPU data (MPU metadata, MF metadata, and sample data) in the storage device, it accumulates the data after rearranging it in the MPU configuration described in FIG. 19 or FIG. 33.

[0370] On the transmitting side, a packet sequence number is assigned to packets having the same packet ID in the MMT packet. At this time, the packet sequence number may be assigned after the MMT packets including the MPU metadata, MF metadata, and sample data are sorted in the transmission order, or may be assigned in the order before sorting.

[0371] When the packet sequence number is assigned in the order before sorting, in the receiving device 20, the data can be sorted in the configuration order of the MPU based on the packet sequence number, which facilitates accumulation.

[0372] [Method for Detecting the Start of an Access Unit and the Start of a Slice Segment] A method for detecting the start of an access unit and the start of a slice segment based on the information in the MMT packet header and the MMT payload header will be described.

[0373] Here, two examples are shown: the case where non-VCL NAL units (such as access unit delimiters, VPS, SPS, PPS, and SEI) are collectively stored in the MMT payload as data units, and the case where non-VCL NAL units are each used as data units and the data units are aggregated and stored in one MMT payload.

[0374] FIG. 38 is a diagram showing the case where non-VCL NAL units are used as individual data units and aggregated.

[0375] In the case of FIG. 38, the start of the access unit is the start data of the MMT payload including the data unit in which the fragment_type value is MFU, the aggregation_flag value is 1, and the offset value is 0 in the MMT packet. At this time, the Fragmentation_indicator value is 0.

[0376] Also, in the case of FIG. 38, the start of the slice segment is the start data of the MMT payload of the MMT packet with the fragment_type value being MFU, and the aggregation_flag value being 0, and the fragmentation_indicator value being 00 or 01.

[0377] FIG. 39 is a diagram showing the case where non-VCL NAL units are grouped into data units. Note that the field values of the packet header are as shown in FIG. 17 (or FIG. 18).

[0378] In the case of FIG. 39, the start of the access unit is the start data of the payload in the packet with the Offset value being 0.

[0379] Also, in the case of FIG. 39, the start of the slice segment is the start data of the payload of the packet with the Offset value being different from 0 and the fragmentation indicator value being 00 or 01.

[0380] [Receiving Process in Case of Packet Loss] Normally, when transmitting MP4 format data in an environment where packet loss occurs, the receiving device 20 restores the packets by means of ALFEC (Application Layer FEC), packet retransmission control, etc.

[0381] However, when packet loss occurs in streaming such as broadcasting where AL-FEC is not used, the packets cannot be restored.

[0382] After the receiving device 20 loses data due to packet loss, it is necessary to resume decoding video and audio again. For this purpose, the receiving device 20 needs to detect the start of the access unit or NAL unit and start decoding from the start of the access unit or NAL unit.

[0383] However, since the start code is not attached to the head of the NAL unit in the MP4 format, the receiving device 20 cannot detect the head of the access unit or the NAL unit even if it analyzes the stream.

[0384] Figure 40 is a flowchart of the operation of the receiving device 20 when packet loss occurs.

[0385] The receiving device 20 detects packet loss by means of the Packet sequence number, packet counter, fragment counter, etc. in the header of the MMT packet or MMT payload (S701), and determines which packet has disappeared from the context before and after (S702).

[0386] If it is determined that no packet loss has occurred (No in S702), the receiving device 20 constructs an MP4 file and decodes the access unit or NAL unit (S703).

[0387] If it is determined that packet loss has occurred (Yes in S702), the receiving device 20 generates an NAL unit corresponding to the NAL unit with packet loss using dummy data, and constructs an MP4 file. When the receiving device 20 inserts dummy data into the NAL unit, it indicates that the NAL unit type is dummy data.

[0388] In addition, the receiving device 20 can resume decoding by detecting the head of the next access unit or NAL unit based on the methods described in FIGS. 17, 18, 38, and 39, and inputting the head data to the decoder (S705).

[0389] In the case where packet loss occurs, the receiving device 20 may resume decoding from the beginning of the access unit and the NAL unit based on the information detected based on the packet header, or may resume decoding from the beginning of the access unit and the NAL unit based on the header information of the reconstructed MP4 file including the NAL unit of dummy data.

[0390] When the receiving device 20 stores the MP4 file (MPU), the packet data (such as NAL units) lost due to packet loss may be separately acquired from broadcasting or communication and stored (replaced).

[0391] At this time, when the receiving device 20 acquires the lost packet from communication, it notifies the server of the information of the lost packet (such as packet ID, MPU sequence number, packet sequence number, IP data flow number, and IP address), and acquires the packet. The receiving device 20 may acquire not only the lost packet but also a group of packets before and after the lost packet at the same time.

[0392] [Method for Configuring Movie Fragments] Here, the method for configuring movie fragments will be described in detail.

[0393] As described with reference to FIG. 33, the number of samples constituting the movie fragment and the number of movie fragments constituting one MPU are arbitrary. For example, the number of samples constituting the movie fragment and the number of movie fragments constituting one MPU may be a predetermined fixed number or may be dynamically determined.

[0394] Here, by configuring the movie fragment so as to satisfy the following conditions on the transmission side (transmission device 15), low-latency decoding in the receiving device 20 can be guaranteed.

[0395] The conditions are as follows.

[0396] The transmitting device 15 generates and transmits MF metadata with a movie fragment as a unit obtained by dividing sample data so that the receiving device 20 can always receive an MF meta including information of the sample before the decoding time (DTS(i)) of an arbitrary sample (Sample(i)).

[0397] Specifically, the transmitting device 15 constructs a movie fragment using the encoded sample (including the i-th sample) before DTS(i).

[0398] As a method for dynamically determining the number of samples constituting a movie fragment or the number of movie fragments constituting one MPU so as to guarantee low-latency decoding, for example, the following method is used.

[0399] (1) At the start of decoding, the decoding time DTS(0) of the sample Sample(0) at the head of the GOP is a time based on initial_cpb_removal_delay. The transmitting device constructs the first movie fragment using the encoded sample before the time of DTS(0). Further, the transmitting device 15 generates MF metadata corresponding to the first movie fragment and transmits it before the time of DTS(0).

[0400] (2) The transmitting device 15 also constructs movie fragments so as to satisfy the above conditions for the subsequent samples.

[0401] For example, when the sample at the head of the movie fragment is the k-th sample, the MF meta of the movie fragment including the k-th sample is transmitted by the decoding time DTS(k) of the k-th sample. When the encoding completion time of the l-th sample is before DTS(k) and the encoding completion time of the (l + 1)-th sample is after DTS(k), the transmitting device 15 constructs a movie fragment using the k-th sample to the l-th sample.

[0402] Note that the transmission device 15 may also configure a movie fragment using samples from the k-th sample to samples less than the l-th sample.

[0403] (3) After the encoding of the last sample of the MPU is completed, the transmission device 15 configures a movie fragment using the remaining samples, generates MF metadata corresponding to the movie fragment, and transmits it.

[0404] Note that the transmission device 15 may configure a movie fragment using some of the samples that have been encoded instead of using all the samples that have been encoded.

[0405] Note that in the above, an example is shown in which the number of samples for configuring a movie fragment and the number of movie fragments constituting one MPU are dynamically determined based on the above conditions so as to ensure low-latency decoding. However, the method for determining the number of samples and the number of movie fragments is not limited to such a method. For example, the number of movie fragments constituting one MPU may be fixed to a predetermined value, and the number of samples may be determined so as to satisfy the above conditions. Also, the number of movie fragments constituting one MPU and the time for dividing the movie fragment (or the amount of code of the movie fragment) may be fixed to predetermined values, and the number of samples may be determined so as to satisfy the above conditions.

[0406] Also, when the MPU is divided into a plurality of movie fragments, information indicating whether the MPU is divided into a plurality of movie fragments, the attributes of the divided movie fragments, or the attributes of the MF meta for the divided movie fragments may be transmitted.

[0407] Here, the attributes of the movie fragment are information indicating whether the movie fragment is the first movie fragment of the MPU, the last movie fragment of the MPU, or other movie fragments, and so on.

[0408] The attribute of the MF meta indicates information such as whether the MF meta corresponds to the movie fragment at the beginning of the MPU, the movie fragment at the end of the MPU, or the movie fragment corresponding to other movie fragments.

[0409] Note that the transmitting device 15 may store and transmit, as control information, the number of samples constituting the movie fragment and the number of movie fragments constituting one MPU.

[0410] [Operation of the receiving device] The operation of the receiving device 20 based on the movie fragment configured as described above will be described.

[0411] The receiving device 20 determines the absolute time of each of the PTS and DTS based on the absolute time signaled from the transmitting side, such as the MPU time stamp descriptor, and the relative time of the PTS and DTS included in the MF meta.

[0412] Based on the information on whether the MPU is divided into a plurality of movie fragments, when the MPU is divided, the receiving device 20 performs processing as follows based on the attributes of the divided movie fragments.

[0413] (1) When the movie fragment is the movie fragment at the beginning of the MPU, the receiving device 20 uses the absolute time of the PTS of the first sample included in the MPU time stamp descriptor and the relative time of the PTS and DTS included in the MF meta to generate the absolute time of the PTS and DTS.

[0414] (2) When the movie fragment is not the movie fragment at the beginning of the MPU, the receiving device 20 generates the absolute time of the PTS and DTS using the relative time of the PTS and DTS included in the MF meta without using the information in the MPU time stamp descriptor.

[0415] (3) When the movie fragment is the last movie fragment of the MPU, after calculating the absolute time of the PTS and DTS of all samples, the receiving device 20 resets the calculation process (relative time addition process) of the PTS and DTS. Note that the reset process may be performed in the movie fragment at the head of the MPU.

[0416] The receiving device 20 may determine whether the movie fragment is divided as follows. Also, the receiving device 20 may acquire the attribute information of the movie fragment as follows.

[0417] For example, the receiving device 20 may determine whether it is divided based on the identifier movie_fragment_sequence_number field value indicating the order of the movie fragments shown in the MMTP payload header.

[0418] Specifically, when the number of movie fragments included in one MPU is 1, and the movie_fragment_sequence_number field value is 1, and there are values of 2 or more for the field value, the receiving device 20 may determine that the MPU is divided into a plurality of movie fragments.

[0419] Also, when the number of movie fragments included in one MPU is 1, and the movie_fragment_sequence_number field value is 0, and there are values other than 0 for the field value, the receiving device 20 may determine that the MPU is divided into a plurality of movie fragments.

[0420] Similarly, the attribute information of the movie fragment may be determined based on the movie_fragment_sequence_number.

[0421] Note that even without using the movie_fragment_sequence_number, by counting the transmission of movie fragments and MF meta included in one MPU, it may be determined whether the movie fragments are divided and the attribute information of the movie fragments.

[0422] With the configurations of the transmission device 15 and the reception device 20 as described above, the reception device 20 can receive movie fragment metadata at intervals shorter than that of the MPU, enabling decoding start with low latency. Also, by using a decoding process based on the method of MP4 parsing, it is possible to perform decoding with low latency.

[0423] The reception operation in the case where the MPU is divided into a plurality of movie fragments as described above will be described using a flowchart. FIG. 41 is a flowchart of the reception operation in the case where the MPU is divided into a plurality of movie fragments. Note that this flowchart illustrates the operation of step S604 in FIG. 37 in more detail.

[0424] First, based on the data type indicated in the MMTP payload header, when the data type is MF meta, the reception device 20 acquires the MF metadata (S801).

[0425] Next, the reception device 20 determines whether the MPU is divided into a plurality of movie fragments (S802). If the MPU is divided into a plurality of movie fragments (Yes in S802), it is determined whether the received MF metadata is the metadata at the head of the MPU (S803). When the received MF metadata is the MF metadata at the head of the MPU (Yes in S803), the reception device 20 calculates the absolute times of PTS and DTS from the absolute time of PTS indicated in the MPU timestamp descriptor and the relative times of PTS and DTS indicated in the MF metadata (S804), and determines whether it is the last metadata of the MPU (S805).

[0426] On the other hand, if the received MF metadata is not the MF metadata at the beginning of the MPU (No in S803), the receiving device 20 calculates the absolute times of PTS and DTS using the relative times of PTS and DTS indicated in the MF metadata without using the information of the MPU timestamp descriptor (S808), and proceeds to the process of step S805.

[0427] In step S805, if it is determined that the MF metadata is the last MF metadata of the MPU (Yes in S805), the receiving device 20 calculates the absolute times of PTS and DTS for all samples and then resets the calculation process of PTS and DTS. If it is determined in step S805 that the MF metadata is not the last MF metadata of the MPU (No in S805), the receiving device 20 ends the process.

[0428] Also, if it is determined in step S802 that the MPU is not divided into a plurality of movie fragments (No in S802), the receiving device 20 acquires sample data and determines PTS and DTS based on the MF metadata transmitted after the MPU (S807).

[0429] Then, although not shown, the receiving device 20 finally performs decoding processing and presentation processing based on the determined PTS and DTS.

[0430] [Problems Arising When Movie Fragments are Divided and Their Solutions] So far, a method for shortening the End-to-End delay by dividing movie fragments has been described. From here, problems newly arising when movie fragments are divided and their solutions will be described.

[0431] First, as background, the picture structure in encoded data will be described. FIG. 42 is a diagram showing an example of the prediction structure of pictures in each TemporalId when realizing temporal scalability.

[0432] In coding methods such as MPEG-4 AVC and HEVC (High Efficiency Video Coding), temporal scalability (temporal scalability) can be achieved by using B pictures (bidirectional reference prediction pictures) that can be referenced from other pictures.

[0433] The TemporalId shown in Fig. 42(a) is an identifier for the hierarchy of the coding structure, and the larger the value of TemporalId, the deeper the hierarchy. The square blocks represent pictures, and Ix within the block represents an I picture (intra prediction picture), Px represents a P picture (forward reference prediction picture), and Bx and bx represent B pictures (bidirectional reference prediction pictures). The x in Ix / Px / Bx indicates the display order and represents the order in which the pictures are displayed. The arrows between the pictures indicate the reference relationship. For example, the picture B4 generates a predicted image using I0 and B8 as reference images. Here, it is prohibited for one picture to use another picture with a TemporalId larger than its own as a reference image. The hierarchy is defined to provide temporal scalability. For example, when all the pictures in Fig. 42 are decoded, a video of 120 fps (frames per second) can be obtained, but when only the hierarchy with TemporalId from 0 to 3 is decoded, a video of 60 fps can be obtained.

[0434] Fig. 43 is a diagram showing the relationship between the decoding time (DTS) and the presentation time (PTS) for each picture in Fig. 42. For example, the picture I0 shown in Fig. 43 is presented after the decoding of B4 is completed so that no gap occurs in decoding and presentation.

[0435] As shown in Fig. 43, when the prediction structure includes B pictures, etc., since the decoding order and the presentation order are different, the receiving device 20 needs to perform delay processing of the pictures and rearrangement (reorder) processing of the pictures after decoding the pictures.

[0436] The above described an example of the prediction structure of pictures in the scalability in the time direction. However, even when the scalability in the time direction is not used, depending on the prediction structure, picture delay processing and reordering processing may be required. FIG. 44 is a diagram showing an example of the prediction structure of a picture for which picture delay processing and reordering processing are required. Note that the numbers in FIG. 44 indicate the decoding order.

[0437] As shown in FIG. 44, depending on the prediction structure, the sample that is the first in the decoding order and the sample that is the first in the presentation order may be different. In FIG. 44, the sample that is the first in the presentation order is the fourth sample in the decoding order. Note that FIG. 44 shows an example of the prediction structure, and the prediction structure is not limited to such a structure. In other prediction structures as well, the sample that is the first in the decoding order and the sample that is the first in the presentation order may be different.

[0438] Similar to FIG. 33, FIG. 45 is a diagram showing an example in which an MPU configured in the MP4 format is divided into a plurality of movie fragments and stored in an MMTP payload and an MMTP packet. Note that the number of samples constituting the MPU and the number of samples constituting the movie fragment are arbitrary. For example, the number of samples constituting the MPU may be set to the number of samples in a GOP unit, and two movie fragments may be configured with the number of samples that is half of the GOP unit as the movie fragment. One sample may be used as one movie fragment, or the samples constituting the MPU may not be divided.

[0439] FIG. 45 shows an example in which one MPU includes two movie fragments (moof box and mdat box), but the number of movie fragments included in one MPU does not have to be two. The number of movie fragments included in one MPU may be three or more, or may be the number of samples included in the MPU. Also, the samples stored in the movie fragment do not have to be equally divided, and may be divided into an arbitrary number of samples.

[0440] Movie fragment metadata (MF metadata) includes information on the PTS, DTS, offset, and size of samples included in a movie fragment. When decoding a sample, the receiving device 20 extracts the PTS and DTS from the MF metadata including the information on the sample, and determines the decoding timing and presentation timing.

[0441] Hereinafter, for detailed explanation, the absolute value of the decoding time of the i-th sample is denoted as DTS(i), and the absolute value of the presentation time is denoted as PTS(i).

[0442] Specifically, the information on the i-th sample among the timestamp information stored in moof in the MF metadata is the relative value of the decoding times of the i-th sample and the (i + 1)-th sample, and the relative value of the decoding time and the presentation time of the i-th sample. These are hereinafter denoted as DT(i) and CT(i).

[0443] Movie fragment metadata #1 includes DT(i) and CT(i) of samples #1 - #3, and movie fragment metadata #2 includes DT(i) and CT(i) of samples #4 - #6.

[0444] Also, the absolute value of the PTS of the access unit at the head of the MPU is stored in the MPU timestamp descriptor or the like, and the receiving device 20 calculates the PTS and DTS based on the PTS_MPU of the access unit at the head of the MPU, and CT and DT.

[0445] FIG. 46 is a diagram for explaining a method and problems of calculating PTS and DTS when an MPU is composed of samples #1 - #10.

[0446] FIG. 46(a) shows an example where the MPU is not divided into movie fragments, FIG. 46(b) shows an example where the MPU is divided into two movie fragments in units of 5 samples, and FIG. 46(c) shows an example where the MPU is divided into 10 movie fragments in units of samples.

[0447] As described with reference to FIG. 45, when PTS and DTS are calculated using the MPU time stamp descriptor and the time stamp information (CT and DT) in the MP4, the sample that is the first in the presentation order in FIG. 44 is the fourth in the decoding order. Therefore, the PTS stored in the MPU time stamp descriptor is the PTS (absolute value) of the fourth sample in the decoding order. Hereinafter, this sample will be referred to as Sample A. Also, the sample that is the first in the decoding order will be referred to as Sample B.

[0448] Since the absolute time information related to the time stamp is only the information in the MPU time stamp descriptor, the receiving apparatus 20 cannot calculate the PTS (absolute time) and DTS (absolute time) of other samples until Sample A arrives. The receiving apparatus 20 also cannot calculate the PTS and DTS of Sample B.

[0449] In the example of FIG. 46(a), Sample A is included in the same movie fragment as Sample B and is stored in one MF meta. Therefore, after receiving the MF meta, the receiving apparatus 20 can immediately determine the DTS of Sample B.

[0450] In the example of FIG. 46(b), Sample A is included in the same movie fragment as Sample B and is stored in one MF meta. Therefore, after receiving the MF meta, the receiving apparatus 20 can immediately determine the DTS of Sample B.

[0451] In the example of FIG. 46(c), Sample A is included in a movie fragment different from Sample B. Therefore, the receiving apparatus 20 cannot determine the DTS of Sample B until it receives the MF meta including the CT and DT of the movie fragment including Sample A.

[0452] Therefore, in the case of the example of FIG. 46(c), the receiving apparatus 20 cannot start decoding immediately after Sample B arrives.

[0453] As described above, when the movie fragment containing the B sample does not contain the A sample, the receiving device 20 cannot start decrypting the B sample until it has received the MF meta related to the movie fragment containing the A sample.

[0454] This problem occurs when the first sample in the presentation order does not match the first sample in the decoding order, and the movie fragment is split before the A sample and the B sample are no longer stored in the same movie fragment. Also, this problem occurs regardless of whether the MF meta is sent later or earlier.

[0455] As described above, when the first sample in the presentation order does not match the first sample in the decoding order, and the A sample and the B sample are not stored in the same movie fragment, the DTS cannot be determined immediately after receiving the B sample. Therefore, the transmitting device 15 separately transmits the DTS (absolute value) of the B sample, or information that can be calculated on the receiving side for the DTS (absolute value) of the B sample. Such information may be transmitted using control information, packet headers, etc.

[0456] The receiving device 20 calculates the DTS (absolute value) of the B sample using such information. FIG. 47 is a flowchart of the receiving operation when the DTS is calculated using such information.

[0457] The receiving device 20 receives the movie fragment at the head of the MPU (S901), and determines whether the A sample and the B sample are stored in the same movie fragment (S902). If they are stored in the same movie fragment (Yes in S902), the receiving device 20 calculates the DTS using only the information of the MF meta without using the DTS (absolute time) of the B sample, and starts decoding (S904). Note that in step S904, the receiving device 20 may determine the DTS using the DTS of the B sample.

[0458] On the other hand, if in step S902, the A sample and the B sample are not stored in the same movie fragment (No in S902), the receiving device 20 acquires the DTS (absolute time) of the B sample, determines the DTS, and starts decoding (S903).

[0459] In the above description, an example of calculating the absolute value of the decoding time and the absolute value of the presentation time of each sample using MF meta (timestamp information stored in moof in MP4 format) in the MMT standard has been described. However, it goes without saying that MF meta can be replaced with any control information that can be used to calculate the absolute value of the decoding time and the absolute value of the presentation time of each sample and implemented. Examples of such control information include control information in which the relative value CT(i) of the decoding times of the i-th sample and the (i + 1)-th sample described above is replaced with the relative value of the presentation times of the i-th sample and the (i + 1)-th sample, and control information including both the relative value CT(i) of the decoding times of the i-th sample and the (i + 1)-th sample and the relative value of the presentation times of the i-th sample and the (i + 1)-th sample.

[0460] [Supplementary Explanation] As described above, a transmitting device that transmits the DTS (absolute value) of the B sample, or information that can calculate the DTS (absolute value) of the B sample on the receiving side as control information, can also be configured as shown in FIG. 48. FIG. 48 is a diagram showing another example of the configuration of the transmitting device.

[0461] The transmitting device 300 includes an encoding data generation unit 301, a packet generation unit 302, a first transmitting unit 303, an information generation unit 304, and a second transmitting unit 305. As shown in FIG. 48, the packet generation unit 302 and the information generation unit 304 may be realized as one generation unit 306, and the first transmitting unit 303 and the second transmitting unit 305 may be realized as one transmitting unit 307.

[0462] The encoding unit 301 encodes the video signal to generate encoded data including a plurality of access units.

[0463] The packet generation unit 302 stores a plurality of access units in packets in units of access units or units obtained by dividing an access unit, and generates a packet group.

[0464] The first transmission unit 303 transmits the generated packet group as data.

[0465] The information generation unit 304 generates first information indicating the presentation time of the first presented access unit among the plurality of access units, and second information used for calculating the decoding time of the plurality of access units. Here, the first information is, for example, an MPU timestamp descriptor, and the second information is auxiliary information such as, for example, timestamp information of MF metadata (or information in which the timestamp information is partially modified).

[0466] The second transmission unit 305 transmits the generated first information and second information as control information.

[0467] Further, the receiving device corresponding to the transmitting device 300 may be configured as shown in FIG. 49, for example. FIG. 49 is a diagram showing another example of the configuration of the receiving device.

[0468] The receiving device 400 includes a first receiving unit 401, a second receiving unit 402, and a decoding unit 403. As shown in FIG. 49, the first receiving unit 401 and the second receiving unit 402 may be realized as one receiving unit 404.

[0469] The first receiving unit 401 receives a packet group in which encoded data including a plurality of access units is packetized in units of access units or units obtained by dividing an access unit.

[0470] The second receiving unit 402 receives control information including first information indicating the presentation time of the first presented access unit among the plurality of access units, and second information used for calculating the decoding time of the plurality of access units.

[0471] The decoding unit 403 decodes the access units included in the received packet group based on the first information and the second information.

[0472] (Other embodiments) As described above, the transmission device, the reception device, the transmission method, and the reception method according to the embodiments have been described. However, the present invention is not limited to these embodiments.

[0473] In addition, each processing unit included in the transmission device and the reception device according to the above embodiments is typically realized as an LSI which is an integrated circuit. These may be individually formed into one chip, or may be formed into one chip so as to include some or all of them.

[0474] In addition, the integration into an integrated circuit is not limited to an LSI, and it may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) which can be programmed after manufacturing the LSI, or a reconfigurable processor which can reconfigure the connection and setting of circuit cells inside the LSI may be used.

[0475] In each of the above embodiments, each component may be configured by dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0476] In other words, the transmitting apparatus and the receiving apparatus include a processing circuitry and a storage electrically connected to the processing circuitry (accessible from the control circuitry). The processing circuitry includes at least one of dedicated hardware and a program execution unit. Further, when the processing circuitry includes a program execution unit, the storage stores a software program executed by the program execution unit. The processing circuitry uses the storage to execute the transmitting method or the receiving method according to the above embodiment.

[0477] Furthermore, the present invention may be the above software program or a non-transitory computer-readable recording medium on which the above program is recorded. Needless to say, the above program can be distributed via a transmission medium such as the Internet.

[0478] Also, all the numbers used above are for illustrative purposes to specifically describe the present invention, and the present invention is not limited to the illustrated numbers.

[0479] Also, the division of the functional blocks in the block diagram is an example, and a plurality of functional blocks may be realized as one functional block, one functional block may be divided into a plurality, or some functions may be transferred to other functional blocks. Further, the functions of a plurality of functional blocks having similar functions may be processed by a single piece of hardware or software in parallel or in time division.

[0480] Also, the order in which the steps included in the above transmitting method or receiving method are executed is for illustrative purposes to specifically describe the present invention, and may be an order other than the above. Also, some of the above steps may be executed simultaneously (in parallel) with other steps.

[0481] As described above, the transmission device, reception device, transmission method, and reception method according to one or more aspects of the present invention have been described based on the embodiments. However, the present invention is not limited to these embodiments. As long as it does not depart from the gist of the present invention, various modifications conceived by those skilled in the art applied to these embodiments, or forms constructed by combining components in different embodiments may also be included within the scope of one or more aspects of the present invention.

Industrial Applicability

[0482] The present invention can be applied to devices or equipment that perform media transport such as video data and audio data.

Explanation of Signs

[0483] 15, 100, 300 Transmission device 16, 101, 301 Encoding unit 17, 102 Multiplexing unit 18, 104, 307 Transmission unit 20, 200, 400 Reception device 21 Packet filtering unit 22 Transmission order type determination unit 23 Random access unit 24, 212 Control information acquisition unit 25 Data acquisition unit 26 Calculation unit 27 Initialization information acquisition unit 28, 206 Decoding command unit 29, 403, 204A, 204B, 204C, 204D Decoding unit 30 Presentation unit 201 Tuner 202 Demodulation unit 203 Demultiplexing unit 205 Display unit 211 Type determination unit 213 Slice information acquisition unit 214 Decoded data generation unit 302 Packet generation unit 303 First transmission unit 304 Information generation unit 305 Second Transmission Unit 306 Generation Unit 401 First Reception Unit 402 Second Reception Unit 404 Reception Unit

Claims

Claim 1 Encode a video signal to generate encoded data including a plurality of access units, correspond the plurality of access units to the configuration of an MFU (Movie Fragment Unit), store them in packets in units of access units or units obtained by dividing an access unit, and store the sequence number in the MFU in the packets to generate a packet group, store a plurality of parameters when encoding the video signal in the payload of one packet different from the packet in which the video signal is stored, and include type information for identifying the data stored in the payload in the header of the packet, transmit the generated packet group as data, generate first information indicating the presentation time of the first presented access unit among the plurality of access units, second information used for calculating the decoding time of the plurality of access units, and third information which is a relative value of the presentation time with respect to the access unit presented one before in the presentation order of the access units, transmit the generated first information, second information, and third information in a second packet different from the first packet storing the plurality of access units in units of the access units or units obtained by dividing the access units, The packetization for generating the packet group is performed in the MMT (MPEG Media Transport) format Transmission method.

Images