Receiving method and receiving device

Abstract

This provides a receiving method that enables the execution of an application consisting of predetermined data units at an intended time. [Solution] A receiving method for receiving a predetermined data unit containing data constituting an application, wherein the receiving method receives (i) the predetermined data unit and (ii) control information which stores time specification information indicating the operating time of the application and identification information indicating whether or not the time specification information is time information before leap second adjustment, and executes the application stored in the received predetermined data unit based on the received control information, wherein the time specification information is UTC (Universal Time Coordinated) time and NPT (Normal Play Time), the control information is a UTC-NPT reference descriptor showing the relationship between UTC time and NPT time, and the reference time information for generating the time specification information is NTP.

Description

Technical Field

[0004]

[0001] The present invention relates to a reception method and a reception apparatus.

Background Art

[0002] With the advancement of broadcast and communication services, the introduction of ultra-high-definition moving image contents such as 8K (7680×4320 pixels: hereinafter also referred to as 8K4K) and 4K (3840×2160 pixels: hereinafter also referred to as 4K2K) is being considered. A receiving apparatus needs to decode and display the encoded data of the received ultra-high-definition moving image in real time. However, in particular, moving images with a resolution such as 8K have a large processing load during decoding, and it is difficult to decode such a moving image in real time with a single decoder. Therefore, a method of reducing the processing load per decoder and achieving real-time processing by parallelizing the decoding process using a plurality of decoders has been considered.

[0003] Also, the encoded data is multiplexed based on a multiplexing method such as MPEG-2 TS (Transport Stream) or MMT (MPEG Media Transport) and then transmitted. For example, Non-Patent Document 1 discloses a technique of transmitting encoded media data for each packet according to MMT.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Incidentally, with the advancement of broadcasting and communication services, the introduction of ultra-high-definition video content such as 8K and 4K (3840x2160 pixels) is being considered. In the MMT / TLV method, the transmitter's reference clock is synchronized to a 64-bit long-format NTP as defined in RFC 5905, and timestamps such as PTS (Presentation Time Stamp) and DTS (Decode Time Stamp) are added to the synchronization medium based on the reference clock. Furthermore, the transmitter sends the reference clock information to the receiver, and the receiver generates its own system clock based on the reference clock information.

[0006] However, with such MMT transmission methods, if leap second adjustments are made to reference time information such as NTP, the receiving device may not be able to decode or present the MPU, which is a predetermined data unit contained in the received packet, at the intended time, even if it follows the DTS or PTS associated with the MPU.

[0007] The present invention provides a transmission method that enables a receiving device to reproduce a predetermined data unit at the intended time, even when leap second adjustments are made to the reference time information that serves as the reference clock for the transmitting and receiving devices. [Means for solving the problem]

[0008] To achieve the above objective, a receiving method according to one aspect of the present invention is a receiving method for receiving a predetermined data unit in which data constituting an application is stored, the receiving method receiving (i) the predetermined data unit and (ii) control information which stores time specification information indicating the operating time of the application and identification information indicating whether or not the time specification information is time information before leap second adjustment, and the receiving method executes the application stored in the received predetermined data unit based on the received control information, wherein the time specification information is UTC (Universal Time Coordinated) time and NPT (Normal Play Time), the control information is a UTC-NPT reference descriptor indicating the relationship between UTC time and NPT time, and the reference time information for generating the time specification information is NTP (Network Time Protocol).

[0009] To achieve the above objective, a transmission method according to one aspect of the present invention is a transmission method for storing and transmitting data constituting an application in a predetermined data unit, wherein time specification information indicating the operating time of the application is generated based on reference time information, and (i) the predetermined data unit and (ii) control information indicating the generated time specification information are transmitted, wherein the control information stores the generated time specification information and identification information indicating whether or not the time specification information is time information before leap second adjustment, wherein the time specification information is UTC (Universal Time Coordinated) time and NPT (Normal Play Time), the control information is a UTC-NPT reference descriptor indicating the relationship between UTC time and NPT time, and the reference time information is NTP (Network Time Coordinated). This is the Time Protocol.

[0010] To achieve the above objective, a transmission method according to one aspect of the present invention is a transmission method for storing and transmitting data constituting an application in a predetermined data unit, wherein time specification information indicating the operating time of the application is generated based on reference time information, and (i) the predetermined data unit and (ii) control information indicating the generated time specification information are transmitted, wherein the control information stores the generated time specification information and identification information indicating whether or not the time specification information is time information before leap second adjustment, the time specification information in the predetermined data unit is time information to which a predetermined time has been added, the time specification information is UTC (Universal Time Coordinated) time and NPT (Normal Play Time), and the control information is a UTC-NPT reference descriptor indicating the relationship between UTC time and NPT time.

[0011] Furthermore, a receiving method according to one aspect of the present invention is a receiving method for receiving a predetermined data unit in which data constituting an application is stored, the receiving method receiving (i) the predetermined data unit and (ii) control information which stores time specification information indicating the operating time of the application and identification information indicating whether or not the time specification information is time information before leap second adjustment, and the receiving method executes the application stored in the received predetermined data unit based on the received control information, wherein the time specification information of the predetermined data unit is time information in which a predetermined time has been added to the time information, the time specification information is UTC (Universal Time Coordinated) time and NPT (Normal Play Time), and the control information is a UTC-NPT reference descriptor indicating the relationship between UTC time and NPT time.

[0012] To achieve the above objective, a transmission method according to one aspect of the present invention is a transmission method for storing and transmitting data constituting an application in a predetermined data unit, wherein time specification information indicating the operating time of the application is generated based on reference time information, and (i) the predetermined data unit and (ii) control information indicating the generated time specification information are transmitted, wherein the control information stores the generated time specification information and identification information indicating whether or not the time specification information is time information before leap second adjustment, wherein the time specification information is UTC (Universal Time Coordinated) time and NPT (Normal Play Time), and the control information is a UTC-NPT reference descriptor indicating the relationship between UTC time and NPT time.

[0013] Furthermore, a receiving method according to one aspect of the present invention is a receiving method for receiving a predetermined data unit in which data constituting an application is stored, the receiving method receiving (i) the predetermined data unit and (ii) control information which stores time specification information indicating the operating time of the application and identification information indicating whether or not the time specification information is time information before leap second adjustment, and the receiving method for executing the application stored in the received predetermined data unit based on the received control information, wherein the time specification information is UTC (Universal Time Coordinated) time and NPT (Normal Play Time), and the control information is a UTC-NPT reference descriptor indicating the relationship between UTC time and NPT time.

[0014] To achieve the above objective, a transmission method according to one aspect of the present invention is a transmission method for storing and transmitting data constituting an application in a predetermined data unit, wherein time specification information indicating the operating time of the application is generated based on reference time information, and (i) the predetermined data unit and (ii) control information indicating the generated time specification information are transmitted, wherein the control information stores the generated time specification information and identification information indicating whether or not the time specification information is time information before leap second adjustment, wherein the identification information indicates whether or not the time specification information was generated based on time information from a predetermined period before the time immediately before the leap second adjustment to the time immediately before the leap second adjustment, and the reference time information is NTP (Network Time Protocol).

[0015] Furthermore, a receiving method according to one aspect of the present invention is a receiving method for receiving a predetermined data unit in which data constituting an application is stored, the receiving method receiving (i) the predetermined data unit and (ii) control information which stores time specification information indicating the operating time of the application and identification information indicating whether or not the time specification information is time information before leap second adjustment, and the receiving method executes the application stored in the received predetermined data unit based on the received control information, wherein the identification information indicates whether or not the time specification information was generated based on time information from a predetermined period before the time immediately before leap second adjustment to the time immediately before, and the time specification information is generated based on NTP (Network Time Protocol).

[0016] To achieve the above objective, a transmission method according to one aspect of the present invention is a transmission method for storing and transmitting data constituting an application in a predetermined data unit, wherein the transmission method generates time specification information indicating the operating time of the application based on reference time information, and transmits (i) the predetermined data unit and (ii) control information indicating the generated time specification information, wherein the control information stores the generated time specification information and identification information indicating whether or not the time specification information is time information before leap second adjustment, wherein the identification information indicates whether or not the time specification information was generated based on time information from a predetermined period before the time immediately before leap second adjustment to the time immediately before, and the time specification information in the predetermined data unit is time information to which a predetermined time has been added.

[0017] Furthermore, a receiving method according to one aspect of the present invention is a receiving method for receiving a predetermined data unit in which data constituting an application is stored, the receiving method receiving (i) the predetermined data unit and (ii) control information which stores time specification information indicating the operating time of the application and identification information indicating whether or not the time specification information is time information before leap second adjustment, and the receiving method executes the application stored in the received predetermined data unit based on the received control information, wherein the identification information indicates whether or not the time specification information was generated based on time information from a time a predetermined period before the time immediately before leap second adjustment to the time immediately before, and the time specification information of the predetermined data unit is time information generated by adding a predetermined time to the time information.

[0018] In order to achieve the above object, a transmission method according to an aspect of the present invention is a transmission method for storing and transmitting data constituting an application in a predetermined data unit, generating time specification information indicating an operation time of the application based on reference time information received from the outside, and transmitting (i) the predetermined data unit and (ii) control information indicating the generated time specification information, wherein the control information stores the generated time specification information and identification information indicating whether the time specification information is time information before leap second adjustment.

[0019] Note that these general or specific aspects may be implemented by a system, a device, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be implemented by any combination of a system, a device, an integrated circuit, a computer program, and a recording medium.

Advantages of the Invention

[0020] Even when leap second adjustment is performed on the reference time information serving as a reference for the reference clocks of the transmission side and the receiving device, the receiving device can execute an application composed of a predetermined data unit at the intended time.

Brief Description of the Drawings

[0021] [Figure 1] FIG. 1 is a diagram showing an example of dividing a picture into slice segments. [Figure 2] FIG. 2 is a diagram showing an example of a PES packet sequence in which picture data is stored. [Figure 3] FIG. 3 is a diagram showing an example of dividing a picture according to Embodiment 1. [Figure 4] FIG. 4 is a diagram showing an example of dividing a picture according to a comparative example of Embodiment 1. [Figure 5] FIG. 5 is a diagram showing an example of data of an access unit according to Embodiment 1. [Figure 6] FIG. 6 is a block diagram of a transmission device according to Embodiment 1. [Figure 7] FIG. 7 is a block diagram of the receiving apparatus according to Embodiment 1. [Figure 8] FIG. 8 is a diagram showing an example of an MMT packet according to Embodiment 1. [Figure 9] FIG. 9 is a diagram showing another example of an MMT packet according to Embodiment 1. [Figure 10] FIG. 10 is a diagram showing an example of data input to each decoding unit according to Embodiment 1. [Figure 11] FIG. 11 is a diagram showing an example of an MMT packet and header information according to Embodiment 1. [Figure 12] FIG. 12 is a diagram showing another example of data input to each decoding unit according to Embodiment 1. [Figure 13] FIG. 13 is a diagram showing an example of picture segmentation according to Embodiment 1. [Figure 14] FIG. 14 is a flowchart of the transmission method according to Embodiment 1. [Figure 15] FIG. 15 is a block diagram of the receiving apparatus according to Embodiment 1. [Figure 16] FIG. 16 is a flowchart of the receiving method according to Embodiment 1. [Figure 17] FIG. 17 is a diagram showing an example of an MMT packet and header information according to Embodiment 1. [Figure 18] FIG. 18 is a diagram showing an example of an MMT packet and header information according to Embodiment 1. [Figure 19] FIG. 19 is a diagram showing the configuration of the MPU. [Figure 20] FIG. 20 is a diagram showing the configuration of the MF metadata. [Figure 21] FIG. 21 is a diagram for explaining the data transmission order. [Figure 22] FIG. 22 is a diagram showing an example of a method of performing decoding without using header information. [Figure 23] FIG. 23 is a block diagram of the transmission apparatus according to Embodiment 2. [Figure 24]Figure 24 is a flowchart of the transmission method according to Embodiment 2. [Figure 25] Figure 25 is a block diagram of the receiving device according to Embodiment 2. [Figure 26] Figure 26 is a flowchart of the operation for identifying the MPU starting position and the NAL unit position. [Figure 27] Figure 27 is a flowchart illustrating the operation of obtaining initialization information based on the transmission order type and decoding media data based on the initialization information. [Figure 28] Figure 28 is a flowchart showing the operation of the receiving device when a low-latency presentation mode is provided. [Figure 29] Figure 29 shows an example of the transmission order of MMT packets when auxiliary data is transmitted. [Figure 30] Figure 30 illustrates an example in which a transmitter generates auxiliary data based on the MOOF configuration. [Figure 31] Figure 31 is a diagram illustrating the reception of auxiliary data. [Figure 32] Figure 32 is a flowchart of the receiving operation using auxiliary data. [Figure 33] Figure 33 shows the configuration of an MPU, which is composed of multiple movie fragments. [Figure 34] Figure 34 is a diagram illustrating the transmission order of MMT packets when the MPU configuration shown in Figure 33 is transmitted. [Figure 35] Figure 35 is the first diagram illustrating an example of the operation of a receiver when one MPU is composed of multiple movie fragments. [Figure 36] Figure 36 is a second diagram illustrating an example of the operation of a receiver when one MPU is composed of multiple movie fragments. [Figure 37] Figure 37 is a flowchart illustrating the operation of the receiving method described in Figures 35 and 36. [Figure 38]Figure 38 shows a case where non-VCL NAL units are treated as individual data units and aggregated. [Figure 39] Figure 39 shows a case where non-VCL NAL units are grouped together to form a data unit. [Figure 40] Figure 40 is a flowchart showing the operation of the receiving device when packet loss occurs. [Figure 41] Figure 41 is a flowchart of the receiving operation when the MPU is divided into multiple movie fragments. [Figure 42] Figure 42 shows an example of the picture prediction structure for each TemporalId when achieving temporal scalability. [Figure 43] Figure 43 shows the relationship between the decoded time (DTS) and the displayed time (PTS) for each picture in Figure 42. [Figure 44] Figure 44 shows an example of a picture prediction structure that requires picture delay processing and reordering processing. [Figure 45] Figure 45 shows an example where an MPU composed in MP4 format is divided into multiple movie fragments and stored in an MMTP payload and MMTP packets. [Figure 46] Figure 46 is a diagram illustrating the calculation methods and challenges of PTS and DTS. [Figure 47] Figure 47 is a flowchart of the receiving operation when the DTS is calculated using the information used for DTS calculation. [Figure 48] Figure 48 is a diagram illustrating how data units are stored in the payload in MMT. [Figure 49] Figure 49 shows the operation flow of the transmitting device according to Embodiment 3. [Figure 50] Figure 50 shows the operation flow of the receiving device according to Embodiment 3. [Figure 51] Figure 51 is a diagram showing an example of a specific configuration of the transmitting device according to Embodiment 3. [Figure 52]Figure 52 is a diagram showing an example of a specific configuration of the receiving device according to Embodiment 3. [Figure 53] Figure 53 shows the method for storing non-timed media in the MPU and the method for transmitting it using MMTP packets. [Figure 54] Figure 54 shows an example of packetizing and transmitting each of the multiple data segments obtained by splitting a file. [Figure 55] Figure 55 shows another example where each of the multiple data segments obtained by splitting a file is packetized and transmitted. [Figure 56] Figure 56 shows the syntax for a file-by-file loop in the asset management table. [Figure 57] Figure 57 shows the operation flow for identifying the segmented data number in the receiving device. [Figure 58] Figure 58 shows the operation flow for determining the number of divided data segments in the receiving device. [Figure 59] Figure 59 shows the operation flow for determining whether or not to operate a fragment counter in the transmitting device. [Figure 60] Figure 60 is a diagram illustrating how to identify the number of fragmented data points and the fragmented data number (when using a fragment counter). [Figure 61] Figure 61 shows the operation flow of the transmitting device when a fragment counter is used. [Figure 62] Figure 62 shows the operation flow of the receiving device when a fragment counter is used. [Figure 63] Figure 63 shows the service configuration when the same program is transmitted using multiple IP data flows. [Figure 64] Figure 64 shows an example of a specific configuration of a transmitting device. [Figure 65] Figure 65 shows an example of a specific configuration of a receiving device. [Figure 66] Figure 66 shows the operation flow of the transmitting device. [Figure 67]Figure 67 shows the operation flow of the receiving device. [Figure 68] Figure 68 shows the receive buffer model based on the receive buffer model specified in ARIB STD-B60, specifically for the case where only the broadcast transmission line is used. [Figure 69] Figure 69 shows an example of aggregating multiple data units and storing them in a single payload. [Figure 70] Figure 70 shows an example of aggregating multiple data units and storing them in a single payload, specifically an example where a video signal in NAL size format is treated as a single data unit. [Figure 71] Figure 71 shows the payload structure of an MMTP packet where the data unit length is not indicated. [Figure 72] Figure 72 shows an example of the extend area assigned to each packet. [Figure 73] Figure 73 shows the operation flow of the receiving device. [Figure 74] Figure 74 shows an example of a specific configuration of a transmitting device. [Figure 75] Figure 75 shows an example of a specific configuration of a receiving device. [Figure 76] Figure 76 shows the operation flow of the transmitting device. [Figure 77] Figure 77 shows the operation flow of the receiving device. [Figure 78] Figure 78 shows the protocol stack for the MMT / TLV scheme as defined in ARIB STD-B60. [Figure 79] Figure 79 shows the structure of a TLV packet. [Figure 80] Figure 80 shows an example of a block diagram of a receiving device. [Figure 81] Figure 81 is a diagram illustrating the timestamp descriptor. [Figure 82] Figure 82 is a diagram illustrating leap second adjustments. [Figure 83]Figure 83 shows the relationship between NTP time, MPU timestamp, and MPU presentation timing. [Figure 84] Figure 84 is a diagram illustrating a correction method for correcting timestamps on the transmitting side. [Figure 85] Figure 85 is a diagram illustrating a correction method for correcting timestamps in a receiving device. [Figure 86] Figure 86 shows the operation flow by the transmitting device when correcting the MPU timestamp on the transmitting side. [Figure 87] Figure 87 shows the operation flow by the receiving device when the transmitting device corrects the MPU timestamp. [Figure 88] Figure 88 shows the operation flow by the transmitting side (transmitting device) when correcting the MPU timestamp in the receiving device. [Figure 89] Figure 89 shows the operation flow of the receiving device when correcting the MPU timestamp. [Figure 90] Figure 90 shows an example of a specific configuration of a transmitting device. [Figure 91] Figure 91 shows an example of a specific configuration of a receiving device. [Figure 92] Figure 92 shows the operation flow of the transmitting device. [Figure 93] Figure 93 shows the operation flow of the receiving device. [Figure 94] Figure 94 shows an example of an extended MPU timestamp descriptor. [Figure 95] Figure 95 illustrates the case where discontinuities occur in the MPU sequence numbers due to adjustments being made to the MPU sequence numbers. [Figure 96] Figure 96 illustrates the case where packet sequence numbers become discontinuous when switching from a standard system to a redundant system. [Figure 97]Figure 97 shows the operation flow of the receiving device when discontinuities occur in MPU sequence numbers or packet sequence numbers. [Modes for carrying out the invention]

[0022] A transmission method according to one aspect of the present invention is a transmission method for storing and transmitting data constituting an application in a predetermined data unit, wherein, based on reference time information received from an external source, time specification information indicating the operating time of the application is generated, and (i) the predetermined data unit and (ii) control information indicating the generated time specification information are transmitted, the control information storing the generated time specification information and identification information indicating whether or not the time specification information is time information before leap second adjustment.

[0023] This transmission method allows an application consisting of a predetermined data unit to be executed at the intended time, even if leap second adjustments are made to the reference time information that serves as the basis for the reference clock of the transmitting and receiving devices.

[0024] Furthermore, in the generation process, the time information presented by the MPU may be generated by adding a predetermined time to the time information.

[0025] Furthermore, the identification information may indicate whether the presented time information was generated based on time information from a predetermined period prior to the time immediately preceding the leap second adjustment.

[0026] Furthermore, the time specification information may be UTC (Universal Time Coordinated) time and NPT (Normal Play Time), and the control information may be a UTC-NPT reference descriptor that shows the relationship between UTC time and NPT time.

[0027] Furthermore, the aforementioned reference time information may be NTP (Network Time Protocol).

[0028] Furthermore, a receiving method according to one aspect of the present invention is a receiving method for receiving a predetermined data unit in which data constituting an application is stored, the receiving method receiving (i) the predetermined data unit and (ii) control information which stores time specification information indicating the operating time of the application and identification information indicating whether or not the time specification information is time information before leap second adjustment, and the receiving method for executing the application stored in the received predetermined data unit based on the received control information.

[0029] This receiving method allows an application consisting of a predetermined data unit to be executed at the intended time, even if leap second adjustments are made to the reference time information that serves as the basis for the reference clock.

[0030] These comprehensive or specific embodiments may be implemented as systems, devices, integrated circuits, computer programs, or recording media such as computer-readable CD-ROMs, or as any combination of systems, devices, integrated circuits, computer programs, or recording media.

[0031] The embodiments will be described in detail below with reference to the drawings.

[0032] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples only and are not intended to limit the present invention. Furthermore, among the components in the following embodiments, those not described in the independent claim representing the highest-level concept will be described as optional components.

[0033] (Knowledge that formed the basis of this invention) In recent years, the resolution of displays in devices such as TVs, smartphones, and tablet devices has been increasing. In particular, 8K4K (8K x 4K resolution) services are planned for broadcasting in Japan in 2020. For ultra-high-resolution video such as 8K4K, real-time decoding is difficult with a single decoder, so methods that use multiple decoders to perform decoding in parallel are being considered.

[0034] Since encoded data is transmitted after being multiplexed based on multiplexing schemes such as MPEG-2 TS and MMT, the receiving device needs to separate the encoded video data from the multiplexed data before decoding. Hereafter, the process of separating the encoded data from the multiplexed data will be referred to as demultiplexing.

[0035] When parallelizing the decoding process, it is necessary to distribute the encoded data to be decoded to each decoder. Distributing the encoded data requires analyzing the encoded data itself, and especially with 8K content, the bitrate is very high, resulting in a heavy processing load for the analysis. Therefore, the demultiplexing process becomes a bottleneck, preventing real-time playback.

[0036] Incidentally, in video encoding schemes such as H.264 and H.265, standardized by MPEG and the ITU, the transmitting device can divide the picture into multiple areas called slices or slice segments, and encode each divided area so that it can be decoded independently. Therefore, for example, in the case of H.265, the receiving device that receives the broadcast can separate the data for each slice segment from the received data and output the data for each slice segment to separate decoders, thereby achieving parallelization of the decoding process.

[0037] Figure 1 shows an example of dividing a single picture into four slice segments in HEVC. For example, the receiving device has four decoders, and each decoder decodes one of the four slice segments.

[0038] In conventional broadcasting, the transmitting device stores one picture (an access unit in the MPEG system standard) in one PES packet and multiplexes the PES packets into a sequence of TS packets. Therefore, the receiving device had to separate the payload of the PES packets, analyze the data of the access units stored in the payload to separate each slice segment, and output the data of each separated slice segment to the decoder.

[0039] However, the inventors have found that there is a problem in that it is difficult to perform this process in real time because the amount of processing required to analyze the data of the access unit and separate the slice segments is large.

[0040] Figure 2 shows an example where picture data divided into slice segments is stored in the payload of a PES packet.

[0041] As shown in Figure 2, for example, data from multiple slice segments (slice segments 1 to 4) is stored in the payload of a single PES packet. Furthermore, the PES packet is multiplexed into a sequence of TS packets.

[0042] (Embodiment 1) The following explanation uses H.265 as the encoding scheme for video as an example, but this embodiment can also be applied when using other encoding schemes such as H.264.

[0043] Figure 3 shows an example of how an access unit (picture) in this embodiment is divided into segments. The access unit is divided into two equal parts horizontally and vertically, resulting in a total of four tiles, by a feature called tiling, which was introduced by H.265. Furthermore, there is a one-to-one correspondence between the slice segments and the tiles.

[0044] Let's explain why the data is divided into two equal parts horizontally and vertically. First, during decoding, line memory is generally required to store data for one horizontal line. However, with ultra-high resolutions such as 8K and 4K, the horizontal size increases, thus increasing the size of the line memory. In the implementation of receiving devices, it is desirable to reduce the size of the line memory. To reduce the size of the line memory, vertical division is necessary. Vertical division requires a data structure called a tile. For these reasons, tiles are used.

[0045] On the other hand, since images generally have high horizontal correlation, encoding efficiency improves when a wider range can be referenced horizontally. Therefore, from the standpoint of encoding efficiency, it is desirable for the access units to be divided horizontally.

[0046] By dividing the access unit into two equal parts horizontally and vertically, these two characteristics can be reconciled, taking into account both implementation aspects and coding efficiency. If a single decoder can decode 4K2K video in real time, then by dividing an 8K4K image into four equal parts, with each slice segment being 4K2K, the receiver can decode the 8K4K image in real time.

[0047] Next, we will explain why the tiles and slice segments obtained by dividing the access unit horizontally and vertically are mapped one-to-one. In H.265, an access unit consists of multiple units called NAL (Network Adaptation Layer) units.

[0048] The payload of a NAL unit stores one of the following: an access unit delimiter indicating the starting position of the access unit; SPS (Sequence Parameter Set), which is initialization information used commonly at the sequence level during decoding; PPS (Picture Parameter Set), which is initialization information used commonly within a picture during decoding; SEI (Supplemental Enhancement Information), which is not required for the decoding process itself but is necessary for processing and displaying the decoding results; and encoded data of the slice segment. The header of the NAL unit contains type information to identify the data stored in the payload.

[0049] Here, the transmitting device encodes the data in MPEG-2 TS, MMT (MPEG Media Transport), and MPEG DASH (Dynamic Adaptive When multiplexing using multiplexing formats such as Streaming over HTTP or RTP (Real-time Transport Protocol), the basic unit can be set to a NAL unit. In order to store one slice segment in one NAL unit, it is desirable to divide the access unit into slice segment units when dividing the area. For this reason, the transmitting device establishes a one-to-one correspondence between tiles and slice segments.

[0050] As shown in Figure 4, the transmitting device can also configure tiles 1 through 4 together as a single slice segment. However, in this case, all tiles are stored in a single NAL unit, making it difficult for the receiving device to separate the tiles in the multiplexing layer.

[0051] Note that there are two types of slice segments: independent slice segments that can be decoded independently, and reference slice segments that refer to independent slice segments. Here, we will explain the case where independent slice segments are used.

[0052] Figure 5 shows an example of access unit data divided so that the boundaries between tiles and slice segments coincide, as shown in Figure 3. The access unit data includes the NAL unit that stores the access unit delimiter placed at the beginning, the NAL units of the SPS, PPS, and SEI placed thereafter, and the slice segment data that stores the data for tiles 1 to 4 placed thereafter. Note that the access unit data does not have to include some or all of the NAL units of the SPS, PPS, and SEI.

[0053] Next, the configuration of the transmitting device 100 according to this embodiment will be described. Figure 6 is a block diagram showing an example of the configuration of the transmitting device 100 according to this embodiment. This transmitting device 100 comprises an encoding unit 101, a multiplexing unit 102, a modulation unit 103, and a transmitting unit 104.

[0054] The encoding unit 101 generates encoded data by encoding the input image, for example, according to H.265. Furthermore, the encoding unit 101 divides the access unit into four slice segments (tiles), as shown in Figure 3, and encodes each slice segment.

[0055] The multiplexing unit 102 multiplexes the encoded data generated by the encoding unit 101. The modulation unit 103 modulates the data obtained through multiplexing. The transmission unit 104 transmits the modulated data as a broadcast signal.

[0056] Next, the configuration of the receiving device 200 according to this embodiment will be described. Figure 7 is a block diagram showing an example of the configuration of the receiving device 200 according to this embodiment. This receiving device 200 includes a tuner 201, a demodulation unit 202, a demultiplexing unit 203, a plurality of decoding units 204A to 204D, and a display unit 205.

[0057] The tuner 201 receives the broadcast signal. The demodulation unit 202 demodulates the received broadcast signal. The demodulated data is input to the demultiplexing unit 203.

[0058] The demultiplexing unit 203 separates the demodulated data into division units and outputs the data for each division unit to the decoding units 204A to 204D. Here, a division unit is a divided region obtained by dividing the access unit, for example, a slice segment in H.265. In this case, an 8K4K image is divided into four 4K2K images. Therefore, there are four decoding units 204A to 204D.

[0059] Multiple decoding units 204A to 204D operate synchronously with each other based on a predetermined reference clock. Each decoding unit decodes the encoded data in division units according to the access unit's DTS (Decoding Time Stamp) and outputs the decoding result to the display unit 205.

[0060] The display unit 205 generates an 8K4K output image by integrating multiple decoding results output from multiple decoding units 204A to 204D. The display unit 205 displays the generated output image according to the PTS (Presentation Time Stamp) of the access unit, which is acquired separately. When integrating the decoding results, the display unit 205 may perform filtering such as a deblocking filter on the boundary areas of adjacent division units, such as tile boundaries, so that the boundaries become less noticeable.

[0061] In the above explanation, a transmitting device 100 and a receiving device 200 that transmit or receive broadcasts were used as examples, but the content may also be transmitted and received via a communication network. When the receiving device 200 receives content via a communication network, the receiving device 200 separates the multiplexed data from the IP packets received via a network such as Ethernet.

[0062] In broadcasting, the transmission path delay between the transmission of the broadcast signal and its arrival at the receiving device 200 is constant. On the other hand, in communication networks such as the Internet, the transmission path delay between the data transmitted from the server and its arrival at the receiving device 200 is not constant due to congestion. Therefore, the receiving device 200 often does not perform strict synchronized playback based on a reference clock, such as PCR in broadcast MPEG-2 TS. For this reason, the receiving device 200 may display the 8K4K output image according to the PTS in the display unit without strictly synchronizing each decoding unit.

[0063] Furthermore, due to network congestion or other factors, the decoding process for all segment units may not be completed by the time indicated on the access unit's PTS. In this case, the receiving device 200 either skips the access unit's display or delays the display until decoding of at least four segment units is complete and the generation of the 8K4K image is finished.

[0064] Furthermore, content may be transmitted and received using both broadcasting and telecommunications. This method can also be applied when playing back multiplexed data stored on recording media such as hard disks or memory.

[0065] Next, we will explain the method of multiplexing access units divided into slice segments when MMT is used as the multiplexing scheme.

[0066] Figure 8 shows an example of packetizing HEVC access unit data into MMT. SPS, PPS, and SEI do not necessarily need to be included in the access unit, but this example illustrates the case where they are present.

[0067] NAL units located before the first slice segment within an access unit, such as the access unit delimiter, SPS, PPS, and SEI, are bundled together and stored in MMT packet #1. Subsequent slice segments are stored in separate MMT packets, one for each slice segment.

[0068] As shown in Figure 9, NAL units located before the first slice segment within an access unit may be stored in the same MMT packet as the first slice segment.

[0069] Furthermore, if NAL units such as End-of-Sequence or End-of-Bitstream, which indicate the end of a sequence or stream, are appended after the final slice segment, these are stored in the same MMT packet as the final slice segment. However, since NAL units such as End-of-Sequence or End-of-Bitstream are inserted at the end point of the decoding process or at the connection point of two streams, it may be desirable for the receiving device 200 to be able to easily acquire these NAL units at the multiplexing layer. In this case, these NAL units may be stored in a separate MMT packet from the slice segment. This allows the receiving device 200 to easily separate these NAL units at the multiplexing layer.

[0070] Furthermore, TS, DASH, or RTP may be used as the multiplexing method. In these methods as well, the transmitting device 100 stores different slice segments in different packets. This ensures that the receiving device 200 can separate the slice segments at the multiplexing layer.

[0071] For example, when TS is used, encoded data is packetized into PES packets on a slice segment basis. When RTP is used, encoded data is packetized into RTP packets on a slice segment basis. In these cases as well, the NAL unit and the slice segment, which are placed before the slice segment, may be packetized separately, as shown in MMT packet #1 in Figure 8.

[0072] When TS is used, the transmitting device 100 indicates the unit of data to be stored in the PES packet by using a data alignment descriptor, etc. Also, since DASH is a method that downloads MP4 data units called segments via HTTP, etc., the transmitting device 100 does not packetize the encoded data when transmitting. For this reason, the transmitting device 100 may create subsamples for each slice segment so that the receiving device 200 can detect slice segments at the multiplexing layer in MP4, and store information indicating the storage location of the subsamples in the MP4 header.

[0073] The following provides a detailed explanation of MMT packetization of slice segments.

[0074] As shown in Figure 8, the encoded data is packetized, and data that is commonly referenced when decoding all slice segments within access units such as SPS and PPS is stored in MMT packet #1. In this case, the receiving device 200 concatenates the payload data of MMT packet #1 with the data of each slice segment and outputs the resulting data to the decoding unit. In this way, the receiving device 200 can easily generate input data for the decoding unit by concatenating the payloads of multiple MMT packets.

[0075] Figure 10 shows an example of how input data is generated from the MMT packets shown in Figure 8 for the decoding units 204A to 204D. The demultiplexing unit 203 generates the data necessary for decoding unit 204A to decode slice segment 1 by concatenating the payload data of MMT packet #1 and MMT packet #2. The demultiplexing unit 203 similarly generates input data for decoding units 204B to 204D. That is, the demultiplexing unit 203 generates the input data for decoding unit 204B by concatenating the payload data of MMT packet #1 and MMT packet #3. The demultiplexing unit 203 generates the input data for decoding unit 204C by concatenating the payload data of MMT packet #1 and MMT packet #4. The demultiplexing unit 203 generates the input data for decoding unit 204D by concatenating the payload data of MMT packet #1 and MMT packet #5.

[0076] The demultiplexing unit 203 may remove NAL units that are not necessary for the decoding process, such as the access unit delimiter and SEI, from the payload data of MMT packet #1, and separate only the SPS and PPS NAL units that are necessary for the decoding process and add them to the slice segment data.

[0077] As shown in Figure 9, when encoded data is packetized, the demultiplexing unit 203 outputs MMT packet #1, which contains the leading data of the access unit, to the first decoding unit 204A at the multiplexing layer. The demultiplexing unit 203 also analyzes the MMT packet containing the leading data of the access unit at the multiplexing layer, separates the SPS and PPS NAL units, and adds the separated SPS and PPS NAL units to the data of the second and subsequent slice segments to generate input data for each of the second and subsequent decoding units.

[0078] Furthermore, it is desirable that the receiving device 200 can use the information contained in the header of the MMT packet to identify the type of data stored in the MMT payload, and, if a slice segment is stored in the payload, the index number of that slice segment within the access unit. Here, the data type refers to either pre-sliced-segment data (a collective term for NAL units that are placed before the first slice segment within the access unit) or slice segment data. When storing fragmented units of an MPU, such as a slice segment, in an MMT packet, a mode for storing an MFU (Media Fragment Unit) is used. When using this mode, the transmitting device 100 can, for example, set the Data Unit, which is the basic unit of data in an MFU, to a sample (a data unit in MMT, corresponding to an access unit) or a subsample (a unit obtained by dividing a sample).

[0079] In this case, the header of the MMT packet includes a field called the Fragmentation indicator and a field called the Fragment counter.

[0080] The Fragmentation indicator indicates whether the data stored in the payload of an MMT packet is a fragmented Data unit, and if so, whether the fragment is the first or last fragment in the Data unit, or neither the first nor the last. In other words, the Fragmentation indicator included in the header of a packet is identification information that indicates whether (1) the packet is the only Data unit, which is the basic data unit; (2) the Data unit is divided into multiple packets and the packet in question is the first packet of the Data unit; (3) the Data unit is divided into multiple packets and the packet in question is a packet other than the first or last packet of the Data unit; or (4) the Data unit is divided into multiple packets and the packet in question is the last packet of the Data unit.

[0081] The Fragment counter is an index number that indicates which fragment in the Data unit the data stored in the MMT packet corresponds to.

[0082] Therefore, by setting the samples in the MMT as Data units, and setting the pre-slice segment data and each slice segment as fragment units of Data units, the receiving device 200 can identify the type of data to be stored in the payload using the information contained in the header of the MMT packet. In other words, the demultiplexing unit 203 can generate input data for each decoding unit 204A to 204D by referring to the header of the MMT packet.

[0083] Figure 11 shows an example where a sample is set in a Data unit, and the pre-slice segment data and slice segment are packetized as fragments of the Data unit.

[0084] The data before the slice segment, and the slice segment itself, are divided into five fragments, from Fragment #1 to Fragment #5. Each fragment is stored in a separate MMT packet. The values ​​of the Fragmentation indicator and Fragment counter included in the header of the MMT packet are as shown in the diagram.

[0085] For example, the Fragment indicator is a 2-bit binary value. The Fragment indicators for MMT packet #1, the first packet in a Data unit, MMT packet #5, the last packet in the unit, and MMT packets #2 through #4 in between are each set to different values. Specifically, the Fragment indicator for MMT packet #1 is set to 01, the Fragment indicator for MMT packet #5 is set to 11, and the Fragment indicators for MMT packets #2 through #4 are set to 10. If a Data unit contains only one MMT packet, the Fragment indicator is set to 00.

[0086] Furthermore, the Fragment counter is 4 in MMT packet #1, which is the total number of fragments (5) minus 1. It decreases by 1 in subsequent packets, and is 0 in the final MMT packet #5.

[0087] Therefore, the receiving device 200 can identify MMT packets containing pre-slice segment data using either the Fragment indicator or the Fragment counter. Furthermore, the receiving device 200 can identify MMT packets containing the Nth slice segment by referring to the Fragment counter.

[0088] The header of the MMT packet separately includes the sequence number within the MPU of the Movie Fragment to which the Data unit belongs, the sequence number of the MPU itself, and the sequence number within the Movie Fragment of the sample to which the Data unit belongs. The demultiplexer 203 can uniquely determine the sample to which the Data unit belongs by referring to these.

[0089] Furthermore, the demultiplexing unit 203 can determine the index number of the fragment within the Data unit from the Fragment counter, etc., so even if packet loss occurs, it can uniquely identify the slice segment stored in the fragment. For example, even if fragment #4 shown in Figure 11 could not be obtained due to packet loss, the demultiplexing unit 203 can determine that the fragment received after fragment #3 is fragment #5, and can therefore correctly output the slice segment 4 stored in fragment #5 to the decoding unit 204D instead of the decoding unit 204C.

[0090] Furthermore, if a transmission path is used that guarantees no packet loss, the demultiplexing unit 203 does not need to refer to the header of the MMT packet to determine the type of data stored in the MMT packet or the index number of the slice segment, but can process the arriving packets periodically. For example, if an access unit transmits using a total of five MMT packets—pre-slicing data and four slice segments—the receiving device 200 can obtain the pre-slicing data and the data of the four slice segments sequentially by processing the received MMT packets sequentially after determining the pre-slicing data of the access unit to begin decoding.

[0091] The following describes some variations of packetization.

[0092] The slice segments do not necessarily have to divide the access unit both horizontally and vertically within its plane; as shown in Figure 1, the access unit may be divided only horizontally or only vertically.

[0093] Furthermore, if the access unit is divided only horizontally, tiles do not need to be used.

[0094] Furthermore, the number of in-plane divisions in an access unit is arbitrary and not limited to four. However, the area size of slice segments and tiles must be greater than or equal to the lower limit of the encoding standard such as H.265.

[0095] The transmitting device 100 may store identification information indicating the method of dividing the access unit in the plane in an MMT message or a TS descriptor. For example, information indicating the number of divisions in the horizontal and vertical directions in the plane may be stored. Alternatively, unique identification information may be assigned to the division method, such as being divided into two equal parts in the horizontal and vertical directions as shown in Figure 3, or being divided into four equal parts in the horizontal direction as shown in Figure 1. For example, if the access unit is divided as shown in Figure 3, the identification information indicates mode 1, and if the access unit is divided as shown in Figure 1, the identification information indicates mode 1.

[0096] Furthermore, the multiplexing layer may include information indicating constraints on the encoding conditions related to the method of dividing the plane. For example, information indicating that one slice segment consists of one tile may be used. Alternatively, information indicating that the reference block used when motion compensation is performed during decoding of a slice segment or tile is limited to slice segments or tiles at the same position in the screen, or limited to blocks within a predetermined range in adjacent slice segments may be used.

[0097] Furthermore, the transmitting device 100 may switch whether or not to divide the access unit into multiple slice segments depending on the resolution of the video. For example, the transmitting device 100 may not perform in-plane division when the video to be processed has a resolution of 4K2K, but may divide the access unit into four when the video to be processed has a resolution of 8K4K. By pre-defining the division method for 8K4K video, the receiving device 200 can determine whether or not to perform in-plane division and the division method by acquiring the resolution of the video to be received, and switch the decoding operation accordingly.

[0098] Furthermore, the receiving device 200 can detect whether or not in-plane fragmentation has occurred by referring to the header of the MMT packet. For example, if the access unit is not fragmented, and the MMT Data unit is set to sample, then no fragmentation of the Data unit will occur. Therefore, the receiving device 200 can determine that the access unit is not fragmented if the value of the Fragment counter included in the header of the MMT packet is always zero. Alternatively, the receiving device 200 may detect whether the value of the Fragmentation indicator is always 01. The receiving device 200 can also determine that the access unit is not fragmented if the value of the Fragmentation indicator is always 01.

[0099] Furthermore, the receiving device 200 can also handle cases where the number of in-plane divisions in the access unit does not match the number of decoding units. For example, if the receiving device 200 has two decoding units 204A and 204B that can decode 8K2K encoded data in real time, the demultiplexing unit 203 outputs two of the four slice segments that make up the 8K4K encoded data to the decoding unit 204A.

[0100] Figure 12 shows an example of operation when MMT packetized data, as shown in Figure 8, is input to two decoding units 204A and 204B. Here, it is desirable that the receiving device 200 can directly integrate and output the decoding results from decoding units 204A and 204B. Therefore, the demultiplexing unit 203 selects the slice segments to output to decoding units 204A and 204B so that the decoding results of each decoding unit 204A and 204B are spatially continuous.

[0101] Furthermore, the demultiplexing unit 203 may select the decoding unit to use depending on the resolution or frame rate of the encoded video data. For example, if the receiving device 200 has four 4K2K decoding units, the receiving device 200 will use all four decoding units to perform decoding if the resolution of the input image is 8K4K. Alternatively, if the resolution of the input image is 4K2K, the receiving device 200 will use only one decoding unit to perform decoding. Or, even if the image is divided into four parts, if 8K4K can be decoded in real time by a single decoding unit, the demultiplexing unit 203 will integrate all the division units and output them to a single decoding unit.

[0102] Furthermore, the receiving device 200 may determine which decoding unit to use by considering the frame rate. For example, if the receiving device 200 has two decoding units, each with an upper limit of 60fps that can be decoded in real time when the resolution is 8K4K, there may be cases where encoded data at 120fps in 8K4K is input. In this case, if the plane is composed of four division units, then, as in the example in Figure 12, slice segment 1 and slice segment 2 are input to decoding unit 204A, and slice segment 3 and slice segment 4 are input to decoding unit 204B. Since each decoding unit 204A and 204B can decode up to 120fps in real time if the resolution is 8K2K (half the resolution of 8K4K), the decoding process is performed by these two decoding units 204A and 204B.

[0103] Furthermore, even if the resolution and frame rate are the same, the amount of processing will differ depending on the encoding profile or level, or the encoding method itself, such as H.264 or H.265. Therefore, the receiving device 200 may select the decoding unit to use based on this information. If the receiving device 200 cannot decode all of the encoded data received by broadcast or communication, or if all slice segments or tiles constituting the user-selected area cannot be decoded, the receiving device 200 may automatically determine the slice segments or tiles that can be decoded within the processing range of the decoding unit. Alternatively, the receiving device 200 may provide a user interface for the user to select the area to decode. In this case, the receiving device 200 may display a warning message indicating that not all areas can be decoded, or it may display information indicating the number of decodeable areas, slice segments, or tiles.

[0104] Furthermore, the above method can also be applied when MMT packets containing slice segments of the same encoded data are transmitted and received using multiple transmission paths, such as broadcasting and telecommunications.

[0105] Furthermore, the transmitting device 100 may perform encoding so that the regions of each slice segment overlap in order to make the boundaries between division units less noticeable. In the example shown in Figure 13, an 8K4K picture is divided into four slice segments 1 to 4. Each of slice segments 1 to 3 is, for example, 8K × 1.1K, and slice segment 4 is 8K × 1K. Also, adjacent slice segments overlap each other. In this way, motion compensation during encoding can be efficiently performed at the boundaries of the four divisions shown by the dotted lines, thus improving the image quality at the boundaries. In this way, image quality degradation at the boundaries is reduced.

[0106] In this case, the display unit 205 extracts an 8K × 1K region from the 8K × 1.1K region and integrates the resulting regions. The transmitting device 100 may also include information indicating whether the slice segments are encoded with overlap and the extent of the overlap in the multiplexing layer or within the encoded data and transmit it separately.

[0107] Furthermore, the same method can be applied when tiles are used.

[0108] The following describes the operation flow of the transmitting device 100. Figure 14 is a flowchart showing an example of the operation of the transmitting device 100.

[0109] First, the encoding unit 101 divides the picture (access unit) into multiple slice segments (tiles), which are multiple regions (S101). Next, the encoding unit 101 encodes each of the multiple slice segments in a way that allows for independent decoding, thereby generating encoded data corresponding to each of the multiple slice segments (S102). The encoding unit 101 may encode the multiple slice segments with a single encoding unit, or it may process them in parallel with multiple encoding units.

[0110] Next, the multiplexing unit 102 multiplexes the multiple encoded data generated by the encoding unit 101 by storing them in multiple MMT packets (S103). Specifically, as shown in Figures 8 and 9, the multiplexing unit 102 stores the multiple encoded data in multiple MMT packets so that no single MMT packet contains encoded data corresponding to different slice segments. Also, as shown in Figure 8, the multiplexing unit 102 stores control information commonly used for all decoding units within the picture in MMT packet #1, which is different from the multiple MMT packets #2 to #5 in which the multiple encoded data are stored. Here, the control information includes at least one of the access unit delimiter, SPS, PPS, and SEI.

[0111] The multiplexing unit 102 may store the control information in the same MMT packet as any of the multiple MMT packets that store the multiple encoded data. For example, as shown in Figure 9, the multiplexing unit 102 may store the control information in the first MMT packet (MMT packet #1 in Figure 9) of the multiple MMT packets that store the multiple encoded data.

[0112] Finally, the transmitting device 100 transmits multiple MMT packets. Specifically, the modulation unit 103 modulates the data obtained through multiplexing, and the transmitting unit 104 transmits the modulated data (S104).

[0113] Figure 15 is a block diagram showing an example configuration of the receiving device 200, and is a diagram that shows in detail the configuration of the demultiplexing unit 203 and subsequent stages shown in Figure 7. As shown in Figure 15, the receiving device 200 further includes a decoding command unit 206. The demultiplexing unit 203 also includes a type determination unit 211, a control information acquisition unit 212, a slice information acquisition unit 213, and a decoded data generation unit 214.

[0114] The following describes the operation flow of the receiving device 200. Figure 16 is a flowchart showing an example of the operation of the receiving device 200. Here, the operation for one access unit is shown. If decoding processing is performed for multiple access units, the process in this flowchart is repeated.

[0115] First, the receiving device 200 receives, for example, multiple packets (MMT packets) generated by the transmitting device 100 (S201).

[0116] Next, the type determination unit 211 obtains the type of encoded data stored in the received packet by analyzing the header of the received packet (S202).

[0117] Next, the type determination unit 211 determines, based on the type of the acquired encoded data, whether the data stored in the received packet is data before the slice segment or data from the slice segment (S203).

[0118] If the data stored in the received packet is pre-slice segment data (Yes in S203), the control information acquisition unit 212 acquires the pre-slice segment data of the access unit to be processed from the payload of the received packet and stores the pre-slice segment data in memory (S204).

[0119] On the other hand, if the data stored in the received packet is slice segment data (No in S203), the receiving device 200 uses the header information of the received packet to determine which of the multiple regions the data stored in the received packet belongs to. Specifically, the slice information acquisition unit 213 analyzes the header of the received packet to obtain the index number Idx of the slice segment stored in the received packet (S205). Specifically, the index number Idx is the index number within the Movie Fragment of the access unit (sample in MMT).

[0120] Note that the processing in step S205 may be performed together in step S202.

[0121] Next, the decryption data generation unit 214 determines the decryption unit to decrypt the slice segment (S206). Specifically, an index number Idx is pre-associated with multiple decryption units, and the decryption data generation unit 214 determines the decryption unit to decrypt the slice segment by selecting the decryption unit corresponding to the index number Idx obtained in step S205.

[0122] Furthermore, as explained in the example in Figure 12, the decoded data generation unit 214 may determine which decoding unit to decode a slice segment based on at least one of the resolution of the access unit (picture), the method of dividing the access unit into multiple slice segments (tiles), and the processing capacity of the multiple decoding units provided in the receiving device 200. For example, the decoded data generation unit 214 determines the method of dividing the access unit based on identification information in a descriptor such as an MMT message or a TS section.

[0123] Next, the decryption data generation unit 214 generates multiple input data (combined data) to be input to multiple decoding units by combining control information, which is included in any of the multiple packets and is used in common for all decoding units within the picture, with each of the multiple encoded data of the multiple slice segments. Specifically, the decryption data generation unit 214 obtains slice segment data from the payload of the received packet. The decryption data generation unit 214 generates the input data to the decoding unit determined in step S206 (S207) by combining the pre-slice segment data stored in memory in step S204 with the obtained slice segment data.

[0124] If, after step S204 or S207, the data of the received packet is not the final data for the access unit (No in S208), the processing from step S201 onwards is repeated. In other words, the above processing is repeated until input data for multiple decoding units 204A to 204D corresponding to all slice segments contained in the access unit is generated.

[0125] Note that the timing of packet reception is not limited to the timing shown in Figure 16; multiple packets may be received in advance or sequentially and stored in memory or elsewhere.

[0126] On the other hand, if the data of the received packet is the final data of the access unit (Yes in S208), the decoding instruction unit 206 outputs the multiple input data generated in step S207 to the corresponding decoding units 204A to 204D (S209).

[0127] Next, the multiple decoding units 204A to 204D generate multiple decoded images by decoding multiple input data in parallel according to the access unit's DTS (S210).

[0128] Finally, the display unit 205 generates a display image by combining multiple decoded images generated by the multiple decoding units 204A to 204D, and displays the display image according to the access unit's PTS (S211).

[0129] The receiving device 200 obtains the access unit's DTS and PTS by analyzing the payload data of the MMT packet, which contains the MPU header information or the Movie Fragment header information. Furthermore, if TS is used as the multiplexing method, the receiving device 200 obtains the access unit's DTS and PTS from the PES packet header. If RTP is used as the multiplexing method, the receiving device 200 obtains the access unit's DTS and PTS from the RTP packet header.

[0130] Furthermore, when the display unit 205 integrates the decoding results of multiple decoding units, it may perform filtering, such as deblocking, at the boundaries of adjacent division units. Note that filtering is not necessary when displaying the decoding result of a single decoding unit, so the display unit 205 may switch processing depending on whether or not filtering is performed at the boundaries of the decoding results of multiple decoding units. Whether or not filtering is necessary may be predetermined depending on whether or not there is division. Alternatively, information indicating whether or not filtering is necessary may be stored separately in the multiplexing layer. In addition, information necessary for filtering, such as filter coefficients, may be stored in the SPS, PPS, SEI, or slice segments. The decoding units 204A to 204D, or the demultiplexing unit 203, acquire this information by analyzing the SEI and output the acquired information to the display unit 205. The display unit 205 performs filtering using this information. Note that if this information is stored in slice segments, it is desirable that the decoding units 204A to 204D acquire this information.

[0131] In the above explanation, an example was shown where there are two types of data stored in the fragment: data before the slice segment and the slice segment itself. However, there may be three or more types of data. In this case, step S203 will perform a case distinction according to the type.

[0132] Furthermore, the transmitting device 100 may fragment the slice segment and store it in an MMT packet if the data size of the slice segment is large. In other words, the transmitting device 100 may fragment the pre-slice segment data and the slice segment. In this case, if the access unit and the data unit are set to be equal, as in the packetization example shown in Figure 11, the following problems arise.

[0133] For example, if slice segment 1 is divided into three fragments, slice segment 1 will be divided into three packets with Fragment counter values ​​ranging from 1 to 3 and transmitted. Furthermore, in slice segments 2 and beyond, the Fragment counter value will be 4 or higher, making it impossible to associate the Fragment counter value with the data stored in the payload. Consequently, the receiving device 200 cannot identify the packet containing the first data of a slice segment from the information in the header of the MMT packet.

[0134] In such cases, the receiving device 200 may analyze the data of the MMT packet payload to determine the starting position of the slice segment. Here, in H.264 or H.265, there are two formats for storing NAL units in the multiplexing layer: a format called the byte stream format, in which a start code consisting of a specific bit sequence is added immediately before the NAL unit header, and a format called the NAL size format, in which a field indicating the size of the NAL unit is added.

[0135] The byte stream format is used in MPEG-2 systems and RTP, among others. The NAL size format is used in MP4, and in DASH and MMT, which use MP4.

[0136] When a byte stream format is used, the receiving device 200 analyzes whether the leading data of the packet matches the start code. If the leading data of the packet matches the start code, the receiving device 200 can determine whether the data contained in the packet is slice segment data by obtaining the type of NAL unit from the subsequent NAL unit header.

[0137] On the other hand, in the case of the NAL size format, the receiving device 200 cannot detect the starting position of the NAL unit based on the bit sequence. Therefore, in order to obtain the starting position of the NAL unit, the receiving device 200 needs to shift the pointer by reading data equal to the size of the NAL unit, starting from the first NAL unit of the access unit.

[0138] However, if the header of the MPU or Movie Fragment in MMT indicates the size in units of subsamples, and the subsamples correspond to pre-slicing data or slice segments, the receiving device 200 can determine the starting position of each NAL unit based on the subsample size information. Therefore, the transmitting device 100 may include information indicating whether subsample-level information exists in the MPU or Movie Fragment in the information that the receiving device 200 acquires at the start of data reception, such as the MPT in MMT.

[0139] The MPU data is an extension based on the MP4 format. In MP4, there are modes that allow the storage of parameter sets such as SPS and PPS for H.264 or H.265 as sample data, and modes that do not. Information to identify this mode is indicated by the entry name of SampleEntry. If the mode that allows storage is used and the parameter set is included in the sample, the receiver 200 acquires the parameter set using the method described above.

[0140] On the other hand, if a mode that does not allow storage is used, the parameter set is stored as Decoder Specific Information within SampleEntry, or stored using a stream for the parameter set. Here, since the stream for the parameter set is not generally used, it is desirable for the transmitting device 100 to store the parameter set in Decoder Specific Information. In this case, the receiving device 200 parses the SampleEntry that is transmitted in the MMT packet as MPU metadata or Movie Fragment metadata to obtain the parameter set that the access unit refers to.

[0141] If the parameter set is stored as sample data, the receiver 200 can obtain the parameter set necessary for decoding by referring only to the sample data without referring to SampleEntry. In this case, the transmitter 100 does not need to store the parameter set in SampleEntry. This allows the transmitter 100 to use the same SampleEntry on different MPUs, thereby reducing the processing load on the transmitter 100 during MPU generation. Furthermore, there is the advantage that the receiver 200 does not need to refer to the parameter set in SampleEntry.

[0142] Alternatively, the transmitting device 100 may store one default parameter set in SampleEntry and store the parameter set referenced by the access unit in the sample data. In conventional MP4, it was common to store parameter sets in SampleEntry, so there is a possibility that some receiving devices may stop playback if no parameter set exists in SampleEntry. This problem can be solved by using the method described above.

[0143] Alternatively, the transmitter 100 may store the parameter set in the sample data only if a parameter set different from the default parameter set is used.

[0144] In both modes, it is possible to store the parameter set in SampleEntry. Therefore, the transmitter 100 may always store the parameter set in VisualSampleEntry, and the receiver 200 may always retrieve the parameter set from VisualSampleEntry.

[0145] In the MMT standard, MP4 header information such as Moov and Moof is transmitted as MPU metadata or movie fragment metadata, but the transmitting device 100 does not necessarily have to transmit MPU metadata and movie fragment metadata. Furthermore, the receiving device 200 can also determine whether SPS and PPS are stored in the sample data based on the service or asset type according to the ARIB (Association of Radio Industries and Businesses) standard, or whether or not MPU metadata is transmitted.

[0146] Figure 17 shows an example where the data before the slice segment and each slice segment are set to different data units.

[0147] In the example shown in Figure 17, the data size of the data before the slice segment, and the data sizes of slice segments 1 through 4, are Length#1 through Length#5, respectively. The field values ​​of Fragmentation indicator, Fragment counter, and Offset included in the MMT packet header are as shown in the figure.

[0148] Here, Offset is offset information indicating the bit length (offset) from the beginning of the encoded data of the sample (access unit or picture) to which the payload data belongs, to the first byte of the payload data (encoded data) contained in the MMT packet. Note that while the Fragment counter value is explained as starting from the total number of fragments minus 1, it may start from other values.

[0149] Figure 18 shows an example of when a data unit is fragmented. In the example shown in Figure 18, slice segment 1 is divided into three fragments, each stored in MMT packet #2 to MMT packet #4. In this case, if the data size of each fragment is Length#2_1 to Length#2_3, the values ​​of each field are as shown in the figure.

[0150] Thus, when data units such as slice segments are set as Data units, the beginning of the access unit and the beginning of the slice segment can be determined as follows based on the field values ​​in the MMT packet header.

[0151] In packets where the Offset value is 0, the beginning of the payload is the beginning of the access unit.

[0152] The beginning of a packet payload where the Offset value is not 0 and the Fragmentation indicator value is 00 or 01 is the beginning of the slice segment.

[0153] Furthermore, if no fragmentation of the data unit occurs and no packet loss occurs, the receiving device 200 can identify the index number of the slice segment to be stored in the MMT packet based on the number of slice segments obtained after detecting the beginning of the access unit.

[0154] Similarly, even when the data unit of the data before the slice segment is fragmented, the receiving device 200 can detect the access unit and the beginning of the slice segment.

[0155] Furthermore, even if packet loss occurs, or if the SPS, PPS, and SEI included in the pre-slice segment data are set to separate Data units, the receiving device 200 can identify the MMT packet containing the first data of the slice segment based on the analysis results of the MMT header, and then identify the starting position of the slice segment or tile within the picture (access unit) by analyzing the slice segment header. The processing load involved in analyzing the slice header is small, so processing load is not a problem.

[0156] Thus, each of the multiple encoded data in multiple slice segments is associated one-to-one with a basic data unit (DATA unit), which is the unit of data stored in one or more packets. Furthermore, each of the multiple encoded data is stored in one or more MMT packets.

[0157] The header information of each MMT packet includes a Fragmentation indicator (identification information) and an Offset (offset information).

[0158] The receiving device 200 determines that the beginning of the payload data contained in a packet having header information that includes a Fragmentation indicator with a value of 00 or 01 is the beginning of the encoded data for each slice segment. Specifically, it determines that the beginning of the payload data contained in a packet having header information that includes an Offset with a non-zero value and a Fragmentation indicator with a value of 00 or 01 is the beginning of the encoded data for each slice segment.

[0159] Furthermore, in the example in Figure 17, the beginning of the Data unit is either the beginning of the access unit or the beginning of the slice segment, and the Fragmentation indicator value is 00 or 01. In addition, the receiving device 200 can also detect the beginning of the access unit or the beginning of the slice segment without referring to the Offset by referring to the type of NAL unit and determining whether the beginning of the Data Unit is an access unit delimiter or a slice segment.

[0160] Thus, by the transmitting device 100 packetizing the data so that the beginning of the NAL unit always starts from the beginning of the MMT packet payload, the receiving device 200 can detect the access unit or the beginning of the slice segment by analyzing the Fragmentation indicator and the NAL unit header, even when the pre-slice segment data is divided into multiple data units. The type of the NAL unit is found in the first byte of the NAL unit header. Therefore, when the receiving device 200 analyzes the header portion of the MMT packet, it can obtain the type of the NAL unit by analyzing an additional byte of data.

[0161] In the case of audio, the receiving device 200 only needs to be able to detect the beginning of the access unit and make a determination based on whether the value of the Fragmentation indicator is 00 or 01.

[0162] Furthermore, as mentioned above, when encoding data that has been encoded in a way that allows for segmental decoding is stored in an MPEG-2 TS PES packet, the transmitting device 100 can use a data alignment descriptor. Below, an example of how to store the encoded data in a PES packet will be described in detail.

[0163] For example, in HEVC, the transmitter 100 can use a data alignment descriptor to indicate whether the data stored in the PES packet is an access unit, a slice segment, or a tile. The alignment types in HEVC are defined as follows:

[0164] Alignment type 8 indicates an HEVC slice segment. Alignment type 9 indicates an HEVC slice segment or access unit. Alignment type 12 indicates an HEVC slice segment or tile.

[0165] Therefore, the transmitting device 100 can indicate, for example, that the data of a PES packet is either a slice segment or pre-slice segment data by using type 9. Since a separate type is also defined for indicating a slice instead of a slice segment, the transmitting device 100 may also use a type that indicates a slice instead of a slice segment.

[0166] Furthermore, the DTS and PTS included in the PES packet header are set only in PES packets that contain the initial data of an access unit. Therefore, the receiving device 200 can determine that if the type is 9 and the PES packet contains a DTS or PTS field, the PES packet contains the entire access unit or the initial segment of the access unit.

[0167] Furthermore, the transmitting device 100 may use a field such as transport_priority, which indicates the priority of the TS packet containing the PES packet containing the access unit's leading data, to enable the receiving device 200 to distinguish the data contained in the packets. Alternatively, the receiving device 200 may determine the data contained in the packets by analyzing whether the payload of the PES packet is an access unit delimiter. Additionally, the data_alignment_indicator in the PES packet header indicates whether the data in the PES packet is stored according to these types. If this flag (data_alignment_indicator) is set to 1, it is guaranteed that the data stored in the PES packet conforms to the type indicated in the data alignment descriptor.

[0168] Furthermore, the transmitting device 100 may use a data alignment descriptor only when PES packetizing in a decomposable unit such as a slice segment. This allows the receiving device 200 to determine that the encoded data is PES packetized in a decomposable unit if a data alignment descriptor is present, and that the encoded data is PES packetized in an access unit unit if a data alignment descriptor is not present. Note that the MPEG-2 TS standard specifies that if data_alignment_indicator is set to 1 and no data alignment descriptor is present, the unit of PES packetization is an access unit.

[0169] If the PMT contains a data alignment descriptor, the receiving device 200 determines that the data is packetized into PES packets in units that can be divided and decoded, and can generate input data for each decoding unit based on the packetized units. If the PMT does not contain a data alignment descriptor, and the receiving device 200 determines that parallel decoding of the encoded data is necessary based on program information or other descriptor information, it generates input data for each decoding unit by analyzing the slice header of the slice segment, etc. Furthermore, if the encoded data can be decoded by a single decoding unit, the receiving device 200 decodes the data of the entire access unit with that decoding unit. If information indicating whether the encoded data consists of units that can be divided and decoded, such as slice segments or tiles, is separately indicated by the PMT descriptor, etc., the receiving device 200 may determine whether the encoded data can be decoded in parallel based on the analysis results of the descriptor.

[0170] Furthermore, since the DTS and PTS included in the PES packet header are set only in the PES packet containing the access unit's initial data, if the access unit is split and converted into PES packets, the second and subsequent PES packets will not contain information indicating the access unit's DTS and PTS. Therefore, when the decoding process is performed in parallel, each decoding unit 204A to 204D and the display unit 205 use the DTS and PTS stored in the header of the PES packet containing the access unit's initial data.

[0171] (Embodiment 2) Embodiment 2 describes a method for storing NAL-sized data in an MP4-format-based MPU in MMT. While the following description presents an example of a storage method for an MPU used in MMT, this storage method is also applicable to DASH, which is also MP4-format-based.

[0172] [How to store data on the MPU] In the MP4 format, multiple access units are grouped together and stored in a single MP4 file. In MMT, the MPU (Multi-Purpose Unit) stores data for each media within a single MP4 file, and this data can contain any number of access units. Since an MPU is a self-decrypting unit, for example, an MPU can store access units in the GOP (Group of Processors) unit.

[0173] Figure 19 shows the configuration of the MPU. At the top of the MPU are ftyp, mmpu, and moov, which are collectively defined as MPU metadata. moov stores common initialization information and MMT hint tracks in a file.

[0174] Furthermore, moof stores initialization information and size for each sample and subsample, information that identifies the presentation time (PTS) and decoding time (DTS) (sample_duration, sample_size, sample_composition_time_offset), and data_offset which indicates the data's position.

[0175] Furthermore, multiple access units are each stored as samples in an mdat (mdat box). Data in moof and mdat, excluding the samples, is defined as movie fragment metadata (hereinafter referred to as MF metadata), and the sample data in mdat is defined as media data.

[0176] Figure 20 shows the structure of MF metadata. As shown in Figure 20, MF metadata consists, more specifically, the type, length, and data of the moof box (moof) and the type and length of the mdat box (mdat).

[0177] When storing access units in MP4 data, there are modes that allow the storage of parameter sets such as H.264 and H.265 SPS and PPS as sample data, and modes that do not.

[0178] In the modes where storage is not possible, the parameter set is stored in the Decoder Specific Information of SampleEntry in moov. In the modes where storage is possible, the parameter set is included within the sample.

[0179] MPU metadata, MF metadata, and media data are stored in the MMT payload, and the MMT payload header contains the Fragment Type (FT) as an identifier to identify this data. FT=0 indicates MPU metadata, FT=1 indicates MF metadata, and FT=2 indicates media data.

[0180] Figure 19 illustrates an example where MPU metadata units and MF metadata units are stored as data units in the MMT payload. However, units such as ftyp, mmpu, moov, and moof may also be stored as data units in the MMT payload on a data unit basis. Similarly, Figure 19 illustrates an example where sample units are stored as data units in the MMT payload. However, data units may be composed of sample units or NAL units, and such data units may be stored in the MMT payload on a data unit basis. Such data units may also be stored in the MMT payload in further fragmented units.

[0181] [Traditional transmission methods and challenges] Traditionally, when encapsulating multiple access units in MP4 format, the moov and moof files were created only after all the samples to be stored in the MP4 were available.

[0182] When transmitting MP4 format in real time using broadcasting or other methods, if, for example, the samples stored in a single MP4 file are in GOP units, then moov and moof files are created after the GOP-unit time samples have been accumulated, resulting in a delay due to encapsulation. This encapsulation on the transmitting side always increases the end-to-end delay by the amount of time in GOP units. This makes it difficult to provide services in real time, and in particular, when live content is transmitted, it leads to a degradation of service for viewers.

[0183] Figure 21 is a diagram illustrating the data transmission order. When MMT is applied to broadcasting, as shown in Figure 21(a), if the MMT packets are transmitted in the order of the MPU configuration (MMT packets #1, #2, #3, #4, #5, #6 are transmitted in that order), a delay occurs in the transmission of the MMT packets due to encapsulation.

[0184] To prevent delays caused by this encapsulation, a method has been proposed, as shown in Figure 21(b), that does not send MPU header information such as MPU metadata and MF metadata (packets #1 and #2 are not sent, and packets #3-#6 are sent in that order). Alternatively, as shown in Figure 20(c), a method can be considered in which media data is sent first without waiting for the creation of MPU header information, and then the MPU header information is sent after the media data has been sent (sent in the order of #3-#6, #1, #2).

[0185] If MPU header information is not transmitted, the receiving device will decode without using the MPU header information. Also, if the MPU header information is sent later than the media data, the receiving device will wait to obtain the MPU header information before decoding.

[0186] However, conventional MP4-compliant receivers are not guaranteed to decode without using MPU header information. Furthermore, if the receiver performs decoding without using the MPU header through special processing, using conventional transmission methods would make the decoding process complicated, making real-time decoding highly unlikely. In addition, if the receiver waits to acquire MPU header information before decoding, buffering of media data is necessary until the receiver acquires the header information, but the buffer model is not defined, and decoding is not guaranteed.

[0187] Therefore, the transmitting device according to Embodiment 2 transmits the MPU metadata before the media data by storing only common information in the MPU metadata, as shown in Figure 20(d). The transmitting device according to Embodiment 2 transmits the MF metadata, which has a delay in generation, after the media data. This provides a transmission or reception method that can guarantee the decoding of media data.

[0188] The following describes the receiving methods when using each of the transmission methods shown in Figure 21 (a)-(d).

[0189] In each transmission method shown in Figure 21, the MPU data is first composed in the following order: MPU metadata, MFU metadata, and media data.

[0190] After configuring the MPU data, if the transmitting device transmits the data in the order of MPU metadata, MF metadata, and media data as shown in Figure 21(a), the receiving device can decode it using either of the following methods (A-1) or (A-2).

[0191] (A-1) The receiving device acquires MPU header information (MPU metadata and MF metadata) and then decodes the media data using the MPU header information.

[0192] (A-2) The receiving device decodes the media data without using the MPU header information.

[0193] While all of these methods introduce encapsulation-induced delays on the transmitting side, they have the advantage of eliminating the need for buffering media data in the receiving device to acquire the MPU header. Without buffering, there is no need for buffering memory, and furthermore, no buffering delay occurs. Additionally, method (A-1) uses MPU header information for decoding, making it applicable to conventional receiving devices as well.

[0194] If the transmitting device transmits only media data, as shown in Figure 21(b), the receiving device can decode it using the method described in (B-1) below.

[0195] (B-1) The receiving device decodes the media data without using the MPU header information.

[0196] Furthermore, although not shown in the diagram, if the MPU metadata is transmitted before the media data transmission shown in Figure 21(b), decryption can be performed using the method described in (B-2) below.

[0197] (B-2) The receiving device decodes the media data using the MPU metadata.

[0198] The advantages of methods (B-1) and (B-2) above are that they do not cause delays due to encapsulation on the transmitting side and do not require buffering of media data to obtain the MPU header. However, since neither method (B-1) nor (B-2) performs decoding using MPU header information, special processing may be required for decoding.

[0199] When the transmitting device transmits data in the order of media data, MPU metadata, and MF metadata, as shown in Figure 21(c), the receiving device can decode the data using either of the following methods (C-1) or (C-2).

[0200] (C-1) The receiving device acquires the MPU header information (MPU metadata and MF metadata) and then decodes the media data.

[0201] (C-2) The receiving device decodes the media data without using the MPU header information.

[0202] When method (C-1) above is used, media data needs to be buffered in order to obtain MPU header information. In contrast, when method (C-2) above is used, buffering is not required to obtain MPU header information.

[0203] Furthermore, neither method (C-1) nor (C-2) above causes any delay due to encapsulation on the transmitting side. Also, method (C-2) does not use MPU header information, so special processing may be required.

[0204] When the transmitting device transmits data in the order of MPU metadata, media data, and MF metadata, as shown in Figure 21(d), the receiving device can decode the data using either of the following methods (D-1) or (D-2).

[0205] (D-1) The receiving device acquires MPU metadata, then acquires MF metadata, and then decodes the media data.

[0206] (D-2) After acquiring the MPU metadata, the receiving device decodes the media data without using the MF metadata.

[0207] When method (D-1) above is used, media data needs to be buffered in order to obtain MF metadata, but when method (D-2) above is used, buffering for obtaining MF metadata is not required.

[0208] The method described above (D-2) does not perform decryption using MF metadata, so special processing may be required.

[0209] As explained above, if decoding is possible using MPU metadata and MF metadata, there is the advantage that it can also be decoded by conventional MP4 receivers.

[0210] In Figure 21, the MPU data is structured in the order of MPU metadata, MFU metadata, and media data. In moof, the positional information (offset) for each sample and subsample is determined based on this structure. The MF metadata also includes data other than the media data in the mdat box (such as the size and type of the box).

[0211] Therefore, when the receiving device identifies media data based on MF metadata, it reconstructs the data in the order in which the MPU data was composed, regardless of the order in which the data was transmitted, and then decodes it using the moov of the MPU metadata or the moof of the MF metadata.

[0212] In Figure 21, the MPU data is structured in the order of MPU metadata, MFU metadata, and media data. However, the MPU data may be structured in a different order than shown in Figure 21, and location information (offset) may be defined accordingly.

[0213] For example, MPU data may be composed of MPU metadata, media data, and MF metadata in that order, and negative positional information (offset) may be indicated in the MF metadata. In this case as well, regardless of the order in which the data is transmitted, the receiving device reconstructs the data in the order in which the MPU data was composed on the transmitting side, and then decodes it using moov or moof.

[0214] The transmitting device may signal information indicating the order in which the MPU data is composed, and the receiving device may reconstruct the data based on the signaled information.

[0215] As explained above, the receiving device receives the packetized MPU metadata, packetized media data (sample data), and packetized MF metadata in this order, as shown in Figure 21(d). Here, the MPU metadata is an example of the first metadata, and the MF metadata is an example of the second metadata.

[0216] Next, the receiving device reconstructs the MPU data (a file in MP4 format) containing the received MPU metadata, received MF metadata, and received sample data. Then, it decodes the sample data contained in the reconstructed MPU data using the MPU metadata and MF metadata. The MF metadata is metadata that can only be generated by the transmitting side after the sample data has been generated (for example, the length stored in the mbox).

[0217] The operation of the receiving device described above is, in more detail, carried out by each component that makes up the receiving device. For example, the receiving device comprises a receiving unit that receives the above data, a reconstruction unit that reconstructs the above MPU data, and a decoding unit that decodes the above MPU data. The receiving unit, generation unit, and decoding unit are each implemented by a microcomputer, processor, dedicated circuit, etc.

[0218] [Method for decryption without using header information] Next, we will explain a method for decoding without using header information. Here, we will describe a method for decoding without using header information in the receiving device, regardless of whether the sender sends header information or not. In other words, this method is applicable to any of the transmission methods explained using Figure 21. However, some decoding methods are applicable only to specific transmission methods.

[0219] Figure 22 shows an example of a method for decoding without using header information. In Figure 22, only MMT payloads and MMT packets containing only media data are shown; MMT payloads and MMT packets containing MPU metadata or MF metadata are not shown. Furthermore, in the following explanation of Figure 22, it is assumed that media data belonging to the same MPU is transmitted continuously. In addition, the explanation uses the example where a sample is stored in the payload as media data, but in the following explanation of Figure 22, it is of course possible that a NAL unit or a fragmented NAL unit is stored.

[0220] To decode media data, the receiving device must first acquire the initialization information necessary for decoding. Furthermore, if the media is video, the receiving device must acquire initialization information for each sample, identify the starting position of the MPU (Mass Processing Unit), and obtain the starting positions of the sample and NAL (Non-Aligned Log) unit. The receiving device also needs to identify the decoding time (DTS) and presentation time (PTS) for each sample.

[0221] Therefore, the receiving device can decode the data without using header information, for example, by using the method described below. Note that if the payload contains NAL units or fragmented NAL units, then in the following explanation, "sample" should be read as "NAL unit in the sample."

[0222] <Random access (=identifying the first sample on the MPU)> If header information is not transmitted, the receiving device can identify the first sample of the MPU using the following methods 1 and 2. If header information is transmitted, method 3 can be used.

[0223] [Method 1] The receiving device obtains samples contained in MMT packets where the MMT packet header has 'RAP_flag=1'.

[0224] [Method 2] The receiving device acquires a sample with'sample number = 0' in the MMT payload header.

[0225] [Method 3] When at least one of the MPU metadata and the MF metadata is transmitted before or after the media data, the receiving device acquires a sample included in the MMT payload in which the fragment type (FT) in the MMT payload header has switched to the media data.

[0226] In Methods 1 and 2, when multiple samples belonging to different MPUs are mixed in one payload, it is impossible to determine which NAL unit is a random access point (RAP_flag = 1 or sample number = 0). For this reason, restrictions such as not mixing samples of different MPUs in one payload, or when samples of different MPUs are mixed in one payload, setting the RAP_flag to 1 when the last (or first) sample is a random access point are necessary.

[0227] Also, in order for the receiving device to acquire the start position of the NAL unit, it is necessary to shift the data read pointer by the size of the NAL unit in order from the first NAL unit of the sample.

[0228] When the data is fragmented, the receiving device can identify the data unit by referring to the fragment_indicator and the fragment_number.

[0229] <Determination of the DTS of the Sample> There are the following Method 1 and Method 2 for determining the DTS of the sample.

[0230] [Method 1] The receiving device determines the DTS of the first sample based on the predicted structure. However, this method requires analysis of the encoded data, and real-time decoding may be difficult, so Method 2 is preferable.

[0231] [Method 2] The receiving device separately transmits the DTS of the first sample and obtains the transmitted DTS of the first sample. Methods for transmitting the DTS of the first sample include, for example, transmitting the DTS of the MPU first sample using MMT-SI, or transmitting the DTS for each sample using the MMT packet header extension area. The DTS may be an absolute value or a relative value to the PTS. The transmitting side may also signal whether the DTS of the first sample is included.

[0232] Note that in both Method 1 and Method 2, the DTS of the following samples will be calculated assuming a fixed frame rate.

[0233] In addition to using the extended area to store the DTS for each sample in the packet header, another method is to store the DTS of the samples contained in the MMT packet in the 32-bit NTP timestamp field of the MMT packet header. If the DTS cannot be represented with the number of bits (32 bits) of a single packet header, the DTS may be represented using multiple packet headers. Furthermore, the DTS may be represented by a combination of the NTP timestamp field and the extended area of ​​the packet header. If no DTS information is included, a known value (e.g., ALL0) is used.

[0234] <Determination of PTS for the sample> The receiving device obtains the PTS of the first sample from the MPU timestamp descriptor for each asset included in the MPU. For subsequent sample PTSs, the receiving device calculates them based on parameters indicating the display order of samples such as POCs, assuming a fixed frame rate. Thus, in order to calculate DTS and PTS without using header information, transmission at a fixed frame rate is essential.

[0235] Furthermore, if MF metadata is transmitted, the receiving device can calculate the absolute values ​​of the DTS and PTS from the relative time information of the DTS and PTS from the first sample indicated in the MF metadata and the absolute value of the timestamp of the first MPU sample indicated in the MPU timestamp descriptor.

[0236] Furthermore, when calculating DTS and PTS by analyzing encoded data, the receiving device may use the SEI information included in the access unit for calculation.

[0237] <Initialization Information (Parameter Set)> [For video] In the case of video, the parameter set is stored in the sample data. Furthermore, if MPU metadata and MF metadata are not transmitted, it is guaranteed that the parameter set necessary for decoding can be obtained by referring only to the sample data.

[0238] Furthermore, as shown in Figures 21(a) and (d), if the MPU metadata is transmitted before the media data, it may be specified that the parameter set is not stored in SampleEntry. In this case, the receiving device will refer only to the parameter set within the sample and not to the parameter set in SampleEntry.

[0239] Furthermore, if MPU metadata is transmitted before media data, SampleEntry will store a common parameter set for the MPU and a default parameter set, and the receiving device may refer to the parameter set in SampleEntry and the parameter set within the sample. By storing the parameter set in SampleEntry, decoding becomes possible even for conventional receiving devices that cannot play back if the parameter set is not present in SampleEntry.

[0240] [For audio] In the case of audio, a LATM header is required for decoding, and in MP4, the LATM header must be included in the sample entry. However, if the header information is not transmitted, it is difficult for the receiving device to obtain the LATM header, so the LATM header is included separately in control information such as SI. The LATM header may also be included in the message, table, or descriptor. In addition, the LATM header may be included within the sample.

[0241] The receiving device obtains the LATM header from the SI or other source before starting decoding, and then begins decoding the audio. Alternatively, as shown in Figures 21(a) and 21(d), if the MPU metadata is transmitted before the media data, the receiving device can receive the LATM header before the media data. Therefore, if the MPU metadata is transmitted before the media data, decoding can be performed even using conventional receiving devices.

[0242] <Other> The transmission order and transmission order type may be notified as control information in the MMT packet header, payload header, or MPT or other tables, messages, descriptors, etc. The transmission order type here refers to, for example, the four types of transmission orders shown in Figure 21 (a) to (d), and it is sufficient that an identifier for each type is stored in a location that can be obtained before decryption begins.

[0243] Furthermore, different transmission order types may be used for audio and video, or a common type may be used for both audio and video. Specifically, for example, audio may be transmitted in the order of MPU metadata, MF metadata, and media data, as shown in Figure 21(a), while video may be transmitted in the order of MPU metadata, media data, and MF metadata, as shown in Figure 21(d).

[0244] By the method described above, the receiving device can perform decoding without using the header information. Also, when the MPU metadata is transmitted before the media data (in (a) and (d) of FIG. 21), even a conventional receiving device can perform decoding.

[0245] In particular, since the MF metadata is transmitted after the media data (in (d) of FIG. 21), it is possible to avoid the delay caused by encapsulation and enable decoding even by a conventional receiving device.

[0246] [Configuration and Operation of Transmitting Device] Next, the configuration and operation of the transmitting device will be described. FIG. 23 is a block diagram of the transmitting device according to Embodiment 2, and FIG. 24 is a flowchart of the transmitting method according to Embodiment 2.

[0247] As shown in FIG. 23, the transmitting device 15 includes an encoding unit 16, a multiplexing unit 17, and a transmitting unit 18.

[0248] The encoding unit 16 generates encoded data by encoding the video or audio to be encoded, for example, according to H.265 (S10).

[0249] The multiplexing unit 17 multiplexes (packetizes) the encoded data generated by the encoding unit 16 (S11). Specifically, the multiplexing unit 17 packetizes each of the sample data, MPU metadata, and MF metadata that constitute a file in the MP4 format. The sample data is data obtained by encoding a video signal or an audio signal, the MPU metadata is an example of the first metadata, and the MF metadata is an example of the second metadata. Both the first metadata and the second metadata are metadata used for decoding the sample data, but the difference between them is that the second metadata includes data that can be generated only after the generation of the sample data.

[0250] Here, data that can only be generated after the generation of sample data is, for example, data other than the sample data stored in the mdat file in the MP4 format (data in the mdat header, i.e., type and length as shown in Figure 20). Here, the second metadata only needs to include at least a part of this data, namely length.

[0251] The transmitting unit 18 transmits a packetized MP4 format file (S12). The transmitting unit 18 transmits the MP4 format file in the manner shown in Figure 21(d), for example. That is, it transmits the packetized MPU metadata, packetized sample data, and packetized MF metadata in that order.

[0252] The encoding unit 16, the multiplexing unit 17, and the transmission unit 18 are each implemented by a microcomputer, processor, or dedicated circuitry.

[0253] [Configuration of the receiving device] Next, the configuration and operation of the receiving device will be described. Figure 25 is a block diagram of the receiving device according to Embodiment 2.

[0254] As shown in Figure 25, the receiving device 20 includes a packet filtering unit 21, a transmission order type determination unit 22, a random access unit 23, a control information acquisition unit 24, a data acquisition unit 25, a PTS / DTS calculation unit 26, an initialization information acquisition unit 27, a decoding command unit 28, a decoding unit 29, and a presentation unit 30.

[0255] [Receiver operation 1] First, we will describe the operation of the receiving device 20 to identify the MPU starting position and the NAL unit position when the media is video. Figure 26 is a flowchart of this operation of the receiving device 20. Here, we assume that the transmission order type of the MPU data is stored in the SI information by the transmitting device 15 (multiplexing unit 17).

[0256] First, the packet filtering unit 21 performs packet filtering on the received file. The transmission order type determination unit 22 analyzes the SI information obtained by packet filtering to obtain the transmission order type of the MPU data (S21).

[0257] Next, the transmission order type determination unit 22 determines (determines) whether the data after packet filtering contains MPU header information (at least one of MPU metadata or MF metadata) (S22). If MPU header information is included (Yes in S22), the random access unit 23 identifies the first MPU sample by detecting that the fragment type of the MMT payload header switches to media data (S23).

[0258] On the other hand, if MPU header information is not included (No in S22), the random access unit 23 identifies the first MPU sample based on the RAP_flag in the MMT packet header or the sample number in the MMT payload header (S24).

[0259] Furthermore, the transmission order type determination unit 22 determines whether or not MF metadata is included in the packet-filtered data (S25). If it is determined that MF metadata is included (Yes in S25), the data acquisition unit 25 acquires NAL units by reading NAL units based on the sample, subsample offset, and size information included in the MF metadata (S26). On the other hand, if it is determined that MF metadata is not included (No in S25), the data acquisition unit 25 acquires NAL units by reading the size data of the NAL units in order from the first NAL unit of the sample (S27).

[0260] Furthermore, even if the receiving device 20 determines in step S22 that MPU header information is included, it may use the processing in step S24 instead of step S23 to identify the first MPU sample. Alternatively, if it is determined that MPU header information is included, the processing in step S23 and the processing in step S24 may be used in combination.

[0261] Furthermore, even if the receiving device 20 determines in step S25 that MF metadata is included, it may acquire the NAL unit using the process in step S27 without using the process in step S26. Also, if it is determined that MF metadata is included, the processes in steps S23 and S24 may be used in combination.

[0262] Furthermore, it is assumed that in step S25, it is determined that MF metadata is included, and that the MF data is transmitted after the media data. In this case, the receiving device 20 may buffer the media data and wait until the MF metadata is acquired before proceeding with the processing in step S26, or the receiving device 20 may decide whether or not to proceed with the processing in step S27 without waiting for the MF metadata to be acquired.

[0263] For example, the receiving device 20 may decide whether to wait to acquire MF metadata based on whether it has a buffer with a buffer size capable of buffering media data. Alternatively, the receiving device 20 may decide whether to wait to acquire MF metadata based on whether the end-to-end delay is reduced. Furthermore, the receiving device 20 may primarily use the processing in step S26 to perform the decoding process, and use the processing in step S27 when a processing mode occurs, such as when packet loss occurs.

[0264] If the transmission order type is predetermined, steps S22 and S26 may be omitted. In this case, the receiving device 20 may determine how to identify the first sample of the MPU and how to identify the NAL unit, taking into consideration the buffer size and end-to-end delay.

[0265] Furthermore, if the transmission sequence type is known in advance, the transmission sequence type determination unit 22 in the receiving device 20 is unnecessary.

[0266] Furthermore, although not shown in Figure 26 above, the decoding command unit 28 outputs the data acquired by the data acquisition unit to the decoding unit 29 based on the PTS and DTS calculated by the PTS and DTS calculation unit 26 and the initialization information acquired by the initialization information acquisition unit 27. The decoding unit 29 decodes the data, and the presentation unit 30 presents the decoded data.

[0267] [Operation of the receiving device 2] Next, we will describe the operation in which the receiving device 20 acquires initialization information based on the transmission order type and decodes the media data based on the initialization information. Figure 27 is a flowchart of this operation.

[0268] First, the packet filtering unit 21 performs packet filtering on the received file. The transmission order type determination unit 22 analyzes the SI information obtained by packet filtering and obtains the transmission order type (S301).

[0269] Next, the transmission order type determination unit 22 determines whether or not MPU metadata has been transmitted (S302). If it is determined that MPU metadata has been transmitted (Yes in S302), the transmission order type determination unit 22 determines, based on the analysis in step S301, whether or not MPU metadata has been transmitted before media data (S303). If MPU metadata has been transmitted before media data (Yes in S303), the initialization information acquisition unit 27 decodes the media data based on the common initialization information included in the MPU metadata and the initialization information of the sample data (S304).

[0270] On the other hand, if it is determined that the MPU metadata is transmitted after the media data (No in S303), the data acquisition unit 25 buffers the media data until the MPU metadata is acquired (S305), and then performs the processing in step S304 after the MPU metadata has been acquired.

[0271] Furthermore, if it is determined in step S302 that MPU metadata has not been transmitted (No in S302), the initialization information acquisition unit 27 decodes the media data based only on the initialization information of the sample data (S306).

[0272] Furthermore, if the decoding of media data is guaranteed only when the initialization information of the sample data is used on the transmitting side, the processing based on the determinations in steps S302 and S303 is not performed, and the processing in step S306 is used instead.

[0273] Furthermore, the receiving device 20 may determine whether or not to buffer the media data before step S305. In this case, if the receiving device 20 determines to buffer the media data, it proceeds to the process of step S305; if it determines not to buffer the media data, it proceeds to the process of step S306. The determination of whether or not to buffer the media data may be based on the buffer size and occupancy of the receiving device 20, or it may be determined by considering the end-to-end delay, for example, by selecting the option with the smaller end-to-end delay.

[0274] [Operation of the receiving device 3] This section details the transmission and reception methods when MF metadata is transmitted after media data (Figures 21(c) and 21(d)). The following explanation uses Figure 21(d) as an example. Note that only the method shown in Figure 21(d) is used for transmission, and no transmission order type signaling is performed.

[0275] As mentioned above, when transmitting data in the order of MPU metadata, media data, and MF metadata, as shown in Figure 21(d), (D-1) The receiving device 20 decodes the media data after acquiring the MPU metadata and then the MF metadata. (D-2) After acquiring the MPU metadata, the receiving device 20 decodes the media data without using the MF metadata. Two decryption methods are possible.

[0276] Here, D-1 requires buffering of media data for MF metadata acquisition, but since decoding can be performed using MPU header information, it can be decoded by conventional MP4-compliant receivers. On the other hand, D-2 does not require buffering of media data for MF metadata acquisition, but since it cannot be decoded using MF metadata, special processing is required for decoding.

[0277] Furthermore, the method shown in Figure 21(d) has the advantage that, since MF metadata is transmitted after the media data, no delay occurs due to encapsulation, and end-to-end delay can be reduced.

[0278] The receiving device 20 can select one of the two decoding methods described above, depending on the capabilities of the receiving device 20 and the quality of service it provides.

[0279] The transmitting device 15 must ensure that the decoding operation in the receiving device 20 can be performed with reduced occurrence of buffer overflow and underflow. For example, the following parameters can be used as elements to define the decoder model when decoding using the D-1 method.

[0280] • Buffer size for reconfiguring the MPU (MPU buffer) For example, buffer size = maximum rate × maximum MPU time × α, where the maximum rate is the upper limit rate of the encoding data profile and level + the overhead of the MPU header. Also, the maximum MPU time is the maximum duration of a GOP (video) assuming 1 MPU = 1 GOP.

[0281] Here, the audio may be in the same GOP units as the video above, or it may be in a different unit. α is a margin to prevent overflow and may be multiplied or added to the maximum rate × maximum MPU time. If multiplied, α ≥ 1, and if added, α ≥ 0.

[0282] • The upper limit of the decoding delay time from when data is input to the MPU buffer until it is decoded (TSTD_delay in MPEG-TS STD) For example, during transmission, the DTS is set so that the time of completion of MPU data acquisition at the receiver is less than or equal to the DTS, taking into account the maximum MPU time and the upper limit of the decoding delay time.

[0283] Furthermore, the transmitting device 15 may add DTS and PTS according to the decoder model when decoding using method D-1. This allows the transmitting device 15 to guarantee the decoding performed by the receiving device using method D-1, while simultaneously transmitting auxiliary information necessary when decoding is performed using method D-2.

[0284] For example, the transmitting device 15 can guarantee the operation of the receiving device decoding using the D-2 method by signaling the pre-buffering time in the decoder buffer when decoding using the D-2 method.

[0285] The pre-buffering time may be included in SI control information such as messages, tables, and descriptors, or in the headers of MMT packets and MMT payloads. The SEI in the encoded data may also be overwritten. The DTS and PTS for decoding using method D-1 are stored in the MPU timestamp descriptor, SampleEntry, while the DTS and PTS for decoding using method D-2, or the pre-buffering time, may be described in the SEI.

[0286] If the receiving device 20 supports only MP4-compliant decoding using the MPU header, it may select decoding method D-1. If it supports both D-1 and D-2, it may select either one.

[0287] The transmitting device 15 may provide DTS and PTS to ensure the decoding operation of one of the devices (D-1 in this description), and may also transmit auxiliary information to assist the decoding operation of the other device.

[0288] Furthermore, when the D-2 method is used, the end-to-end delay is likely to be larger compared to when the D-1 method is used, due to delays caused by pre-buffering of MF metadata. Therefore, the receiver 20 may select the D-2 method for decoding when it wants to reduce the end-to-end delay. For example, the receiver 20 may always use the D-2 method when it always wants to reduce the end-to-end delay. Alternatively, the receiver 20 may use the D-2 method only when operating in a low-latency presentation mode where it wants to present live content, channel selection, or zapping operations with low latency.

[0289] Figure 28 is a flowchart of this receiving method.

[0290] First, the receiving device 20 receives the MMT packet and acquires the MPU data (S401). Then, the receiving device 20 (transmission order type determination unit 22) determines whether to present the program in low-latency presentation mode (S402).

[0291] If the program is not presented in low-latency presentation mode (No in S402), the receiving device 20 (random access unit 23 and initialization information acquisition unit 27) acquires random access and initialization information using the header information (S405). The receiving device 20 (PTS, DTS calculation unit 26, decoding command unit 28, decoding unit 29, presentation unit 30) performs decoding and presentation processing based on the PTS and DTS assigned by the transmitting side (S406).

[0292] On the other hand, when the program is presented in low-latency presentation mode (Yes in S402), the receiving device 20 (random access unit 23 and initialization information acquisition unit 27) acquires random access and initialization information using a decoding method that does not use header information (S403). The receiving device 20 also performs decoding and presentation processing based on auxiliary information for decoding without using PTS, DTS, and header information provided by the transmitting side (S404). Note that in steps S403 and S404, processing may be performed using MPU metadata.

[0293] [Method of sending and receiving data using auxiliary data] The above describes the transmission and reception operations when MF metadata is transmitted after media data (in the cases of Figure 21(c) and Figure 21(d)). Next, we will describe a method in which the transmitting device 15 can start decoding earlier and reduce end-to-end delay by transmitting auxiliary data that has some of the functions of MF metadata. Here, we will describe an example in which auxiliary data is further transmitted based on the transmission method shown in Figure 21(d), but the method using auxiliary data is also applicable to the transmission methods shown in Figure 21(a) to (c).

[0294] Figure 29(a) shows an MMT packet transmitted using the method shown in Figure 21(d). In other words, the data is transmitted in the order of MPU metadata, media data, and MF metadata.

[0295] Here, Sample #1, Sample #2, Sample #3, and Sample #4 are samples included in the media data. While this example describes media data being stored in MMT packets on a sample basis, media data may also be stored in MMT packets on a NAL unit basis, or in units obtained by dividing a NAL unit. Furthermore, multiple NAL units may be aggregated and stored in an MMT packet.

[0296] As explained in D-1 above, in the case of the method shown in Figure 21(d), that is, when data is transmitted in the order of MPU metadata, media data, and MF metadata, there is a method in which the MPU metadata is acquired first, then the MF metadata is acquired, and then the media data is decoded. In this D-1 method, buffering of media data is required for acquiring the MF metadata, but since decoding is performed using the MPU header information, the D-1 method has the advantage that it can be applied to conventional MP4-compliant receivers as well. On the other hand, the receiver 20 has the disadvantage that it must wait to start decoding until the MF metadata is acquired.

[0297] In contrast, as shown in Figure 29(b), in methods that use auxiliary data, the auxiliary data is transmitted before the MF metadata.

[0298] MF metadata contains information indicating the DTS, PTS, offset, and size of all samples included in the movie fragment. In contrast, auxiliary data contains information indicating the DTS, PTS, offset, and size of some of the samples included in the movie fragment.

[0299] For example, MF metadata contains information for all samples (sample #1-sample #4), while auxiliary data contains information for only some samples (sample #1-#2).

[0300] In the case shown in Figure 29(b), the use of auxiliary data enables decoding of sample #1 and sample #2, thus reducing the end-to-end delay compared to the D-1 transmission method. The auxiliary data may be combined and included in any way, and the auxiliary data may be transmitted repeatedly.

[0301] For example, in Figure 29(c), when the auxiliary information is transmitted at timing A, the transmitting device 15 includes the information of sample #1 in the auxiliary information; when the auxiliary information is transmitted at timing B, the auxiliary information includes the information of sample #1 and sample #2; and when the transmitting device 15 transmits the auxiliary information at timing C, the auxiliary information includes the information of sample #1, sample #2, and sample #3.

[0302] The MF metadata includes information on sample #1, sample #2, sample #3, and sample #4 (information on all samples within the movie fragment).

[0303] Auxiliary data does not necessarily need to be sent immediately after it is generated.

[0304] Furthermore, the headers of MMT packets and MMT payloads specify a type that indicates the presence of auxiliary data.

[0305] For example, if auxiliary data is stored in the MMT payload using MPU mode, the fragment_type field value (e.g., FT=3) specifies a data type indicating that it is auxiliary data. The auxiliary data may be based on the MOOF configuration or in other configurations.

[0306] If auxiliary data is stored in the MMT payload as control signals (descriptors, tables, messages), a descriptor tag, table ID, and message ID are specified to indicate that it is auxiliary data.

[0307] Additionally, PTS or DTS may be stored in the header of the MMT packet or MMT payload.

[0308] [Example of generating supplementary data] The following describes an example in which the transmitting device generates auxiliary data based on the MOOF configuration. Figure 30 is a diagram illustrating an example in which the transmitting device generates auxiliary data based on the MOOF configuration.

[0309] In a standard MP4 file, a MOO file is created for each movie fragment, as shown in Figure 20. The MOO file contains information indicating the DTS, PTS, offset, and size of the samples contained within the movie fragment.

[0310] Here, the transmitting device 15 uses only a portion of the sample data that make up the MPU to construct an MP4 (MP4 file) and generates auxiliary data.

[0311] For example, as shown in Figure 30(a), the transmitter 15 generates an MP4 using only sample #1 from the samples #1-#4 that make up the MPU, and uses the moof+mdat header as auxiliary data.

[0312] Next, as shown in Figure 30(b), the transmitter 15 generates an MP4 using sample #1 and sample #2 from the samples #1-#4 that constitute the MPU, and uses the moof+mdat header as the following auxiliary data.

[0313] Next, as shown in Figure 30(c), the transmitter 15 generates an MP4 using sample #1, sample #2, and sample #3 from the samples #1-#4 that constitute the MPU, and uses the moof+mdat header as the following auxiliary data.

[0314] Next, as shown in Figure 30(d), the transmitter 15 generates all MP4s from samples #1-#4 that make up the MPU, and the moof+mdat header of these MP4s becomes the movie fragment metadata.

[0315] In this example, the transmitter 15 generates auxiliary data for every sample, but it may also generate auxiliary data for every N samples. The value of N is any number; for example, if auxiliary data is transmitted M times when transmitting one MPU, then N = total samples / M.

[0316] Note that the information indicating the sample offset in moof may be the offset value after the sample entry area for the subsequent number of samples has been allocated as a NULL area.

[0317] Furthermore, auxiliary data may be generated in a configuration that fragments the MF metadata.

[0318] [Example of receiving operation using auxiliary data] The reception of the auxiliary data generated as explained in Figure 30 will now be described. Figure 31 is a diagram illustrating the reception of auxiliary data. In Figure 31(a), the number of samples constituting the MPU is 30, and auxiliary data is generated and transmitted every 10 samples.

[0319] In Figure 30(a), Auxiliary Data #1 contains sample information for samples #1-#10, Auxiliary Data #2 contains sample information for samples #1-#20, and MF metadata contains sample information for samples #1-#30.

[0320] Although samples #1-#10, #11-#20, and #21-#30 are stored in a single MMT payload, they may also be stored on a sample-by-sample or NAL-by-AL basis, or as fragments or aggregated units.

[0321] The receiving device 20 receives packets of MPU metadata, sample data, MF metadata, and auxiliary data, respectively.

[0322] The receiver 20 concatenates the sample data in the order it is received (from end to end), and updates the previous auxiliary data after receiving the latest auxiliary data. Furthermore, the receiver 20 can configure a complete MPU by finally replacing the auxiliary data with MF metadata.

[0323] When the receiving device 20 receives auxiliary data #1, it concatenates the data as shown in the upper part of Figure 31(b) to form an MP4. This allows the receiving device 20 to parse samples #1-#10 using the MPU metadata and information from auxiliary data #1, and to decode based on the PTS, DTS, offset, and size information contained in the auxiliary data.

[0324] Furthermore, upon receiving auxiliary data #2, the receiving device 20 concatenates the data as shown in the middle section of Figure 31(b) to form an MP4. This allows the receiving device 20 to parse samples #1-#20 using the MPU metadata and information from auxiliary data #2, and to perform decoding based on the PTS, DTS, offset, and size information contained in the auxiliary data.

[0325] Furthermore, upon receiving the MF metadata, the receiving device 20 concatenates the data as shown in the lower part of Figure 31(b) to form an MP4 file. This allows the receiving device 20 to parse samples #1-#30 using the MPU metadata and MF metadata, and to perform decoding based on the PTS, DTS, offset, and size information contained in the MF metadata.

[0326] In the absence of auxiliary data, the receiving device 20 could only acquire sample information after receiving the MF metadata, and therefore had to start decoding after receiving the MF metadata. However, by having the transmitting device 15 generate and transmit auxiliary data, the receiving device 20 can acquire sample information using the auxiliary data without waiting for the MF metadata to be received, thus shortening the decoding start time. Furthermore, by having the transmitting device 15 generate auxiliary data based on MOOF as explained using Figure 30, the receiving device 20 can use its conventional MP4 parser to parse the data.

[0327] Furthermore, the newly generated auxiliary data and MF metadata include sample information that overlaps with previously transmitted auxiliary data. Therefore, even if past auxiliary data cannot be obtained due to packet loss or other reasons, it is possible to reconstruct the MP4 file and obtain sample information (PTS, DTS, size, and offset) using the newly acquired auxiliary data and MF metadata.

[0328] Note that the auxiliary data does not necessarily need to include information from past sample data. For example, auxiliary data #1 may correspond to sample data #1-#10, and auxiliary data #2 may correspond to sample data #11-#20. For example, as shown in Figure 31(c), the transmitting device 15 may sequentially send out complete MF metadata as a data unit, and fragmented units of the data unit as auxiliary data.

[0329] Furthermore, the transmitting device 15 may repeatedly transmit auxiliary data or repeatedly transmit MF metadata in order to prevent packet loss.

[0330] Furthermore, the MMT packets and MMT payloads containing the auxiliary data include the MPU sequence number and asset ID, similar to the MPU metadata, MF metadata, and sample data.

[0331] The reception operation using the auxiliary data described above will be explained using the flowchart in Figure 32. Figure 32 is a flowchart of the reception operation using auxiliary data.

[0332] First, the receiving device 20 receives the MMT packet and analyzes the packet header and payload header (S501). Next, the receiving device 20 analyzes whether the fragment type is auxiliary data or MF metadata (S502). If the fragment type is auxiliary data, it overwrites and updates the past auxiliary data (S503). If there is no past auxiliary data for the same MPU, the receiving device 20 uses the received auxiliary data as new auxiliary data. Then, the receiving device 20 acquires a sample based on the MPU metadata, auxiliary data, and sample data and performs decoding (S507).

[0333] On the other hand, if the fragment type is MF metadata, the receiving device 20 overwrites the past auxiliary data with MF metadata in step S505 (S505). Then, the receiving device 20 acquires the sample in complete MPU form based on the MPU metadata, MF metadata, and sample data, and performs decoding (S506).

[0334] Although not shown in Figure 32, in step S502, the receiving device 20 stores the data in a buffer if the fragment type is MPU metadata, and stores the data concatenated after each sample in the buffer if it is sample data.

[0335] If auxiliary data cannot be obtained due to packet loss, the receiving device 20 can decode the sample by overwriting it with the latest auxiliary data or by using past auxiliary data.

[0336] The transmission cycle and number of transmissions of auxiliary data may be predetermined values. Information on the transmission cycle and number of transmissions (count, countdown) may be transmitted together with the data. For example, the data unit header may store the transmission cycle, number of transmissions, and timestamps such as initial_cpb_removal_delay.

[0337] By sending auxiliary data containing information about the MPU's initial sample at least once before initial_cpb_removal_delay, it becomes possible to follow the CPB buffer model. In this case, the MPU timestamp descriptor stores a value based on the picture timing SEI.

[0338] Furthermore, the transmission method used in receiving operations where such auxiliary data is used is not limited to the MMT method, but can also be applied to streaming transmission of packets composed of ISOBMFF file formats, such as MPEG-DASH.

[0339] [How to send multiple movie fragments when one MPU is composed of them] In the explanations from Figure 19 onwards, one MPU was assumed to consist of one movie fragment. However, here we will explain the case where one MPU is composed of multiple movie fragments. Figure 33 shows the configuration of an MPU composed of multiple movie fragments.

[0340] In Figure 33, the samples (#1-#6) stored in one MPU are divided and stored into two movie fragments. The first movie fragment is generated based on samples #1-#3, and the corresponding moof box is generated. The second movie fragment is generated based on samples #4-#6, and the corresponding moof box is generated.

[0341] The headers of the moof and mdat boxes in the first movie fragment are stored in the MMT payload and MMT packet as movie fragment metadata #1. On the other hand, the headers of the moof and mdat boxes in the second movie fragment are stored in the MMT payload and MMT packet as movie fragment metadata #2. Note that in Figure 33, the MMT payload containing the movie fragment metadata is hatched.

[0342] The number of samples that make up the MPU and the number of samples that make up the movie fragments are arbitrary. For example, the number of samples that make up the MPU may be the number of samples per GOP, and half the number of samples per GOP may be used as the number of movie fragments, resulting in two movie fragments.

[0343] Note that this example shows two movie fragments (moof box and mdat box) in one MPU, but one MPU may contain three or more movie fragments, not just two. Also, the samples stored in the movie fragments do not have to be divided equally; they may be divided into any number of samples.

[0344] In Figure 33, the MPU metadata unit and the MF metadata unit are stored in the MMT payload as data units. However, the transmitter 15 may store units such as ftyp, mmpu, moov, and moof as data units in the MMT payload, or it may store the data units in fragmented units in the MMT payload. Alternatively, the transmitter 15 may store the data units in aggregated units in the MMT payload.

[0345] Furthermore, in Figure 33, samples are stored in the MMT payload on a sample-by-sample basis. However, the transmitter 15 may configure data units not on a sample-by-sample basis, but on a NAL unit basis or on a unit formed by combining multiple NAL units, and store them in the MMT payload on a data unit basis. Alternatively, the transmitter 15 may store the data units in the MMT payload in fragmented units, or in aggregated units.

[0346] In Figure 33, the MPU is configured in the order of moof#1, mdat#1, moof#2, and mdat#2, and moof#1 is assigned an offset as if the corresponding mdat#1 is appended to it. However, the offset may also be assigned as if mdat#1 is appended to it before moof#1. In this case, however, it is not possible to generate movie fragment metadata in the form of moof+mdat, and the headers of moof and mdat are transmitted separately.

[0347] Next, we will explain the transmission order of MMT packets when the MPU configuration described in Figure 33 is transmitted. Figure 34 is a diagram illustrating the transmission order of MMT packets.

[0348] Figure 34(a) shows the transmission order when sending MMT packets in the configuration order of the MPU shown in Figure 33. Specifically, Figure 34(a) shows an example in which the following are transmitted in this order: MPU metadata, MF metadata #1, media data #1 (samples #1-#3), MF metadata #2, and media data #2 (samples #4-#6).

[0349] Figure 34(b) shows an example of sending MPU metadata, media data #1 (samples #1-#3), MF metadata #1, media data #2 (samples #4-#6), and MF metadata #2 in that order.

[0350] Figure 34(c) shows an example of sending media data #1 (samples #1-#3), MPU metadata, MF metadata #1, media data #2 (samples #4-#6), and MF metadata #2 in that order.

[0351] MF Meta #1 is generated using samples #1-#3, and MF Meta #2 is generated using samples #4-#6. Therefore, when the transmission method shown in Figure 34(a) is used, a delay occurs in the transmission of sample data due to encapsulation.

[0352] In contrast, when the transmission methods shown in Figure 34(b) and Figure 34(c) are used, samples can be transmitted without waiting for MF metadata to be generated, thus eliminating encapsulation delays and reducing end-to-end delays.

[0353] Furthermore, in the transmission sequence shown in Figure 34(a), one MPU is divided into multiple movie fragments, and the number of samples stored in the MF metadata is smaller than in Figure 19. Therefore, the delay due to encapsulation can be reduced compared to Figure 19.

[0354] In addition to the methods described herein, for example, the transmitter 15 may concatenate MF Meta #1 and MF Meta #2 and transmit them together at the end of the MPU. In this case, the MF Meta of different movie fragments may be aggregated and stored in a single MMT payload. Alternatively, the MF Meta of different MPUs may be aggregated and stored in the MMT payload.

[0355] [How to receive when a single MPU consists of multiple movie fragments] This section describes an example of the operation of the receiving device 20, which receives and decodes MMT packets transmitted in the transmission order described in Figure 34(b). Figures 35 and 36 are diagrams illustrating this example of operation.

[0356] The receiving device 20 receives MMT packets containing MPU metadata, samples, and MF metadata, transmitted in the order shown in Figure 35. The sample data is concatenated in the order it is received.

[0357] The receiver 20 concatenates the data at T1, the time when MF Meta #1 was received, as shown in (1) of Figure 36, to form an MP4. This allows the receiver 20 to acquire samples #1-#3 based on the MPU metadata and the information in MF Meta #1, and to perform decoding based on the PTS, DTS, offset, and size information contained in the MF Meta.

[0358] Furthermore, the receiver 20 concatenates the data at T2, the time when MF Meta #2 was received, as shown in (2) of Figure 36, to form an MP4. This allows the receiver 20 to acquire samples #4-#6 based on the MPU metadata and the information in MF Meta #2, and to perform decoding based on the PTS, DTS, offset, and size information of the MF Meta. Alternatively, the receiver 20 may acquire samples #1-#6 based on the information in MF Meta #1 and MF Meta #2 by concatenating the data as shown in (3) of Figure 36 to form an MP4.

[0359] By having a single MPU split the movie into multiple fragments, the time it takes for the MPU to acquire the initial MF metadata is reduced, thus speeding up the decoding start time. Additionally, the buffer size required to store samples before decoding can be reduced.

[0360] The transmitting device 15 may also set the movie fragment division unit such that the time from transmitting (or receiving) the first sample in the movie fragment to transmitting (or receiving) the MF meta corresponding to the movie fragment is shorter than the initial_cpb_removal_delay specified by the encoder. By setting it in this way, the receive buffer can follow the cpb buffer, enabling low-latency decoding. In this case, absolute times based on initial_cpb_removal_delay can be used for PTS and DTS.

[0361] Furthermore, the transmitting device 15 may divide the movie fragments at equal intervals, or divide subsequent movie fragments at shorter intervals than the previous movie fragments. This ensures that the receiving device 20 always receives MF metadata containing information about the sample before decoding the sample, enabling continuous decoding.

[0362] The absolute time for PTS and DTS can be calculated using the following two methods.

[0363] (1) The absolute time of PTS and DTS is determined based on the reception time (T1 or T2) of MF Meta #1 and MF Meta #2, and the relative time of PTS and DTS included in the MF Meta.

[0364] (2) The absolute time of the PTS and DTS is determined based on the absolute time signaled by the sender, such as the MPU timestamp descriptor, and the relative time of the PTS and DTS included in the MF metadata.

[0365] Furthermore, (2-A) the absolute time signaled by the transmitter 15 may be an absolute time calculated based on the initial_cpb_removal_delay specified by the encoder.

[0366] Furthermore, (2-B) the absolute time signaled by the transmitter 15 may be an absolute time calculated based on the predicted reception time of the MF meta.

[0367] Furthermore, MF Meta #1 and MF Meta #2 may be transmitted repeatedly. By repeatedly transmitting MF Meta #1 and MF Meta #2, the receiving device 20 can acquire the MF Meta again even if it was previously unable to acquire it due to packet loss or other reasons.

[0368] The payload header of an MFU containing samples that make up a movie fragment can store an identifier indicating the order of the movie fragment. On the other hand, the MMT payload does not include an identifier indicating the order of the MF meta that make up the movie fragment. Therefore, the receiving device 20 identifies the order of the MF meta using packet_sequence_number. Alternatively, the transmitting device 15 may signal by storing an identifier indicating which movie fragment the MF meta belongs to in the control information (message, table, descriptor), MMT header, MMT payload header, or data unit header.

[0369] The transmitting device 15 may transmit MPU metadata, MF metadata, and samples in a predetermined transmission order, and the receiving device 20 may perform reception processing based on the predetermined transmission order. Alternatively, the transmitting device 15 may signal the transmission order, and the receiving device 20 may select (determine) the reception processing based on the signaling information.

[0370] The above-described receiving method will be explained using Figure 37. Figure 37 is a flowchart of the operation of the receiving method described in Figures 35 and 36.

[0371] First, the receiving device 20 determines (identifies) whether the data contained in the MMT payload is MPU metadata, MF metadata, or sample data (MFU) based on the fragment type indicated in the MMT payload (S601, S602). If the data is sample data, the receiving device 20 buffers the sample and waits for the reception of the MF metadata corresponding to the sample and for the start of decoding (S603).

[0372] On the other hand, in step S602, if the data is MF metadata, the receiving device 20 obtains sample information (PTS, DTS, location information, and size) from the MF metadata, obtains a sample based on the obtained sample information, decodes the sample based on the PTS and DTS, and presents it (S604).

[0373] Although not shown in the diagram, if the data is MPU metadata, the MPU metadata contains initialization information necessary for decoding. Therefore, the receiving device 20 stores this information and uses it to decode the sample data in step S604.

[0374] Furthermore, when the receiving device 20 stores the received MPU data (MPU metadata, MF metadata, and sample data) in the storage device, it stores the data after rearranging it according to the MPU configuration as described in Figure 19 or Figure 33.

[0375] On the transmitting side, MMT packets are assigned a packet sequence number to packets that have the same packet ID. At this time, the packet sequence number may be assigned after the MMT packets, including MPU metadata, MF metadata, and sample data, have been sorted in the order of transmission, or it may be assigned in the order before sorting.

[0376] If packet sequence numbers are assigned in the order before sorting, the receiving device 20 can sort the data according to the MPU's configuration order based on the packet sequence numbers, making storage easier.

[0377] [Method for detecting the beginning of an access unit and the beginning of a slice segment] This section describes how to detect the beginning of an access unit or slice segment based on information from the MMT packet header and MMT payload header.

[0378] This section presents two examples: one where non-VCL NAL units (such as access unit delimiters, VPS, SPS, PPS, and SEI) are stored together as a data unit in the MMT payload, and another where each non-VCL NAL unit is treated as a data unit and these data units are aggregated to store in a single MMT payload.

[0379] Figure 38 shows a case where non-VCL NAL units are treated as individual data units and aggregated.

[0380] In the case of Figure 38, the beginning of the access unit is the first data of an MMT payload containing an MMT packet with a fragment_type value of MFU, an aggregation_flag value of 1, and an offset value of 0. In this case, the Fragmentation_indicator value is 0.

[0381] In addition, in the case of Figure 38, the beginning of the slice segment is an MMT packet with a fragment_type value of MFU, and is the first data of an MMT payload with an aggregation_flag value of 0 and a fragmentation_indicator value of 00 or 01.

[0382] Figure 39 shows the case where non-VCL NAL units are grouped together as a data unit. The field values ​​of the packet header are as shown in Figure 17 (or Figure 18).

[0383] In Figure 39, the beginning of the access unit is the first data of the payload in a packet with an Offset value of 0.

[0384] In addition, in the case of Figure 39, the beginning of a slice segment is the first data in the payload of a packet whose Offset value is a value other than 0 and whose fragmentation indicator value is 00 or 01.

[0385] [Reception process when packet loss occurs] Normally, when transmitting MP4 format data in an environment where packet loss occurs, the receiving device 20 restores the packets using ALFEC (Application Layer FEC) or packet retransmission control.

[0386] However, if packet loss occurs in streaming, such as broadcasting, where AL-FEC cannot be used, the packets cannot be recovered.

[0387] The receiving device 20 needs to resume decoding video and audio after data is lost due to packet loss. To do this, the receiving device 20 needs to detect the beginning of the access unit or NAL unit and start decoding from the beginning of the access unit or NAL unit.

[0388] However, since MP4 format NAL units do not have a start code at the beginning, the receiving device 20 cannot detect the beginning of the access unit or NAL unit even when analyzing the stream.

[0389] Figure 40 is a flowchart showing the operation of the receiving device 20 when packet loss occurs.

[0390] The receiving device 20 detects packet loss using the PacketSequence number, packet counter, fragment counter, etc., in the header of the MMT packet or MMT payload (S701), and determines which packet has been lost based on the context (S702).

[0391] If the receiving device 20 determines that no packet loss has occurred (No in S702), it constructs an MP4 file and decodes the access unit or NAL unit (S703).

[0392] If the receiving device 20 determines that packet loss has occurred (Yes in S702), it generates a NAL unit corresponding to the lost NAL unit using dummy data and constructs an MP4 file (S704). When the receiving device 20 puts dummy data into the NAL unit, it indicates that it is dummy data in the NAL unit type.

[0393] Furthermore, the receiving device 20 can detect the beginning of the next access unit or NAL unit based on the method described in Figures 17, 18, 38, and 39, and resume decoding by inputting the beginning data to the decoder (S705).

[0394] If packet loss occurs, the receiving device 20 may resume decoding from the beginning of the access unit and NAL unit based on information detected based on the packet header, or it may resume decoding from the beginning of the access unit and NAL unit based on the header information of the reconstructed MP4 file, including a NAL unit of dummy data.

[0395] When the receiving device 20 stores MP4 files (MPU), packet data (such as NAL units) lost due to packet loss may be separately acquired from broadcasts or communications and stored (replaced).

[0396] In this case, when the receiving device 20 retrieves the lost packet from the communication, it notifies the server of the information of the lost packet (such as the packet ID, MPU sequence number, packet sequence number, IP data flow number, and IP address) and retrieves the packet. The receiving device 20 may retrieve not only the lost packet but also the packets before and after the lost packet simultaneously.

[0397] [How to structure movie fragments] This section provides a detailed explanation of how to construct movie fragments.

[0398] As explained in Figure 33, the number of samples constituting a movie fragment and the number of movie fragments constituting one MPU are arbitrary. For example, the number of samples constituting a movie fragment and the number of movie fragments constituting one MPU may be a fixed predetermined number or may be determined dynamically.

[0399] Here, by configuring the movie fragments on the transmitting side (transmitting device 15) to satisfy the following conditions, low-latency decoding in the receiving device 20 can be guaranteed.

[0400] The conditions are as follows:

[0401] The transmitting device 15 generates and transmits MF metadata as movie fragments, dividing the sample data into units such that the receiving device 20 can always receive MF metadata containing information about any given sample (Sample(i)) before the decoding time (DTS(i)) of that sample.

[0402] Specifically, the transmitting device 15 constructs a movie fragment using samples (including the i-th sample) that have been encoded before DTS(i).

[0403] To ensure low-latency decoding, the following methods can be used to dynamically determine the number of samples that make up a movie fragment and the number of movie fragments that make up one MPU.

[0404] (1) At the start of decoding, the decoding time DTS(0) of the first sample Sample(0) in the GOP is the time based on initial_cpb_removal_delay. The transmitting device constructs the first movie fragment using the encoded samples at a time before DTS(0). The transmitting device 15 also generates MF metadata corresponding to the first movie fragment and transmits it at a time before DTS(0).

[0405] (2) The transmitting device 15 configures movie fragments in subsequent samples in such a way that the above conditions are met.

[0406] For example, if the first sample of a movie fragment is the k-th sample, the MF metadata of the movie fragment containing the k-th sample is transmitted by the decoding time DTS(k) of the k-th sample. If the encoding completion time of the l-th sample is before DTS(k) and the encoding completion time of the (l+1)-th sample is after DTS(k), the transmitting device 15 constructs a movie fragment using samples k through l.

[0407] The transmitting device 15 may also construct a movie fragment using samples from the kth sample down to the lth sample.

[0408] (3) After the encoding of the last sample of the MPU is completed, the transmitting device 15 constructs a movie fragment using the remaining samples, generates MF metadata corresponding to the movie fragment, and transmits it.

[0409] The transmitting device 15 may construct a movie fragment using only some of the encoded samples, rather than using all of the encoded samples.

[0410] In the above example, we showed how the number of samples constituting a movie fragment and the number of movie fragments constituting one MPU are dynamically determined based on the above conditions to ensure low-latency decoding. However, the method for determining the number of samples and movie fragments is not limited to this method. For example, the number of movie fragments constituting one MPU may be fixed to a predetermined value, and the number of samples may be determined to satisfy the above conditions. Alternatively, the number of movie fragments constituting one MPU and the time at which the movie fragments are divided (or the code amount of the movie fragments) may be fixed to predetermined values, and the number of samples may be determined to satisfy the above conditions.

[0411] Additionally, if the MPU is divided into multiple movie fragments, information indicating whether the MPU is divided into multiple movie fragments, the attributes of the divided movie fragments, or the attributes of the MF meta for the divided movie fragments may be transmitted.

[0412] Here, movie fragment attributes refer to information indicating whether a movie fragment is the first movie fragment in the MPU, the last movie fragment in the MPU, or any other movie fragment.

[0413] Furthermore, the attributes of an MF meta indicate whether the MF meta corresponds to the first movie fragment of the MPU, the last movie fragment of the MPU, or any other movie fragment.

[0414] The transmitting device 15 may also store and transmit the number of samples constituting a movie fragment and the number of movie fragments constituting one MPU as control information.

[0415] [Operation of the receiving device] The operation of the receiving device 20 based on the movie fragment configured as described above will now be explained.

[0416] The receiving device 20 determines the absolute time of the PTS and DTS based on the absolute time signaled from the transmitting side, such as the MPU timestamp descriptor, and the relative time of the PTS and DTS included in the MF metadata.

[0417] Based on information indicating whether the MPU is divided into multiple movie fragments, the receiving device 20 processes the data as follows, based on the attributes of the divided movie fragments, if the MPU is divided.

[0418] (1) If the movie fragment is the first movie fragment of the MPU, the receiving device 20 generates the absolute times of the PTS and DTS using the absolute time of the first sample PTS included in the MPU timestamp descriptor and the relative times of the PTS and DTS included in the MF metadata.

[0419] (2) If the movie fragment is not the first movie fragment of the MPU, the receiving device 20 generates the absolute times of the PTS and DTS using the relative times of the PTS and DTS included in the MF metadata, without using the information of the MPU timestamp descriptor.

[0420] (3) If the movie fragment is the last movie fragment of the MPU, the receiving device 20 calculates the absolute time of the PTS and DTS for all samples and then resets the PTS and DTS calculation process (relative time addition process). The reset process may also be performed for the first movie fragment of the MPU.

[0421] The receiving device 20 may determine whether the movie fragment is divided as shown below. The receiving device 20 may also obtain attribute information of the movie fragment as shown below.

[0422] For example, the receiving device 20 may determine whether the movie fragments are split based on the value of the movie_fragment_sequence_number field, an identifier indicating the order of the movie fragments shown in the MMTP (MMT Protocol) payload header.

[0423] Specifically, the receiving device 20 may determine that an MPU is divided into multiple movie fragments if the number of movie fragments contained in one MPU is 1, the value of the movie_fragment_sequence_number field is 1, and there are other values ​​of 2 or more for that field.

[0424] Furthermore, the receiving device 20 may determine that an MPU is divided into multiple movie fragments if the number of movie fragments contained in one MPU is 1, the value of the movie_fragment_sequence_number field is 0, and there is a value other than 0 for the field value.

[0425] Similarly, the attribute information of a movie fragment may also be determined based on the movie_fragment_sequence_number.

[0426] Alternatively, instead of using movie_fragment_sequence_number, it is also possible to determine whether a movie fragment is divided and to identify the attribute information of the movie fragment by counting the transmissions of movie fragments and MF metadata included in a single MPU.

[0427] With the configuration of the transmitter 15 and receiver 20 described above, the receiver 20 can receive movie fragment metadata at shorter intervals than the MPU, enabling low-latency decoding. Furthermore, low-latency decoding can be performed using decoding processing based on the MP4 parsing method.

[0428] As explained above, the receiving operation when the MPU is divided into multiple movie fragments will now be explained using a flowchart. Figure 41 is a flowchart of the receiving operation when the MPU is divided into multiple movie fragments. This flowchart illustrates the operation of step S604 in Figure 37 in more detail.

[0429] First, the receiving device 20 acquires MF metadata based on the data type indicated in the MMTP payload header if the data type is MF metadata (S801).

[0430] Next, the receiving device 20 determines whether the MPU is divided into multiple movie fragments (S802). If the MPU is divided into multiple movie fragments (Yes in S802), it determines whether the received MF metadata is the metadata at the beginning of the MPU (S803). If the received MF metadata is the MF metadata at the beginning of the MPU (Yes in S803), the receiving device 20 calculates the absolute time of the PTS and DTS from the absolute time of the PTS indicated in the MPU timestamp descriptor and the relative time of the PTS and DTS indicated in the MF metadata (S804), and determines whether it is the metadata at the end of the MPU (S805).

[0431] On the other hand, if the received MF metadata is not the MF metadata at the beginning of the MPU (No in S803), the receiving device 20 does not use the information from the MPU timestamp descriptor, but instead calculates the absolute time of the PTS and DTS using the relative time of the PTS and DTS indicated in the MF metadata (S808), and proceeds to the processing in step S805.

[0432] In step S805, if it is determined that this is the last MF metadata for the MPU (Yes in S805), the receiving device 20 calculates the absolute time of the PTS and DTS for all samples and then resets the PTS and DTS calculation process. In step S805, if it is determined that this is not the last MF metadata for the MPU (No in S805), the receiving device 20 terminates the process.

[0433] Furthermore, if it is determined in step S802 that the MPU has not been divided into multiple movie fragments (No in S802), the receiving device 20 acquires sample data based on the MF metadata transmitted after the MPU and determines the PTS and DTS (S807).

[0434] Then, although not shown in the diagram, the receiving device 20 finally performs decoding and presentation processing based on the determined PTS and DTS.

[0435] [Challenges that arise when splitting movie fragments, and their solutions] So far, we have explained how to reduce end-to-end delay by splitting movie fragments. From here, we will explain the new challenges that arise when splitting movie fragments, and how to solve them.

[0436] First, as background information, we will explain the picture structure in encoded data. Figure 42 shows an example of the picture prediction structure for each TemporalId when achieving temporal scalability.

[0437] In encoding schemes such as MPEG-4 AVC and HEVC (High Efficiency Video Coding), temporal scalability can be achieved by using B-pictures (bidirectional reference prediction pictures) that can be referenced by other pictures.

[0438] The TemporalId shown in Figure 42(a) is an identifier for the hierarchy of the coding structure, where a larger TemporalId value indicates a deeper hierarchy. The square blocks represent pictures, with Ix representing an I-picture (in-screen prediction picture), Px representing a P-picture (forward-referencing prediction picture), and Bx and bx representing B-pictures (bidirectional-referencing prediction pictures). The x in Ix / Px / Bx indicates the display order, representing the order in which the pictures are displayed. The arrows between pictures indicate reference relationships; for example, picture B4 generates prediction images using I0 and B8 as reference images. Here, it is forbidden for one picture to use another picture with a TemporalId greater than its own as a reference image. The hierarchy is defined to provide temporal scalability. For example, in Figure 42, decoding all pictures yields a 120fps (frames per second) video, but decoding only the layers with TemporalId from 0 to 3 yields a 60fps video.

[0439] Figure 43 shows the relationship between the decryption time (DTS) and display time (PTS) for each picture in Figure 42. For example, picture I0 shown in Figure 43 is displayed after the decryption of B4 is complete, so that no gap occurs during decryption and display.

[0440] As shown in Figure 43, when the prediction structure includes a B-picture, the decoding order and the display order are different, so the receiving device 20 needs to perform a delay process on the picture and a reordering process on the picture after decoding it.

[0441] The above describes an example of a picture prediction structure in terms of time scalability. However, even when time scalability is not used, depending on the prediction structure, picture delay processing and reordering may be necessary. Figure 44 shows an example of a picture prediction structure that requires picture delay processing and reordering. The numbers in Figure 44 indicate the decoding order.

[0442] As shown in Figure 44, depending on the prediction structure, the first sample in the decoding order may differ from the first sample in the presentation order. In Figure 44, the first sample in the presentation order is the fourth sample in the decoding order. Note that Figure 44 is just one example of a prediction structure, and prediction structures are not limited to this one. In other prediction structures as well, the first sample in the decoding order may differ from the first sample in the presentation order.

[0443] Figure 45, similar to Figure 33, shows an example where an MPU composed of MP4 format is divided into multiple movie fragments and stored in an MMTP payload and MMTP packets. Note that the number of samples that make up the MPU and the number of samples that make up the movie fragments are arbitrary. For example, the number of samples that make up the MPU may be the number of samples per GOP, and half the number of samples per GOP may be used as movie fragments to form two movie fragments. One sample may be one movie fragment, or the samples that make up the MPU may not be divided at all.

[0444] Figure 45 shows an example where one MPU contains two movie fragments (a moof box and an mdat box), but an MPU does not have to contain two movie fragments. An MPU may contain three or more movie fragments, or it may be the number of samples contained in the MPU. Furthermore, the samples stored in the movie fragments do not have to be divided equally; they may be divided into any number of samples.

[0445] Movie fragment metadata (MF metadata) contains information about the PTS, DTS, offset, and size of the sample contained in the movie fragment. When decoding a sample, the receiving device 20 extracts the PTS and DTS from the MF metadata containing the sample information and determines the decoding timing and presentation timing.

[0446] From here on, for the sake of detailed explanation, we will denote the absolute value of the decoding time of sample i as DTS(i) and the absolute value of the presentation time as PTS(i).

[0447] In the MF metadata, the timestamp information stored within moof for the i-th sample specifically consists of the relative decoding times of the i-th sample and the (i+1)-th sample, and the relative decoding time of the i-th sample and its presentation time, which will be hereafter referred to as DT(i) and CT(i).

[0448] Movie fragment metadata #1 contains DT(i) and CT(i) for samples #1-#3, and movie fragment metadata #2 contains DT(i) and CT(i) for samples #4-#6.

[0449] Furthermore, the absolute value of the PTS of the access unit at the beginning of the MPU is stored in the MPU timestamp descriptor, and the receiving device 20 calculates the PTS and DTS based on the PTS_MPU of the access unit at the beginning of the MPU, the CT, and the DT.

[0450] Figure 46 is a diagram illustrating the calculation method and challenges of PTS and DTS when the MPU is configured using samples #1-#10.

[0451] Figure 46(a) shows an example where the MPU is not divided into movie fragments, Figure 46(b) shows an example where the MPU is divided into two movie fragments of 5 samples each, and Figure 46(c) shows an example where the MPU is divided into 10 movie fragments of sample units each.

[0452] As explained in Figure 45, when PTS and DTS are calculated using the MPU timestamp descriptor and the timestamp information (CT and DT) within the MP4, the sample that appears first in the presentation order in Figure 44 is the fourth sample in the decoding order. Therefore, the PTS stored in the MPU timestamp descriptor is the PTS (absolute value) of the fourth sample in the decoding order. Hereafter, this sample will be referred to as sample A. The first sample in the decoding order will be referred to as sample B.

[0453] Since the absolute time information related to the timestamp is only from the MPU timestamp descriptor, the receiving device 20 cannot calculate the PTS (absolute time) and DTS (absolute time) of the other samples until sample A arrives. The receiving device 20 also cannot calculate the PTS and DTS of sample B.

[0454] In the example shown in Figure 46(a), sample A is included in the same movie fragment as sample B and is stored in a single MF meta. Therefore, the receiving device 20 can immediately determine the DTS of sample B after receiving the MF meta.

[0455] In the example shown in Figure 46(b), sample A is included in the same movie fragment as sample B and is stored in a single MF meta. Therefore, the receiving device 20 can immediately determine the DTS of sample B after receiving the MF meta.

[0456] In the example shown in Figure 46(c), sample A is contained in a different movie fragment than sample B. Therefore, the receiving device 20 cannot determine the DTS of sample B until it has received the MF metadata, including the CT and DT, of the movie fragment containing sample A.

[0457] Therefore, in the example of Figure 46(c), the receiving device 20 cannot start decoding immediately after the arrival of the B sample.

[0458] Thus, if a movie fragment containing sample B does not contain sample A, the receiving device 20 cannot start decoding sample B until it has received the MF metadata relating to the movie fragment containing sample A.

[0459] This issue arises when the first sample presented and the first sample decoded do not match, and the movie fragment is split before sample A and sample B are stored in the same movie fragment. This issue also occurs regardless of whether the MF metadata is delayed or delayed.

[0460] Thus, when the first sample in the presentation order does not match the first sample in the decoding order, and when sample A and sample B are not stored in the same movie fragment, the DTS cannot be determined immediately after receiving sample B. Therefore, the transmitting device 15 separately transmits the DTS (absolute value) of sample B, or information that allows the receiving side to calculate the DTS (absolute value) of sample B. Such information may be transmitted using control information or a packet header.

[0461] The receiving device 20 uses this information to calculate the DTS (absolute value) of sample B. Figure 47 is a flowchart of the receiving operation when the DTS is calculated using this information.

[0462] The receiving device 20 receives the first movie fragment of the MPU (S901) and determines whether sample A and sample B are stored in the same movie fragment (S902). If they are stored in the same movie fragment (Yes in S902), the receiving device 20 calculates the DTS using only the MF metadata information, without using the DTS (absolute time) of sample B, and starts decoding (S904). In step S904, the receiving device 20 may also determine the DTS using the DTS of sample B.

[0463] On the other hand, if sample A and sample B are not stored in the same movie fragment in step S902 (No in S902), the receiving device 20 acquires the DTS (absolute time) of sample B, determines the DTS, and starts decoding (S903).

[0464] In the above explanation, we described an example of calculating the absolute value of the decoding time and the absolute value of the presentation time for each sample using the MF metadata (timestamp information stored in the moof file of the MP4 format) in the MMT standard. However, it goes without saying that the MF metadata can also be replaced with any control information that can be used to calculate the absolute value of the decoding time and the absolute value of the presentation time for each sample. Examples of such control information include control information in which the relative value CT(i) of the decoding times of the i-th sample and the (i+1)-th sample is replaced with the relative value of the presentation times of the i-th sample and the (i+1)-th sample, or control information that includes both the relative value CT(i) of the decoding times of the i-th sample and the (i+1)-th sample and the relative value of the presentation times of the i-th sample and the (i+1)-th sample.

[0465] (Embodiment 3) [overview] Embodiment 3 describes a method for transmitting content and a data structure when transmitting content such as video, audio, subtitles, and data broadcasting via broadcast. In other words, it describes a method for transmitting content and a data structure specifically for playback of broadcast streams.

[0466] In Embodiment 3, an example is described in which the MMT method (hereinafter also simply referred to as MMT) is used as the multiplexing method, but other multiplexing methods such as MPEG-DASH or RTP may also be used.

[0467] First, we will explain in detail how Data Units (DUs) are stored in the payload in MMT. Figure 48 is a diagram illustrating how Data Units are stored in the payload in MMT.

[0468] In MMT, the transmitting device stores a portion of the data constituting the MPU as a data unit in the MMTP payload and transmits it with a header. The header includes the MMTP payload header and the MMTP packet header. The data unit can be either a NAL unit or a sample.

[0469] Figure 48(a) shows an example in which a transmitting device aggregates multiple data units and stores them in a single payload. In the example in Figure 48(a), a Data Unit Header (DUH) and Data Unit Length (DUL) are added to the beginning of each of the multiple data units, and multiple data units with data unit headers and data unit lengths are stored together in the payload.

[0470] Figure 48(b) shows an example where one data unit is stored in one payload. In the example in Figure 48(b), a data unit header is added to the beginning of the data unit before it is stored in the payload. Figure 48(c) shows an example where one data unit is split, and each split data unit is given a data unit header before being stored in the payload.

[0471] Data units include timed-MFUs (media containing synchronization information such as video, audio, or subtitles), non-timed-MFUs (media not containing synchronization information such as files), MPU metadata, and MF metadata. A data unit header is defined according to the type of data unit. Note that MPU metadata and MF metadata do not have data unit headers.

[0472] Furthermore, while transmitting devices are generally not allowed to aggregate different types of data units, they may be specified to allow such aggregation. For example, if the size of MF metadata is small, such as when it is divided into movie fragments for each sample, the number of packets can be reduced by aggregating the MF metadata and media data, and furthermore, the transmission capacity can be reduced.

[0473] If the data unit is an MFU, some information about the MPU, such as information for configuring the MPU (MP4), is stored as a header.

[0474] For example, the header of a timed-MFU includes fields such as movie_fragment_sequence_number, sample_number, offset, priority, and dependency_counter, while the header of a non-timed-MFU includes item_iD. The meaning of each field is shown in standards such as ISO / IEC 23008-1 or ARIB STD-B60. The meaning of each field as defined in such standards will be explained below.

[0475] movie_fragment_sequence_number indicates the sequence number of the movie fragment to which the MFU belongs, and is also specified in ISO / IEC 14496-12.

[0476] The `sample_number` indicates the sample number to which the MFU belongs, and is also specified in ISO / IEC 14496-12.

[0477] The offset indicates the offset amount of the MFU in the sample to which the MFU belongs, in bytes.

[0478] The priority value indicates the relative importance of the MFU within the MPU to which it belongs. An MFU with a higher priority value is more important than an MFU with a lower priority value.

[0479] The dependency_counter indicates the number of MFUs whose decoding process depends on the MFU in question (i.e., the number of MFUs that cannot be decoded unless the MFU in question is decoded first). For example, if an MFU is HEVC and a B-picture or P-picture references an I-picture, then the B-picture or P-picture cannot be decoded unless the I-picture is decoded first.

[0480] Therefore, if the MFU is in sample units, the dependency_counter in the I-picture's MFU will show the number of pictures that reference that I-picture. If the MFU is in NAL unit units, the dependency_counter in the MFU belonging to the I-picture will show the number of NAL units belonging to the pictures that reference that I-picture. Furthermore, in the case of time-hierarchically encoded video signals, the MFU of the extension layer depends on the MFU of the base layer, so the dependency_counter in the base layer's MFU will show the number of MFUs in the extension layer. This field cannot be generated until the number of dependent MFUs has been determined.

[0481] The item_iD is an identifier that uniquely identifies the item.

[0482] [MP4 unsupported mode] As explained in Figures 19 and 21, the transmitting device can transmit the MPU in MMT in several ways: by sending the MPU metadata or MF metadata before or after the media data, or by sending only the media data. The receiving device can decode using an MP4-compliant receiving device or method, or by decoding without using a header.

[0483] One method of transmitting data specifically for broadcast stream playback is, for example, a transmission method that does not support MP4 reconstruction in the receiving device.

[0484] Transmission methods that do not support MP4 reconstruction in the receiving device include, for example, methods that do not transmit metadata (MPU metadata and MF metadata), as shown in Figure 21(b). In this case, the field value of the fragment type (information indicating the type of data unit) included in the MMTP packet is fixed at 2 (=MFU).

[0485] If metadata is not transmitted, as explained above, MP4-compliant receivers cannot decode the received data as MP4, but they can decode it without using metadata (headers).

[0486] Therefore, metadata is not necessarily essential information for broadcast stream decoding and playback. Similarly, the data unit header information in the timed-MFU, as explained in Figure 48, is information for reconstructing the MP4 in the receiving device. Since it is not necessary to reconstruct the MP4 during broadcast stream playback, the data unit header information in the timed-MFU (hereinafter also referred to as the timed-MFU header) is not necessarily essential information for broadcast stream playback.

[0487] The receiving device can easily reconstruct the MP4 file using metadata and information for reconstructing the MP4 in the data unit header (hereinafter also referred to as MP4 configuration information). However, the receiving device cannot reconstruct the MP4 file if only one of the metadata or the MP4 configuration information in the data unit header is transmitted. There is little benefit to transmitting only one of the metadata or the information for reconstructing the MP4, and generating and transmitting unnecessary information leads to increased processing load and decreased transmission efficiency.

[0488] Therefore, the transmitting device controls the data structure and transmission of MP4 configuration information using the following method. The transmitting device decides whether or not to include MP4 configuration information in the data unit header based on whether or not metadata is transmitted. Specifically, if metadata is transmitted, the transmitting device includes MP4 configuration information in the data unit header; if metadata is not transmitted, it does not include MP4 configuration information in the data unit header.

[0489] One way to avoid displaying MP4 configuration information in the data unit header is to use methods such as those described below.

[0490] 1. The transmitting device reserves the MP4 configuration information and does not use it. This reduces the processing load on the sending side (the processing load on the transmitting device) that generates the MP4 configuration information.

[0491] 2. The transmitting device removes MP4 configuration information and compresses the header. This reduces the processing load on the sending side that generates MP4 configuration information, and also reduces transmission capacity.

[0492] Furthermore, if the transmitting device removes MP4 configuration information and compresses the header, it may indicate a flag that the MP4 configuration information has been removed (compressed). This flag may be shown in the header (MMTP packet header, MMTP payload header, data unit header) or in the control information, etc.

[0493] Furthermore, information regarding whether metadata is transmitted may be predetermined, or it may be signaled separately in the header or control information and transmitted to the receiving device.

[0494] For example, the MFU header may contain information indicating whether metadata corresponding to the MFU has been transmitted.

[0495] On the other hand, the receiving device can determine whether MP4 configuration information is displayed based on whether metadata is being transmitted.

[0496] If the order of data transmission is predetermined (for example, MPU metadata, MF metadata, media data), the receiving device may determine whether the metadata was received before the media data.

[0497] If MP4 configuration information is provided, the receiving device can use the MP4 configuration information to reconstruct the MP4. Alternatively, the receiving device can use the MP4 configuration information to detect the beginning of other access units or NAL units, etc.

[0498] Note that the MP4 configuration information may consist of all or part of the timed-MFU header.

[0499] Furthermore, the transmitting device also determines whether metadata is being transmitted in the non-timed-MFU header, based on the same criteria. You may decide whether or not to include an ID.

[0500] The transmitting device may display MP4 configuration information in either the timed-MFU or the non-timed-MFU, but not both. If MP4 configuration information is displayed in only one of them, the transmitting device decides whether to display the MP4 configuration information based on whether metadata is transmitted, as well as whether it is a timed-MFU or a non-timed-MFU. The receiving device can determine whether metadata is transmitted and whether MP4 configuration information is displayed based on the timed / non-timed flag.

[0501] In the above explanation, the transmitting device determined whether to display MP4 configuration information based on whether metadata (both MPU metadata and MF metadata) was transmitted. However, the transmitting device may choose not to display MP4 configuration information if a portion of the metadata (either MPU metadata or MF metadata) is not transmitted.

[0502] Furthermore, the transmitting device may decide whether to display MP4 configuration information based on information other than metadata.

[0503] For example, modes such as MP4 support mode and MP4 non-support mode may be defined, and the transmitting device may indicate MP4 configuration information in the data unit header when in MP4 support mode, and not indicate MP4 configuration information in the data unit header when in MP4 non-support mode. Alternatively, the transmitting device may transmit metadata and indicate MP4 configuration information in the data unit header when in MP4 support mode, and not transmit metadata and not indicate MP4 configuration information in the data unit header when in MP4 non-support mode.

[0504] [Transmitter Operation Flow] Next, we will explain the operation flow of the transmitting device. Figure 49 shows the operation flow of the transmitting device.

[0505] The transmitting device first determines whether to transmit metadata (S1001). If the transmitting device determines to transmit metadata (Yes in S1002), it proceeds to step S1003, generates MP4 configuration information, stores it in the header, and transmits it (S1003). In this case, the transmitting device also generates and transmits metadata.

[0506] On the other hand, if the transmitting device determines that metadata should not be transmitted (No in S1002), it transmits the data without generating MP4 configuration information or storing it in the header (S1004). In this case, the transmitting device does not generate or transmit metadata.

[0507] In step S1001, whether or not metadata is transmitted may be predetermined, or it may be determined based on whether metadata is generated within the transmitting device or whether metadata is being transmitted within the transmitting device.

[0508] [Operation Flow of the Receiving Device] Next, we will explain the operation flow of the receiving device. Figure 50 shows the operation flow of the receiving device.

[0509] The receiving device first determines whether metadata is being transmitted (S1101). Whether metadata is being transmitted can be determined by monitoring the fragment type in the MMTP packet payload. Alternatively, whether or not metadata is being transmitted may be predetermined.

[0510] If the receiving device determines that metadata has been transmitted (Yes in S1102), it reconstructs the MP4 and performs decoding using the MP4 configuration information (S1103). On the other hand, if it determines that metadata has not been transmitted (No in S1102), it does not reconstruct the MP4 and performs decoding without using the MP4 configuration information (S1104).

[0511] Furthermore, the receiving device can use the methods described above to detect random access points, the beginning of access units, the beginning of NAL units, etc., without using MP4 configuration information, and can perform decoding, packet loss detection, and recovery from packet loss.

[0512] For example, the first access unit is the first data of an MMT payload with an aggregation_flag value of 1. In this case, the Fragmentation_indicator value is 0.

[0513] Furthermore, the beginning of a slice segment is the first data point of the MMT payload where the aggregation_flag value is 0 and the fragmentation_indicator value is 00 or 01.

[0514] Based on the information described above, the receiving device can detect the beginning of the access unit and the slice segment.

[0515] The receiving device may also analyze the NAL unit header in packets containing the beginning of a data unit with a fragmentation_indicator value of 00 or 01, and detect that the type of NAL unit is an AU delimiter and that the type of NAL unit is a slice segment.

[0516] [Simple Broadcast Mode] Up to this point, we have described a method of transmitting data specifically for broadcast stream playback that does not support MP4 configuration information in the receiving device. However, this is not the only method of transmitting data specifically for broadcast stream playback.

[0517] For example, the following methods may be used as data transmission methods specifically for broadcast stream playback.

[0518] • Transmitting equipment does not need to use AL-FEC in a fixed broadcast reception environment. If AL-FEC is not used, the FEC_type in the MMTP packet header is always fixed at 0.

[0519] • Transmitting devices may always use AL-FEC in mobile broadcasting reception environments and in UDP communication transmission mode. When AL-FEC is used, the FEC_type in the MMTP packet header is always 0 or 1.

[0520] • The transmitting device does not need to perform bulk transmission of assets. If bulk transmission of assets is not performed, location_infolocation, which indicates the number of transmission locations for the assets within MPT, may be fixed at 1.

[0521] • The transmitting device does not need to perform hybrid transmission of assets, programs, and messages.

[0522] Furthermore, for example, if a broadcast simple mode is defined, the transmitting device may either set it to an MP4 non-support mode when broadcast simple mode is enabled, or use the data transmission method specifically for broadcast stream playback as described above. Whether or not it is broadcast simple mode may be predetermined, or the transmitting device may store a flag indicating that it is broadcast simple mode as control information and transmit it to the receiving device.

[0523] Furthermore, the transmitting device may, based on whether it is in MP4 non-support mode (i.e., whether metadata is being transmitted), use the data transmission method specifically for broadcast stream playback as described above, in broadcast simple mode if it is in MP4 non-support mode, as explained in Figure 49.

[0524] When the receiving device is in broadcast simple mode, it can decode the MP4 without reconstructing it, treating it as an MP4 non-support mode.

[0525] Furthermore, when the receiving device is in broadcast simple mode, it can determine that it has functions specifically for broadcasting and perform reception processing specifically for broadcasting.

[0526] As a result, in broadcast-simple mode, by using only broadcast-specific functions, unnecessary processing for both the transmitting and receiving devices can be reduced, and transmission overhead can be reduced by not compressing and transmitting unnecessary information.

[0527] Furthermore, if an MP4-unsupported mode is used, hint information supporting storage methods other than MP4 configuration may be provided.

[0528] Other storage methods besides MP4 format include, for example, directly storing MMT packets or IP packets, or converting MMT packets into MPEG-2 TS packets.

[0529] In addition, in non-MP4 supported modes, formats that do not conform to the MP4 structure may be used.

[0530] For example, in the case of MP4 non-support mode, the data stored in the MFU may be in a format that includes a byte start code, rather than a format that includes the size of the NAL unit at the beginning of the NAL unit, which is in MP4 format.

[0531] In MMT, the asset type, which indicates the type of asset, is described by a 4CC registered in MP4REG (http: / / www.mp4ra.org). When HEVC is used as the video signal, 'HEV1' or 'HVC1' is used. 'HVC1' is a format in which the sample may contain a parameter set, while 'HEV1' is a format in which the sample does not contain a parameter set, but the parameter set is included in the sample entry in the MPU metadata.

[0532] In broadcast simple mode or MP4 non-support mode, if MPU metadata and MF metadata are not transmitted, it may be specified that the parameter set must always be included in the sample. Furthermore, it may be specified that the 'HVC1' format must always be used, regardless of whether 'HEV1' or 'HVC1' is indicated in the asset type.

[0533] [Supplement 1: Transmitter] As described above, if metadata is not transmitted, the MP4 configuration information will be set to reserved, and the transmitting device that is not in operation can be configured as shown in Figure 51. Figure 51 is a diagram showing a specific example of the configuration of the transmitting device.

[0534] The transmitting device 300 comprises an encoding unit 301, an assignment unit 302, and a transmitting unit 303. Each of the encoding unit 301, assignment unit 302, and transmitting unit 303 is implemented, for example, by a microcomputer, processor, or dedicated circuit.

[0535] The encoding unit 301 encodes a video signal or an audio signal to generate sample data. Specifically, the sample data is a data unit.

[0536] The assignment unit 302 assigns header information, including MP4 configuration information, to the sample data, which is data in which a video signal or audio signal has been encoded. The MP4 configuration information is information that allows the receiving side to reconstruct the sample data as an MP4 format file, and its content differs depending on whether or not the presentation time of the sample data is specified.

[0537] As described above, the assignment unit 302 includes MP4 configuration information such as movie_fragment_sequence_number, sample_number, offset, priority, and dependency_counter in the header (header information) of timed-MFU, which is an example of sample data (sample data containing information about synchronization) for which a presentation time has been determined.

[0538] On the other hand, the assignment unit 302 includes MP4 configuration information such as item_id in the header (header information) of timed-MFU, which is an example of sample data for which the presentation time is not specified (sample data that does not contain information about synchronization).

[0539] Then, if metadata corresponding to the sample data is not transmitted by the transmission unit 303 (for example, in the case shown in Figure 21(b)), the assignment unit 302 assigns header information that does not include MP4 configuration information to the sample data, depending on whether or not a presentation time for the sample data has been specified.

[0540] Specifically, the assignment unit 302 assigns header information that does not include the first MP4 configuration information to the sample data when a presentation time for the sample data is specified, and assigns header information that includes the second MP4 configuration information to the sample data when a presentation time for the sample data is not specified.

[0541] For example, as shown in step S1004 of Figure 49, if the transmission unit 303 does not transmit metadata corresponding to the sample data, the assignment unit 302 sets the MP4 configuration information to reserved (fixed value), thereby effectively not generating MP4 configuration information and not effectively storing it in the header (header information). The metadata includes MPU metadata and movie fragment metadata.

[0542] The transmitting unit 303 transmits sample data to which header information has been added. More specifically, the transmitting unit 303 packets the sample data to which header information has been added using the MMT method and transmits it.

[0543] As mentioned above, in transmission and reception methods specifically designed for broadcast stream playback, the receiving device does not need to reconstruct the data units into MP4 format. When the receiving device does not need to reconstruct into MP4, the processing load on the transmitting device is reduced by not generating unnecessary information such as MP4 configuration information.

[0544] On the other hand, while the transmitting device must send the necessary information, it also needs to maintain consistency with the standard so as not to have to send any unnecessary additional information separately.

[0545] With a configuration like that of the transmitter 300, by fixing the area where MP4 configuration information is stored, it is possible to transmit only the necessary information according to the standard without transmitting the MP4 configuration information, thus eliminating the need to transmit unnecessary additional information. In other words, the configuration of the transmitter and the processing load of the transmitter can be reduced. Furthermore, by not transmitting unnecessary data, transmission efficiency can be improved.

[0546] [Supplement 2: Receiving device] Furthermore, the receiving device corresponding to the transmitting device 300 may be configured as shown in Figure 52, for example. Figure 52 is a diagram showing another example of the configuration of the receiving device.

[0547] The receiving device 400 comprises a receiving unit 401 and a decoding unit 402. The receiving unit 401 and the decoding unit 402 are implemented, for example, by a microcomputer, a processor, or a dedicated circuit.

[0548] The receiving unit 401 receives sample data which is data in which a video signal or audio signal has been encoded, and which has header information attached that includes MP4 configuration information for reconstructing the sample data as an MP4 format file.

[0549] The decoding unit 402 decodes the sample data without using MP4 configuration information if the receiving unit has not received metadata corresponding to the sample data and the presentation time of the sample data has been determined.

[0550] For example, as shown in step S1104 of Figure 50, the decoding unit 402 performs the decoding process without using MP4 configuration information if the receiving unit 401 does not receive metadata corresponding to the sample data.

[0551] This makes it possible to reduce the configuration of the receiving device 400 and the processing load in the receiving device 400.

[0552] (Embodiment 4) [overview] Embodiment 4 describes a method for storing asynchronous (non-timed) media, such as files, that do not contain synchronization information, into the MPU, and a method for transmitting them using MMTP packets. Although Embodiment 4 uses the MPU in MMT as an example, it is also applicable to DASH, which is also MP4-based.

[0553] First, we will explain the details of how non-timed media (hereinafter also referred to as "asynchronous media data") is stored in the MPU using Figure 53. Figure 53 is a diagram showing how non-timed media is stored in the MPU and how it is transmitted using MMTP packets.

[0554] The MPU, which stores non-timed media, consists of boxes such as ftyp, mmpu, moov, and meta, which store information about the files to be stored in the MPU. A meta box can contain multiple idat boxes, and each idat box stores one file as an item.

[0555] Some of the ftyp, mmpu, moov, and meta boxes constitute a single data unit as MPU metadata, while the item or idat boxes constitute a data unit as an MFU.

[0556] After data units are aggregated or fragmented, they are transmitted as MMTP packets with a data unit header, MMTP payload header, and MMTP packet header added.

[0557] Figure 53 shows an example where File#1 and File#2 are stored in a single MPU. The MPU metadata is not fragmented, and the MFU is fragmented and stored in MMTP packets, but this is not the only option; aggregation or fragmentation may be used depending on the size of the data units. Furthermore, the MPU metadata does not have to be transmitted; in that case, only the MFU is transmitted.

[0558] Header information such as the data unit header includes the itemID (an identifier that uniquely identifies the item), while the MMTP payload header and MMTP packet header include the packet sequence number (a sequence number for each packet) and the MPU sequence number (a sequence number for the MPU, a unique number within the asset).

[0559] Note that the data structures of the MMTP payload header and the MMTP packet header other than the data unit header are the same as those of the timed media (hereinafter also referred to as "synchronized media data") described so far, and include an aggregation_flag, a fragmentation_indicator, a fragment_counter, etc.

[0560] Next, a specific example of header information when a file (= Item = MFU) is split and packetized will be described using FIGS. 54 and 55.

[0561] FIGS. 54 and 55 are diagrams showing examples of packetizing and transmitting each of a plurality of split data obtained by splitting a file. FIGS. 54 and 55 specifically show information (packet sequence number, fragment counter, fragmentation indicator, MPU sequence number, item ID) included in any of the data unit header, the MMTP payload header, and the MMTP packet header, which is the header information for each split MMTP packet. Note that FIG. 54 shows an example in which File #1 is split into M (M <= 256) parts, and FIG. 55 shows an example in which File #2 is split into N (256 <N) parts.

[0562] The split data number indicates the index of the split data from the beginning of the file, and this information is not transmitted. That is, the split data number is not included in the header information. Also, the split data number is a number assigned to each packet corresponding to a plurality of split data obtained by splitting the file, and is a number assigned by adding 1 in ascending order from the first packet.

[0563] The packet sequence number is the sequence number of packets having the same packet ID. In FIGS. 54 and 55, assuming that the split data at the beginning of the file is A, consecutive numbers are assigned up to the split data at the end of the file. The packet sequence number is a number assigned by adding 1 in ascending order from the split data at the beginning of the file, and is a number corresponding to the split data number.

[0564] The fragment counter indicates the number of fragments that occur after the fragment in question, out of the multiple fragments obtained when a single file is split. Furthermore, if the number of fragments (the number of multiple fragments obtained when splitting a single file) exceeds 256, the fragment counter displays the remainder when the number of fragments is divided by 256. In the example in Figure 54, since the number of fragments is 256 or less, the field value of the fragment counter is (M - fragment number). On the other hand, in the example in Figure 55, since the number of fragments exceeds 256, the value is ((N - fragment number)%256), which is obtained by dividing (N - fragment number) by 256.

[0565] The fragmentation indicator indicates the state of data fragmentation stored in the MMTP packet. It shows whether the data is the first fragment, the last fragment, any other fragment, or one or more unfragmented data units. Specifically, the fragmentation indicator is "01" for the first fragment, "11" for the last fragment, "10" for the remaining fragments, and "00" for unfragmented data units.

[0566] In this embodiment, when the number of divided data points exceeds 256, the remainder obtained by dividing the number of divided data points by 256 is used for explanation. However, the number of divided data points is not limited to 256 and may be any other number (a predetermined number).

[0567] As shown in Figures 54 and 55, when a file is split and conventional header information is added to each of the multiple split data obtained by splitting the file and then transmitted, the receiving device does not have information that allows it to determine which split data number the data stored in the received MMTP packet corresponds to in the original file (split data number), the number of split data in the file, or the split data number and the number of split data. Therefore, with conventional transmission methods, even if an MMTP packet is received, it is not possible to uniquely detect the split data number or the number of split data stored in the received MMTP packet.

[0568] For example, as shown in Figure 54, if the number of divided data is 256 or less, and it is known in advance that the number of divided data is 256 or less, it is possible to identify the divided data number and the number of divided data by referring to the fragment counter. However, if the number of divided data is 256 or more, it is not possible to identify the divided data number and the number of divided data.

[0569] Furthermore, if the number of data segments in a file is limited to 256 or less, and the data size that can be transmitted in one packet is x [bytes], then the maximum size of a file that can be transmitted is limited to x * 256 [bytes]. For example, in broadcasting, x = 4k [bytes] is assumed, in which case the maximum size of a file that can be transmitted is limited to 4k * 256 = 1M [bytes]. Therefore, if you want to transmit a file larger than 1 [Mbytes], you cannot limit the number of data segments in the file to 256 or less.

[0570] Furthermore, for example, by referring to the fragmentation indicator, it is possible to detect the fragmented data at the beginning or end of a file. Therefore, it is possible to count the number of MMTP packets until the MMTP packet containing the last fragmented data of the file is received, or, after receiving the MMTP packet containing the last fragmented data of the file, calculate the fragmented data number and the number of fragmented data by combining it with the packet sequence number. Thus, signaling the fragmented data number and the number of fragmented data by combining the fragmentation indicator and the packet sequence number is also possible. However, if reception starts with an MMTP packet containing fragmented data in the middle of the file (i.e., fragmented data that is neither the first nor the last fragmented data of the file), the fragmented data number and the number of fragmented data for that fragment cannot be determined. The fragmented data number and the number of fragmented data for that fragment can only be determined after receiving the MMTP packet containing the last fragmented data of the file.

[0571] To address the problem described in Figures 54 and 55, namely, to uniquely determine the segmentation data number and the number of segmented data when a packet containing segmented file data is received midway through the process, the following method is used.

[0572] First, let's explain the segmented data numbers.

[0573] For the segmented data number, the packet sequence number in the segmented data at the beginning of the file (item) is used as the signaling element.

[0574] As a signaling method, the data is stored in the control information that manages the file. Specifically, in Figures 54 and 55, the packet sequence number A of the first segmented data in the file is stored in the control information. The receiving device obtains the value of A from the control information and calculates the segmented data number from the packet sequence number shown in the packet header.

[0575] The segment data number of the segmented data is obtained by subtracting the packet sequence number A of the first segmented data from the packet sequence number of the segmented data in question.

[0576] One example of control information for managing files is the asset management table defined in ARIB STD-B60. The asset management table contains information such as file size and version for each file, and this information is stored and transmitted in the data transmission message. Figure 56 shows the syntax of the file-by-file loop in the asset management table.

[0577] If the existing asset management table's space cannot be expanded, signaling may be performed using a 32-bit area of ​​the item_info_byte field, which contains information about the item. A flag indicating whether the packet sequence number in the split data at the beginning of the file (item) is shown in a portion of the item_info_byte may be indicated in the reserved_future_use field of the control information.

[0578] When repeatedly transmitting files, such as in a data carousel, multiple packet sequence numbers may be indicated, or the packet sequence number of the first file to be transmitted immediately afterward may be indicated.

[0579] The information doesn't have to be limited to the packet sequence number of the split data at the beginning of the file; any information that links the file's split data number to the packet sequence number is acceptable.

[0580] Next, I will explain the number of data segments.

[0581] The order of loops for each file included in the asset management table may be defined as the file transmission order. This allows us to determine the packet sequence number of the first two consecutive files in transmission order, and by subtracting the first packet sequence number of the previously transmitted file from the first packet sequence number of the later transmitted file, we can determine the number of data segments in the previously transmitted file. In other words, for example, if File#1 shown in Figure 54 and File#2 shown in Figure 55 are consecutive files in this order, then the last packet sequence number of File#1 and the first packet sequence number of File#2 are assigned consecutive numbers.

[0582] Alternatively, the method of splitting the file may be specified to allow for the determination of the number of data segments within a file. For example, if the number of data segments is N, the size of each segment from the 1st to the (N-1)th segment can be defined as L, and the size of the Nth segment can be defined as the remainder (item_size - L * (N-1)). In this case, the number of data segments can be calculated from the item_size shown in the asset management table. In this case, the number of data segments will be an integer value obtained by rounding up (item_size / L). Note that this is not the only method for splitting a file.

[0583] Alternatively, the number of divided data points may be stored directly in the asset management table.

[0584] The receiving device receives control information using the method described above and calculates the number of divided data based on the control information. It can also calculate the packet sequence number corresponding to the file's divided data number based on the control information. If the timing of receiving the divided data packets is earlier than the timing of receiving the control information, the divided data number and the number of divided data may be calculated at the time the control information is received.

[0585] Furthermore, when signaling the fragment data number or the number of fragment data using the method described above, the fragment data number or the number of fragment data is not identified based on the fragment counter, making the fragment counter unnecessary data. Therefore, in the transmission of asynchronous media, if information that can identify the fragment data number and the number of fragment data is signaled using the method described above, the fragment counter may be disabled or the header may be compressed. This can reduce the processing load on the transmitting and receiving devices and improve transmission efficiency. In other words, when transmitting asynchronous media, the fragment counter may be reserved (disabled). Specifically, the value of the fragment counter may be set to a fixed value, for example, "0". Also, when receiving asynchronous media, the fragment counter may be ignored.

[0586] When storing synchronized media such as video or audio, the transmission order of MMTP packets at the transmitting device matches the arrival order of MMTP packets at the receiving device, and packets are not retransmitted. In such cases, if there is no need to detect and reconstruct packet loss, the fragment counter may not be operated. In other words, in this case, the fragment counter may be reserved (disabled).

[0587] Furthermore, even without using a fragment counter, it is possible to detect random access points, the beginning of access units, and the beginning of NAL units, enabling decoding, packet loss detection, and recovery from packet loss.

[0588] Furthermore, in the transmission of real-time content such as live broadcasts, lower latency transmission is required, and it is necessary to sequentially packetize and send data as encoding is completed. However, in the transmission of real-time content, conventional fragment counters cannot determine the number of fragments when sending the first fragmented data. Therefore, the first fragmented data is sent only after all data unit encoding is complete and the number of fragments has been determined, resulting in a delay. Even in such cases, this delay can be reduced by not using a fragment counter using the method described above.

[0589] Figure 57 shows the operation flow for identifying the segmented data number in the receiving device.

[0590] The receiving device obtains control information containing file information (S1201). The receiving device determines whether the control information indicates the packet sequence number of the beginning of the file (S1202). If the control information indicates the packet sequence number of the beginning of the file (Yes in S1202), it calculates the packet sequence number corresponding to the segmented data number of the segmented data of the file (S1203). Then, after obtaining the MMTP packet containing the segmented data, the receiving device identifies the segmented data number of the file from the packet sequence number stored in the packet header of the obtained MMTP packet (S1204). On the other hand, if the control information does not indicate the packet sequence number of the beginning of the file (No in S1202), the receiving device obtains the MMTP packet containing the last segmented data of the file, and then identifies the segmented data number using the fragment indicator and packet sequence number stored in the packet header of the obtained MMTP packet (S1205).

[0591] Figure 58 shows the operation flow for determining the number of divided data segments in the receiving device.

[0592] The receiving device obtains control information containing file information (S1301). The receiving device determines whether the control information contains information that allows for the calculation of the number of divided data files (S1302). If it determines that the control information contains information that allows for the calculation of the number of divided data files (Yes in S1302), it calculates the number of divided data files based on the information contained in the control information (S1303). On the other hand, if the receiving device determines that it is not possible to calculate the number of divided data files (No in S1302), it obtains the MMTP packet containing the last divided data file, and then identifies the number of divided data files using the fragment indicator stored in the packet header of the obtained MMTP packet and the packet sequence number (S1304).

[0593] Figure 59 shows the operation flow for determining whether or not to operate a fragment counter in the transmitting device.

[0594] First, the transmitting device determines whether the medium to be transmitted (hereinafter also referred to as "media data") is a synchronous medium or an asynchronous medium (S1401).

[0595] If the result of the determination in step S1401 is that it is a synchronous medium (synchronous medium in S1402), the transmitting device determines whether the order of transmitted and received MMTP packets matches in the environment in which the synchronous medium is transmitted, and whether packet reconstruction is unnecessary in the event of packet loss (S1403). If the transmitting device determines that it is unnecessary (Yes in S1403), it does not operate the fragment counter (S1404). On the other hand, if the transmitting device determines that it is not unnecessary (No in S1403), it operates the fragment counter (S1405).

[0596] If the result of the determination in step S1401 is that it is an asynchronous medium (asynchronous medium in S1402), the transmitting device decides whether or not to operate the fragment counter based on whether the fragment data number and the number of fragment data are signaled using the method described above. Specifically, if the fragment data number and the number of fragment data are signaled (Yes in S1406), the transmitting device does not operate the fragment counter (S1404). On the other hand, if the fragment data number and the number of fragment data are not signaled (No in S1406), the transmitting device operates the fragment counter (S1405).

[0597] If the transmitting device does not operate a fragment counter, it may set the fragment counter value to "reserved" or compress the header.

[0598] Furthermore, the transmitting device may decide whether or not to signal the aforementioned fragment data number and the number of fragment data based on whether or not it operates a fragment counter.

[0599] Furthermore, if the synchronous media does not operate a fragment counter, the transmitting device may signal the fragment data number and the number of fragment data using the method described above in the asynchronous media. Conversely, the operation of the synchronous media may be determined based on whether or not the asynchronous media operates a fragment counter. In this case, the operation of fragmentation can be the same for both the synchronous and asynchronous media.

[0600] Next, we will explain how to identify the number of divided data and the number of divided data (when using a fragment counter). Figure 60 is a diagram illustrating how to identify the number of divided data and the number of divided data (when using a fragment counter).

[0601] As explained using Figure 54, if the number of divided data is 256 or less, and it is known in advance that the number of divided data is 256 or less, it is possible to identify the divided data number and the number of divided data by referring to the fragment counter.

[0602] If the number of data segments in a file is limited to 256 or less, and the data size that can be transmitted in one packet is x [bytes], then the maximum size of a file that can be transmitted is limited to x * 256 [bytes]. For example, in broadcasting, x = 4k [bytes] is assumed, in which case the maximum size of a file that can be transmitted is limited to 4k * 256 = 1M [bytes].

[0603] If the file size exceeds the maximum size of a file that can be transmitted, the file is pre-divided so that the size of each divided file is x*256 [bytes] or less. Each of the multiple divided files obtained by splitting the file is treated as a single file (item), and is further divided into 256 or less sections. The resulting divided data is then stored in MMTP packets and transmitted.

[0604] Furthermore, information indicating that an item is a split file, the number of split files, and the sequence number of the split files may be stored in the control information and sent to the receiving device. Alternatively, this information may be stored in the asset management table, or it may be indicated using a part of the existing field item_info_byte.

[0605] If an item is one of several split files obtained by splitting a single file, the receiving device can identify the other split files and reconstruct the original file. Furthermore, the receiving device can uniquely identify the number of split data items and the split data item number by using the number of split files, the index of the split files, and the fragment counter in the control information. It can also uniquely identify the number of split data items and the split data item number without using packet sequence numbers or similar information.

[0606] Here, it is desirable that the item_ids of each of the multiple split files obtained by splitting a single file are the same. If different item_ids are assigned, the item_id of the first split file may be used to uniquely refer to the file from other control information, etc.

[0607] Furthermore, multiple split files may always belong to the same MPU. When storing multiple files in an MPU, different types of files may not be stored, and all files stored may always be split versions of a single file. The receiving device can detect file updates by checking the version information for each MPU, without having to check the version information for each item.

[0608] Figure 61 shows the operation flow of the transmitting device when a fragment counter is used.

[0609] First, the transmitting device checks the size of the file to be transmitted (S1501). Next, the transmitting device determines whether the file size exceeds x*256 [bytes] (where x is the data size that can be transmitted in one packet, for example, the MTU size) (S1502). If the file size exceeds x*256 [bytes] (Yes in S1502), the device splits the file so that the size of each split file is less than x*256 [bytes] (S1503). Then, the split files are transmitted as items, and information about the split files (for example, that they are split files, the sequence number in the split files, etc.) is stored in the control information and transmitted (S1504). On the other hand, if the file size is less than x*256 [bytes] (No in S1502), the file is transmitted as an item as usual (S1505).

[0610] Figure 62 shows the operation flow of the receiving device when a fragment counter is used.

[0611] First, the receiving device acquires and analyzes control information related to file transmission, such as the asset management table (S1601). Next, the receiving device determines whether the desired item is a split file (S1602). If the receiving device determines that the desired file is a split file (Yes in S1602), it acquires information from the control information to reconstruct the file, such as the split file and the index of the split file (S1603). Then, the receiving device acquires the items that make up the split file and reconstructs the original file (S1604). On the other hand, if the receiving device determines that the desired file is not a split file (No in S1602), it acquires the file as usual (S1605).

[0612] In short, the transmitting device signals the packet sequence number of the fragmented data at the beginning of the file. It also signals information that allows for the identification of the number of fragmented data. Alternatively, the transmitting device defines fragmentation rules that allow for the identification of the number of fragmented data. Furthermore, the transmitting device either reserves or compresses the header without using a fragment counter.

[0613] If the packet sequence number of the data at the beginning of the file is signaled, the receiving device will identify the segmented data number and the number of segmented data from the packet sequence number of the segmented data at the beginning of the file and the packet sequence number of the MMTP packet.

[0614] From another perspective, the transmitting device splits the file and transmits the data in separate files. It signals information linking the split files (sequence number, number of splits, etc.).

[0615] The receiving device identifies the segmented data number and the number of segmented data using the fragment counter and the sequence number of the segmented file.

[0616] This allows for the unique identification of segmented data numbers and segmented data. Furthermore, since the segmented data number of a segmented data is identified as soon as it is received, waiting time and memory usage can be reduced.

[0617] Furthermore, by not using a fragment counter, the configuration of the transceiver can reduce the processing load and improve transmission efficiency.

[0618] Figure 63 shows the service configuration when the same program is transmitted using multiple IP data flows. Here, a portion of the data (video and audio) of the program with service ID=2 is transmitted using an IP data flow with the MMT method, while data with the same service ID but different from that portion is transmitted using an IP data flow with the advanced BS data transmission method (in this example, the file transmission protocol is different but could be the same protocol).

[0619] The transmitting device performs IP data multiplexing to ensure that the receiving device receives data composed of multiple IP data flows by the time of decoding.

[0620] The receiving device can achieve guaranteed receiver operation by processing data composed of multiple IP data flows based on the decoding time.

[0621] [Note: Transmitting device and receiving device] As described above, a transmitting device that transmits data without operating a fragment counter can be configured as shown in Figure 64. Similarly, a receiving device that receives data without operating a fragment counter can be configured as shown in Figure 65. Figure 64 is a diagram showing a specific example of the configuration of a transmitting device. Figure 65 is a diagram showing a specific example of the configuration of a receiving device.

[0622] The transmitting device 500 comprises a divided section 501, a component section 502, and a transmitting section 503. Each of the divided section 501, component section 502, and transmitting section 503 is implemented by, for example, a microcomputer, a processor, or a dedicated circuit.

[0623] The receiving device 600 comprises a receiving unit 601, a determination unit 602, and a component unit 603. Each of the receiving unit 601, the determination unit 602, and the component unit 603 is implemented by, for example, a microcomputer, a processor, or a dedicated circuit.

[0624] Detailed descriptions of each component of the transmitting device 500 and the receiving device 600 will be provided in the descriptions of the transmitting method and the receiving method, respectively.

[0625] First, the transmission method will be explained using Figure 66. Figure 66 shows the operation flow (transmission method) by the transmitting device.

[0626] First, the splitting unit 501 of the transmitting device 500 divides the data into multiple split data (S1701).

[0627] Next, component 502 of the transmitting device 500 constructs multiple packets by adding header information to each of the multiple divided data and packaging them (S1702).

[0628] Then, the transmitting unit 503 of the transmitting device 500 transmits the configured plurality of packets (S1703). The transmitting unit 503 transmits the fragmentation data information and the value of the invalidated fragment counter. The fragmentation data information is information for identifying the fragmentation data number and the number of fragmentation data. The fragmentation data number is a number indicating which of the plurality of fragmentation data the fragmentation data is. The number of fragmentation data is the number of the plurality of fragmentation data.

[0629] This reduces the processing load on the transmitting device 500.

[0630] Next, the receiving method will be explained using Figure 67. Figure 67 shows the operation flow (receiving method) by the receiving device.

[0631] First, the receiving unit 601 of the receiving device 600 receives multiple packets (S1801).

[0632] Next, the determination unit 602 of the receiving device 600 determines whether or not segmented data information has been acquired from the multiple packets received (S1802).

[0633] Then, if the receiving device 600 component 603 determines that it has acquired fragmented data information by the determination unit 602 (Yes in S1802), it constructs data from the multiple received packets without using the value of the fragment counter included in the header information (S1803).

[0634] On the other hand, if the determination unit 602 determines that it has not acquired fragmented data information, the component 603 may use the value of the fragment counter included in the header information to construct data from the multiple received packets (S1804).

[0635] This reduces the processing load on the receiving device 600.

[0636] (Embodiment 5) [overview] Embodiment 5 describes a method for transmitting transmission packets (TLV packets) when NAL units are stored in the multiplexing layer in NAL size format.

[0637] As described in Embodiment 1, there are two formats for storing H.264 and H.265 NAL units in the multiplexing layer. One is a format called the byte stream format, which adds a start code consisting of a specific bit sequence immediately before the NAL unit header. The other is a format called the NAL size format, which adds a field indicating the size of the NAL unit. The byte stream format is used in MPEG-2 systems and RTP, while the NAL size format is used in MP4, or in DASH and MMT, which use MP4.

[0638] In byte stream format, the start code consists of 3 bytes, and any additional byte (a byte with a value of 0) can be added.

[0639] On the other hand, in the NAL size format of typical MP4 files, size information is represented by either 1 byte, 2 bytes, or 4 bytes. This size information is shown in the `lengthSizeMinusOne` field in the HEVC sample entry. A value of "0" in this field indicates 1 byte, "1" indicates 2 bytes, and "3" indicates 4 bytes.

[0640] In ARIB STD-B60, "Media Transport Method by MMT in Digital Broadcasting," standardized in July 2014, when storing NAL units in the multiplexing layer, if the output of the HEVC encoder is a byte stream, the byte start code is removed, and the size of the NAL unit in bytes, represented by a 32-bit (unsigned integer), is added as length information immediately before the NAL unit. Note that MPU metadata, including HEVC sample entries, is not transmitted, and the size information is fixed at 32 bits (4 bytes).

[0641] Furthermore, ARIB STD-B60 "Media Transport System by MMT in Digital Broadcasting" specifies that in the receive buffer model considered by the transmitting device during transmission to guarantee buffer operation in the receiving device, the pre-decoded buffer for the video signal is a CPB (Critical Point Buffer).

[0642] However, there are the following challenges. CPB in MPEG-2 systems and HRD in HEVC are defined on the assumption that the video signal is in byte stream format. For example, if rate control of transmission packets is performed on the assumption that it is in byte stream format with a 3-byte start code, a receiving device that receives a transmission packet in NAL size format with an added 4-byte size area may not be able to satisfy the receive buffer model in ARIB STD-B60. Furthermore, since the receive buffer model in ARIB STD-B60 does not specify the buffer size and extraction rate, it is difficult to guarantee the buffer operation in the receiving device.

[0643] Therefore, in order to solve the above problems, a receive buffer model to guarantee buffer operation in the receiver is defined as follows.

[0644] Figure 68 shows the receive buffer model based on the receive buffer model specified in ARIB STD-B60, specifically for the case where only the broadcast transmission line is used.

[0645] The receive buffer model includes a TLV packet buffer (first buffer), an IP packet buffer (second buffer), an MMTP buffer (third buffer), and a pre-decode buffer (fourth buffer). Note that in broadcast transmission lines, digitter buffers and buffers for FEC are not necessary and are therefore omitted.

[0646] The TLV packet buffer receives TLV packets (transmission packets) from the broadcast transmission path, converts the IP packets, which consist of a variable-length packet header (IP packet header, full header during IP packet compression, compressed header during IP packet compression) and a variable-length payload stored in the received TLV packets, into IP packets (first packets) with a fixed-length IP packet header after header decompression, and outputs the resulting IP packets at a constant bitrate.

[0647] The IP packet buffer converts IP packets into MMTP packets (second packets) that have a packet header and a variable-length payload, and outputs the resulting MMTP packets at a constant bitrate. The IP packet buffer may also be merged into the MMTP buffer.

[0648] The MMTP buffer converts the output MMTP packets into NAL units and outputs the resulting NAL units at a constant bitrate.

[0649] The pre-decryption buffer sequentially stores the output NAL units, generates access units from the multiple stored NAL units, and outputs the generated access units to the decoder at the timing corresponding to the decoding time of the access unit.

[0650] In the receive buffer model shown in Figure 68, a distinctive feature is that the MMTP buffer and pre-decode buffer, which are buffers other than the preceding TLV packet buffer and IP packet buffer, follow the receive buffer model in MPEG-2 TS.

[0651] For example, the MMTP buffer (MMTP B1) in video consists of buffers equivalent to the transport buffer (TB) and multiplexing buffer (MB) in MPEG-2 TS. Similarly, the MMTP buffer (MMTP Bn) in audio consists of a buffer equivalent to the transport buffer (TB) in MPEG-2 TS.

[0652] The transport buffer size is fixed, similar to that of MPEG-2 TS. For example, it can be n times the MTU size (where n can be a decimal or an integer, and is greater than or equal to 1).

[0653] Furthermore, the MMTP packet size is defined such that the overhead rate of the MMTP packet header is smaller than that of the PES packet header. This allows the same extraction rates RX1, RXn, and RXs as those used in MPEG-2 TS to be applied to the transport buffer.

[0654] Furthermore, the size of the multiplexing buffer and the extraction rate are, respectively, those of MPEG-2. The MB size in TS is specified as RBX1.

[0655] In addition to the above receive buffer model, the following constraints are imposed to solve the problem.

[0656] The HEVC HRD specification assumes a byte stream format, and MMT is a NAL-sized format that adds a 4-byte size area to the beginning of the NAL unit. Therefore, during encoding, rate control is performed to satisfy the HRD in the NAL-sized format.

[0657] In other words, the transmitting device controls the transmission packet rate based on the above-described receive buffer model and constraints.

[0658] The receiving device can perform decoding operations that prevent underflow or overflow by using the above-mentioned signals for reception processing.

[0659] Even if the size area at the beginning of the NAL unit is not 4 bytes, rate control is performed to satisfy HRD by taking the size area at the beginning of the NAL unit into consideration.

[0660] The TLV packet buffer extraction rate (the bitrate at which the TLV packet buffer outputs IP packets) should be set considering the transmission rate after IP header decompression.

[0661] In other words, it takes a TLV packet with variable data size as input, removes the TLV header, expands (restores) the IP header, and then considers the transmission rate of the output IP packet. To put it another way, it takes into account the increase or decrease in header size relative to the input transmission rate.

[0662] Specifically, because the data size is variable, packets with and without IP header compression coexist, and the IP header size differs depending on the packet type (IPv4, IPv6, etc.), the transmission rate of output IP packets is not unique. Therefore, the average packet length of the variable-length data is defined, and the transmission rate of IP packets output from TLV packets is determined.

[0663] Here, in order to define the maximum transmission speed after decompression of the IP header, the transmission rate is determined assuming that the IP header is always compressed.

[0664] Furthermore, when IPv4 and IPv6 packet types are mixed, or when packet types are defined without distinction, the transmission rate is determined assuming IPv6 packets, which have a larger header size and a larger increase rate after header expansion.

[0665] For example, if the average packet length of TLV packets input to the TLV packet buffer is S, and all IP packets stored in the TLV packets are IPv6 packets with header compression, then the maximum output transmission rate after removing the TLV header and decompressing the IP header is: Input rate × {S / (S + IPv6 header compression amount)} This is the result.

[0666] More specifically, the average packet length S of TLV packets is S = 0.75 × 1500 (1500 is assumed to be the maximum MTU size) Set based on this, IPv6 header compression amount = TLV header length - IPv6 header length - UDP header length =3-40-8 In that case, the maximum output transmission rate after removing the TLV header and decompressing the IP header is: Input rate × 1.0417 ≈ Input rate × 1.05 This is the result.

[0667] Figure 69 shows an example of aggregating multiple data units and storing them in a single payload.

[0668] In the MMT method, when data units are aggregated, the data unit length and data unit header are added before the data unit, as shown in Figure 69.

[0669] However, for example, when storing a video signal in NAL size format as a single data unit, as shown in Figure 70, there are two fields indicating the size for a single data unit, resulting in redundant information. Figure 70 is an example of aggregating multiple data units and storing them in a single payload, specifically showing an example where a video signal in NAL size format is treated as a single data unit. Specifically, both the size area at the beginning of the NAL size format (hereinafter referred to as the "size area") and the data unit length field located before the data unit header in the MMTP payload header are fields indicating size, and therefore the information is redundant. For example, if the length of the NAL unit is L bytes, the size area will show L bytes, and the data unit length field will show L bytes + "length of the size area" (bytes). The values ​​shown in the size area and the data unit length field do not perfectly match, but since the other value can be easily calculated from the other, it can be said that they are redundant.

[0670] Thus, when data containing size information is stored as a data unit, and multiple such data units are aggregated and stored in a single payload, there is a problem of high overhead and poor transmission efficiency due to the duplication of size information.

[0671] Therefore, in a transmission device, data containing size information is stored as a data unit, and when multiple such data units are aggregated and stored in a single payload, it is conceivable to store them as shown in Figures 71 and 72.

[0672] As shown in Figure 71, it is conceivable to store the NAL unit, including the size area, as a data unit, and not indicate the data unit length that is conventionally included in the MMTP payload header. Figure 71 is a diagram showing the payload structure of an MMTP packet in which the data unit length is not indicated.

[0673] Furthermore, as shown in Figure 72, a flag indicating whether the data unit length is indicated, and information indicating the length of the size area may be newly stored in the header. The location where the flag and information indicating the length of the size area are stored may be indicated on a data unit basis, such as in the data unit header, or it may be indicated on a unit of aggregation of multiple data units (packet unit). Figure 72 shows an example of this being indicated in the extend area assigned to a packet unit. Note that the storage location of the newly indicated information is not limited to this, and may also be in the MMTP payload header, MMTP packet header, or control information.

[0674] On the receiving end, if a flag indicating whether the data unit length is compressed indicates that the data unit length is compressed, the length information of the size area inside the data unit is obtained, and based on the length information of the size area, the size area is obtained, and the data unit length can be calculated using the obtained length information of the size area and the size area.

[0675] By using the above method, the amount of data can be reduced on the sending side, thereby improving transmission efficiency.

[0676] Alternatively, overhead can be reduced by reducing the size area rather than reducing the data unit length. If the size area is reduced, information indicating whether the size area has been reduced, and information indicating the length of the data unit length field, may be stored.

[0677] The MMTP payload header also includes length information.

[0678] When storing NAL units that include the size area as data units, the payload size area in the MMTP payload header may be reduced, regardless of whether aggregation is performed or not.

[0679] Furthermore, even when storing data that does not include the size area as a data unit, if it is aggregated and the data unit length is indicated, the payload size area in the MMTP payload header may be reduced.

[0680] When reducing the payload size area, a flag indicating whether or not it was reduced, or length information of the reduced size field, or length information of the size field that was not reduced, may be indicated, as described above.

[0681] Figure 73 shows the operation flow of the receiving device.

[0682] As described above, the sending device stores the NAL unit, including the size area, as a data unit, and the data unit length included in the MMTP payload header is not indicated in the MMTP packet.

[0683] The following sections will explain, using flags to indicate whether the data unit length is shown, and examples of cases where the length information of the size field is shown in the MMTP packet.

[0684] The receiving device determines, based on the information transmitted from the sending side, whether the data unit includes a size area and whether the data unit length has been reduced (S1901).

[0685] If it is determined that the data unit length has been reduced (Yes in S1902), the length information of the size area inside the data unit is obtained, and then the size area inside the data unit is analyzed to calculate the data unit length (S1903).

[0686] On the other hand, if it is determined that the data unit length has not been reduced (No in S1902), the data unit length is calculated as usual from either the data unit length or the size area inside the data unit (S1904).

[0687] Note that if the receiving device already knows the flag indicating whether the data unit length has been reduced, or the length information of the size area, it does not need to be transmitted. In this case, the receiving device performs the processing shown in Figure 73 based on the predetermined information.

[0688] [Note: Transmitting device and receiving device] As described above, a transmitting device that performs rate control to satisfy the specifications of the receive buffer model during encoding can also be configured as shown in Figure 74. Furthermore, a receiving device that receives and decodes transmission packets sent from the transmitting device can also be configured as shown in Figure 75. Figure 74 shows an example of a specific configuration of a transmitting device. Figure 75 shows an example of a specific configuration of a receiving device.

[0689] The transmitting device 700 comprises a generation unit 701 and a transmitting unit 702. Each of the generation unit 701 and the transmitting unit 702 is implemented, for example, by a microcomputer, a processor, or a dedicated circuit.

[0690] The receiving device 800 comprises a receiving unit 801, a first buffer 802, a second buffer 803, a third buffer 804, a fourth buffer 805, and a decoding unit 806. Each of the receiving unit 801, the first buffer 802, the second buffer 803, the third buffer 804, the fourth buffer 805, and the decoding unit (decoder) 806 is implemented by, for example, a microcomputer, a processor, or a dedicated circuit.

[0691] Detailed descriptions of each component of the transmitting device 700 and the receiving device 800 will be provided in the descriptions of the transmitting method and the receiving method, respectively.

[0692] First, the transmission method will be explained using Figure 76. Figure 76 shows the operation flow (transmission method) by the transmitting device.

[0693] First, the generation unit 701 of the transmitting device 700 generates an encoded stream by performing rate control to satisfy the specifications of a predetermined receive buffer model in order to guarantee the buffer operation of the receiving device (S2001).

[0694] Next, the transmitting unit 702 of the transmitting device 700 packets the generated encoded stream and transmits the transmission packets obtained by packetization (S2002).

[0695] The receiving buffer model used in the transmitting device 700 is configured to include the first to fourth buffers 802 to 805 of the receiving device 800, so its explanation is omitted.

[0696] This allows the transmitting device 700 to guarantee the buffer operation of the receiving device 800 when transmitting data using a method such as MMT.

[0697] Next, the receiving method will be explained using Figure 77. Figure 77 shows the operation flow (receiving method) by the receiving device.

[0698] First, the receiving unit 801 of the receiving device 800 receives a transmission packet consisting of a fixed-length packet header and a variable-length payload (S2101).

[0699] Next, the first buffer 802 of the receiving device 800 converts the packet, which consists of a variable-length packet header and a variable-length payload stored in the received transmission packet, into a first packet having a fixed-length packet header with the header expanded, and outputs the first packet obtained by the conversion at a constant bit rate (S2102).

[0700] Next, the second buffer 803 of the receiving device 800 converts the first packet obtained by the conversion into a second packet consisting of a packet header and a variable-length payload, and outputs the second packet obtained by the conversion at a constant bit rate (S2103).

[0701] Next, the third buffer 804 of the receiving device 800 converts the output second packet into a NAL unit and outputs the resulting NAL unit at a constant bit rate (S2104).

[0702] Next, the fourth buffer 805 of the receiving device 800 sequentially stores the output NAL units, generates an access unit from the multiple stored NAL units, and outputs the generated access unit to the decoder at the timing of the decoding time corresponding to the access unit (S2105).

[0703] Then, the decoding unit 806 of the receiving device 800 decodes the access unit output by the fourth buffer (S2106).

[0704] This allows the receiving device 800 to perform decoding operations without underflow or overflow.

[0705] (Embodiment 6) [overview] Embodiment 6 describes a transmission method and a reception method when leap second adjustments are made to the reference time information of the reference clock in the MMT / TLV transmission method.

[0706] Figure 78 shows the protocol stack for the MMT / TLV scheme as defined in ARIB STD-B60.

[0707] In the MMT method, packets contain data such as video and audio, which are then processed by multiple MPUs (Media Processing Units). Data is stored in designated data units such as Presentation Units (Presentation Units) and Media Fragment Units (MFUs), and an MMTP packet header is added to generate an MMTP packet as a designated packet (MMTP packetization). Similarly, control information such as control messages in MMTP can also be converted into an MMTP packet by adding an MMTP packet header. The MMTP packet header includes a field for storing a 32-bit short-format NTP (Network Time Protocol: defined in IETF RFC 5905), which can be used for QoS control of communication lines, etc.

[0708] Furthermore, the transmitter's (transmitting device's) reference clock is synchronized to a 64-bit long-format NTP as defined in RFC 5905, and timestamps such as PTS (Presentation Time Stamp) and DTS (Decode Time Stamp) are added to the synchronization medium based on this synchronized reference clock. In addition, the transmitter sends the reference clock information to the receiver, and the receiver generates its own system clock based on the reference clock information received from the transmitter.

[0709] Specifically, PTS and DTS are stored in MPU timestamp descriptors and MPU extended timestamp descriptors, which are MMTP control information, and are stored in MP tables for each asset. After being packaged as MMTP packets as control messages, they are transmitted.

[0710] Data that has been packetized using MMTP is encapsulated into an IP packet with a UDP or IP header added. In this process, an IP data flow is defined as a collection of packets where the source IP address, destination IP address, source port number, destination port number, and protocol type are the same in the IP or UDP header. Note that because the headers of IP packets in the same IP data flow are redundant, some IP packets undergo header compression.

[0711] Furthermore, a 64-bit NTP timestamp is stored as reference clock information in the NTP packet and then in the IP packet. In this case, the source IP address, destination IP address, source port number, destination port number, and protocol type in the IP packet containing the NTP packet are fixed values, and the IP packet header is not compressed.

[0712] Figure 79 shows the structure of a TLV packet.

[0713] As shown in Figure 79, TLV packets can contain data such as IP packets, compressed IP packets, AMT (Address Map Table), and NIT (Network Information Table), and these data are identified using an 8-bit data type. In addition, the TLV packet indicates the data length (in bytes) using a 16-bit field, followed by the data value. The TLV packet also has 1 byte of header information before the data type, and this header information is stored in a header area totaling 4 bytes. Furthermore, the TLV packet is mapped to a transmission slot in the advanced BS transmission system, and the mapping information is stored in the TMCC (Transmission and Multiplexing Configuration Control) control information.

[0714] Figure 80 shows an example of a block diagram of a receiving device.

[0715] In the receiving device, the broadcast signal received by the tuner is first decoded by the demodulation means, the transmission path encoded data is corrected, and errors are applied to extract TLV packets. Then, the TLV / IP DEMUX means performs TLV demux processing and IP demux processing. TLV demux processing is performed according to the data type of the TLV packet. For example, if the TLV packet contains a compressed IP packet, the compressed header of the compressed IP packet is restored. IP demux processing performs processing such as header analysis of IP packets and UDP packets and extracts MMTP packets and NTP packets.

[0716] The NTP clock generation means reconstructs the NTP clock from the extracted NTP packets. The MMTP DEMUX performs filtering of components such as video and audio, as well as control information, based on the packet ID stored in the extracted MMTP packet header. The control information acquisition means retrieves timestamp descriptors stored in the MP table, and the PTS / DTS calculation means calculates the PTS and DTS for each access unit. The timestamp descriptors include both MPU timestamp descriptors and MPU extended timestamp descriptors.

[0717] The access unit playback means converts the video and audio filtered from the MMTP packets into data units to be presented. Specifically, these data units include the NAL unit and access unit of the video signal, audio frames, and subtitle presentation units. The decoding and presentation means decodes and presents the access unit at the time when the PTS / DTS of the access unit match, based on the reference time information of the NTP clock.

[0718] However, the configuration of the receiving device is not limited to this.

[0719] Next, we will explain timestamp descriptors.

[0720] Figure 81 is a diagram illustrating the timestamp descriptor.

[0721] PTS and DTS are stored in MMT control information, specifically in the MPU timestamp descriptor as first control information and the MPU extended timestamp descriptor as second control information. These are stored in the MP table for each asset, then packaged into MMTP packets as control messages and transmitted.

[0722] Figure 81(a) shows the configuration of an MPU timestamp descriptor as defined in ARIB STD-B60. For each of the multiple MPUs, the MPU timestamp descriptor stores the PTS (absolute value represented by a 64-bit NTP) of the first AU (hereinafter referred to as the "first AU") among the multiple AUs contained in that MPU, in the order of presentation. In other words, the presentation time information of the MPU assigned to it is stored in the control information of the MMTP packet and transmitted.

[0723] Figure 81(b) shows the configuration of the MPU extended timestamp descriptor. The MPU extended timestamp descriptor stores information for calculating the PTS and DTS of AUs contained within each of the multiple MPUs. The MPU extended timestamp descriptor contains relative information from the PTS of the first AU of the MPU stored in the MPU timestamp descriptor, and the PTS and DTS of AUs contained within the MPU can be calculated based on both the MPU timestamp descriptor and the MPU extended timestamp descriptor. In other words, the PTS and DTS of AUs other than the first AU contained within the MPU can be calculated based on the PTS of the first AU stored in the MPU timestamp descriptor and the relative information stored in the MPU extended timestamp descriptor.

[0724] NTP is a standard time information based on Coordinated Universal Time (UTC). UTC performs leap second adjustments (hereinafter referred to as "leap second adjustments") to correct the difference with astronomical time, which is based on the Earth's rotation speed. Specifically, leap second adjustments are performed at 9:00 AM Japan time and involve the insertion or deletion of one second.

[0725] Figure 82 is a diagram illustrating leap second adjustments.

[0726] Figure 82(a) shows an example of leap second insertion in Japan Standard Time. As shown in Figure 82(a), with leap second insertion, after 8:59:59 Japan Standard Time, the time becomes 8:59:59 at the point where it would normally be 9:00:00, and the 8:59:59 time range is repeated twice.

[0727] Figure 82(b) shows an example of leap second elimination in Japan Standard Time. As shown in Figure 82(b), with leap second elimination, after 8:59:58 Japan Standard Time, the time that would normally be 8:59:59 becomes 9:00:00, and one second in the 8:59:59 range is eliminated.

[0728] NTP packets contain a 64-bit timestamp as well as a 2-bit leap_indicator. The leap_indicator is a flag used to notify in advance that a leap second adjustment will be performed. A leap_indicator=1 indicates a leap second insertion, and a leap_indicator=2 indicates a leap second deletion. Advance notification can be initiated at the beginning of the month in which the leap second adjustment will be performed, 24 hours in advance, or at any other arbitrary time. The leap_indicator becomes 0 at the time the leap second adjustment is completed (9:00:00). For example, if advance notification is given 24 hours in advance, the leap_indicator will be shown as "1" or "2" from 9:00 on the day before the leap second adjustment in Japan Standard Time until just before the leap second adjustment is performed on the day of the adjustment (i.e., the time in the 8:59:59 range for the first leap second insertion, and the time in the 8:59:58 range for leap second deletion).

[0729] Next, I will explain the challenges involved in leap second adjustments.

[0730] Figure 83 shows the relationship between NTP time, MPU timestamp, and MPU presentation timing. The NTP time is the time indicated by NTP. The MPU timestamp is the timestamp indicating the PTS of the first AU in the MPU. The MPU presentation timing is the timing at which the receiving device should present the MPU according to the MPU timestamp. Specifically, Figures 83(a) to (c) show the relationship between NTP time, MPU timestamp, and MPU presentation time in the cases where no leap second adjustment occurs, a leap second is inserted, and a leap second is removed, respectively.

[0731] This explanation assumes that the transmitting NTP time (reference clock) is synchronized with the NTP server, and the receiving NTP time (system clock) is synchronized with the transmitting NTP time. In this case, the receiving device reconstructs the time based on the timestamp stored in the NTP packet transmitted from the transmitting side. Furthermore, since both the transmitting and receiving NTP times are synchronized with the NTP server in this case, a ±1 second adjustment is made when adjusting for leap seconds. Also, the NTP time in Figure 83 is assumed to be the same for both the transmitting and receiving NTP times. It is also assumed that there is no transmission delay.

[0732] The MPU timestamp in Figure 83 shows the timestamp of the first AU in the presentation order among the multiple AUs contained in each of the multiple MPUs, and is generated (set) based on the NTP time indicated by the arrow. Specifically, the MPU presentation time information is generated by adding a predetermined time (for example, 1.5 seconds in Figure 83) to the NTP time, which is the reference time information at the time the MPU presentation time information is generated. The generated MPU timestamp is stored in the MPU timestamp descriptor.

[0733] The receiving device presents the MPU at the MPU presentation time based on the timestamp stored in the MPU timestamp descriptor.

[0734] Note that in Figure 83, the playback time for one MPU is assumed to be 1 second, but the playback time for one MPU may be any other time, for example, 0.5 seconds or 0.1 seconds.

[0735] In the example shown in Figure 83(a), the receiving device can sequentially present MPUs #1-#5 based on the timestamps stored in the MPU timestamp descriptor.

[0736] However, in Figure 83(b), the presentation times of MPU#2 and MPU#3 overlap due to leap second insertion. Therefore, if the receiving device presents the MPU based on the timestamp stored in the MPU timestamp descriptor, there will be two MPUs presenting in the same 9:00:00 time range, making it impossible to determine which of the two MPUs to present. Furthermore, the receiving device will have two instances of the MPU presentation time (8:59:59) indicated by the timestamp of MPU#1, making it impossible to determine which of the two MPU presentation times to use for presentation.

[0737] Furthermore, in Figure 83(c), the receiving device is unable to present MPU#3 because, due to the removal of leap seconds, the MPU presentation time (8:59:59) indicated by the MPU timestamp of MPU#3 does not exist in NTP time.

[0738] The receiving device can solve the above problems if it performs MPU decoding and presentation processing that is not based on timestamps. However, it is difficult for a receiving device that performs processing based on timestamps to perform different processing (processing that is not based on timestamps) only when a leap second occurs.

[0739] Next, we will explain how to solve the problem of leap second adjustments by correcting the timestamp on the sending side.

[0740] Figure 84 illustrates a correction method for correcting timestamps on the sending side. Specifically, Figure 84(a) shows an example of leap second insertion, and Figure 84(b) shows an example of leap second deletion.

[0741] First, let's explain the case of leap second insertion.

[0742] As shown in Figure 84(a), when a leap second is inserted, the time up to immediately before the leap second insertion (i.e., up to the first 8:59:59 in NTP time) is designated as area A, and the time after the leap second insertion (i.e., from the second 8:59:59 in NTP time onward) is designated as area B. Areas A and B are temporal regions, representing time zones or periods. The MPU timestamp in Figure 84(a) is the same as the timestamp explained in Figure 83, and is a timestamp generated (set) based on the NTP time at the time the MPU timestamp is assigned.

[0743] This section will specifically explain how to correct the timestamp on the sending side when a leap second is inserted.

[0744] The transmitting device performs the following processing.

[0745] 1. If the timing for assigning the MPU timestamp (the timing indicated by the arrow in (a) of Figure 84) is included in area A, and the MPU timestamp (the value of the MPU timestamp before correction) is 9:00:00 or later, the MPU timestamp is corrected by subtracting 1 second, and the corrected MPU timestamp is stored in the MPU timestamp descriptor. In other words, if the MPU timestamp was generated based on the NTP time included in area A, and the MPU timestamp is 9:00:00 or later, the MPU timestamp is corrected by -1 second. Note that "9:00:00" here refers to the time when the leap second adjustment is based on Japan Standard Time (i.e., the time calculated by adding 9 hours to the UTC time). In addition, correction information, which indicates that a correction has been made, is sent separately to the receiving device.

[0746] 2. If the timing for assigning the MPU timestamp (the timing indicated by the arrow in Figure 84(a)) is in area B, the MPU timestamp will not be corrected. In other words, if the MPU timestamp was generated based on the NTP time included in area B, the MPU timestamp will not be corrected.

[0747] The receiving device presents the MPU based on the MPU timestamp and correction information indicating whether or not the MPU timestamp has been corrected (i.e., whether or not it contains information indicating that it has been corrected).

[0748] If the receiving device determines that the MPU timestamp has not been corrected (i.e., that it does not contain information indicating that it has been corrected), it will present the MPU at the time when the timestamp stored in the MPU timestamp descriptor matches the receiving device's NTP time (including both pre- and post-correction times). In other words, if the MPU timestamp stored in the MPU timestamp descriptor of an MPU sent before the corrected MPU, the receiving device will present the MPU at the time when that MPU timestamp matches the NTP time before the leap second insertion (i.e., before the first 8:59:59). If the MPU timestamp stored in the MPU timestamp descriptor of a received MPU is a timestamp sent after the corrected MPU timestamp, the receiving device will present the MPU at the time when it matches the NTP time after the leap second insertion (i.e., after the second 8:59:59).

[0749] Furthermore, if the MPU timestamp stored in the MPU timestamp descriptor of the received MPU has been corrected, the receiving device will present the MPU based on the NTP time after the leap second insertion (i.e., from the second 8:59:59 onwards) in the MPU timestamp descriptor.

[0750] Furthermore, information indicating that the MPU timestamp value has been corrected is stored and transmitted in control messages, descriptors, tables, MPU metadata, MF metadata, MMTP packet headers, etc.

[0751] Next, we will explain the case of removing leap seconds.

[0752] As shown in Figure 84(b), when a leap second is removed, the time up to the moment immediately before the leap second removal (i.e., just before 9:00:00 in NTP time) is designated as area C, and the time after the leap second removal (i.e., from 9:00:00 in NTP time) is designated as area D. Areas C and D are temporal areas, representing time zones or periods. The MPU timestamp in Figure 84(b) is the same as the timestamp explained in Figure 83, and is a timestamp generated (set) based on the NTP time at the time the MPU timestamp is assigned.

[0753] This section will specifically explain how to correct the timestamp on the sending side when a leap second is removed.

[0754] The transmitting device performs the following processing.

[0755] 1. If the timing for assigning the MPU timestamp (the timing indicated by the arrow in Figure 84(b)) is included in the C area, and the MPU timestamp (the value of the MPU timestamp before correction) is 8:59:59 or later, the MPU timestamp is corrected by adding 1 second, and the corrected MPU timestamp is stored in the MPU timestamp descriptor. In other words, if the MPU timestamp was generated based on the NTP time included in the C area, and the MPU timestamp is 8:59:59 or later, the MPU timestamp is corrected by adding 1 second. Note that "8:59:59" here is the time obtained by subtracting 1 second from the time that corresponds to the time on which leap second adjustments are made in Japan Standard Time (i.e., the time calculated by adding 9 hours to the UTC time). In addition, correction information, which indicates that a correction has been made, is sent to the receiving device. Note that in this case, the correction information does not necessarily have to be sent.

[0756] 2. If the timing for assigning the MPU timestamp (the timing indicated by the arrow in Figure 84(b)) is in the D region, the MPU timestamp will not be corrected. In other words, if the MPU timestamp was generated based on the NTP time included in the D region, the MPU timestamp will not be corrected.

[0757] The receiving device will present the MPU based on the MPU timestamp. If correction information is available indicating whether the MPU timestamp has been corrected, the MPU may be presented based on the MPU timestamp and the correction information.

[0758] Through the above process, even if leap second adjustments are made to the NTP time, the receiving device can use the MPU timestamp stored in the MPU timestamp descriptor to display a normal MPU.

[0759] Furthermore, the receiving end may be notified of whether the timing of the MPU timestamp assignment is in area A, area B, area C, or area D. In other words, the receiving end may be notified of whether the MPU timestamp was generated based on an NTP time included in area A, area B, area C, or area D. To put it another way, identification information indicating whether the MPU timestamp (presented time) was generated based on the reference time information (NTP time) before leap second adjustment may be transmitted. This identification information is assigned based on the leap_indicator included in the NTP packet, and therefore indicates whether the MPU timestamp is set based on a time between 9:00 the day before the leap second adjustment in Japan Standard Time and the time immediately before the leap second adjustment on the day the leap second adjustment is performed (i.e., the time in the 8:59:59 range for the first leap second insertion, and the time in the 8:59:58 range for leap second deletion). In other words, the identification information indicates whether the MPU timestamp was generated based on NTP time from a predetermined period (e.g., 24 hours) prior to the time immediately preceding the leap second adjustment.

[0760] Next, we will explain a method to solve the problems associated with leap second adjustments by correcting the timestamp in the receiving device.

[0761] Figure 85 is a diagram illustrating a correction method for correcting timestamps in a receiving device. Specifically, Figure 85(a) shows an example of leap second insertion, and Figure 85(b) shows an example of leap second deletion.

[0762] First, let's explain the case of leap second insertion.

[0763] As shown in Figure 85(a), similar to Figure 84(a), when a leap second is inserted, the time up to immediately before the leap second insertion (i.e., up to the first 8:59:59 in NTP time) is designated as area A, and the time after the leap second insertion (i.e., from the second 8:59:59 in NTP time onward) is designated as area B. Areas A and B are temporal regions, representing time zones or periods. The MPU timestamp in Figure 85(a) is the same as the timestamp explained in Figure 83, and is a timestamp generated (set) based on the NTP time at the time the MPU timestamp is assigned.

[0764] This section will specifically explain how the receiving device corrects the timestamp in the case of leap second insertion.

[0765] The transmitting device performs the following processing.

[0766] The generated MPU timestamp is stored in the MPU timestamp descriptor without correction and sent to the receiving device.

[0767] • The system sends identification information to the receiving device indicating whether the MPU timestamp was assigned in area A or area B. In other words, it sends identification information to the receiving device indicating whether the MPU timestamp was generated based on an NTP time contained in area A or an NTP time contained in area B.

[0768] The receiving device performs the following processing:

[0769] The receiving device corrects the MPU timestamp based on the MPU timestamp and identification information indicating whether the timing at which the MPU timestamp was assigned was in area A or area B.

[0770] Specifically, the following processes are performed.

[0771] 1. If the timing for assigning the MPU timestamp (the timing indicated by the arrow in Figure 85(a)) is included in area A, and the MPU timestamp (the value of the MPU timestamp before correction) is 9:00:00 or later, the MPU timestamp is corrected by subtracting 1 second. In other words, if the MPU timestamp was generated based on an NTP time included in area A, and the MPU timestamp is 9:00:00 or later, the MPU timestamp is corrected by subtracting 1 second. Note that "9:00:00" here refers to the time when the leap second adjustment is based on Japan Standard Time (i.e., the time calculated by adding 9 hours to UTC time).

[0772] 2. If the timing for assigning the MPU timestamp (the timing indicated by the arrow in Figure 85(a)) is in area B, the MPU timestamp will not be corrected. In other words, if the MPU timestamp was generated based on the NTP time included in area B, the MPU timestamp will not be corrected.

[0773] If the MPU timestamp is not corrected, the MPU will be presented at the time when the MPU timestamp stored in the MPU timestamp descriptor matches the NTP time of the receiving device (including both before and after correction).

[0774] In other words, if an MPU timestamp is received before the MPU timestamp to be corrected, the MPU will be presented at a time that matches the NTP time before the leap second insertion (i.e., before the first 8:59:59). If an MPU timestamp is received after the MPU timestamp to be corrected, the MPU will be presented at a time that matches the NTP time after the leap second insertion (after the second 8:59:59).

[0775] When correcting the MPU timestamp, the corrected MPU timestamp will be used by the receiving device to present the MPU based on the NTP time after the leap second insertion (i.e., from the second 8:59:59 onwards).

[0776] Furthermore, the sender stores identification information indicating whether the timing of the MPU timestamp assignment was in area A or area B in control messages, descriptors, tables, MPU metadata, MF metadata, MMTP packet headers, etc., and transmits it.

[0777] Next, we will explain the case of removing leap seconds.

[0778] As shown in Figure 85(b), when a leap second is removed, the time up to just before the leap second removal (i.e., just before 9:00:00 in NTP time) is designated as area C, and the time after the leap second removal (i.e., from 9:00:00 in NTP time onward) is designated as area D, similar to Figure 84(a). Areas C and D are temporal areas, representing time zones or periods. The MPU timestamp in Figure 85(b) is the same as the timestamp explained in Figure 83, and is a timestamp generated (set) based on the NTP time at the time the MPU timestamp is assigned.

[0779] This section specifically explains how the receiving device corrects the timestamp when a leap second is removed.

[0780] The transmitting device performs the following processing.

[0781] The generated MPU timestamp is stored in the MPU timestamp descriptor without correction and sent to the receiving device.

[0782] • The system sends identification information to the receiving device indicating whether the MPU timestamp was assigned in area C or area D. In other words, it sends identification information to the receiving device indicating whether the MPU timestamp was generated based on an NTP time contained in area C or an NTP time contained in area B.

[0783] The receiving device performs the following processing:

[0784] The receiving device corrects the MPU timestamp based on the MPU timestamp and identification information indicating whether the timing at which the MPU timestamp was assigned was in area C or area D.

[0785] Specifically, the following processes are performed.

[0786] 1. If the timing for assigning the MPU timestamp (the timing indicated by the arrow in Figure 85(b)) is included in the C region, and the MPU timestamp (the value of the MPU timestamp before correction) is 8:59:59 or later, then the MPU timestamp value is corrected by adding 1 second. In other words, if the MPU timestamp was generated based on the NTP time included in the C region, and the MPU timestamp is 8:59:59 or later, then the MPU timestamp is corrected by adding 1 second. Note that "8:59:59" here is the time obtained by subtracting 1 second from the time that corresponds to Japan Standard Time (i.e., the time calculated by adding 9 hours to UTC time), which is the reference time for leap second adjustments.

[0787] 2. If the timing for assigning the MPU timestamp (the timing indicated by the arrow in Figure 85(b)) is in the D region, the MPU timestamp will not be corrected. In other words, if the MPU timestamp was generated based on the NTP time included in the D region, the MPU timestamp will not be corrected.

[0788] The receiving device presents the MPU based on the MPU timestamp and the corrected MPU timestamp.

[0789] Through the above process, even if leap second adjustments are made to the NTP time, the receiving device can use the MPU timestamp stored in the MPU timestamp descriptor to display a normal MPU.

[0790] Even in this case, similar to the case where the MPU timestamp is corrected on the transmitting side as explained in Figure 84, the receiving side may be notified whether the timing of the MPU timestamp assignment is in area A, area B, area C, or area D. The details of the notification are the same, so the explanation will be omitted.

[0791] Furthermore, additional information (identification information), such as the information indicating whether the MPU timestamp has been corrected or the information indicating whether the MPU timestamp was added in area A, area B, area C, or area D, as explained in Figures 84 and 85, may be enabled when the leap_indicator of the NTP packet indicates the deletion (leap_indicator=2) or insertion (leap_indicator=1) of a leap second. This information may be enabled from a predetermined time (for example, 3 seconds before the leap second adjustment) or it may be enabled dynamically.

[0792] Furthermore, the timing of when the validity period of the additional information ends and when the signaling of the additional information ends can be set to coincide with the leap_indicator in the NTP packet, or it can be set to become effective from a predetermined arbitrary time (for example, 3 seconds before leap second adjustment), or it can be enabled dynamically.

[0793] Furthermore, since the 32-bit timestamp stored in MMTP packets and the timestamp information stored in TMCC are also generated and added based on NTP time, similar issues arise. For this reason, even in the case of 32-bit timestamps and timestamp information stored in TMCC, the timestamp can be corrected using the same method as in Figures 84 and 85, and the receiving device can perform processing based on the timestamp. For example, when storing the above additional information for a 32-bit timestamp stored in an MMTP packet, it may be indicated using the extended area of ​​the MMTP packet header. In this case, the extended type of the multi-header type indicates that it is additional information. Also, the time when leap_indicator is set may indicate the additional information using some bits of the 32-bit or 64-bit timestamp.

[0794] In the example shown in Figure 84, the corrected MPU timestamp is stored in the MPU timestamp descriptor. However, both the pre-correction and post-correction MPU timestamps (i.e., the uncorrected MPU timestamp and the corrected MPU timestamp) may be sent to the receiving device. For example, the pre-correction MPU timestamp descriptor and the post-correction MPU timestamp descriptor may be stored in the same MPU timestamp descriptor, or they may be stored in two separate MPU timestamp descriptors. In this case, the order in which the two MPU timestamp descriptors are placed, or the order in which they are written within the MPU timestamp descriptor, may be used to identify whether it is the pre-correction MPU timestamp or the post-correction MPU timestamp. Alternatively, the MPU timestamp descriptor may always store the pre-correction MPU timestamp, and when correction is performed, the post-correction timestamp may be stored in the MPU extended timestamp descriptor.

[0795] In this embodiment, the NTP time was explained using Japan Standard Time (9:00 AM) as an example, but it is not limited to Japan Standard Time. Leap second adjustments are based on UTC time and are corrected simultaneously worldwide. Japan Standard Time is 9 hours ahead of UTC time and is expressed as the value (+9) relative to UTC time. Thus, a time adjusted to a different time depending on the time difference depending on the location may be adopted.

[0796] As described above, by correcting the timestamp at the transmitting or receiving device based on information indicating the timing of timestamp assignment, normal reception processing using timestamps becomes possible.

[0797] Furthermore, while a receiving device that continues the decoding process well before the leap second adjustment time may be able to perform decoding and presentation processing without using a timestamp, a receiving device that selects a station immediately before the leap second adjustment time may not be able to determine a timestamp and may not be able to present the station until after the leap second adjustment is complete. Even in such cases, by using the correction method in this embodiment, it becomes possible to perform receiving processing using a timestamp, and station selection becomes possible even immediately before the leap second adjustment time.

[0798] Figure 86 shows the operation flow of the transmitting device when correcting the MPU timestamp on the transmitting side (transmitting device), as explained in Figure 84, and Figure 87 shows the operation flow of the receiving device.

[0799] First, the operation flow of the transmitting side (transmitting device) will be explained using Figure 86.

[0800] When leap seconds are inserted or removed, it is determined whether the timing of the MPU timestamp assignment is in area A, area B, area C, or area D (S2201). Cases where leap seconds are not inserted or removed are not shown.

[0801] Here, regions A through D are defined as follows:

[0802] Area A: Time up to just before the leap second insertion (up to the first 8:59:59 interval) Area B: Time after leap second insertion (after the second 8:59:59) Area C: Time up to just before the removal of leap seconds (up to 9:00:00) Area D: Time after the removal of leap seconds (after 9:00:00)

[0803] In step S2201, if it is determined to be area A and the MPU timestamp indicates 9:00:00 or later, the MPU timestamp is corrected by -1 second and the corrected MPU timestamp is stored in the MPU timestamp descriptor (S2202).

[0804] Then, correction information indicating that the MPU timestamp has been corrected is signaled and sent to the receiving device (S2203).

[0805] Furthermore, if it is determined in step S2201 that the area is C and the MPU timestamp indicates 8:59:59 or later, the MPU timestamp is corrected by +1 second and the corrected MPU timestamp is stored in the MPU timestamp descriptor (S2205).

[0806] Furthermore, if it is determined in step S2201 that the data is in area B or area D, the MPU timestamp is stored in the MPU timestamp descriptor without correction (S2204).

[0807] Next, the operation flow of the receiving device will be explained using Figure 87.

[0808] Based on the information signaled by the sender, it is determined whether or not the MPU timestamp has been corrected (S2301).

[0809] If it is determined that the MPU timestamp has been corrected (Yes in S2301), the receiving device presents the MPU based on the MPU timestamp in the NTP time after leap second adjustment has been performed (S2302).

[0810] If it is determined that the MPU timestamp has not been corrected (No in S2301), the MPU will be presented based on the MPU timestamp in NTP time.

[0811] When correcting an MPU timestamp, the MPU corresponding to that MPU timestamp is presented in the section where a leap second is inserted.

[0812] Conversely, if the MPU timestamp is not corrected, the MPU corresponding to that MPU timestamp will not be presented in the section where a leap second is inserted, but will be presented in the section where a leap second is not inserted.

[0813] Figure 88 shows the operation flow of the transmitting side when the MPU timestamp is corrected in the receiving device, as explained in Figure 85, and Figure 89 shows the operation flow of the receiving device.

[0814] First, the operation flow of the transmitting side (transmitting device) will be explained using Figure 88.

[0815] Based on the leap_indicator in the NTP packet, it is determined whether leap second adjustments (insertion or deletion) have been made (S2401).

[0816] If it is determined that a leap second adjustment will be made (Yes in S2401), the timing for assigning the MPU timestamp is determined, identification information is signaled, and transmitted to the receiving device (S2402).

[0817] On the other hand, if it is determined that no leap second adjustment is required (No in S2041), the process terminates in normal operation.

[0818] Next, the operation flow of the receiving device will be explained using Figure 89.

[0819] Based on the identification information signaled by the transmitting side (transmitting device), it is determined whether the timing of the MPU timestamp assignment is in area A, area B, area C, or area D (S2501). Here, areas A to D are the same as defined above, so their explanation is omitted. Note that, as with Figure 87, cases where no leap seconds are inserted or deleted are not shown.

[0820] If it is determined in step S2501 that the area is A, and the MPU timestamp indicates 9:00:00 or later, the MPU timestamp is corrected by -1 second (S2502).

[0821] If it is determined in step S2501 that the area is C, and the MPU timestamp indicates 8:59:59 or later, the MPU timestamp is corrected by +1 second (S2504).

[0822] If it is determined in step S2501 that the region is either area B or area D, the MPU timestamp is not corrected (S2503).

[0823] The receiving device further presents the MPU based on the corrected MPU timestamp in a process not shown.

[0824] When correcting MPU timestamps, the MPU corresponding to the MPU timestamp will be displayed based on the MPU timestamp at the NTP time after the leap second adjustment has been performed.

[0825] Conversely, if the MPU timestamp is not corrected, the MPU corresponding to that MPU timestamp will not be presented in the section where a leap second is inserted, but will be presented in the section where a leap second is not inserted.

[0826] In short, the transmitting device determines the timing for assigning an MPU timestamp to each of the multiple MPUs. If the determination shows that the timing is just before the insertion of a leap second and the MPU timestamp indicates 9:00:00 or later, the MPU timestamp is corrected by -1 second. The transmitting device also signals correction information indicating that the MPU timestamp has been corrected and sends it to the receiving device. If the determination shows that the timing is just before the deletion of a leap second and the MPU timestamp indicates 8:59:59 or later, the MPU timestamp is corrected by +1 second.

[0827] Furthermore, the receiving device, based on whether or not the MPU timestamp indicated by the correction information signaled by the transmitting device has been corrected, will present the MPU based on the MPU timestamp at the NTP time after the leap second adjustment if the MPU timestamp has been corrected. If the MPU timestamp has not been corrected, the receiving device will present the MPU based on the MPU timestamp at the NTP time before the leap second adjustment.

[0828] Furthermore, the transmitting side (transmitting device) determines the timing for assigning an MPU timestamp to each of the multiple MPUs and signals the result of the determination. The receiving device performs the following processing based on the information indicating the timing for assigning the MPU timestamp, which is signaled by the transmitting side. Specifically, if the information indicating the timing indicates a time just before the insertion of a leap second, and the MPU timestamp indicates 9:00:00 or later, the MPU timestamp is corrected by -1 second. Also, if the information indicating the timing indicates a time just before the deletion of a leap second, and the MPU timestamp indicates 8:59:59 or later, the MPU timestamp is corrected by +1 second.

[0829] If the MPU timestamp has been corrected, the receiving device will present the MPU based on the MPU timestamp at the NTP time after the leap second adjustment has been performed. If the MPU timestamp has not been corrected, the receiving device will present the MPU based on the MPU timestamp at the NTP time before the leap second adjustment has been performed.

[0830] In this way, by determining the timing of MPU timestamp assignment during leap second adjustment, it becomes possible to correct the MPU timestamp, allowing the receiving device to determine which MPU to present and perform appropriate reception processing using MPU timestamp descriptors and MPU extended timestamp descriptors. In other words, even if leap second adjustment is performed on NTP time, the receiving device can present a normal MPU using the MPU timestamp stored in the MPU timestamp descriptor.

[0831] [Note: Transmitting device and receiving device] As described above, the transmitting device that stores and transmits the data constituting the encoded stream on the MPU can also be configured as shown in Figure 90. Similarly, the receiving device that receives the MPU containing the data constituting the encoded stream can also be configured as shown in Figure 91. Figure 90 is a diagram showing a specific example of the configuration of the transmitting device. Figure 91 is a diagram showing a specific example of the configuration of the receiving device.

[0832] The transmitting device 900 comprises a generation unit 901 and a transmitting unit 902. Each of the generation unit 901 and the transmitting unit 902 is implemented, for example, by a microcomputer, a processor, or a dedicated circuit.

[0833] The receiving device 1000 comprises a receiving unit 1001 and a playback unit 1002. Each of the receiving unit 1001 and the playback unit 1002 is implemented, for example, by a microcomputer, a processor, or a dedicated circuit.

[0834] Detailed descriptions of each component of the transmitting device 900 and the receiving device 1000 will be provided in the descriptions of the transmitting method and the receiving method, respectively.

[0835] First, the transmission method will be explained using Figure 92. Figure 92 shows the operation flow (transmission method) by the transmitting device.

[0836] First, the generation unit 901 of the transmitting device 900 generates presentation time information (MPU timestamp) indicating the presentation time of the MPU as a predetermined data unit, based on the NTP time as reference time information received from an external source (S2601).

[0837] Next, the transmitting unit 902 of the transmitting device 900 transmits the MPU, the presentation time information generated by the generation unit 901, and identification information indicating whether or not the presentation time information (MPU timestamp) was generated based on the NTP time before leap second adjustment (S2602).

[0838] As a result, the receiving device that receives the information transmitted from the transmitting device 900 can regenerate the MPU based on the identification information, even if a leap second adjustment has been made, and thus can regenerate the MPU at the intended time.

[0839] Next, the receiving method will be explained using Figure 93. Figure 93 shows the operation flow (receiving method) by the receiving device.

[0840] First, the receiving unit 1001 of the receiving device 1000 receives the MPU, presentation time information indicating the presentation time of the MPU, and identification information indicating whether or not the presentation time information (MPU timestamp) was generated based on the NTP time before leap second adjustment (S2701).

[0841] Next, the playback unit 1002 of the receiving device 1000 plays back the MPU received by the receiving unit 1001 based on the presentation time information (MPU timestamp) and identification information received by the receiving unit 1001 (S2702).

[0842] This allows the receiving device 1000 to play the MPU at the intended time, even if leap second adjustments are made.

[0843] In Embodiment 6, "reproduction" refers to a process that includes at least one of the processes of decoding and presentation.

[0844] (Modified version of Embodiment 6) Next, we will explain specific examples of signaling methods for additional information (identification information) as described in Figures 84 and 85.

[0845] This section describes a signaling method using a combination of the MPU timestamp descriptor shown in Figure 81(a) and the MPU extended timestamp descriptor shown in Figure 81(b).

[0846] Figure 94 shows an example of an extended MPU timestamp descriptor. The underlined fields in Figure 94 are new fields added to the MPU extended timestamp descriptor shown in Figure 81(b).

[0847] In Figure 94, NTP_leap_indicator is a flag indicating whether or not to include additional information (identification information) related to NTP leap second adjustment in the MPU extended timestamp descriptor, which is the second control information. If this flag is set, the mpu_presentation_time_type becomes effective within the loop for each of the multiple MPUs.

[0848] In Figure 94, `mpu_presentation_time_type` indicates whether the `mpu_presentation_time` with the same sequence number recorded in the MPU timestamp stamp descriptor has been corrected when the MPU timestamp correction is performed on the transmitting side (transmitting device) as described in Figure 84. Furthermore, `mpu_presentation_time_type` indicates whether the timing at which the `mpu_presentation_time` with the same sequence number recorded in the MPU timestamp stamp descriptor is assigned is in area A, area B, area C, or area D when the MPU timestamp correction is performed on the receiving device as described in Figure 85.

[0849] In other words, the MPU extended timestamp descriptor stores identification information indicating whether the MPU timestamp was generated based on time information before leap second adjustments (NTP time). Furthermore, the MPU extended timestamp descriptor stores identification information corresponding to each MPU in each loop of multiple MPUs.

[0850] In step S2602 above, the transmitting unit 902 of the transmitting device 900 transmits the MPU, an MPU timestamp descriptor as first control information which stores the presentation time information (MPU timestamp) generated by the generation unit 901, and an MPU extended timestamp descriptor as second control information which stores identification information indicating whether or not the presentation time information was generated based on time information before leap second adjustment (NTP time).

[0851] In step S2701, the receiving unit 1001 of the receiving device 1000 receives the MPU, the MPU timestamp descriptor, and the MPU extended timestamp descriptor. Then, the playback unit 1002 of the receiving device 1000 plays back the MPU received by the receiving unit 1001 based on the MPU timestamp descriptor and the MPU extended timestamp descriptor received by the receiving unit 1001. In this way, the receiving device 1000 analyzes both the MPU timestamp descriptor and the MPU extended timestamp descriptor, enabling decoding using the MPU timestamp even during leap second adjustments. Therefore, the receiving device 1000 can play back the MPU at the intended time even when leap second adjustments are performed. Furthermore, since identification information is stored in the MPU extended timestamp descriptor, the function can be extended to signal whether the MPU timestamp was generated based on time information before leap second adjustments (NTP time), while maintaining the configuration of the MPU timestamp descriptor to be compatible with existing standards (MPEG). In this way, since the conventional configuration can be used, even when the signaling function is expanded, design changes can be minimized, and the manufacturing costs of the transmitting side (transmitter) and receiving device can be reduced.

[0852] The timing for setting the NTP_leap_indicator can be the same as the leap_indicator in the NTP packet, or it can be any time while the leap_indicator is set. In other words, the value of the NTP_leap_indicator is determined according to the value of the leap_indicator in the NTP packet corresponding to the NTP time on which the MPU timestamp was generated.

[0853] (Modification 2 of Embodiment 6) The problems related to leap second adjustment described in this embodiment are not limited to cases where NTP time is used, but also occur when UTC time is used.

[0854] If the time system is based on UTC, and the presented time information (MPU timestamp) is generated and assigned based on that time system, then by using the same method as previously described, the receiving device can correct and process the presented time information using the presented time information and signaling information (additional information, identification information) even during leap second adjustments.

[0855] For example, in ARIB STD-B60, when transmitting an event message to an application running on a receiver for the purpose of notifying the broadcasting station of the time (time specification information) that the broadcasting station will use, an event message descriptor is stored in the MMTP packet, and the time at which ...

Claims

1. A receiving method for receiving a predetermined data unit containing data that constitutes an application, (i) The predetermined data unit and (ii) Control information which stores time specification information indicating the operating time of the application and identification information indicating whether or not the time specification information is time information before leap second adjustment, Based on the received control information, the application stored in the received predetermined data unit is executed. The aforementioned time specification information is UTC (Universal Time Coordinated) time and NPT (Normal Play Time), The aforementioned control information is a UTC-NPT reference descriptor that shows the relationship between UTC time and NPT time. The reference time information for generating the aforementioned time specification information is NTP (Network Time Protocol). Reception method.

2. A receiving device that receives a predetermined data unit containing data that constitutes an application, (i) the predetermined data unit, and (ii) control information that stores time specification information indicating the operating time of the application and identification information indicating whether or not the time specification information is time information before leap second adjustment, The system comprises an execution unit that executes the application stored in the predetermined data unit received by the receiving unit based on the control information received by the receiving unit, The aforementioned time specification information is UTC (Universal Time Coordinated) time and NPT (Normal Play Time), The aforementioned control information is a UTC-NPT reference descriptor that shows the relationship between UTC time and NPT time. The reference time information for generating the aforementioned time specification information is NTP (Network Time Protocol). Receiving device.

Images