Receiving method and receiving device

Abstract

The delay time from when data is received until video and audio are reproduced in a receiving device is reduced. [Solution] The receiving method includes a video control information receiving step of receiving video control information for playing video data, a video data receiving step of receiving video data, an audio control information receiving step of receiving audio control information for playing audio data of a playback section corresponding to the playback section of the video data, and an audio data receiving step of receiving audio data, wherein the audio data includes a plurality of audio samples, the audio control information includes first time information indicating a first leading playback start time which is the playback start time of the audio sample to be played first among the plurality of audio samples, and the audio data is divided into a plurality of packets each including one of the plurality of audio samples and is included in a packet different from the audio control information.

Description

[Technical Field]

[0001] The present invention relates to a transmission method, a reception method, a transmission device, and a reception device. [Background technology]

[0002] Currently, it is being considered to segment coded data such as audio and video using formats such as MPEG-DASH (Dynamic Adaptive Streaming over HTTP) or MMT (MPEG Media Transport), divide the resulting segments into packets, and transmit the resulting packets via broadcasting or communications.

[0003] For example, Non-Patent Document 1 discloses a technique for transmitting encoded media data packet by packet in accordance with MMT. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Information technology - High efficiency coding and media delivery in heterogeneous environment - Part1:MPEG media transport(MMT), ISO / IEC DIS 23008-1 Summary of the Invention [Problem to be solved by the invention]

[0005] In such a transmission method and a reception method, it is desirable to reduce the delay time from when the data is received until the video and audio are reproduced in the receiving device.

[0006] Therefore, an object of the present invention is to provide a transmission method or a reception method that can reduce the delay time from when data is received until video and audio are reproduced in a receiving device. [Means for solving the problem]

[0007] In order to achieve the above object, a receiving method according to one aspect of the present invention includes a video control information receiving step of receiving video control information for playing video data, a video data receiving step of receiving the video data, an audio control information receiving step of receiving audio control information for playing audio data of a playback section corresponding to a playback section of the video data, and an audio data receiving step of receiving the audio data, wherein the audio data includes a plurality of audio samples, the audio control information includes first time information indicating a first leading playback start time which is the playback start time of an audio sample to be played first among the plurality of audio samples, and the audio data is divided into a plurality of packets each including one of the plurality of audio samples. , is included in a packet different from the audio control information, each of the plurality of packets includes sequence information indicating the ordinal number of the packet among the plurality of packets, the video data includes a plurality of video samples, and the video control information includes second time information indicating a second leading playback start time which is the playback start time of the video sample that is to be played first among the plurality of video samples, and the receiving method further includes a playback time acquisition step of acquiring the playback time of each of the plurality of audio samples, and a generation step of generating the playback start time of the audio sample included in the packet to be processed using the first leading playback start time, the playback time, and the sequence information included in the packet to be processed.

[0008] These general or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium. [Effects of the Invention]

[0009] As described above, the present invention can provide a transmission method or a reception method that can reduce the delay time from when data is received until video and audio are reproduced in a receiving device. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of the structure of an MP4 file. [Figure 2] FIG. 10 is a diagram illustrating an example of the configuration of MMT data. [Figure 3] FIG. 10 shows audio samples in MMT data where decoding can begin. [Figure 4] FIG. 1 shows audio samples in MPEG-2 TS where decoding can begin. [Figure 5] FIG. 3 is a diagram illustrating an example of a configuration of transmission data according to the first embodiment. [Figure 6] FIG. 3 is a diagram showing a data transmission order according to the first embodiment. [Figure 7] 1 is a block diagram of a transmission device according to a first embodiment. [Figure 8] 4 is a flowchart of a transmission process according to the first embodiment. [Figure 9] 4 is a flowchart of a transmission process according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a configuration of transmission data according to the second embodiment. [Figure 11] FIG. 10 is a block diagram of a transmitting device according to a second embodiment. [Figure 12] 10 is a flowchart of a transmission process according to the second embodiment. [Figure 13] FIG. 11 is a diagram illustrating an example of the configuration of transmission data according to the third embodiment. [Figure 14] FIG. 11 is a diagram illustrating an example of the configuration of transmission data according to the third embodiment. [Figure 15] FIG. 11 is a block diagram of a transmitting device according to a third embodiment. [Figure 16] 11 is a flowchart of a transmission process according to the third embodiment. [Figure 17] FIG. 10 is a block diagram of a receiving device according to a fourth embodiment. [Figure 18] 11 is a flowchart of a reception process according to the fourth embodiment. [Figure 19] FIG. 11 is a block diagram of a receiving device according to a fifth embodiment. [Figure 20] 13 is a flowchart of a reception process according to the fifth embodiment. [Figure 21] 13 is a flowchart of another example of the reception process according to the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] (Findings that form the basis of the present invention) Each segment consists of a header and a storage section for encoded data. The receiving device separates access units (called samples in MP4-based formats such as DASH and MMT) from the encoded data. The receiving device must also analyze the header to obtain the PTS (Presentation Time Stamp) and DTS (Decoding Time Stamp) of the access unit.

[0012] In conventional broadcasting, video and audio are packetized for each access unit and multiplexed into an MPEG-2 Transport Stream (TS) before being transmitted. This allows a receiving device to separate the access units and obtain the PTS and DTS for each packet.

[0013] On the other hand, when segments are packetized and transmitted, such as in DASH and MMT, the receiving device must acquire the header of the segment even during random access. Therefore, if the receiving device starts receiving a signal from the middle of a segment, the receiving device cannot decode the access unit of that segment. Therefore, the receiving device can only decode from the next segment. Thus, the present inventor has discovered a problem in that random access increases the delay time until decoding begins.

[0014] Hereinafter, with reference to FIGS. 1 to 4, a description will be given of problems that arise during random access in a conventional method of packet multiplexing MPUs (Media Processing Units) that constitute MMT.

[0015] First, a method for storing video and audio data in the conventional MP4 format and an example of data storage in MMT, a transmission format based on MP4, will be described.

[0016] Figure 1 shows an example of data storage in MP4. As shown in Figure 1, an MP4 file includes multiple MP4 fragments. Each MP4 fragment includes a pair of moof and mdat. mdat includes multiple video samples and multiple audio samples that are arranged alternately.

[0017] Generally, MP4 fragments are used as random access points and store samples that make up a video random access unit (RAU). Audio can be decoded from any sample, but samples corresponding to the playback section of the video random access unit are stored.

[0018] For example, if the playback section of a video in a random access unit is one second from 10 seconds to 11 seconds, the corresponding audio sample is stored as a sample for one second from 10 seconds to 11 seconds.

[0019] Depending on the video frame rate and audio sampling frequency, the playback intervals of the two may not exactly match. In this case, for example, the audio sample with the PTS immediately preceding the PTS of the first sample of the video playback interval to the audio sample with the playback end time immediately preceding the playback end time of the last sample of the video playback interval is selected.

[0020] Fig. 2 is a diagram showing an example of how data is stored in an MPU in MMT. As in the case of Fig. 1, MPUs are generated so that the playback intervals of the video and audio MPUs are equal.

[0021] Decoding video and audio samples requires parsing the header (moof, etc.) of the MPU to which the sample belongs.

[0022] Figure 3 is a diagram showing a problem with random access when MMT data such as that shown in Figure 2 is multiplexed into MMT packets and transmitted. Figure 3 shows the arrangement (transmission order) of video and audio data stored in MMT packets and transmitted. Here, for video, the RAU starts from video sample V4. In other words, video can be decoded from video sample V4.

[0023] In this example, the Nth audio MPU #N starts from audio sample A1, and MPU #N+1 starts from audio sample A10. The audio sample with the PTS immediately following the PTS of video sample V4 is audio sample A2.

[0024] In this case, if the receiving device starts receiving from video sample V4, it cannot obtain the header of audio MPU #N and therefore cannot decode audio samples A2, A3, etc. contained in that MPU #N. Therefore, the receiving device can only decode audio from audio sample A10, which is the first sample of MPU #N+1.

[0025] On the other hand, in conventional TS that uses MPEG-2 TS (Transport Stream) as the transmission format, a receiving device can decode from the audio access unit that starts immediately after the video random access point. For example, as shown in Figure 4, when data is received in the same order as in Figure 3, the receiving device can decode audio from audio sample A2.

[0026] As described above, when an MMT stream in which video and audio are multiplexed is transmitted via broadcasting or communication, there is a problem in that there may be a long delay time between when the receiving device starts decoding from the video random access point and when it can start playing the audio.

[0027] Here, with regard to video, there is a restriction that playback can only be started in RAU units. Therefore, with regard to video data, it is inevitable that there will be a certain amount of delay time before the image is displayed. On the other hand, although audio can be played back from any sample, in the above case, a delay time occurs before playback can start. Thus, the present inventors have discovered a problem in that even though a receiving device receives playable data, it cannot play the data because it is unable to receive control information for the data.

[0028] A transmission method according to one aspect of the present invention includes a video control information transmission step of transmitting video control information for playing individually playable units of video data, a video data transmission step of transmitting the video data, an audio control information transmission step of determining the transmission order of audio control information for playing audio data of a playback section corresponding to the playback section of the video data after the video control information and transmitting the audio control information in the determined transmission order, and an audio data transmission step of transmitting the audio data.

[0029] According to this transmission method, audio control information for audio data is transmitted after video control information for video data corresponding to the audio data. This allows the receiving device to start decoding audio at the same time as it can start decoding video during random access. This reduces the delay time until the receiving device starts decoding audio data. Furthermore, overhead can be reduced because there is no need to subdivide audio segments.

[0030] For example, the audio data may include a plurality of audio samples each of which can be played back individually, and the audio control information may include time information indicating the playback start time of each of the plurality of audio samples.

[0031] For example, the playback section of the audio data may be approximately equal to the playback section of the video data.

[0032] For example, in the audio data transmission step, the audio data may be divided into a plurality of packets, each of which contains one of the plurality of audio samples, and transmitted, and each of the plurality of packets may contain sequence information indicating the number of the packet among the plurality of packets.

[0033] This allows the receiving device to calculate the playback start time of each audio sample and play the audio sample using the calculated playback start time even when it cannot receive audio control information, thereby reducing the delay time until the receiving device starts decoding the audio data.

[0034] For example, the video data may be one of a plurality of video data obtained by dividing a video signal, the audio data may be one of a plurality of audio data obtained by dividing an audio signal, and the playback time of the audio data may be shorter than the playback time of the video data.

[0035] This reduces the delay time until the receiving device starts decoding the audio data.

[0036] For example, the plurality of audio data may include a plurality of audio data whose playback start times are approximately the same as the playback start times of each of the plurality of video data.

[0037] A receiving method according to an aspect of the present invention receives the video control information, the video data, the audio control information, and the audio data transmitted by the transmitting method.

[0038] This allows the receiving device to start decoding audio at the same time that it can start decoding video during random access, thereby reducing the delay time until the receiving device starts decoding audio data.

[0039] In addition, a receiving method according to one aspect of the present invention is a receiving method for receiving the video control information, the video data, the audio control information, and the audio data transmitted by the transmission method, and includes a playback start time acquisition step for acquiring a first playback start time, which is the playback start time of the audio sample that is to be played first among the plurality of audio samples, a playback time acquisition step for acquiring the playback time of each of the plurality of audio samples, and a determination step for determining the playback start time of the audio sample included in the packet to be processed using the first playback start time, the playback time, and the order information included in the packet to be processed.

[0040] This allows the receiving device to calculate the playback start time of each audio sample and play the audio sample using the calculated playback start time even when it cannot receive audio control information, thereby reducing the delay time until the receiving device starts decoding the audio data.

[0041] In addition, a transmitting device according to one embodiment of the present invention includes a video control information transmitting unit that transmits video control information for playing video data in units that can be played back individually, a video data transmitting unit that transmits the video data, an audio control information transmitting unit that determines the transmission order of audio control information for playing audio data in a playback section corresponding to the playback section of the video data after the video control information and transmits the audio control information in the determined transmission order, and an audio data transmitting unit that transmits the audio data.

[0042] According to this, the transmitting device transmits audio control information for audio data after video control information for the video data corresponding to the audio data. This allows the receiving device to start decoding audio at the same time as it can start decoding video during random access. This reduces the delay time until the receiving device starts decoding audio data. Furthermore, overhead can be reduced because there is no need to subdivide audio segments.

[0043] A receiving device according to an aspect of the present invention receives the video control information, the video data, the audio control information, and the audio data transmitted by the transmitting device.

[0044] This allows the receiving device to start decoding audio at the same time that it can start decoding video during random access, thereby reducing the delay time until the receiving device starts decoding audio data.

[0045] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0046] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0047] It should be noted that the embodiments described below each illustrate a specific example of the present invention. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present invention. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept are described as optional components.

[0048] (Embodiment 1) A transmission method and a transmission device according to this embodiment will be described below. The transmission method according to this embodiment transmits an MMT packet storing the MPU header of an audio MPU having the same playback section as the video MPU, after an MMT packet storing the MPU header of the video MPU.

[0049] This allows the receiving device to start decoding audio at the same time as it can start decoding video during random access, thereby reducing the delay time until the receiving device starts decoding audio.In addition, since audio segments are not subdivided, overhead can also be reduced.

[0050] First, the data structure in the transmission method according to this embodiment will be described.

[0051] Fig. 5 is a diagram showing the configuration of an MPU according to this embodiment. As shown in Fig. 5, MMT data includes a plurality of video MPUs (V_MPUs) and a plurality of audio MPUs (A_MPUs).

[0052] Each MPU (video MPU and audio MPU) includes an MPU header and an MPU payload. The MPU header includes MPU metadata (ftyp, mmpu, moov, etc.) and FRG metadata (headers for each movie fragment, such as moof and mdat). The MPU header also includes information indicating at least one of the PTS and DTS for each of the multiple samples included in the MPU metadata.

[0053] In the following, the MPU header of a video MPU is referred to as the video MPU header, and the MPU payload of a video MPU is referred to as the video MPU payload. Also, the MPU header of an audio MPU is referred to as the audio MPU header, and the MPU payload of an audio MPU is referred to as the audio MPU payload.

[0054] For example, each of the multiple audio MPUs corresponds to one of the multiple video MPUs. In Figure 5, the corresponding video MPU and audio MPU are represented as V_MPU#n (n is a natural number) and A_MPU#n. Furthermore, the corresponding audio MPU and video MPU have, for example, the same playback section.

[0055] Depending on the video frame rate or audio sampling frequency, the playback intervals of the two may not exactly match. In this case, for example, the audio MPU corresponding to the video MPU may be selected from the audio sample having a PTS immediately before the PTS of the first sample in the playback interval of the video MPU to the audio sample having a playback end time immediately before the playback end time of the last sample in the playback interval of the video MPU. Note that the audio MPU corresponding to the video MPU may also be selected from the sample having a PTS immediately after the PTS of the first sample in the playback interval of the video MPU to the sample having a playback end time immediately after the playback end time of the last sample in the playback interval of the video MPU.

[0056] That is, the corresponding video MPU and audio MPU have approximately the same playback period. In other words, the corresponding video MPU and audio MPU have at least a part of their playback periods overlap.

[0057] FIG. 6 is a diagram showing an example of the transmission order (multiplexing order) of MTT packets in this embodiment.

[0058] The video MPU payload includes multiple video samples (access units) V1 to V8. The video MPU is divided into multiple MMT packets (hereinafter also referred to as video MMT packets). The multiple video MMT packets include a packet (V_MPU(n)_H) that stores a video MPU header and multiple packets (V_MPU(n)_P) that store multiple video samples included in the video MPU payload. The multiple packets (V_MPU(n)_P) store multiple divided video samples (V1 to V8).

[0059] The audio MPU payload includes multiple audio samples (access units) A1 to A10. The audio MPU is divided into multiple MMT packets (hereinafter also referred to as audio MMT packets). The multiple audio MMT packets include a packet (A_MPU(n)_H) that stores an audio MPU header and multiple packets (A_MPU(n)_P) that store multiple audio samples included in the audio MPU payload. The multiple packets (A_MPU(n)_P) store the multiple divided audio samples (A1 to A10).

[0060] In this embodiment, as shown in Figure 6, an MMT packet V_MPU(n)_H storing the MPU header of the audio MPU corresponding to the video MPU is transmitted after an MMT packet V_MPU(n)_H storing the MPU header of the video MPU.

[0061] This allows the receiving device to start decoding audio at the same time as it can start decoding video during random access, thereby reducing the delay time until the receiving device starts decoding audio.In addition, since audio segments are not subdivided, overhead can also be reduced.

[0062] In video encoding, the amount of code of the first access unit (sample in MP4) of an RAU is generally larger than that of subsequent access units. On the other hand, the variation in the amount of code of audio access units is smaller than that of video. For this reason, particularly when a packet sequence in which audio and video encoded data are multiplexed is transmitted at a fixed rate, the transmission of the first access unit of a video RAU begins earlier to ensure that all data of the access unit has been received by the decoding time. Therefore, even if a system decoder model such as STD (System Target Decoder) is specified in MMT as in MPEG-2 TS (Transport Stream), an MMT packet storing the nth audio MPU header can be transmitted after an MMT packet storing the nth video MPU header, and the system decoder model can be satisfied.

[0063] The configuration and operation of the transmitting device according to this embodiment will be described below.

[0064] Fig. 7 is a block diagram of a transmitting device 100 according to this embodiment. The transmitting device 100 shown in Fig. 7 includes an MPU generation unit 101 and a packet multiplexing unit 102. The packet multiplexing unit 102 includes a video header determination unit 111, an audio header determination unit 112, and a multiplexing unit 113.

[0065] FIG. 8 is a flowchart of the transmission process (multiplexing process) according to this embodiment.

[0066] The MPU generation unit 101 divides the input video signal and audio signal to generate a plurality of video MPUs and a plurality of audio MPUs.

[0067] Next, the packet multiplexing unit 102 multiplexes the video MPU and audio MPU.

[0068] Specifically, first, the video header determination unit 111 determines whether the transmission order (VH_LOC) of the MMT packet storing the MPU header of V_MPU(n), which is the n-th video MPU, has been determined (S101).

[0069] If VH_LOC has been determined (Yes in S101), the audio header determination unit 112 determines the transmission order so that the MMT packet storing the MPU header of A_MPU(n), the nth audio MPU, is transmitted after VH_LOC (S102).

[0070] Next, multiplexing unit 113 stores the audio MPU and video MPU data in MMT packets and multiplexes the packets to generate a transmission stream (S103). At this time, multiplexing unit 113 transmits the MMT packets storing the audio MPU headers in the transmission order determined in step S102. In other words, multiplexing unit 113 transmits the MMT packet storing the MPU header of A_MPU(n), the nth audio MPU, after the MMT packet storing the MPU header of V_MPU(n), the nth video MPU.

[0071] On the other hand, if VH_LOC has not been determined (No in S101), the multiplexing unit 113 stores the data of the audio MPU and the video MPU in MMT packets and multiplexes the packets to generate a transmission stream (S103).

[0072] Furthermore, the above series of processes is repeated until all MMT packets are transmitted (S104).

[0073] Note that, here, transmitting device 100 determines the transmission order of the MMT packets of the audio MPU headers based on the MMT packet of the video MPU header so that the transmission order of the MMT packets of the audio MPU headers comes after the MMT packet of the video MPU header, but may also determine the transmission order of the MMT packets of the video MPU headers based on the MMT packet of the audio MPU header so that the transmission order of the MMT packets of the video MPU headers comes before the MMT packet of the audio MPU header. In other words, multiplexing unit 113 may transmit an MMT packet storing the MPU header of V_MPU(n), the nth video MPU, before an MMT packet storing the MPU header of A_MPU(n), the nth audio MPU.

[0074] In the above description, transmitting device 100 multiplexes the MPU header and MPU payload included in each MPU into a single stream and transmits the multiplexed stream, but it is also possible to transmit the MPU header (or part of the information included in the MPU header) as a signal separate from the stream into which the MPU payload is multiplexed. Even in this case, transmitting device 100 transmits the audio MPU header after the video MPU header corresponding to the audio MPU header in the separate signal.

[0075] Furthermore, even when transmitting only an MPU payload without transmitting an MPU header, the transmitting device 100 may align the playback sections of the audio MPU and the video MPU. This allows the index numbers of the video MPU and the audio MPU that are played back synchronously to be the same. In MMT transmission, the absolute values ​​of the PTS and DTS for each MPU are provided to the receiving device. Therefore, since the index numbers of the video MPU and the audio MPU that are played back synchronously are the same, the amount of processing required for searching for an audio access unit that has a PTS corresponding to the PTS of the video access unit that starts playback can be reduced in the receiving device.

[0076] As described above, transmitting apparatus 100 according to the present embodiment performs the processing shown in FIG.

[0077] First, the transmitting device 100 transmits video control information (video MPU header) for playing back video data (video MPU) in units that can be played back individually (RAU units) (S121).

[0078] Next, the transmitting device 100 transmits the video data (video MPU) in the individually playable units (RAU units) (S122).

[0079] Next, the transmitting device 100 transmits audio control information (audio MPU header) for reproducing audio data (audio MPU) of a playback section corresponding to the playback section of the video data after transmitting the video control information (S123). That is, the transmitting device 100 determines the transmission order of the audio control information (audio MPU header) for reproducing audio data (audio MPU) of a playback section corresponding to the playback section of the video data to be after the video control information, and transmits the audio control information in the determined transmission order. In other words, the transmitting device 100 determines the transmission order of the audio control information so that the transmission order of the audio control information does not come before the video control information, and transmits the audio control information in the determined transmission order. As a result, the audio control information is always transmitted after the video control information. In other words, the transmitting device 100 prohibits the audio control information from being transmitted before the video control information.

[0080] Next, the transmitting device 100 transmits the audio data (S124).

[0081] For example, the audio data includes a plurality of audio samples that can be individually played back, and the audio control information includes time stamps (PTSs) indicating the playback start times of the respective audio samples. The playback duration of the audio data is approximately equal to the playback duration of the video data.

[0082] As described above, when transmitting segments by packet multiplexing, the transmitting device 100 according to this embodiment transmits the header of an audio segment (MPU) after the header of a video segment whose starting PTS matches that of the audio segment. This allows the receiving device to start audio decoding at the same time as video decoding can be started during random access. This reduces the delay time from the start of video decoding to the start of audio decoding. Furthermore, since there is no need to subdivide audio segments, overhead can also be reduced.

[0083] (Embodiment 2) In this embodiment, the transmitting device generates a stream that allows the receiving device to determine the PTS of the audio sample based on information included in the audio sample data, etc. This allows the receiving device to play the audio sample even if it cannot obtain the audio MPU header.

[0084] For example, by using the method of this embodiment, even in the case shown in FIG. 3, the receiving device can start decoding from audio sample A2.

[0085] FIG. 10 is a diagram showing the configuration of an audio MPU according to this embodiment.

[0086] 10, one audio sample (any of A1 to A8) is stored in each of multiple MMT packets (A_MPU(n)_P1 to AMPU(n)_P8) that store audio samples. In other words, multiple MMT packets correspond one-to-one to multiple audio samples.

[0087] Each MMT packet includes a header and a payload in which audio sample data is stored. The header includes a fragment SN. The fragment SN is an index number of the unit (fragment) into which the MPU is divided. In other words, the fragment SN indicates the ordinal number of the fragment (packet) in the MPU. In this embodiment, one audio sample is stored in one fragment (packet), so the fragment SN indicates the decoding order of the audio samples included in the fragment in the MPU.

[0088] Note that the MMT packets that store the audio samples do not include an MPU header.

[0089] Furthermore, the PTS (Presentation Time Stamp) of the first sample in the audio MPU that starts decoding is separately transmitted to the receiving device by an MMT message or the like.

[0090] For example, when decoding is started after tuning in to a broadcast channel, the PTS of the first sample of the MPU transmitted within a certain period after tuning in is indicated in the PA message of the MMT acquired at the time of tuning.

[0091] The sample data also includes information indicating the sampling frequency. For example, in the ADTS (Audio Data Transport Stream) format of AAC in MPEG-2 and MPEG-4, the ADTS header includes information indicating the sampling frequency and channel configuration.

[0092] Furthermore, when storing AAC ADTS and LATM (Low-overhead MPEG-4 Audio Transport Multiplex) in MP4, header information such as an ADTS header may be removed from the sample data and stored in the moov stsd. In this case, information indicating whether the AAC sample data includes header information may be stored in the MP4 file brand or a separately defined box, or whether the AAC sample data includes header information may be predetermined in operational regulations for broadcasting, etc. Alternatively, information indicating whether the sample data includes header information may be included in MMT SI information, etc. Furthermore, when the multiplexing method is MMT, specifying ADTS or LATM as the stream type may indicate that the sample data includes header information. If header information is included in the sample data, a receiving device can decode and play audio based on the header information even if an MPU header is not transmitted or cannot be obtained.

[0093] If the header information is not included in the sample data, the receiving device must obtain the playback time of the sample in advance. If the header information is not included in the sample data, the receiving device may start decoding from an MPU from which the MPU header can be obtained, without using the method described below.

[0094] The receiving device calculates the PTS of the audio sample using the PTS of the first sample of the MPU, the playback time of one audio sample, and the fragment SN.

[0095] Specifically, the receiving device determines the PTS of any audio sample using the following (Equation 1).

[0096] sample(i).pts=MPU(n).pts+(num_spl / sampling_rate)*frg_sn ···(Formula 1)

[0097] Here, sample(i).pts is the PTS of the ith sample in the nth MPU, and MPU(n).pts is the PTS of the first sample in display order in the nth MPU.

[0098] num_spl is the number of samples contained in an audio frame. Note that the samples in this case are different from the samples in MP4. In the case of MPEG-2 or MPEG-4 AAC (Advanced Audio Coding), the number of samples per frame is 1024.

[0099] sampling_rate is the sampling frequency of the audio, and frg_sn is the fragment SN.

[0100] That is, the receiving device calculates the playback time of one audio sample by dividing the number of samples included in an audio frame by the sampling frequency.The receiving device calculates the PTS of an audio sample by adding the product of the playback time of one audio sample and the fragment SN (the index number of the sample within the MPU) to the PTS of the first sample in the MPU.

[0101] If the playback time of the audio sample can be obtained by a method other than the above, the receiving device may use the other method.

[0102] For example, if the playback time of a sample is constant, such as if the audio sampling frequency is constant, the receiving device may determine the PTS of the audio sample based on the playback time obtained in advance.

[0103] Also, information indicating the sampling frequency or the playback time of the samples may be transmitted by an MMT message, etc. Furthermore, when MMT packets are transmitted directly or after being multiplexed into IP packets and stored in TS packets, information indicating the sampling frequency or the playback time of the samples may be transmitted by an MPEG-2 system descriptor, etc.

[0104] This method can also be applied to video if the frame rate is known or can be obtained separately using an MMT message or the like.

[0105] Furthermore, in MPEG-4 AVC (Advanced Video Coding) and MPEG HEVC (High Efficiency Video Coding), sample data must include initialization information for decoding, such as SPS (Sequence Parameter Set) and PPS (Picture Parameter Set). Whether the SPS and PPS are included in a sample can be identified by the MP4 brand name or sample entry type, so this information may be transmitted separately.

[0106] As a result, even if the receiving device cannot receive the audio MPU header, it can obtain the PTS of any audio sample in the MPU. This reduces the delay until audio is played back without imposing restrictions on the transmission order of the video MPU header and the audio MPU header.

[0107] The configuration and operation of the transmitting device according to this embodiment will be described below.

[0108] 11 is a block diagram of transmitting apparatus 200 according to this embodiment. Transmitting apparatus 200 shown in FIG. 11 includes a video MPU generation unit 201, an audio MPU generation unit 202, an audio packetization unit 203, and a video packetization unit 204.

[0109] FIG. 12 is a flowchart of the transmission process (multiplexing process) according to this embodiment.

[0110] First, the video MPU generation unit 201 determines the playback section of V_MPU(n), which is the nth video MPU (S201). Then, the video packetization unit 204 packetizes the determined V_MPU(n) into multiple MMT packets. At this time, the video packetization unit 204 generates V_MPU(n) so that the first sample of the V_MPU(n) becomes the random access point. In other words, V_MPU(n) is made up of one or more video random access units.

[0111] Next, the audio MPU generation unit 202 determines the playback section of A_MPU(n), which is the n-th audio MPU, so that the playback section coincides with the playback section of V_MPU(n) (S202).

[0112] Next, the audio packetizer 203 generates a plurality of MMT packets so that each sample included in the A_MPU(n) determined in step S202 is stored in one MMT packet (S203).

[0113] Then, transmitting device 200 multiplexes the audio MMT packets generated by audio packetization unit 203 and the video MMT packets generated by video packetization unit 204, and transmits the multiplexed packets.

[0114] In the above description, the playback intervals of V_MPU(n) and A_MPU(n) match, but the playback intervals of V_MPU(n) and A_MPU(n) may be different. In this embodiment, even if the receiving device receives an audio MPU partway through, it can start decoding from the sample acquired immediately after reception, so the playback intervals of V_MPU(n) and A_MPU(n) do not necessarily have to match.

[0115] As described above, when transmitting an audio segment by packet multiplexing, transmitting apparatus 200 according to this embodiment stores the header and data storage portion of the segment in separate packets, and further packetizes the data storage portion for each access unit. Furthermore, transmitting apparatus 200 stores an index number indicating the decoding order of the access unit within the segment in the header of the packet storing the access unit, and transmits the packet. This allows the receiving apparatus to calculate the PTS of the access unit. Therefore, the delay time until the receiving apparatus starts decoding is reduced without restricting the packet multiplexing operation of the segment.

[0116] That is, transmitting device 200 divides audio data (audio MPU) into a plurality of packets (MMT packets), each including one of a plurality of audio samples (audio samples), and transmits the divided packets. Each of the plurality of packets includes sequence information (fragment SN) indicating the ordinal number of the packet among the plurality of packets.

[0117] (Embodiment 3) In this embodiment, the playback duration of the audio MPU is set shorter than that of the video MPU without aligning the playback durations of the video MPU and the audio MPU. This increases the transmission frequency of the audio MPU, thereby improving random accessibility.

[0118] Assuming random access in MPU units, for video, the RAU in the coded data is the smallest unit of the MPU. On the other hand, for audio, decoding is possible from any sample (access unit), so the number of samples that make up the MPU can be set freely.

[0119] For example, the playback time of a video RAU is generally about 0.5 to 1 second, taking into account the balance between coding efficiency and random accessibility. The playback time of a video MPU also follows the playback time of the RAU.

[0120] On the other hand, for audio, the MPU playback time can be set to around 0.1 seconds. For example, if the audio MPU playback time is set to 0.1 seconds, audio playback can begin at the latest 0.1 seconds after video playback begins. In this way, by setting the audio MPU playback time to a short time, the delay between when video playback begins and when audio playback can begin can be reduced.

[0121] 13 is a diagram showing an example of a video MPU and an audio MPU according to the present embodiment. As shown in FIG. 13, the playback time of each of the plurality of audio MPUs is set to be shorter than the playback time of each of the plurality of video MPUs.

[0122] Alternatively, the playback sections of multiple audio MPUs may be set so that the first PTS of one of the audio MPUs corresponds to the first PTS of the video MPU. Fig. 14 shows an example of the video MPU and audio MPU in this case.

[0123] For example, when the playback interval of an audio MPU is set as shown in Fig. 13, there is no audio MPU that has the same starting PTS as the starting PTS of video MPU (1). Thus, in the example shown in Fig. 13, it is not guaranteed that the starting PTS of any audio MPU corresponds to the starting PTS of the video MPU.

[0124] On the other hand, when the playback interval of an audio MPU is set as shown in Fig. 14, there will always be an audio MPU whose starting PTS is the same as the starting PTS of each video MPU. In other words, it is guaranteed that the starting PTS of one of the audio MPUs corresponds to the starting PTS of a video MPU.

[0125] Furthermore, if the playback time of the video RAU is constant, the transmitting device can determine the playback time of the audio MPU by acquiring the playback time of the video MPU prior to starting the MPU generation process. On the other hand, if a change in the frame rate of the video material or a change in the encoding conditions occurs and the playback time of the video RAU becomes variable, the transmitting device determines the playback time of the audio MPU while acquiring the playback time of each video MPU.

[0126] The configuration and operation of the transmitting device according to this embodiment will be described below.

[0127] 15 is a block diagram of a transmitting device 300 according to this embodiment. The transmitting device 300 shown in FIG. 15 includes a video MPU generation unit 301, a section determination unit 302, a first determination unit 303, a second determination unit 304, a start determination unit 305, and an MPU generation unit 306.

[0128] FIG. 16 is a flowchart of the transmission process (multiplexing process) according to this embodiment.

[0129] First, as an initial setting, the transmitting device 300 selects the first MPU by setting n to 0 (S301).

[0130] Next, the video MPU generation unit 301 determines the playback section of the first video MPU (V_MPU(0)) based on the RAU of the video (S302). Here, a video MPU is made up of one or more RAUs.

[0131] Next, the start determining unit 305 determines the start sample of the first audio MPU (A_MPU(0)) so that the PTS of the start sample of V_MPU(0) and that of A_MPU(0) are the same (S303).

[0132] Next, the section determination unit 302 determines whether the MPU (A_MPU) of the audio to be processed is the final MPU in the playback section of V_MPU(n) (S304).

[0133] If A_MPU is the final MPU (Yes in S304), the first determination unit 303 determines the final sample of A_MPU so that the playback end time of the final sample of A_MPU is equal to the playback end time of V_MPU(n) and the playback time of A_MPU is less than or equal to a threshold value (S305).

[0134] On the other hand, if A_MPU is not the final MPU (No in S304), the second determination unit 304 determines the final sample of A_MPU so that the playback time of A_MPU is equal to or less than the threshold (S306).

[0135] The threshold used in steps S305 and S306 is set based on the tolerable delay time between the start of playback of a video random access point and the start of audio playback. However, as the MPU playback time shortens, the proportion of the MPU header in the transmitted data increases. In particular, audio has a lower bit rate than video. Also, the MPU header has a large fixed portion that is independent of the number of samples stored in the MPU. Therefore, reducing the MPU header overhead is important. Therefore, for example, the MPU playback time is determined based on both the delay time and the MPU header overhead.

[0136] The number of audio samples that can be stored in one MPU depends on the playback time of one sample, which in turn depends on the sampling frequency, etc.

[0137] After step S305 or S306, the MPU generation unit 306 generates an audio MPU(n) based on the first sample of the audio MPU determined in step S303 and the last sample of the audio MPU determined in step S305 or S306.

[0138] Next, the transmitting device 300 increments n by 1 to select the next MPU (S307), and repeats the processes of steps S304 to S307 until the playback sections of all MPUs have been determined (S308).

[0139] The first sample in each of the MPUs after the first MPU is set to the sample immediately following the final sample in the immediately preceding MPU.

[0140] Furthermore, the transmitting device 300 may separate MPUs when switching between sampling frequencies or channel configurations (such as monaural, stereo, or 5.1 channels). This allows the audio encoding conditions to be consistent within the same MPU. This allows the receiving device to easily achieve seamless playback at the time of switching, since it only needs to handle switching between encoding conditions on an MPU-by-MPU basis during playback. For example, the receiving device can prepare for switching in advance by pre-reading the encoding conditions (parameters) stored in the header or first sample of the next MPU during playback.

[0141] Also, as shown in FIGS. 13 and 14, it may or may not be guaranteed that the first PTS of any audio MPU corresponds to the first PTS of a video MPU.

[0142] As described above, the transmitting device 300 according to this embodiment generates video segments (MPUs) so that their beginnings become random access points, and generates audio segments so that their playback times are equal to or shorter than a predetermined value. Furthermore, the transmitting device 300 sets the playback times of audio segments to be shorter than those of video segments. While this increases overhead related to audio segments, it makes it possible to eliminate the need for PTS calculation processing in the receiving device. Furthermore, it reduces the delay time until the receiving device starts decoding.

[0143] That is, the video data (video MPU) is one of the multiple pieces of video data obtained by dividing the video signal. The audio data (audio MPU) is one of the multiple pieces of audio data obtained by dividing the audio signal. As shown in Figure 13, the playback time of the audio data (audio MPU) is shorter than the playback time of the video data (video MPU).

[0144] Furthermore, as shown in FIG. 14, the plurality of pieces of audio data (audio MPU) may include a plurality of pieces of audio data with playback start times (PTS) that are approximately equal to the playback start times (PTS) of the plurality of pieces of video data (video MPU).

[0145] (Fourth embodiment) In this embodiment, a receiving device that receives data transmitted by transmitting device 100 according to the first embodiment and reproduces the data will be described.

[0146] 17 is a block diagram of a receiving apparatus 400 according to this embodiment. The receiving apparatus 400 shown in FIG.

[0147] Fig. 18 is a flowchart of a receiving process according to this embodiment. The flowchart shown in Fig. 18 shows an example of an operation when determining an audio sample at which decoding starts during random access.

[0148] First, video MPU determination unit 401 determines the video MPU (V_MPU) that will start playback (S401). For example, the header of an MMT packet contains identification information that indicates whether the payload contains an MPU header or sample data. Furthermore, the packet ID (corresponding to the PID of a TS packet) allows receiving device 400 to identify whether the data stored in the payload is a video or audio asset. Therefore, after starting to receive a packet having a packet ID corresponding to a video asset in broadcasting or communication, video MPU determination unit 401 determines the MPU that first obtains an MPU header as the V_MPU.

[0149] Next, the audio MPU determination unit 402 searches for an audio MPU that has the same starting PTS as the PTS (starting PTS) of the first sample in display order in the V_MPU, from among the MPUs that have received an MPU header after the V_MPU, and determines the audio MPU obtained by the search as the audio MPU (A_MPU) from which playback will start (S402). Here, the PTS being the same does not necessarily mean that the PTSs are completely the same, and as explained in the first embodiment, it is sufficient that the PTSs are approximately the same.

[0150] Furthermore, the first PTSs of the audio and video MPUs are separately indicated by an MMT message or an MPEG-2 system descriptor when MMT packets are multiplexed into a TS, and the audio MPU determination unit 402 can search for an audio MPU having the first PTS of a V_MPU based on this separately indicated information. Note that information indicating the first PTS of an MPU is included in the MPU header, and the audio MPU determination unit 402 may use this information to search for an audio MPU having the first PTS of a V_MPU.

[0151] Next, the sample determination unit 403 determines to start playback from the first sample of A_MPU (S403).

[0152] In step S403, since it is guaranteed that the playback sections of the video MPU and audio MPU are aligned, the sample determination unit 403 can determine to start playback from the first sample of A_MPU. On the other hand, if it is not guaranteed that the playback sections of the video MPU and audio MPU are aligned, the receiving device 400 may perform the following process.

[0153] First, the receiving device 400 acquires the first PTS (V_MPU.pts) of the V_MPU.

[0154] Next, the receiving device 400 acquires the first PTS of the audio MPU acquired first after the V_MPU, and analyzes the MPU header to acquire the playback time of the audio MPU. The receiving device 400 then determines whether the audio MPU contains an audio sample with the same PTS as V_MPU.pts. Note that the PTSs do not necessarily have to match, and the receiving device 400 may determine whether the audio MPU contains an audio sample with a PTS immediately before or after V_MPU.pts.

[0155] Furthermore, if the playback time of the audio MPU is separately indicated like the first PTS of the MPU, the receiving device 400 does not analyze the MPU header, but obtains the playback time of the audio MPU from the separately indicated information.

[0156] If it is determined that there is no audio sample having the same PTS as V_MPU.pts, the receiving device 400 acquires the next audio MPU and performs the above determination process. Then, the receiving device 400 repeats the determination process sequentially for the subsequent audio MPUs until it is determined that there is an audio sample having the same PTS as V_MPU.pts.

[0157] Note that the receiving device 400 may start video playback from the first sample of V_MPU, and may start audio playback from the MPU immediately following A_MPU. Here, if the decoding order and display order of video differ, the receiving device 400 starts decoding from the first sample in the decoding order, and starts playback (display) from the first sample in the display order.

[0158] Also, audio playback may start from A_MPU before the start of video V_MPU playback, and video playback may start from the time of V_MPU.pts.

[0159] As described above, receiving apparatus 400 according to this embodiment determines to start decoding from the video segment whose header is received first after starting reception, and starts decoding from the audio segment whose PTS matches the starting PTS of the video segment from which decoding starts. This allows receiving apparatus 400 to reduce the delay time until decoding starts.

[0160] Also, although the case of receiving a signal transmitted by transmitting device 100 according to embodiment 1 has been described here, a similar method can also be applied to receiving a signal transmitted by transmitting device 300 according to embodiment 3.

[0161] That is, the receiving device 400 determines to start decoding from the video segment whose header is received first after starting reception. Also, the receiving device 400 starts decoding from the audio segment whose first PTS is the same as or later than the PTS of the video segment from which decoding is started and which is received first. This allows the receiving device 400 to reduce the delay time until decoding starts.

[0162] In this way, the receiving device 400 can receive the video control information (video MPU header), video data (video MPU), audio control information (audio MPU header) and audio data (audio MPU) transmitted by the transmitting device 100 or 300 according to embodiment 1 or 3, and reproduce the video data and audio data.

[0163] (Embodiment 5) In this embodiment, a receiving device that receives data transmitted by transmitting device 200 according to the second embodiment and reproduces the data will be described.

[0164] 19 is a block diagram of receiving apparatus 500 according to the present embodiment. Receiving apparatus 500 shown in FIG. 19 includes start PTS determination section 501, judgment section 502, first PTS calculation section 503, second PTS calculation section 504, and sample acquisition section 505.

[0165] Fig. 20 is a flowchart of a receiving process according to this embodiment. The flowchart shown in Fig. 20 shows an example of an operation for acquiring an audio sample from which decoding starts and determining a PTS.

[0166] First, the start PTS determination unit 501 determines the PTS (tgt_pts) of the audio sample from which decoding starts (S501). An audio sample whose PTS is tgt_pts is denoted as tgt_spl. tgt_pts is, for example, the first PTS of the video MPU from which playback starts. This is because, after tuning in to a broadcast channel or when receiving content via a communication network, the video random access point is generally used as the starting point.

[0167] Next, the determination unit 502 determines whether the MPU header of the MPU (mpu_cur) to which tgt_spl belongs has been acquired (S502). For example, the determination unit 502 determines the MPU (mpu_cur) that includes a sample whose PTS is tgt_pts, based on the first PTS of each MPU separately acquired from an MMT message or the like. Note that information indicating the first PTS of the MPU may be included in the MPU header, and the determination unit 502 may determine the MPU (mpu_cur) that includes a sample whose PTS is tgt_pts based on this information. Alternatively, the determination unit 502 may separately download information about the assets that make up the MMT package, and determine the MPU (mpu_cur) that includes a sample whose PTS is tgt_pts based on this information.

[0168] If the MPU header of the MPU (mpu_cur) has been acquired (Yes in S502), the first PTS calculation unit 503 analyzes the MPU header and acquires the index number of tgt_spl within mpu_cur (S503). Here, the PTS of the sample included in the MPU can be determined based on information indicating the DTS and PTS of the sample included in trun within moof. Since the PTS obtained by analyzing moof is a differential value from the first PTS of the MPU, the first PTS calculation unit 503 calculates the actual PTS by adding the first PTS of the MPU to the obtained PTS. Then, the first PTS calculation unit 503 determines the sample whose calculated PTS is the same as tgt_pts as tgt_spl. Note that if there is no sample whose PTS is the same as tgt_pts, the first PTS calculation unit 503 determines the sample whose PST is immediately before or after tgt_pts as tgt_spl.

[0169] Next, the first PTS calculation unit 503 obtains an index number (fragment SN) indicating the ordinal number of the sample tgt_spl in the MPU in decoding order. Note that if the MPU is made up of multiple movie fragments, the fragment SN is a serial number starting from the first sample of the first movie fragment.

[0170] On the other hand, if the MPU header of the MPU (mpu_cur) cannot be obtained (No in S502), the second PTS calculation unit 504 calculates the PTS of the sample based on the playback time per sample and other factors using the above (Equation 1), and obtains the index number of tgt_spl in mpu_cur (S504). Also, if the PTS of the sample can be obtained by a method other than the method using (Equation 1), the second PTS calculation unit 504 may use that method. The PTS calculation method is described in detail in the above-mentioned second embodiment.

[0171] After step S503 or S504, the sample acquisition unit 505 acquires a packet in which the MPU sequence number in the header of the MMT packet matches the sequence number of mpu_cur and the fragment SN is equal to the index number obtained in step S503 or S504, and acquires the audio sample contained in the packet from which playback will begin (S505).

[0172] In some cases, the MPU header of a corresponding audio sample cannot be obtained at a video random access point. If the MPU to which the audio sample belongs is MPU(n), receiving device 500 obtains the PTS for samples included in MPU(n) using the method of step S504, and for samples included in MPU(n+1) and later using the method of step S503.

[0173] Note that the receiving device 500 may always obtain the PTS using the method of step S504 without performing the processes of steps S502 and S503.

[0174] In addition, in the above example, the audio sample from which decoding starts is determined based on the video playback start time in step S501, but this embodiment can also be applied to a case where playback starts from the first audio sample that can be obtained after reception starts. Figure 21 is a flowchart of the reception process in this case.

[0175] First, the receiving apparatus 500 determines the audio sample (tgt_spl) at which to start decoding (S511). For example, the receiving apparatus 500 determines the first received audio sample as tgt_spl.

[0176] Next, the receiving device 500 determines whether or not the MPU header of the MPU (mpu_cur) to which the tgt_spl belongs has been acquired (S512). Note that this process is the same as step S502 above.

[0177] If the MPU header of the MPU (mpu_cur) has been acquired (Yes in S512), the receiving device 500 analyzes the MPU header and calculates the PTS of tgt_spl (S513). Note that the details of the method for calculating the PTS are the same as those in step S503 above.

[0178] On the other hand, if the MPU header of the MPU (mpu_cur) cannot be obtained (No in S512), the receiving device 500 calculates the PTS of the sample based on the playback time per sample and other information using the above (Equation 1) (S514). Note that the details of the PTS calculation method are the same as those in step S504.

[0179] Through the above processing, the receiving device 500 can obtain the PTS of the audio sample even when it is unable to obtain the MPU header of the audio.

[0180] As described above, the receiving device 500 obtains data for one access unit from the payload of one packet, and calculates the difference in PTS between the access unit that is the first in decoding order in the segment and the obtained access unit based on the index number. Furthermore, the receiving device 500 determines the PTS of the access unit by separately obtaining the first PTS of the segment. In this way, by calculating the PTS of the access unit in the receiving device 500, the delay time until the start of decoding can be reduced without restricting the packet multiplexing operation of the segment.

[0181] That is, receiving device 500 receives video control information (video MPU header), video data (video MPU), audio control information (audio MPU header), and audio data (audio MPU) transmitted by transmitting device 200 according to Embodiment 2. Furthermore, receiving device 500 acquires a first playback start time, which is the playback start time (PTS) of the audio sample that is to be played first among a plurality of audio samples included in the audio data (audio MPU).

[0182] Furthermore, the receiving device 500 acquires the playback time of each of the plurality of audio samples. For example, the receiving device 500 acquires the number of samples included in an audio frame and a sampling frequency, and calculates the playback time of each of the plurality of audio samples from the acquired number of samples and sampling frequency.

[0183] Next, the receiving device 500 determines the playback start time (PTS) of the audio sample contained in the packet to be processed using the above-mentioned first playback start time, the above-mentioned playback time, and the sequence information (fragment SN) contained in the packet to be processed.

[0184] (Variation) The multiplexing method applied in the present invention is not limited to MP4, MMT, or DASH, but the present invention can also be applied to other formats that can segment coded data such as audio and video.

[0185] The transmitting device may transmit the MMT packet stored in an IP packet, etc. Furthermore, in broadcasting, the transmitting device may transmit the IP packet stored in a TS packet or a TLV (Time Length Value) packet defined by ARIB (Association of Radio Industries and Businesses), etc.

[0186] Furthermore, the receiving device may multiplex segments such as MPUs into packets other than MMT packets. The present invention can be applied to any protocol that allows packet multiplexing of audio and video MPU data into a single stream.

[0187] In addition, in the first to third embodiments, the transmitting device may multiplex audio and video packets so as to satisfy a system decoder model (such as a model of STD (System Target Decoder) in the MPEG-2 system) when multiplexing packets. Also, the system decoder model can be defined in the processing of MMT packets.

[0188] Furthermore, in the above description, different methods have been described individually in the first to third embodiments, but two or more of these methods may be combined. For example, by combining the method according to the first embodiment with the method according to the second or third embodiment, if the receiving device is unable to acquire the audio MPU header due to a communication error or the like, the delay time until the audio is played can be reduced by the method according to the second or third embodiment. Furthermore, by combining the second and third embodiments, the frequency of the process of calculating the PTS from information other than the MPU header can be reduced.

[0189] Although the transmitting device, receiving device, transmitting method and receiving method according to the embodiments have been described above, the present invention is not limited to these embodiments.

[0190] Furthermore, each processing unit included in the transmitting device and receiving device according to the above-described embodiments is typically realized as an LSI, which is an integrated circuit. These may be individually implemented as single chips, or some or all of them may be integrated into a single chip.

[0191] Furthermore, the integration is not limited to LSI, but may be realized by dedicated circuits or general-purpose processors. FPGAs (Field Programmable Gate Arrays), which can be programmed after LSI fabrication, or reconfigurable processors, which allow the connections and settings of circuit cells within LSIs to be reconfigured, may also be used.

[0192] In each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may also be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0193] In other words, the transmitting device and the receiving device include processing circuitry and storage electrically connected to (accessible from) the processing circuitry. The processing circuitry includes at least one of dedicated hardware and a program execution unit. If the processing circuitry includes a program execution unit, the storage unit stores a software program to be executed by the program execution unit. The processing circuitry uses the storage unit to execute the transmitting method or the receiving method according to the above embodiment.

[0194] Furthermore, the present invention may be the above-mentioned software program, or a non-transitory computer-readable recording medium on which the above-mentioned program is recorded. Needless to say, the above-mentioned program can be distributed via a transmission medium such as the Internet.

[0195] Furthermore, all the numbers used above are merely examples for the purpose of specifically explaining the present invention, and the present invention is not limited to the numbers used as examples.

[0196] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.

[0197] The order in which the steps included in the above-described transmitting method or receiving method are executed is merely an example for specifically explaining the present invention, and an order other than the above may be used. Also, some of the above steps may be executed simultaneously (in parallel) with other steps.

[0198] While the transmitting device, receiving device, transmitting method, and receiving method according to one or more aspects of the present invention have been described based on the embodiments, the present invention is not limited to these embodiments. As long as they do not deviate from the spirit of the present invention, various modifications conceivable by those skilled in the art to the present embodiments, and configurations constructed by combining components of different embodiments, may also be included within the scope of one or more aspects of the present invention. [Industrial Applicability]

[0199] The present invention can be applied to devices or equipment that transport media such as video data and audio data. [Explanation of symbols]

[0200] 100, 200, 300 transmitter 101 MPU generation section 102 Packet multiplexing unit 111 Video header determination unit 112 Audio header determination unit 113 Multiplexer 201, 301 Video MPU generation unit 202 Audio MPU Generation Unit 203 Audio Packetizer 204 Video Packetizer 302 Section Determination Unit 303 First Decision Section 304 Second Decision Section 305 Leading Determination Unit 306 MPU generation section 400, 500 receiving device 401 Video MPU decision unit 402 Audio MPU decision unit 403 Sample Determination Unit 501 Start PTS determination section 502 Judgment section 503 First PTS Calculation Department 504 Second PTS Calculation Department 505 Sample Acquisition Unit

Claims

1. A receiving method, comprising: a video control information receiving step of receiving video control information for reproducing video data; a video data receiving step of receiving the video data; an audio control information receiving step of receiving audio control information for reproducing audio data in a playback section corresponding to the playback section of the video data; a voice data receiving step of receiving the voice data, the audio data includes a plurality of audio samples; the audio control information includes first time information indicating a first leading playback start time, which is a playback start time of an audio sample to be first played back among the plurality of audio samples; the audio data is divided into a plurality of packets each including one of the plurality of audio samples, and the audio control information is included in a packet different from the packet; each of the plurality of packets includes sequence information indicating the ordinal number of the packet among the plurality of packets; the video data includes a plurality of video samples; the video control information includes second time information indicating a second leading playback start time, which is a playback start time of a video sample to be played first among the plurality of video samples; The receiving method further comprises: a playback time acquisition step of acquiring a playback time of each of the plurality of audio samples; a generating step of generating a playback start time of the audio sample included in the packet to be processed using the first leading playback start time, the playback time, and the sequence information included in the packet to be processed. Receiving method.

2. A receiving device, a video control information receiving unit that receives video control information for reproducing video data; a video data receiving unit that receives the video data; an audio control information receiving unit that receives audio control information for reproducing audio data in a playback section corresponding to the playback section of the video data; a voice data receiving unit that receives the voice data, the audio data includes a plurality of audio samples; the audio control information includes first time information indicating a first leading playback start time, which is a playback start time of an audio sample to be first played back among the plurality of audio samples; the audio data is divided into a plurality of packets each including one of the plurality of audio samples, and the audio control information is included in a packet different from the packet; each of the plurality of packets includes sequence information indicating the ordinal number of the packet among the plurality of packets; the video data includes a plurality of video samples; the video control information includes second time information indicating a second leading playback start time, which is a playback start time of a video sample to be played first among the plurality of video samples; The receiving device further a playback time acquisition unit that acquires a playback time of each of the plurality of audio samples; a generation unit that generates a playback start time of the audio sample included in the packet to be processed using the first leading playback start time, the playback time, and the sequence information included in the packet to be processed. Receiving device.

Images