Receiving device and receiving method

Abstract

To provide a receiver or the like capable of guaranteeing a buffer operation of the receiver.SOLUTION: A generator 701 that generates an encoded stream by performing rate control so as to satisfy a regulation by a receiving buffer model, and a transmitter 702 that transmits a transmission packet obtained by packetizing the encoded stream, wherein the receiving buffer model converts a packet having a variable-length packet header of the transmission packet into a first packet having a decompressed header, and converts the first packet into a second packet, first to fourth buffers that convert the second packet into a NAL unit and output an access unit generated from a plurality of NAL units to a decoder at a corresponding decoding time, wherein the NAL unit has a NAL size format in which a 4-byte size area is added to a head of the NAL unit.SELECTED DRAWING: Figure 74

Description

Technical Field

[0001] The present invention relates to a receiving apparatus and a receiving method.

Background Art

[0002] With the advancement of broadcast and communication services, the introduction of ultra-high-definition moving image contents such as 8K (7680×4320 pixels: hereinafter also referred to as 8K4K) and 4K (3840×2160 pixels: hereinafter also referred to as 4K2K) has been considered. A receiving apparatus needs to decode and display the encoded data of the received ultra-high-definition moving image in real time. However, in particular, a moving image with a resolution such as 8K has a large processing load during decoding, and it is difficult to decode such a moving image in real time with a single decoder. Therefore, a method of reducing the processing load per decoder and achieving real-time processing by parallelizing the decoding process using a plurality of decoders has been considered.

[0003] Also, the encoded data is multiplexed based on a multiplexing method such as MPEG-2 TS (Transport Stream) or MMT (MPEG Media Transport) and then transmitted. For example, Non-Patent Document 1 discloses a technique of transmitting encoded media data for each packet according to MMT.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Incidentally, with the advancement of broadcasting and communication services, the introduction of ultra-high-definition video content such as 8K and 4K (3840x2160 pixels) is being considered. In multiplexing methods such as MMT (MPEG Media Transport) and MPEG-DASH, media data such as video, audio, and files are multiplexed into ISOBMFF (ISO based media file format) (MP4 format) files, and after being packetized using MMT, they are transmitted using broadcast transmission lines or communication transmission lines.

[0006] However, with MMT transmission, there is a possibility that the buffer operation of the receiving device cannot be guaranteed.

[0007] The present invention provides a receiving device and a receiving method that can guarantee the buffer operation of the receiving device when transmitting data using a method such as MMT. [Means for solving the problem]

[0008] To achieve the above objective, a transmitting device according to one aspect of the present invention includes: a receiving unit that receives a transmission packet in which an encoded stream is packetized, comprising a fixed-length packet header and a variable-length payload containing a first IP packet having a variable-length packet header; a first buffer that converts the first IP packet contained in the received transmission packet containing the first IP packet into a first packet having a fixed-length packet header with the header expanded, and outputs the first packet obtained by the conversion at a bit rate of 1.05 times the input rate to the first buffer; and a second packet comprising a packet header and a variable-length payload obtained by the conversion. The system comprises: a second buffer that outputs the aforementioned second packet at a constant bitrate; a third buffer that converts the outputted second packet into a NAL unit and outputs the resulting NAL unit at a constant bitrate; a fourth buffer that sequentially stores the outputted NAL units, generates an access unit from the stored NAL units, and outputs the generated access unit at the timing of the decoding time corresponding to the access unit; and a decoder that decodes the access unit output by the fourth buffer. The second buffer and the third buffer are merged into a single buffer, and the NAL units are stored in a multiplexing layer and have a NAL size format with a 4-byte size area added to the beginning.

[0009] To achieve the above objective, a transmitting device according to one aspect of the present invention comprises: a generation unit that generates an encoded stream by performing rate control to satisfy the requirements of a receive buffer model; a transmitting unit that packets the generated encoded stream and transmits the transmission packets obtained by packetization, wherein the transmission packets consist of a fixed-length packet header and a variable-length payload, the variable-length payload includes a first IP packet having a variable-length packet header, the receive buffer model receives the transmission packet including the first IP packet, converts the first IP packet stored in the received transmission packet into a first packet having a fixed-length packet header with the header expanded, and outputs the first packet obtained by conversion at a bit rate of 1.05 times the input rate to the first buffer, and the output first packet consists of a packet header and a variable-length payload The system comprises: a second buffer that converts the second packet into a second packet composed of data and outputs the second packet obtained by the conversion at a constant bitrate; a third buffer that converts the output second packet into a NAL unit and outputs the NAL unit obtained by the conversion at a constant bitrate; and a fourth buffer that sequentially stores the output NAL units, generates an access unit from the stored NAL units, and outputs the generated access unit to the decoder at the timing of the decoding time corresponding to the access unit. The second buffer and the third buffer are merged to form a single buffer. If the second packet contains multiple data units including the NAL unit, information indicating the data unit length is added to the second packet. The NAL unit is stored in a multiplexing layer and has a NAL size format with a 4-byte size area added to the beginning.

[0010] Furthermore, a receiving device according to one aspect of the present invention includes: a receiving unit that receives a transmission packet in which an encoded stream has been packetized, the transmission packet comprising a fixed-length packet header and a variable-length payload containing a first IP packet having a variable-length packet header; a first buffer that converts the first IP packet contained in the received transmission packet, which includes the first IP packet, into a first packet having a fixed-length packet header with the header expanded, and outputs the first packet obtained by the conversion at a bit rate of 1.05 times the input rate to the first buffer; a second buffer that converts the output first packet into a second packet comprising a packet header and a variable-length payload, and outputs the second packet obtained by the conversion at a constant bit rate; and the output The system comprises: a third buffer that converts the second packet into a NAL unit and outputs the resulting NAL unit at a constant bitrate; a fourth buffer that sequentially stores the output NAL units, generates an access unit from the stored NAL units, and outputs the generated access unit at the timing of the decoding time corresponding to the access unit; and a decoder that decodes the access unit output by the fourth buffer. The second buffer and the third buffer are merged into a single buffer. If the second packet contains multiple data units including the NAL unit, information indicating the data unit length is added to the second packet. The NAL unit is stored in a multiplexing layer and has a NAL size format with a 4-byte size area added to the beginning.

[0011] To achieve the above objective, a transmitting device according to one aspect of the present invention comprises: a generation unit that generates an encoded stream by performing rate control to satisfy the requirements of a receive buffer model; a transmitting unit that packets the generated encoded stream and transmits the transmission packets obtained by packetization, wherein the transmission packets consist of a fixed-length packet header and a variable-length payload, the variable-length payload includes a first IP packet having a variable-length packet header or a second IP packet having a fixed-length packet header, the receive buffer model receives the transmission packet including the first IP packet, converts the first IP packet stored in the received transmission packet into a first packet having a fixed-length packet header with the header expanded, and outputs the first packet obtained by conversion at a bit rate of 1.05 times the input rate to the first buffer, and the output first packet is packetized The system comprises: a second buffer that converts the second packet, which consists of a header and a variable-length payload, into a second packet and outputs the resulting second packet at a constant bitrate; a third buffer that converts the output second packet into a NAL unit and outputs the resulting NAL unit at a constant bitrate; and a fourth buffer that sequentially stores the output NAL units, generates an access unit from the stored NAL units, and outputs the generated access unit to the decoder at the timing of the decoding time corresponding to the access unit. The second buffer and the third buffer are merged into a single buffer. If the second packet contains multiple data units including the NAL unit, information indicating the data unit length is added to the second packet. The NAL unit is stored in a multiplexing layer and has a NAL size format with a 4-byte size area added to the beginning.

[0012] Furthermore, a receiving device according to one aspect of the present invention includes: a receiving unit that receives a transmission packet in which an encoded stream is packetized, the transmission packet comprising a fixed-length packet header and a first IP packet having a variable-length packet header, or a variable-length payload including a second IP packet having a fixed-length packet header; a first buffer that converts the first IP packet contained in the received transmission packet, including the first IP packet, into a first packet having a decompressed fixed-length packet header, and outputs the first packet obtained by the conversion at a bit rate of 1.05 times the input rate to the first buffer; and a second packet comprising a packet header and a variable-length payload, and outputs the second packet obtained by the conversion at a constant bit rate. The system comprises a second buffer, a third buffer that converts the output second packet into a NAL unit and outputs the resulting NAL unit at a constant bitrate, a fourth buffer that sequentially stores the output NAL units, generates an access unit from the stored NAL units, and outputs the generated access unit at the timing of the decoding time corresponding to the access unit, and a decoder that decodes the access unit output by the fourth buffer. The second buffer and the third buffer are merged to form a single buffer, and if the second packet contains multiple data units including the NAL unit, information indicating the data unit length is added to the second packet, and the NAL unit is stored in a multiplexing layer and has a NAL size format with a 4-byte size area added to the beginning.

[0013] To achieve the above objective, a transmitting device according to one aspect of the present invention comprises: a generation unit that generates an encoded stream by performing rate control to satisfy the requirements of a receive buffer model; a transmitting unit that packets the generated encoded stream and transmits the transmission packets obtained by packetization, wherein the transmission packets consist of a fixed-length packet header and a variable-length payload, the variable-length payload includes a first IP packet having a variable-length packet header or a second IP packet having a fixed-length packet header, the receive buffer model receives the transmission packet including the first IP packet, converts the first IP packet stored in the received transmission packet into a first packet having a fixed-length packet header with the header expanded, and outputs the first packet obtained by conversion at a bit rate of 1.05 times the input rate to the first buffer, and the output first packet The system comprises: a second buffer that converts a packet into a second packet consisting of a packet header and a variable-length payload, and outputs the resulting second packet at a constant bitrate; a third buffer that converts the output second packet into a NAL unit, and outputs the resulting NAL unit at a constant bitrate; and a fourth buffer that sequentially stores the output NAL units, generates an access unit from the stored NAL units, and outputs the generated access unit to the decoder at the timing of the decoding time corresponding to the access unit. The second buffer and the third buffer are merged to form a single buffer. If the second packet contains multiple data units including the NAL units, information indicating the data unit length is added to the second packet, and the NAL unit is in an NAL size format with a 4-byte size area added to the beginning.

[0014] Furthermore, a receiving device according to one aspect of the present invention includes: a receiving unit that receives a transmission packet in which an encoded stream is packetized, comprising a first IP packet having a fixed-length packet header and a variable-length payload including a first IP packet having a variable-length packet header, or a second IP packet having a fixed-length packet header; a first buffer that converts the first IP packet stored in the received transmission packet, which includes the first IP packet, into a first packet having a fixed-length packet header with the header expanded, and outputs the first packet obtained by the conversion at a bit rate of 1.05 times the input rate to the first buffer; and converts the output first packet into a second packet comprising a packet header and a variable-length payload, and outputs the second packet obtained by the conversion at a constant bit rate The system comprises a second buffer that outputs packets, a third buffer that converts the output second packet into a NAL unit and outputs the resulting NAL unit at a constant bitrate, a fourth buffer that sequentially stores the output NAL units, generates an access unit from the stored NAL units, and outputs the generated access unit at the timing of the decoding time corresponding to the access unit, and a decoder that decodes the access unit output by the fourth buffer. The second buffer and the third buffer are merged to form a single buffer, and if the second packet contains multiple data units including the NAL unit, information indicating the data unit length is added to the second packet, and the NAL unit is in an NAL size format with a 4-byte size area added to the beginning.

[0015] To achieve the above objective, a transmitting device according to one aspect of the present invention comprises: a generation unit that generates an encoded stream by performing rate control to satisfy the requirements of a receive buffer model; and a transmitting unit that packets the generated encoded stream and transmits the transmission packets obtained by packetization, wherein the transmission packets consist of a fixed-length packet header and a variable-length payload, the variable-length payload includes a first IP packet having a variable-length packet header or a second IP packet having a fixed-length packet header, the receive buffer model receives the transmission packet including the first IP packet, converts the first IP packet stored in the received transmission packet into a first packet having a fixed-length packet header with the header expanded, and outputs the first packet obtained by conversion at a bit rate of 1.05 times the input rate to the first buffer. The system comprises: a first buffer; a second buffer that converts the outputted first packet into a second packet consisting of a packet header and a variable-length payload, and outputs the second packet obtained by the conversion at a constant bitrate; a third buffer that converts the outputted second packet into a NAL unit, and outputs the NAL unit obtained by the conversion at a constant bitrate; and a fourth buffer that sequentially stores the outputted NAL units, generates an access unit from the stored NAL units, and outputs the generated access unit to the decoder at the timing of the decoding time corresponding to the access unit. The second buffer and the third buffer are merged to form a single buffer, and if the second packet contains multiple data units including the NAL units, information indicating the data unit length is added to the second packet.

[0016] Furthermore, a receiving device according to one aspect of the present invention includes: a receiving unit that receives a transmission packet in which an encoded stream is packetized, and which consists of a fixed-length packet header and a first IP packet having a variable-length packet header, or a variable-length payload that includes a second IP packet having a fixed-length packet header; a first buffer that converts the first IP packet stored in the received transmission packet, which includes the first IP packet, into a first packet having a decompressed fixed-length packet header, and outputs the first packet obtained by the conversion at a bit rate of 1.05 times the input rate to the first buffer; and converts the output first packet into a second packet consisting of a packet header and a variable-length payload, and The system includes a second buffer that outputs the obtained second packet at a constant bitrate, a third buffer that converts the output second packet into a NAL unit and outputs the obtained NAL unit at a constant bitrate, a fourth buffer that sequentially stores the output NAL units, generates an access unit from the stored NAL units, and outputs the generated access unit at the timing of the decoding time corresponding to the access unit, and a decoder that decodes the access unit output by the fourth buffer, wherein the second buffer and the third buffer are merged into a single buffer, and if the second packet contains multiple data units including the NAL unit, information indicating the data unit length is added to the second packet.

[0017] To achieve the above objective, a transmitting device according to one aspect of the present invention comprises: a generation unit that generates an encoded stream by performing rate control to satisfy the requirements of a predetermined receive buffer model in order to guarantee the buffer operation of a receiving device; and a transmitting unit that packets the generated encoded stream and transmits the transmission packets obtained by packetization, wherein the transmission packets consist of a fixed-length packet header and a variable-length payload, and the receive buffer model receives the transmission packets, converts the packets consisting of a variable-length packet header and a variable-length payload stored in the received transmission packets into a first packet having a fixed-length packet header with the header expanded, and outputs the first packet obtained by conversion at a constant bit rate. The encoded stream comprises a buffer, a second buffer that converts the output first packet into a second packet consisting of a packet header and a variable-length payload, and outputs the second packet obtained by the conversion at a constant bit rate, a third buffer that converts the output second packet into a NAL unit, and outputs the NAL unit obtained by the conversion at a constant bit rate, and a fourth buffer that sequentially stores the output NAL units, generates an access unit from the stored NAL units, and outputs the generated access unit to the decoder at the timing of the decoding time corresponding to the access unit, wherein the encoded stream is a stream containing NAL units in NAL size format with a 4-byte size area appended to the beginning.

[0018] Furthermore, a receiving device according to one aspect of the present invention includes a receiving unit that receives a transmission packet in which the encoded stream is packetized, and which consists of a fixed-length packet header and a variable-length payload; a first buffer that converts the packet consisting of a variable-length packet header and a variable-length payload stored in the received transmission packet into a first packet having a fixed-length packet header with the header expanded, and outputs the first packet obtained by the conversion at a constant bit rate; and converts the output first packet into a second packet consisting of a packet header and a variable-length payload, and outputs the second packet obtained by the conversion at a constant bit rate. The system comprises a second buffer that outputs at a fixed bitrate, a third buffer that converts the output second packet into a NAL unit and outputs the resulting NAL unit at a fixed bitrate, a fourth buffer that sequentially stores the output NAL units, generates an access unit from the stored NAL units, and outputs the generated access unit at the timing of the decoding time corresponding to the access unit, and a decoder that decodes the access unit output by the fourth buffer, wherein the encoded stream is a stream containing NAL units in a NAL size format with a 4-byte size area appended to the beginning.

[0019] To achieve the above objective, a transmitting device according to one aspect of the present invention comprises: a generation unit that generates an encoded stream by performing rate control to satisfy the requirements of a predetermined receive buffer model in order to guarantee the buffer operation of the receiving device; and a transmitting unit that packets the generated encoded stream and transmits the transmission packets obtained by packetization, wherein the transmission packets consist of a fixed-length packet header and a variable-length payload, and the receive buffer model receives the transmission packets and converts the packets consisting of a variable-length packet header and a variable-length payload stored in the received transmission packets into first packets having a fixed-length packet header with the header expanded, by converting them The system includes: a first buffer that outputs the obtained first packet at a constant bitrate; a second buffer that converts the output first packet into a second packet consisting of a packet header and a variable-length payload, and outputs the obtained second packet at a constant bitrate; a third buffer that converts the output second packet into a NAL unit, and outputs the obtained NAL unit at a constant bitrate; and a fourth buffer that sequentially stores the output NAL units, generates an access unit from the stored NAL units, and outputs the generated access unit to the decoder at the timing of the decoding time corresponding to the access unit.

[0020] These general or specific embodiments may be implemented as a system, device, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of a system, device, integrated circuit, computer program, and recording medium. [Effects of the Invention]

[0021] This invention can guarantee the buffer operation of the receiving device when transmitting data using a method such as MMT. [Brief explanation of the drawing]

[0022] [Figure 1] Figure 1 is a diagram showing an example of dividing a picture into slice segments. [Figure 2] Figure 2 is a diagram showing an example of a PES packet sequence in which picture data is stored. [Figure 3] Figure 3 is a diagram showing an example of dividing a picture according to Embodiment 1. [Figure 4] Figure 4 is a diagram showing an example of dividing a picture according to the comparative example of Embodiment 1. [Figure 5] Figure 5 is a diagram showing an example of data of an access unit according to Embodiment 1. [Figure 6] Figure 6 is a block diagram of a transmission device according to Embodiment 1. [Figure 7] Figure 7 is a block diagram of a receiving device according to Embodiment 1. [Figure 8] Figure 8 is a diagram showing an example of an MMT packet according to Embodiment 1. [Figure 9] Figure 9 is a diagram showing another example of an MMT packet according to Embodiment 1. [Figure 10] Figure 10 is a diagram showing an example of data input to each decoding unit according to Embodiment 1. [Figure 11] Figure 11 is a diagram showing an example of an MMT packet and header information according to Embodiment 1. [Figure 12] Figure 12 is a diagram showing another example of data input to each decoding unit according to Embodiment 1. [Figure 13] Figure 13 is a diagram showing an example of dividing a picture according to Embodiment 1. [Figure 14] Figure 14 is a flowchart of a transmission method according to Embodiment 1. [Figure 15] Figure 15 is a block diagram of a receiving device according to Embodiment 1. [Figure 16] Figure 16 is a flowchart of a receiving method according to Embodiment 1. [Figure 17] Figure 17 is a diagram showing an example of an MMT packet and header information according to Embodiment 1. [Figure 18] Figure 18 shows an example of an MMT packet and header information according to Embodiment 1. [Figure 19] Figure 19 shows the configuration of the MPU. [Figure 20] Figure 20 shows the structure of MF metadata. [Figure 21] Figure 21 is a diagram illustrating the order in which data is transmitted. [Figure 22] Figure 22 shows an example of a method for decryption without using header information. [Figure 23] Figure 23 is a block diagram of the transmitting device according to Embodiment 2. [Figure 24] Figure 24 is a flowchart of the transmission method according to Embodiment 2. [Figure 25] Figure 25 is a block diagram of the receiving device according to Embodiment 2. [Figure 26] Figure 26 is a flowchart of the operation for identifying the MPU starting position and the NAL unit position. [Figure 27] Figure 27 is a flowchart illustrating the operation of obtaining initialization information based on the transmission order type and decoding media data based on the initialization information. [Figure 28] Figure 28 is a flowchart showing the operation of the receiving device when a low-latency presentation mode is provided. [Figure 29] Figure 29 shows an example of the transmission order of MMT packets when auxiliary data is transmitted. [Figure 30] Figure 30 illustrates an example in which a transmitter generates auxiliary data based on the MOOF configuration. [Figure 31] Figure 31 is a diagram illustrating the reception of auxiliary data. [Figure 32] Figure 32 is a flowchart of the receiving operation using auxiliary data. [Figure 33] Figure 33 shows the configuration of an MPU, which is composed of multiple movie fragments. [Figure 34]Figure 34 is a diagram illustrating the transmission order of MMT packets when the MPU configuration shown in Figure 33 is transmitted. [Figure 35] Figure 35 is the first diagram illustrating an example of the operation of a receiver when one MPU is composed of multiple movie fragments. [Figure 36] Figure 36 is a second diagram illustrating an example of the operation of a receiver when one MPU is composed of multiple movie fragments. [Figure 37] Figure 37 is a flowchart illustrating the operation of the receiving method described in Figures 35 and 36. [Figure 38] Figure 38 shows a case where non-VCL NAL units are treated as individual data units and aggregated. [Figure 39] Figure 39 shows a case where non-VCL NAL units are grouped together to form a data unit. [Figure 40] Figure 40 is a flowchart showing the operation of the receiving device when packet loss occurs. [Figure 41] Figure 41 is a flowchart of the receiving operation when the MPU is divided into multiple movie fragments. [Figure 42] Figure 42 shows an example of the picture prediction structure for each TemporalId when achieving temporal scalability. [Figure 43] Figure 43 shows the relationship between the decoded time (DTS) and the displayed time (PTS) for each picture in Figure 42. [Figure 44] Figure 44 shows an example of a picture prediction structure that requires picture delay processing and reordering processing. [Figure 45] Figure 45 shows an example where an MPU composed in MP4 format is divided into multiple movie fragments and stored in an MMTP payload and MMTP packets. [Figure 46] Figure 46 is a diagram illustrating the calculation methods and challenges of PTS and DTS. [Figure 47]Figure 47 is a flowchart of the receiving operation when the DTS is calculated using information for DTS calculation. [Figure 48] Figure 48 is a diagram illustrating how data units are stored in the payload in MMT. [Figure 49] Figure 49 shows the operation flow of the transmitting device according to Embodiment 3. [Figure 50] Figure 50 shows the operation flow of the receiving device according to Embodiment 3. [Figure 51] Figure 51 is a diagram showing an example of a specific configuration of the transmitting device according to Embodiment 3. [Figure 52] Figure 52 is a diagram showing an example of a specific configuration of the receiving device according to Embodiment 3. [Figure 53] Figure 53 shows the method for storing non-timed media in the MPU and the method for transmitting it using MMTP packets. [Figure 54] Figure 54 shows an example of packetizing and transmitting each of the multiple data segments obtained by splitting a file. [Figure 55] Figure 55 shows another example where each of the multiple segmented data obtained by splitting a file is packetized and transmitted. [Figure 56] Figure 56 shows the syntax for a file-by-file loop in the asset management table. [Figure 57] Figure 57 shows the operation flow for identifying the segmented data number in the receiving device. [Figure 58] Figure 58 shows the operation flow for determining the number of divided data segments in the receiving device. [Figure 59] Figure 59 shows the operation flow for determining whether or not to operate a fragment counter in the transmitting device. [Figure 60] Figure 60 is a diagram illustrating how to identify the number of divided data and the divided data number (when using a fragment counter). [Figure 61] Figure 61 shows the operation flow of the transmitting device when a fragment counter is used. [Figure 62] Figure 62 shows the operation flow of the receiving device when a fragment counter is used. [Figure 63] Figure 63 shows the service configuration when the same program is transmitted using multiple IP data flows. [Figure 64] Figure 64 shows an example of a specific configuration of a transmitting device. [Figure 65] Figure 65 shows an example of a specific configuration of a receiving device. [Figure 66] Figure 66 shows the operation flow of the transmitting device. [Figure 67] Figure 67 shows the operation flow of the receiving device. [Figure 68] Figure 68 shows the receive buffer model based on the receive buffer model specified in ARIB STD B-60, specifically for the case where only broadcast transmission lines are used. [Figure 69] Figure 69 shows an example of aggregating multiple data units and storing them in a single payload. [Figure 70] Figure 70 shows an example of aggregating multiple data units and storing them in a single payload, specifically an example where a video signal in NAL size format is treated as a single data unit. [Figure 71] Figure 71 shows the payload structure of an MMTP packet where the data unit length is not indicated. [Figure 72] Figure 72 shows an example of the extend area assigned to each packet. [Figure 73] Figure 73 shows the operation flow of the receiving device. [Figure 74] Figure 74 shows an example of a specific configuration of a transmitting device. [Figure 75] Figure 75 shows an example of a specific configuration of a receiving device. [Figure 76] Figure 76 shows the operation flow of the transmitting device. [Figure 77] Figure 77 shows the operation flow of the receiving device. [Modes for carrying out the invention]

[0023] A transmitting device according to one aspect of the present invention comprises a generation unit that generates an encoded stream by performing rate control to satisfy the requirements of a predetermined receive buffer model in order to guarantee the buffer operation of a receiving device, and a transmitting unit that packets the generated encoded stream and transmits the transmission packets obtained by packetization, wherein the transmission packets consist of a fixed-length packet header and a variable-length payload, and the receive buffer model receives the transmission packets and converts the packets consisting of a variable-length packet header and a variable-length payload stored in the received transmission packets into first packets having a fixed-length packet header with the header expanded, and the transmission device obtained by conversion The system includes: a first buffer that outputs the first packet at a constant bitrate; a second buffer that converts the output first packet into a second packet consisting of a packet header and a variable-length payload, and outputs the second packet obtained by the conversion at a constant bitrate; a third buffer that converts the output second packet into a NAL unit, and outputs the NAL unit obtained by the conversion at a constant bitrate; and a fourth buffer that sequentially stores the output NAL units, generates an access unit from the stored NAL units, and outputs the generated access unit to the decoder at the timing of the decoding time corresponding to the access unit.

[0024] Such a transmitting device can guarantee the buffer operation of the receiving device when transmitting data using a method such as MMT.

[0025] Furthermore, for example, in the receiving buffer model, the bitrate at which the first buffer outputs the first packet may be set according to the transmission rate after the packet header of the first packet has been decompressed.

[0026] Furthermore, for example, the encoded stream may be a stream obtained by storing NAL units in a multiplexing layer in a NAL size format with a 4-byte size area appended to the beginning.

[0027] Furthermore, a receiving device according to one aspect of the present invention includes: a receiving unit that receives a transmission packet consisting of a fixed-length packet header and a variable-length payload; a first buffer that converts the packet consisting of a variable-length packet header and a variable-length payload stored in the received transmission packet into a first packet having a fixed-length packet header with the header expanded, and outputs the first packet obtained by the conversion at a constant bit rate; a second buffer that converts the output first packet into a second packet consisting of a packet header and a variable-length payload, and outputs the second packet obtained by the conversion at a constant bit rate; a third buffer that converts the output second packet into a NAL unit, and outputs the NAL unit obtained by the conversion at a constant bit rate; a fourth buffer that sequentially stores the output NAL units, generates an access unit from the stored NAL units, and outputs the generated access unit at the timing of the decoding time corresponding to the access unit; and a decoder that decodes the access unit output by the fourth buffer.

[0028] Such a receiving device can perform decoding operations that avoid underflow or overflow.

[0029] A transmission method according to one aspect of the present invention includes a generation step of generating an encoded stream by performing rate control to satisfy the requirements of a predetermined receive buffer model in order to guarantee the buffer operation of a receiving device, and a transmission step of packetizing the generated encoded stream and transmitting the transmission packets obtained by packetizing, wherein the transmission packets consist of a fixed-length packet header and a variable-length payload, and the receive buffer model receives the transmission packets and converts the packets consisting of a variable-length packet header and a variable-length payload stored in the received transmission packets into first packets having a fixed-length packet header with the header expanded, and obtains by conversion The system includes: a first buffer that outputs the output first packet at a constant bitrate; a second buffer that converts the output first packet into a second packet consisting of a packet header and a variable-length payload, and outputs the second packet obtained by the conversion at a constant bitrate; a third buffer that converts the output second packet into a NAL unit, and outputs the NAL unit obtained by the conversion at a constant bitrate; and a fourth buffer that sequentially stores the output NAL units, generates an access unit from the stored NAL units, and outputs the generated access unit to the decoder at the timing of the decoding time corresponding to the access unit.

[0030] A receiving method according to one aspect of the present invention includes: a receiving step of receiving a transmission packet consisting of a fixed-length packet header and a variable-length payload; a first conversion step of converting the packet consisting of a variable-length packet header and a variable-length payload stored in the received transmission packet into a first packet having a fixed-length packet header with the header expanded, and outputting the first packet obtained by the conversion at a constant bit rate; a second conversion step of converting the output first packet into a second packet consisting of a packet header and a variable-length payload, and outputting the second packet obtained by the conversion at a constant bit rate; a third conversion step of converting the output second packet into a NAL unit, and outputting the NAL unit obtained by the conversion at a constant bit rate; a generation step of sequentially accumulating the output NAL units, generating an access unit from a plurality of accumulated NAL units, and outputting the generated access unit to a decoder at the timing of the decoding time corresponding to the access unit; and a decoding step of decoding the access unit output by the fourth buffer.

[0031] These comprehensive or specific embodiments may be implemented as systems, devices, integrated circuits, computer programs, or recording media such as computer-readable CD-ROMs, or as any combination of systems, devices, integrated circuits, computer programs, or recording media.

[0032] The embodiments will be described in detail below with reference to the drawings.

[0033] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples only and are not intended to limit the present invention. Furthermore, among the components in the following embodiments, those not described in the independent claim representing the highest-level concept will be described as optional components.

[0034] (Knowledge that formed the basis of this invention) In recent years, the resolution of displays in devices such as TVs, smartphones, and tablet devices has been increasing. In particular, 8K4K (8K x 4K resolution) services are planned for broadcasting in Japan in 2020. For ultra-high-resolution video such as 8K4K, real-time decoding is difficult with a single decoder, so methods that use multiple decoders to perform decoding in parallel are being considered.

[0035] Since encoded data is transmitted after being multiplexed based on multiplexing schemes such as MPEG-2 TS and MMT, the receiving device needs to separate the encoded video data from the multiplexed data before decoding. Hereafter, the process of separating the encoded data from the multiplexed data will be referred to as demultiplexing.

[0036] When parallelizing the decoding process, it is necessary to distribute the encoded data to be decoded to each decoder. Distributing the encoded data requires analyzing the encoded data itself, and especially with 8K content, the bitrate is very high, resulting in a heavy processing load for the analysis. Therefore, the demultiplexing process becomes a bottleneck, preventing real-time playback.

[0037] Incidentally, in video encoding schemes such as H.264 and H.265, standardized by MPEG and the ITU, the transmitting device can divide the picture into multiple areas called slices or slice segments, and encode each divided area so that it can be decoded independently. Therefore, for example, in the case of H.265, the receiving device that receives the broadcast can separate the data for each slice segment from the received data and output the data for each slice segment to separate decoders, thereby achieving parallelization of the decoding process.

[0038] Figure 1 shows an example of how HEVC divides one picture into four slice segments. For example, the receiving device has four decoders, and each decoder decodes one of the four slice segments.

[0039] In conventional broadcasting, the transmitting device stores one picture (an access unit in the MPEG system standard) in one PES packet and multiplexes the PES packets into a sequence of TS packets. Therefore, the receiving device had to separate the payload of the PES packets, analyze the data of the access units stored in the payload to separate each slice segment, and output the data of each separated slice segment to the decoder.

[0040] However, the inventors have found that there is a problem in that it is difficult to perform this process in real time because the amount of processing required to analyze the data of the access unit and separate the slice segments is large.

[0041] Figure 2 shows an example where picture data divided into slice segments is stored in the payload of a PES packet.

[0042] As shown in Figure 2, for example, data from multiple slice segments (slice segments 1 to 4) is stored in the payload of a single PES packet. Furthermore, the PES packet is multiplexed into a sequence of TS packets.

[0043] (Embodiment 1) The following explanation uses H.265 as the encoding scheme for video as an example, but this embodiment can also be applied when using other encoding schemes such as H.264.

[0044] Figure 3 shows an example of how an access unit (picture) in this embodiment is divided into segments. The access unit is divided into two equal parts horizontally and vertically, resulting in a total of four tiles, by a feature called tiling, which was introduced by H.265. Furthermore, there is a one-to-one correspondence between the slice segments and the tiles.

[0045] Let's explain why the data is divided into two equal parts horizontally and vertically. First, during decoding, line memory is generally required to store data for one horizontal line. However, with ultra-high resolutions such as 8K and 4K, the horizontal size increases, thus increasing the size of the line memory. In the implementation of receiving devices, it is desirable to reduce the size of the line memory. To reduce the size of the line memory, vertical division is necessary. Vertical division requires a data structure called a tile. For these reasons, tiles are used.

[0046] On the other hand, since images generally have high horizontal correlation, encoding efficiency improves when a wider range can be referenced horizontally. Therefore, from the standpoint of encoding efficiency, it is desirable for the access units to be divided horizontally.

[0047] By dividing the access unit into two equal parts horizontally and vertically, these two characteristics can be reconciled, taking into account both implementation aspects and coding efficiency. If a single decoder can decode 4K2K video in real time, then by dividing an 8K4K image into four equal parts, with each slice segment being 4K2K, the receiver can decode the 8K4K image in real time.

[0048] Next, we will explain why the tiles and slice segments obtained by dividing the access unit horizontally and vertically are mapped one-to-one. In H.265, an access unit consists of multiple units called NAL (Network Adaptation Layer) units.

[0049] The payload of a NAL unit stores one of the following: an access unit delimiter indicating the starting position of the access unit; SPS (Sequence Parameter Set), which is initialization information used commonly at the sequence level during decoding; PPS (Picture Parameter Set), which is initialization information used commonly within a picture during decoding; SEI (Supplemental Enhancement Information), which is not required for the decoding process itself but is necessary for processing and displaying the decoding results; and encoded data of the slice segment. The header of the NAL unit contains type information to identify the data stored in the payload.

[0050] Here, the transmitting device encodes the data in MPEG-2 TS, MMT (MPEG Media Transport), and MPEG DASH (Dynamic Adaptive When multiplexing using multiplexing formats such as Streaming over HTTP or RTP (Real-time Transport Protocol), the basic unit can be set to a NAL unit. In order to store one slice segment in one NAL unit, it is desirable to divide the access unit into slice segment units when dividing the area. For this reason, the transmitting device establishes a one-to-one correspondence between tiles and slice segments.

[0051] As shown in Figure 4, the transmitting device can also configure tiles 1 through 4 together as a single slice segment. However, in this case, all tiles are stored in a single NAL unit, making it difficult for the receiving device to separate the tiles in the multiplexing layer.

[0052] Note that there are two types of slice segments: independent slice segments that can be decoded independently, and reference slice segments that refer to independent slice segments. Here, we will explain the case where independent slice segments are used.

[0053] Figure 5 shows an example of access unit data divided so that the boundaries between tiles and slice segments coincide, as shown in Figure 3. The access unit data includes the NAL unit that stores the access unit delimiter placed at the beginning, the NAL units of the SPS, PPS, and SEI placed thereafter, and the slice segment data that stores the data for tiles 1 to 4 placed thereafter. Note that the access unit data does not have to include some or all of the NAL units of the SPS, PPS, and SEI.

[0054] Next, the configuration of the transmitting device 100 according to this embodiment will be described. Figure 6 is a block diagram showing an example of the configuration of the transmitting device 100 according to this embodiment. This transmitting device 100 comprises an encoding unit 101, a multiplexing unit 102, a modulation unit 103, and a transmitting unit 104.

[0055] The encoding unit 101 generates encoded data by encoding the input image, for example, according to H.265. Furthermore, the encoding unit 101 divides the access unit into four slice segments (tiles), as shown in Figure 3, and encodes each slice segment.

[0056] The multiplexing unit 102 multiplexes the encoded data generated by the encoding unit 101. The modulation unit 103 modulates the data obtained through multiplexing. The transmission unit 104 transmits the modulated data as a broadcast signal.

[0057] Next, the configuration of the receiving device 200 according to this embodiment will be described. Figure 7 is a block diagram showing an example of the configuration of the receiving device 200 according to this embodiment. This receiving device 200 includes a tuner 201, a demodulation unit 202, a demultiplexing unit 203, a plurality of decoding units 204A to 204D, and a display unit 205.

[0058] The tuner 201 receives the broadcast signal. The demodulation unit 202 demodulates the received broadcast signal. The demodulated data is input to the demultiplexing unit 203.

[0059] The demultiplexing unit 203 separates the demodulated data into division units and outputs the data for each division unit to the decoding units 204A to 204D. Here, a division unit is a divided region obtained by dividing the access unit, for example, a slice segment in H.265. In this case, an 8K4K image is divided into four 4K2K images. Therefore, there are four decoding units 204A to 204D.

[0060] Multiple decoding units 204A to 204D operate synchronously with each other based on a predetermined reference clock. Each decoding unit decodes the encoded data in division units according to the access unit's DTS (Decoding Time Stamp) and outputs the decoding result to the display unit 205.

[0061] The display unit 205 generates an 8K4K output image by integrating multiple decoding results output from multiple decoding units 204A to 204D. The display unit 205 displays the generated output image according to the PTS (Presentation Time Stamp) of the access unit, which is acquired separately. When integrating the decoding results, the display unit 205 may perform filtering such as a deblocking filter on the boundary areas of adjacent division units, such as tile boundaries, so that the boundaries become less noticeable.

[0062] In the above explanation, a transmitting device 100 and a receiving device 200 that transmit or receive broadcasts were used as examples, but the content may also be transmitted and received via a communication network. When the receiving device 200 receives content via a communication network, the receiving device 200 separates the multiplexed data from the IP packets received via a network such as Ethernet.

[0063] In broadcasting, the transmission path delay between the transmission of the broadcast signal and its arrival at the receiving device 200 is constant. On the other hand, in communication networks such as the Internet, the transmission path delay between the data transmitted from the server and its arrival at the receiving device 200 is not constant due to congestion. Therefore, the receiving device 200 often does not perform strict synchronized playback based on a reference clock, such as PCR in broadcast MPEG-2 TS. For this reason, the receiving device 200 may display the 8K4K output image according to the PTS in the display unit without strictly synchronizing each decoding unit.

[0064] Furthermore, due to network congestion or other factors, the decoding process for all segment units may not be completed by the time indicated on the access unit's PTS. In this case, the receiving device 200 either skips the access unit's display or delays the display until decoding of at least four segment units is complete and the generation of the 8K4K image is finished.

[0065] Furthermore, content may be transmitted and received using both broadcasting and telecommunications. This method can also be applied when playing back multiplexed data stored on recording media such as hard disks or memory.

[0066] Next, we will explain the method of multiplexing access units divided into slice segments when MMT is used as the multiplexing scheme.

[0067] Figure 8 shows an example of packetizing HEVC access unit data into MMT. SPS, PPS, and SEI do not necessarily need to be included in the access unit, but this example illustrates the case where they are present.

[0068] NAL units located before the first slice segment within an access unit, such as the access unit delimiter, SPS, PPS, and SEI, are bundled together and stored in MMT packet #1. Subsequent slice segments are stored in separate MMT packets, one for each slice segment.

[0069] As shown in Figure 9, NAL units located before the first slice segment within the access unit may be stored in the same MMT packet as the first slice segment.

[0070] Furthermore, if NAL units such as End-of-Sequence or End-of-Bitstream, which indicate the end of a sequence or stream, are appended after the final slice segment, these are stored in the same MMT packet as the final slice segment. However, since NAL units such as End-of-Sequence or End-of-Bitstream are inserted at the end point of the decoding process or at the connection point of two streams, it may be desirable for the receiving device 200 to be able to easily acquire these NAL units at the multiplexing layer. In this case, these NAL units may be stored in a separate MMT packet from the slice segment. This allows the receiving device 200 to easily separate these NAL units at the multiplexing layer.

[0071] Furthermore, TS, DASH, or RTP may be used as the multiplexing method. In these methods as well, the transmitting device 100 stores different slice segments in different packets. This ensures that the receiving device 200 can separate the slice segments at the multiplexing layer.

[0072] For example, when TS is used, encoded data is packetized into PES packets on a slice segment basis. When RTP is used, encoded data is packetized into RTP packets on a slice segment basis. In these cases as well, the NAL unit and the slice segment, which are placed before the slice segment, may be packetized separately, as shown in MMT packet #1 in Figure 8.

[0073] When TS is used, the transmitting device 100 indicates the unit of data to be stored in the PES packet by using a data alignment descriptor, etc. Also, since DASH is a method that downloads MP4 data units called segments via HTTP, etc., the transmitting device 100 does not packetize the encoded data when transmitting. For this reason, the transmitting device 100 may create subsamples for each slice segment so that the receiving device 200 can detect slice segments at the multiplexing layer in MP4, and store information indicating the storage location of the subsamples in the MP4 header.

[0074] The following provides a detailed explanation of MMT packetization of slice segments.

[0075] As shown in Figure 8, the encoded data is packetized, and data that is commonly referenced when decoding all slice segments within access units such as SPS and PPS is stored in MMT packet #1. In this case, the receiving device 200 concatenates the payload data of MMT packet #1 with the data of each slice segment and outputs the resulting data to the decoding unit. In this way, the receiving device 200 can easily generate input data for the decoding unit by concatenating the payloads of multiple MMT packets.

[0076] Figure 10 shows an example of how input data is generated from the MMT packets shown in Figure 8 for the decoding units 204A to 204D. The demultiplexing unit 203 generates the data necessary for decoding unit 204A to decode slice segment 1 by concatenating the payload data of MMT packet #1 and MMT packet #2. The demultiplexing unit 203 similarly generates input data for decoding units 204B to 204D. That is, the demultiplexing unit 203 generates the input data for decoding unit 204B by concatenating the payload data of MMT packet #1 and MMT packet #3. The demultiplexing unit 203 generates the input data for decoding unit 204C by concatenating the payload data of MMT packet #1 and MMT packet #4. The demultiplexing unit 203 generates the input data for decoding unit 204D by concatenating the payload data of MMT packet #1 and MMT packet #5.

[0077] The demultiplexing unit 203 may remove NAL units that are not necessary for the decoding process, such as the access unit delimiter and SEI, from the payload data of MMT packet #1, and separate only the SPS and PPS NAL units that are necessary for the decoding process and add them to the slice segment data.

[0078] As shown in Figure 9, when encoded data is packetized, the demultiplexing unit 203 outputs MMT packet #1, which contains the leading data of the access unit, to the first decoding unit 204A at the multiplexing layer. The demultiplexing unit 203 also analyzes the MMT packet containing the leading data of the access unit at the multiplexing layer, separates the SPS and PPS NAL units, and adds the separated SPS and PPS NAL units to the data of the second and subsequent slice segments to generate input data for each of the second and subsequent decoding units.

[0079] Furthermore, it is desirable that the receiving device 200 can use the information contained in the header of the MMT packet to identify the type of data stored in the MMT payload, and, if a slice segment is stored in the payload, the index number of that slice segment within the access unit. Here, the data type refers to either pre-sliced-segment data (a collective term for NAL units that are placed before the first slice segment within the access unit) or slice segment data. When storing fragmented units of an MPU, such as a slice segment, in an MMT packet, a mode for storing an MFU (Media Fragment Unit) is used. When using this mode, the transmitting device 100 can, for example, set the Data Unit, which is the basic unit of data in an MFU, to a sample (a data unit in MMT, corresponding to an access unit) or a subsample (a unit obtained by dividing a sample).

[0080] In this case, the header of the MMT packet includes a field called the Fragmentation indicator and a field called the Fragment counter.

[0081] The Fragmentation indicator indicates whether the data stored in the payload of an MMT packet is a fragmented Data unit, and if so, whether the fragment is the first or last fragment in the Data unit, or neither the first nor the last. In other words, the Fragmentation indicator included in the header of a packet is identification information that indicates whether (1) the packet is the only Data unit, which is the basic data unit; (2) the Data unit is divided into multiple packets and the packet in question is the first packet of the Data unit; (3) the Data unit is divided into multiple packets and the packet in question is a packet other than the first or last packet of the Data unit; or (4) the Data unit is divided into multiple packets and the packet in question is the last packet of the Data unit.

[0082] The Fragment counter is an index number that indicates which fragment in the Data unit the data stored in the MMT packet corresponds to.

[0083] Therefore, by setting the samples in the MMT as Data units, and setting the pre-slice segment data and each slice segment as fragment units of Data units, the receiving device 200 can identify the type of data to be stored in the payload using the information contained in the header of the MMT packet. In other words, the demultiplexing unit 203 can generate input data for each decoding unit 204A to 204D by referring to the header of the MMT packet.

[0084] Figure 11 shows an example where a sample is set in a Data unit, and the pre-slice segment data and slice segment are packetized as fragments of the Data unit.

[0085] The data before the slice segment, and the slice segment itself, are divided into five fragments, from Fragment #1 to Fragment #5. Each fragment is stored in a separate MMT packet. The values ​​of the Fragmentation indicator and Fragment counter included in the header of the MMT packet are as shown in the diagram.

[0086] For example, the Fragment indicator is a 2-bit binary value. The Fragment indicators for MMT packet #1, the first packet in a Data unit, MMT packet #5, the last packet in the unit, and MMT packets #2 through #4 in between are each set to different values. Specifically, the Fragment indicator for MMT packet #1 is set to 01, the Fragment indicator for MMT packet #5 is set to 11, and the Fragment indicators for MMT packets #2 through #4 are set to 10. If a Data unit contains only one MMT packet, the Fragment indicator is set to 00.

[0087] Furthermore, the Fragment counter is 4 in MMT packet #1, which is the total number of fragments (5) minus 1. It decreases by 1 in subsequent packets, and is 0 in the final MMT packet #5.

[0088] Therefore, the receiving device 200 can identify an MMT packet containing pre-slice segment data using either the Fragment indicator or the Fragment counter. Furthermore, the receiving device 200 can identify an MMT packet containing the Nth slice segment by referring to the Fragment counter.

[0089] The header of the MMT packet separately includes the sequence number within the MPU of the Movie Fragment to which the Data unit belongs, the sequence number of the MPU itself, and the sequence number within the Movie Fragment of the sample to which the Data unit belongs. The demultiplexer 203 can uniquely determine the sample to which the Data unit belongs by referring to these.

[0090] Furthermore, the demultiplexing unit 203 can determine the index number of the fragment within the Data unit from the Fragment counter, etc., so even if packet loss occurs, it can uniquely identify the slice segment stored in the fragment. For example, even if fragment #4 shown in Figure 11 could not be obtained due to packet loss, the demultiplexing unit 203 can determine that the fragment received after fragment #3 is fragment #5, and can therefore correctly output the slice segment 4 stored in fragment #5 to the decoding unit 204D instead of the decoding unit 204C.

[0091] Furthermore, if a transmission path is used that guarantees no packet loss, the demultiplexing unit 203 does not need to refer to the header of the MMT packet to determine the type of data stored in the MMT packet or the index number of the slice segment, but can process the arriving packets periodically. For example, if an access unit transmits using a total of five MMT packets—pre-slicing data and four slice segments—the receiving device 200 can obtain the pre-slicing data and the data of the four slice segments sequentially by processing the received MMT packets sequentially after determining the pre-slicing data of the access unit to begin decoding.

[0092] The following describes some variations of packetization.

[0093] The slice segments do not necessarily have to divide the access unit both horizontally and vertically within its plane; as shown in Figure 1, the access unit may be divided only horizontally or only vertically.

[0094] Furthermore, if the access unit is divided only horizontally, tiles do not need to be used.

[0095] Furthermore, the number of in-plane divisions in an access unit is arbitrary and not limited to four. However, the area size of slice segments and tiles must be greater than or equal to the lower limit of the encoding standard such as H.265.

[0096] The transmitting device 100 may store identification information indicating the method of dividing the access unit in the plane in an MMT message or a TS descriptor. For example, information indicating the number of divisions in the horizontal and vertical directions in the plane may be stored. Alternatively, unique identification information may be assigned to the division method, such as being divided into two equal parts in the horizontal and vertical directions as shown in Figure 3, or being divided into four equal parts in the horizontal direction as shown in Figure 1. For example, if the access unit is divided as shown in Figure 3, the identification information indicates mode 1, and if the access unit is divided as shown in Figure 1, the identification information indicates mode 1.

[0097] Furthermore, the multiplexing layer may include information indicating constraints on the encoding conditions related to the method of dividing the plane. For example, information indicating that one slice segment consists of one tile may be used. Alternatively, information indicating that the reference block used when motion compensation is performed during decoding of a slice segment or tile is limited to slice segments or tiles at the same position in the screen, or limited to blocks within a predetermined range in adjacent slice segments may be used.

[0098] Furthermore, the transmitting device 100 may switch whether or not to divide the access unit into multiple slice segments depending on the resolution of the video. For example, the transmitting device 100 may not perform in-plane division when the video to be processed has a resolution of 4K2K, but may divide the access unit into four when the video to be processed has a resolution of 8K4K. By pre-defining the division method for 8K4K video, the receiving device 200 can determine whether or not to perform in-plane division and the division method by acquiring the resolution of the video to be received, and switch the decoding operation accordingly.

[0099] Furthermore, the receiving device 200 can detect whether or not in-plane fragmentation has occurred by referring to the header of the MMT packet. For example, if the access unit is not fragmented, and the MMT Data unit is set to sample, then no fragmentation of the Data unit will occur. Therefore, the receiving device 200 can determine that the access unit is not fragmented if the value of the Fragment counter included in the header of the MMT packet is always zero. Alternatively, the receiving device 200 may detect whether the value of the Fragmentation indicator is always 01. The receiving device 200 can also determine that the access unit is not fragmented if the value of the Fragmentation indicator is always 01.

[0100] Furthermore, the receiving device 200 can also handle cases where the number of in-plane divisions in the access unit does not match the number of decoding units. For example, if the receiving device 200 has two decoding units 204A and 204B that can decode 8K2K encoded data in real time, the demultiplexing unit 203 outputs two of the four slice segments that make up the 8K4K encoded data to the decoding unit 204A.

[0101] Figure 12 shows an example of operation when MMT packetized data, as shown in Figure 8, is input to two decoding units 204A and 204B. Here, it is desirable that the receiving device 200 can directly integrate and output the decoding results from decoding units 204A and 204B. Therefore, the demultiplexing unit 203 selects the slice segments to output to decoding units 204A and 204B so that the decoding results of each decoding unit 204A and 204B are spatially continuous.

[0102] Furthermore, the demultiplexing unit 203 may select the decoding unit to use depending on the resolution or frame rate of the encoded video data. For example, if the receiving device 200 has four 4K2K decoding units, the receiving device 200 will use all four decoding units to perform decoding if the resolution of the input image is 8K4K. Alternatively, if the resolution of the input image is 4K2K, the receiving device 200 will use only one decoding unit to perform decoding. Or, even if the image is divided into four parts, if 8K4K can be decoded in real time by a single decoding unit, the demultiplexing unit 203 will integrate all the division units and output them to a single decoding unit.

[0103] Furthermore, the receiving device 200 may determine which decoding unit to use by considering the frame rate. For example, if the receiving device 200 has two decoding units, each with an upper limit of 60fps that can be decoded in real time when the resolution is 8K4K, there may be cases where encoded data at 120fps in 8K4K is input. In this case, if the plane is composed of four division units, then, as in the example in Figure 12, slice segment 1 and slice segment 2 are input to decoding unit 204A, and slice segment 3 and slice segment 4 are input to decoding unit 204B. Since each decoding unit 204A and 204B can decode up to 120fps in real time if the resolution is 8K2K (half the resolution of 8K4K), the decoding process is performed by these two decoding units 204A and 204B.

[0104] Furthermore, even if the resolution and frame rate are the same, the amount of processing will differ depending on the encoding profile or level, or the encoding method itself, such as H.264 or H.265. Therefore, the receiving device 200 may select the decoding unit to use based on this information. If the receiving device 200 cannot decode all of the encoded data received by broadcast or communication, or if all slice segments or tiles constituting the user-selected area cannot be decoded, the receiving device 200 may automatically determine the slice segments or tiles that can be decoded within the processing range of the decoding unit. Alternatively, the receiving device 200 may provide a user interface for the user to select the area to decode. In this case, the receiving device 200 may display a warning message indicating that not all areas can be decoded, or it may display information indicating the number of decodeable areas, slice segments, or tiles.

[0105] Furthermore, the above method can also be applied when MMT packets containing slice segments of the same encoded data are transmitted and received using multiple transmission paths, such as broadcasting and telecommunications.

[0106] Furthermore, the transmitting device 100 may perform encoding so that the regions of each slice segment overlap in order to make the boundaries between division units less noticeable. In the example shown in Figure 13, an 8K4K picture is divided into four slice segments 1 to 4. Each of slice segments 1 to 3 is, for example, 8K × 1.1K, and slice segment 4 is 8K × 1K. Also, adjacent slice segments overlap each other. In this way, motion compensation during encoding can be efficiently performed at the boundaries of the four divisions shown by the dotted lines, thus improving the image quality at the boundaries. In this way, image quality degradation at the boundaries is reduced.

[0107] In this case, the display unit 205 extracts an 8K × 1K region from the 8K × 1.1K region and integrates the resulting regions. The transmitting device 100 may also include information indicating whether the slice segments are encoded with overlap and the extent of the overlap in the multiplexing layer or within the encoded data and transmit it separately.

[0108] Furthermore, the same method can be applied when tiles are used.

[0109] The following describes the operation flow of the transmitting device 100. Figure 14 is a flowchart showing an example of the operation of the transmitting device 100.

[0110] First, the encoding unit 101 divides the picture (access unit) into multiple slice segments (tiles), which are multiple regions (S101). Next, the encoding unit 101 encodes each of the multiple slice segments in a way that allows for independent decoding, thereby generating encoded data corresponding to each of the multiple slice segments (S102). The encoding unit 101 may encode the multiple slice segments with a single encoding unit, or it may process them in parallel with multiple encoding units.

[0111] Next, the multiplexing unit 102 multiplexes the multiple encoded data generated by the encoding unit 101 by storing them in multiple MMT packets (S103). Specifically, as shown in Figures 8 and 9, the multiplexing unit 102 stores the multiple encoded data in multiple MMT packets so that no single MMT packet contains encoded data corresponding to different slice segments. Also, as shown in Figure 8, the multiplexing unit 102 stores control information commonly used for all decoding units within the picture in MMT packet #1, which is different from the multiple MMT packets #2 to #5 in which the multiple encoded data are stored. Here, the control information includes at least one of the access unit delimiter, SPS, PPS, and SEI.

[0112] The multiplexing unit 102 may store the control information in the same MMT packet as any of the multiple MMT packets that store the multiple encoded data. For example, as shown in Figure 9, the multiplexing unit 102 may store the control information in the first MMT packet (MMT packet #1 in Figure 9) of the multiple MMT packets that store the multiple encoded data.

[0113] Finally, the transmitting device 100 transmits multiple MMT packets. Specifically, the modulation unit 103 modulates the data obtained through multiplexing, and the transmitting unit 104 transmits the modulated data (S104).

[0114] Figure 15 is a block diagram showing an example configuration of the receiving device 200, and is a diagram that shows in detail the configuration of the demultiplexing unit 203 and subsequent stages shown in Figure 7. As shown in Figure 15, the receiving device 200 further includes a decoding command unit 206. The demultiplexing unit 203 also includes a type determination unit 211, a control information acquisition unit 212, a slice information acquisition unit 213, and a decoded data generation unit 214.

[0115] The following describes the operation flow of the receiving device 200. Figure 16 is a flowchart showing an example of the operation of the receiving device 200. Here, the operation for one access unit is shown. If decoding processing is performed for multiple access units, the process in this flowchart is repeated.

[0116] First, the receiving device 200 receives, for example, multiple packets (MMT packets) generated by the transmitting device 100 (S201).

[0117] Next, the type determination unit 211 obtains the type of encoded data stored in the received packet by analyzing the header of the received packet (S202).

[0118] Next, the type determination unit 211 determines, based on the type of the acquired encoded data, whether the data stored in the received packet is data before the slice segment or data from the slice segment (S203).

[0119] If the data stored in the received packet is pre-slice segment data (Yes in S203), the control information acquisition unit 212 acquires the pre-slice segment data of the access unit to be processed from the payload of the received packet and stores the pre-slice segment data in memory (S204).

[0120] On the other hand, if the data stored in the received packet is slice segment data (No in S203), the receiving device 200 uses the header information of the received packet to determine which of the multiple regions the data stored in the received packet belongs to. Specifically, the slice information acquisition unit 213 analyzes the header of the received packet to obtain the index number Idx of the slice segment stored in the received packet (S205). Specifically, the index number Idx is the index number within the Movie Fragment of the access unit (sample in MMT).

[0121] Note that the processing in step S205 may be performed together in step S202.

[0122] Next, the decryption data generation unit 214 determines the decryption unit to decrypt the slice segment (S206). Specifically, an index number Idx is pre-associated with multiple decryption units, and the decryption data generation unit 214 determines the decryption unit to decrypt the slice segment by selecting the decryption unit corresponding to the index number Idx obtained in step S205.

[0123] Furthermore, as explained in the example in Figure 12, the decoded data generation unit 214 may determine which decoding unit to decode a slice segment based on at least one of the resolution of the access unit (picture), the method of dividing the access unit into multiple slice segments (tiles), and the processing capacity of the multiple decoding units provided in the receiving device 200. For example, the decoded data generation unit 214 determines the method of dividing the access unit based on identification information in a descriptor such as an MMT message or a TS section.

[0124] Next, the decryption data generation unit 214 generates multiple input data (combined data) to be input to multiple decoding units by combining control information, which is included in any of the multiple packets and is used in common for all decoding units within the picture, with each of the multiple encoded data of the multiple slice segments. Specifically, the decryption data generation unit 214 obtains slice segment data from the payload of the received packet. The decryption data generation unit 214 generates the input data to the decoding unit determined in step S206 (S207) by combining the pre-slice segment data stored in memory in step S204 with the obtained slice segment data.

[0125] If, after step S204 or S207, the data of the received packet is not the final data for the access unit (No in S208), the processing from step S201 onwards is repeated. In other words, the above processing is repeated until input data for multiple decoding units 204A to 204D corresponding to all slice segments contained in the access unit is generated.

[0126] Note that the timing of packet reception is not limited to the timing shown in Figure 16; multiple packets may be received in advance or sequentially and stored in memory or elsewhere.

[0127] On the other hand, if the data of the received packet is the final data of the access unit (Yes in S208), the decoding instruction unit 206 outputs the multiple input data generated in step S207 to the corresponding decoding units 204A to 204D (S209).

[0128] Next, the multiple decoding units 204A to 204D generate multiple decoded images by decoding multiple input data in parallel according to the access unit's DTS (S210).

[0129] Finally, the display unit 205 generates a display image by combining multiple decoded images generated by the multiple decoding units 204A to 204D, and displays the display image according to the access unit's PTS (S211).

[0130] The receiving device 200 obtains the access unit's DTS and PTS by analyzing the payload data of the MMT packet, which contains the MPU header information or the Movie Fragment header information. Furthermore, if TS is used as the multiplexing method, the receiving device 200 obtains the access unit's DTS and PTS from the PES packet header. If RTP is used as the multiplexing method, the receiving device 200 obtains the access unit's DTS and PTS from the RTP packet header.

[0131] Furthermore, when the display unit 205 integrates the decoding results of multiple decoding units, it may perform filtering, such as deblocking, at the boundaries of adjacent division units. Note that filtering is not necessary when displaying the decoding result of a single decoding unit, so the display unit 205 may switch processing depending on whether or not filtering is performed at the boundaries of the decoding results of multiple decoding units. Whether or not filtering is necessary may be predetermined depending on whether or not there is division. Alternatively, information indicating whether or not filtering is necessary may be stored separately in the multiplexing layer. In addition, information necessary for filtering, such as filter coefficients, may be stored in the SPS, PPS, SEI, or slice segments. The decoding units 204A to 204D, or the demultiplexing unit 203, acquire this information by analyzing the SEI and output the acquired information to the display unit 205. The display unit 205 performs filtering using this information. Note that if this information is stored in slice segments, it is desirable that the decoding units 204A to 204D acquire this information.

[0132] In the above explanation, an example was shown where there are two types of data stored in the fragment: data before the slice segment and the slice segment itself. However, there may be three or more types of data. In this case, step S203 will perform a case distinction according to the type.

[0133] Furthermore, the transmitting device 100 may fragment the slice segment and store it in an MMT packet if the data size of the slice segment is large. In other words, the transmitting device 100 may fragment the pre-slice segment data and the slice segment. In this case, if the access unit and the data unit are set to be equal, as in the packetization example shown in Figure 11, the following problems arise.

[0134] For example, if slice segment 1 is divided into three fragments, slice segment 1 will be divided into three packets with Fragment counter values ​​ranging from 1 to 3 and transmitted. Furthermore, in slice segments 2 and beyond, the Fragment counter value will be 4 or higher, making it impossible to associate the Fragment counter value with the data stored in the payload. Consequently, the receiving device 200 cannot identify the packet containing the first data of a slice segment from the information in the header of the MMT packet.

[0135] In such cases, the receiving device 200 may analyze the data of the MMT packet payload to determine the starting position of the slice segment. Here, in H.264 or H.265, there are two formats for storing NAL units in the multiplexing layer: a format called the byte stream format, in which a start code consisting of a specific bit sequence is added immediately before the NAL unit header, and a format called the NAL size format, in which a field indicating the size of the NAL unit is added.

[0136] The byte stream format is used in MPEG-2 systems and RTP, among others. The NAL size format is used in MP4, and in DASH and MMT, which use MP4.

[0137] When a byte stream format is used, the receiving device 200 analyzes whether the leading data of the packet matches the start code. If the leading data of the packet matches the start code, the receiving device 200 can determine whether the data contained in the packet is slice segment data by obtaining the type of NAL unit from the subsequent NAL unit header.

[0138] On the other hand, in the case of the NAL size format, the receiving device 200 cannot detect the starting position of the NAL unit based on the bit sequence. Therefore, in order to obtain the starting position of the NAL unit, the receiving device 200 needs to shift the pointer by reading data equal to the size of the NAL unit, starting from the first NAL unit of the access unit.

[0139] However, if the header of the MPU or Movie Fragment in MMT indicates the size in units of subsamples, and the subsamples correspond to pre-slicing data or slice segments, the receiving device 200 can determine the starting position of each NAL unit based on the subsample size information. Therefore, the transmitting device 100 may include information indicating whether subsample-level information exists in the MPU or Movie Fragment in the information that the receiving device 200 acquires at the start of data reception, such as the MPT in MMT.

[0140] The MPU data is an extension based on the MP4 format. In MP4, there are modes that allow the storage of parameter sets such as SPS and PPS for H.264 or H.265 as sample data, and modes that do not. Information to identify this mode is indicated by the entry name of SampleEntry. If the mode that allows storage is used and the parameter set is included in the sample, the receiver 200 acquires the parameter set using the method described above.

[0141] On the other hand, if a mode that does not allow storage is used, the parameter set is stored as Decoder Specific Information within SampleEntry, or stored using a stream for the parameter set. Here, since the stream for the parameter set is not generally used, it is desirable for the transmitting device 100 to store the parameter set in Decoder Specific Information. In this case, the receiving device 200 parses the SampleEntry that is transmitted in the MMT packet as MPU metadata or Movie Fragment metadata to obtain the parameter set that the access unit refers to.

[0142] If the parameter set is stored as sample data, the receiver 200 can obtain the parameter set necessary for decoding by referring only to the sample data without referring to SampleEntry. In this case, the transmitter 100 does not need to store the parameter set in SampleEntry. This allows the transmitter 100 to use the same SampleEntry on different MPUs, thereby reducing the processing load on the transmitter 100 during MPU generation. Furthermore, there is the advantage that the receiver 200 does not need to refer to the parameter set in SampleEntry.

[0143] Alternatively, the transmitting device 100 may store one default parameter set in SampleEntry and store the parameter set referenced by the access unit in the sample data. In conventional MP4, it was common to store parameter sets in SampleEntry, so there is a possibility that some receiving devices may stop playback if no parameter set exists in SampleEntry. This problem can be solved by using the method described above.

[0144] Alternatively, the transmitter 100 may store the parameter set in the sample data only if a parameter set different from the default parameter set is used.

[0145] In both modes, it is possible to store the parameter set in SampleEntry. Therefore, the transmitter 100 may always store the parameter set in VisualSampleEntry, and the receiver 200 may always retrieve the parameter set from VisualSampleEntry.

[0146] In the MMT standard, MP4 header information such as Moov and Moof is transmitted as MPU metadata or movie fragment metadata, but the transmitting device 100 does not necessarily have to transmit MPU metadata and movie fragment metadata. Furthermore, the receiving device 200 can also determine whether SPS and PPS are stored in the sample data based on the service or asset type according to the ARIB (Association of Radio Industries and Businesses) standard, or whether or not MPU metadata is transmitted.

[0147] Figure 17 shows an example where the data before the slice segment and each slice segment are set to different data units.

[0148] In the example shown in Figure 17, the data size of the data before the slice segment, and the data sizes of slice segments 1 through 4, are Length#1 through Length#5, respectively. The field values ​​of Fragmentation indicator, Fragment counter, and Offset included in the MMT packet header are as shown in the figure.

[0149] Here, Offset is offset information indicating the bit length (offset) from the beginning of the encoded data of the sample (access unit or picture) to which the payload data belongs, to the first byte of the payload data (encoded data) contained in the MMT packet. Note that while the Fragment counter value is explained as starting from the total number of fragments minus 1, it may start from other values.

[0150] Figure 18 shows an example of when a data unit is fragmented. In the example shown in Figure 18, slice segment 1 is divided into three fragments, each stored in MMT packet #2 to MMT packet #4. In this case, if the data size of each fragment is Length#2_1 to Length#2_3, the values ​​of each field are as shown in the figure.

[0151] Thus, when data units such as slice segments are set as Data units, the beginning of the access unit and the beginning of the slice segment can be determined as follows based on the field values ​​in the MMT packet header.

[0152] In packets where the Offset value is 0, the beginning of the payload is the beginning of the access unit.

[0153] The beginning of a packet payload where the Offset value is not 0 and the Fragmentation indicator value is 00 or 01 is the beginning of the slice segment.

[0154] Furthermore, if no fragmentation of the data unit occurs and no packet loss occurs, the receiving device 200 can identify the index number of the slice segment to be stored in the MMT packet based on the number of slice segments obtained after detecting the beginning of the access unit.

[0155] Similarly, even when the data unit of the data before the slice segment is fragmented, the receiving device 200 can detect the access unit and the beginning of the slice segment.

[0156] Furthermore, even if packet loss occurs, or if the SPS, PPS, and SEI included in the pre-slice segment data are set to separate Data units, the receiving device 200 can identify the MMT packet containing the first data of the slice segment based on the analysis results of the MMT header, and then identify the starting position of the slice segment or tile within the picture (access unit) by analyzing the slice segment header. The processing load involved in analyzing the slice header is small, so processing load is not a problem.

[0157] Thus, each of the multiple encoded data in multiple slice segments is associated one-to-one with a basic data unit (DATA unit), which is the unit of data stored in one or more packets. Furthermore, each of the multiple encoded data is stored in one or more MMT packets.

[0158] The header information of each MMT packet includes a Fragmentation indicator (identification information) and an Offset (offset information).

[0159] The receiving device 200 determines that the beginning of the payload data contained in a packet having header information that includes a Fragmentation indicator with a value of 00 or 01 is the beginning of the encoded data for each slice segment. Specifically, it determines that the beginning of the payload data contained in a packet having header information that includes an Offset with a non-zero value and a Fragmentation indicator with a value of 00 or 01 is the beginning of the encoded data for each slice segment.

[0160] Furthermore, in the example in Figure 17, the beginning of the Data unit is either the beginning of the access unit or the beginning of the slice segment, and the Fragmentation indicator value is 00 or 01. In addition, the receiving device 200 can also detect the beginning of the access unit or the beginning of the slice segment without referring to the Offset by referring to the type of NAL unit and determining whether the beginning of the Data Unit is an access unit delimiter or a slice segment.

[0161] Thus, by the transmitting device 100 performing packetization such that the beginning of the NAL unit always starts from the beginning of the MMT packet payload, the receiving device 200 can detect the access unit or the beginning of the slice segment by analyzing the Fragmentation indicator and the NAL unit header, even when the pre-slice segment data is divided into multiple data units. The type of the NAL unit is found in the first byte of the NAL unit header. Therefore, when the receiving device 200 analyzes the header portion of the MMT packet, it can obtain the type of the NAL unit by analyzing an additional byte of data. In the case of audio, the receiving device 200 only needs to detect the beginning of the access unit, and can determine this based on whether the value of the Fragmentation indicator is 00 or 01.

[0162] Furthermore, as mentioned above, when encoding data that has been encoded in a way that allows for segmental decoding is stored in an MPEG-2 TS PES packet, the transmitting device 100 can use a data alignment descriptor. Below, an example of how to store the encoded data in a PES packet will be described in detail.

[0163] For example, in HEVC, the transmitter 100 can use a data alignment descriptor to indicate whether the data stored in the PES packet is an access unit, a slice segment, or a tile. The alignment types in HEVC are defined as follows:

[0164] Alignment type 8 indicates an HEVC slice segment. Alignment type 9 indicates an HEVC slice segment or access unit. Alignment type 12 indicates an HEVC slice segment or tile.

[0165] Therefore, the transmitting device 100 can indicate, for example, that the data of a PES packet is either a slice segment or pre-slice segment data by using type 9. Since a separate type is also defined for indicating a slice instead of a slice segment, the transmitting device 100 may also use a type that indicates a slice instead of a slice segment.

[0166] Furthermore, the DTS and PTS included in the PES packet header are set only in PES packets that contain the initial data of an access unit. Therefore, the receiving device 200 can determine that if the type is 9 and the PES packet contains a DTS or PTS field, the PES packet contains the entire access unit or the initial segment of the access unit.

[0167] Furthermore, the transmitting device 100 may use a field such as transport_priority, which indicates the priority of the TS packet containing the PES packet containing the access unit's leading data, to enable the receiving device 200 to distinguish the data contained in the packets. Alternatively, the receiving device 200 may determine the data contained in the packets by analyzing whether the payload of the PES packet is an access unit delimiter. Additionally, the data_alignment_indicator in the PES packet header indicates whether the data in the PES packet is stored according to these types. If this flag (data_alignment_indicator) is set to 1, it is guaranteed that the data stored in the PES packet conforms to the type indicated in the data alignment descriptor.

[0168] Furthermore, the transmitting device 100 may use a data alignment descriptor only when PES packetizing in a decomposable unit such as a slice segment. This allows the receiving device 200 to determine that the encoded data is PES packetized in a decomposable unit if a data alignment descriptor is present, and that the encoded data is PES packetized in an access unit unit if a data alignment descriptor is not present. Note that the MPEG-2 TS standard specifies that if data_alignment_indicator is set to 1 and no data alignment descriptor is present, the unit of PES packetization is an access unit.

[0169] If the PMT contains a data alignment descriptor, the receiving device 200 determines that the data is packetized into PES packets in units that can be divided and decoded, and can generate input data for each decoding unit based on the packetized units. If the PMT does not contain a data alignment descriptor, and the receiving device 200 determines that parallel decoding of the encoded data is necessary based on program information or other descriptor information, it generates input data for each decoding unit by analyzing the slice header of the slice segment, etc. Furthermore, if the encoded data can be decoded by a single decoding unit, the receiving device 200 decodes the data of the entire access unit with that decoding unit. If information indicating whether the encoded data consists of units that can be divided and decoded, such as slice segments or tiles, is separately indicated by the PMT descriptor, etc., the receiving device 200 may determine whether the encoded data can be decoded in parallel based on the analysis results of the descriptor.

[0170] Furthermore, since the DTS and PTS included in the PES packet header are set only in the PES packet containing the access unit's initial data, if the access unit is split and converted into PES packets, the second and subsequent PES packets will not contain information indicating the access unit's DTS and PTS. Therefore, when the decoding process is performed in parallel, each decoding unit 204A to 204D and the display unit 205 use the DTS and PTS stored in the header of the PES packet containing the access unit's initial data.

[0171] (Embodiment 2) Embodiment 2 describes a method for storing NAL-sized data in an MP4-format-based MPU in MMT. While the following description presents an example of a storage method for an MPU used in MMT, this storage method is also applicable to DASH, which is also MP4-format-based.

[0172] [How to store data on the MPU] In the MP4 format, multiple access units are grouped together and stored in a single MP4 file. In MMT, the MPU (Multi-Purpose Unit) stores data for each media within a single MP4 file, and this data can contain any number of access units. Since an MPU is a self-decrypting unit, for example, an MPU can store access units in the GOP (Group of Processors) unit.

[0173] Figure 19 shows the configuration of the MPU. At the top of the MPU are ftyp, mmpu, and moov, which are collectively defined as MPU metadata. moov stores common initialization information and MMT hint tracks in a file.

[0174] Furthermore, moof stores initialization information and size for each sample and subsample, information that identifies the presentation time (PTS) and decoding time (DTS) (sample_duration, sample_size, sample_composition_time_offset), and data_offset which indicates the data's position.

[0175] Furthermore, multiple access units are each stored as samples in an mdat (mdat box). Data in moof and mdat, excluding the samples, is defined as movie fragment metadata (hereinafter referred to as MF metadata), and the sample data in mdat is defined as media data.

[0176] Figure 20 shows the structure of MF metadata. As shown in Figure 20, MF metadata consists, more specifically, the type, length, and data of the moof box (moof) and the type and length of the mdat box (mdat).

[0177] When storing access units in MP4 data, there are modes that allow the storage of parameter sets such as H.264 and H.265 SPS and PPS as sample data, and modes that do not.

[0178] In the modes where storage is not possible, the parameter set is stored in the Decoder Specific Information of SampleEntry in moov. In the modes where storage is possible, the parameter set is included within the sample.

[0179] MPU metadata, MF metadata, and media data are stored in the MMT payload, and the MMT payload header contains the Fragment Type (FT) as an identifier to identify this data. FT=0 indicates MPU metadata, FT=1 indicates MF metadata, and FT=2 indicates media data.

[0180] Figure 19 illustrates an example where MPU metadata units and MF metadata units are stored as data units in the MMT payload. However, units such as ftyp, mmpu, moov, and moof may also be stored as data units in the MMT payload on a data unit basis. Similarly, Figure 19 illustrates an example where sample units are stored as data units in the MMT payload. However, data units may be composed of sample units or NAL units, and such data units may be stored in the MMT payload on a data unit basis. Such data units may also be stored in the MMT payload in further fragmented units.

[0181] [Traditional transmission methods and challenges] Traditionally, when encapsulating multiple access units in MP4 format, the moov and moof files were created only after all the samples to be stored in the MP4 were available.

[0182] When transmitting MP4 format in real time using broadcasting or other methods, if, for example, the samples stored in a single MP4 file are in GOP units, then moov and moof files are created after the GOP-unit time samples have been accumulated, resulting in a delay due to encapsulation. This encapsulation on the transmitting side always increases the end-to-end delay by the amount of time in GOP units. This makes it difficult to provide services in real time, and in particular, when live content is transmitted, it leads to a degradation of service for viewers.

[0183] Figure 21 is a diagram illustrating the data transmission order. When MMT is applied to broadcasting, as shown in Figure 21(a), if the MMT packets are transmitted in the order of the MPU configuration (MMT packets #1, #2, #3, #4, #5, #6 are transmitted in that order), a delay occurs in the transmission of the MMT packets due to encapsulation.

[0184] To prevent delays caused by this encapsulation, a method has been proposed, as shown in Figure 21(b), that does not send MPU header information such as MPU metadata and MF metadata (packets #1 and #2 are not sent, and packets #3-#6 are sent in that order). Alternatively, as shown in Figure 20(c), a method can be considered in which media data is sent first without waiting for the creation of MPU header information, and then the MPU header information is sent after the media data has been sent (sent in the order of #3-#6, #1, #2).

[0185] If MPU header information is not transmitted, the receiving device will decode without using the MPU header information. Also, if the MPU header information is sent later than the media data, the receiving device will wait to obtain the MPU header information before decoding.

[0186] However, conventional MP4-compliant receivers are not guaranteed to decode without using MPU header information. Furthermore, if the receiver performs decoding without using the MPU header through special processing, using conventional transmission methods would make the decoding process complicated, making real-time decoding highly unlikely. In addition, if the receiver waits to acquire MPU header information before decoding, buffering of media data is necessary until the receiver acquires the header information, but the buffer model is not defined, and decoding is not guaranteed.

[0187] Therefore, the transmitting device according to Embodiment 2 transmits the MPU metadata before the media data by storing only common information in the MPU metadata, as shown in Figure 20(d). The transmitting device according to Embodiment 2 transmits the MF metadata, which has a delay in generation, after the media data. This provides a transmission or reception method that can guarantee the decoding of media data.

[0188] The following describes the receiving methods when using each of the transmission methods shown in Figure 21 (a)-(d).

[0189] In each transmission method shown in Figure 21, the MPU data is first composed in the following order: MPU metadata, MFU metadata, and media data.

[0190] After configuring the MPU data, if the transmitting device transmits the data in the order of MPU metadata, MF metadata, and media data as shown in Figure 21(a), the receiving device can decode it using either of the following methods (A-1) or (A-2).

[0191] (A-1) The receiving device acquires MPU header information (MPU metadata and MF metadata) and then decodes the media data using the MPU header information.

[0192] (A-2) The receiving device decodes the media data without using the MPU header information.

[0193] While all of these methods introduce encapsulation-induced delays on the transmitting side, they have the advantage of eliminating the need for buffering media data in the receiving device to acquire the MPU header. Without buffering, there is no need for buffering memory, and furthermore, no buffering delay occurs. Additionally, method (A-1) uses MPU header information for decoding, making it applicable to conventional receiving devices as well.

[0194] If the transmitting device transmits only media data, as shown in Figure 21(b), the receiving device can decode it using the method described in (B-1) below.

[0195] (B-1) The receiving device decodes the media data without using the MPU header information.

[0196] Furthermore, although not shown in the diagram, if the MPU metadata is transmitted before the media data transmission shown in Figure 21(b), decryption can be performed using the method described in (B-2) below.

[0197] (B-2) The receiving device decodes the media data using the MPU metadata.

[0198] The advantages of methods (B-1) and (B-2) above are that they do not cause delays due to encapsulation on the transmitting side and do not require buffering of media data to obtain the MPU header. However, since neither method (B-1) nor (B-2) performs decoding using MPU header information, special processing may be required for decoding.

[0199] When the transmitting device transmits data in the order of media data, MPU metadata, and MF metadata, as shown in Figure 21(c), the receiving device can decode the data using either of the following methods (C-1) or (C-2).

[0200] (C-1) The receiving device acquires the MPU header information (MPU metadata and MF metadata) and then decodes the media data.

[0201] (C-2) The receiving device decodes the media data without using the MPU header information.

[0202] When method (C-1) above is used, media data needs to be buffered in order to obtain MPU header information. In contrast, when method (C-2) above is used, buffering is not required to obtain MPU header information.

[0203] Furthermore, neither method (C-1) nor (C-2) above causes any delay due to encapsulation on the transmitting side. Also, method (C-2) does not use MPU header information, so special processing may be required.

[0204] When the transmitting device transmits data in the order of MPU metadata, media data, and MF metadata, as shown in Figure 21(d), the receiving device can decode the data using either of the following methods (D-1) or (D-2).

[0205] (D-1) The receiving device acquires MPU metadata, then acquires MF metadata, and then decodes the media data.

[0206] (D-2) After acquiring the MPU metadata, the receiving device decodes the media data without using the MF metadata.

[0207] When method (D-1) above is used, media data needs to be buffered in order to obtain MF metadata, but when method (D-2) above is used, buffering for obtaining MF metadata is not required.

[0208] The method described above (D-2) does not perform decryption using MF metadata, so special processing may be required.

[0209] As explained above, if decoding is possible using MPU metadata and MF metadata, there is the advantage that it can also be decoded by conventional MP4 receivers.

[0210] In Figure 21, the MPU data is structured in the order of MPU metadata, MFU metadata, and media data. In moof, the positional information (offset) for each sample and subsample is determined based on this structure. The MF metadata also includes data other than the media data in the mdat box (such as the size and type of the box).

[0211] Therefore, when the receiving device identifies media data based on MF metadata, it reconstructs the data in the order in which the MPU data was composed, regardless of the order in which the data was transmitted, and then decodes it using the moov of the MPU metadata or the moof of the MF metadata.

[0212] In Figure 21, the MPU data is structured in the order of MPU metadata, MFU metadata, and media data. However, the MPU data may be structured in a different order than shown in Figure 21, and location information (offset) may be defined accordingly.

[0213] For example, MPU data may be composed of MPU metadata, media data, and MF metadata in that order, and negative positional information (offset) may be indicated in the MF metadata. In this case as well, regardless of the order in which the data is transmitted, the receiving device reconstructs the data in the order in which the MPU data was composed on the transmitting side, and then decodes it using moov or moof.

[0214] The transmitting device may signal information indicating the order in which the MPU data is composed, and the receiving device may reconstruct the data based on the signaled information.

[0215] As explained above, the receiving device receives the packetized MPU metadata, packetized media data (sample data), and packetized MF metadata in this order, as shown in Figure 21(d). Here, the MPU metadata is an example of the first metadata, and the MF metadata is an example of the second metadata.

[0216] Next, the receiving device reconstructs the MPU data (a file in MP4 format) containing the received MPU metadata, received MF metadata, and received sample data. Then, it decodes the sample data contained in the reconstructed MPU data using the MPU metadata and MF metadata. The MF metadata is metadata that can only be generated by the transmitting side after the sample data has been generated (for example, the length stored in the mbox).

[0217] The operation of the receiving device described above is, in more detail, carried out by each component that makes up the receiving device. For example, the receiving device comprises a receiving unit that receives the above data, a reconstruction unit that reconstructs the above MPU data, and a decoding unit that decodes the above MPU data. The receiving unit, generation unit, and decoding unit are each implemented by a microcomputer, processor, dedicated circuit, etc.

[0218] [Method for decryption without using header information] Next, we will explain a method for decoding without using header information. Here, we will describe a method for decoding without using header information in the receiving device, regardless of whether the sender sends header information or not. In other words, this method is applicable to any of the transmission methods explained using Figure 21. However, some decoding methods are applicable only to specific transmission methods.

[0219] Figure 22 shows an example of a method for decoding without using header information. In Figure 22, only MMT payloads and MMT packets containing only media data are shown; MMT payloads and MMT packets containing MPU metadata or MF metadata are not shown. Furthermore, in the following explanation of Figure 22, it is assumed that media data belonging to the same MPU is transmitted continuously. In addition, the explanation uses the example where a sample is stored in the payload as media data, but in the following explanation of Figure 22, it is of course possible that a NAL unit or a fragmented NAL unit is stored.

[0220] To decode media data, the receiving device must first acquire the initialization information necessary for decoding. Furthermore, if the media is video, the receiving device must acquire initialization information for each sample, identify the starting position of the MPU (Mass Processing Unit), and obtain the starting positions of the sample and NAL (Non-Aligned Log) unit. The receiving device also needs to identify the decoding time (DTS) and presentation time (PTS) for each sample.

[0221] Therefore, the receiving device can decode the data without using header information, for example, by using the method described below. Note that if the payload contains NAL units or fragmented NAL units, then in the following explanation, "sample" should be read as "NAL unit in the sample."

[0222] <Random access (=identifying the first sample on the MPU)> If header information is not transmitted, the receiving device can identify the first sample of the MPU using the following methods 1 and 2. If header information is transmitted, method 3 can be used.

[0223] [Method 1] The receiving device obtains samples contained in MMT packets where the MMT packet header has 'RAP_flag=1'.

[0224] [Method 2] The receiving device obtains a sample in the MMT payload header where 'sample number=0' is specified.

[0225] [Method 3] If at least one of the MPU metadata and MF metadata is transmitted before or after the media data, the receiving device retrieves a sample contained in the MMT payload whose fragment type (FT) in the MMT payload header has been switched to media data.

[0226] In Method 1 and Method 2, if a single payload contains multiple samples belonging to different MPUs, it is impossible to determine which NAL unit is a random access point (RAP_flag=1 or sample number=0). Therefore, constraints such as not mixing samples from different MPUs in a single payload, or, if a single payload contains samples from different MPUs, setting RAP_flag to 1 if the last (or first) sample is a random access point, are necessary.

[0227] Furthermore, in order for the receiving device to obtain the starting position of the NAL units, it is necessary to shift the data read pointer by the size of each NAL unit, starting from the first NAL unit of the sample.

[0228] If the data is fragmented, the receiving device can identify the data unit by referring to the fragment_indicator or fragment_number.

[0229] <DTS determination for the sample> There are two methods for determining the DTS of a sample, Method 1 and Method 2, as described below.

[0230] [Method 1] The receiving device determines the DTS of the first sample based on the predicted structure. However, this method requires analysis of the encoded data, and real-time decoding may be difficult, so Method 2 is preferable.

[0231] [Method 2] The receiving device separately transmits the DTS of the first sample and obtains the transmitted DTS of the first sample. Methods for transmitting the DTS of the first sample include, for example, transmitting the DTS of the MPU first sample using MMT-SI, or transmitting the DTS for each sample using the MMT packet header extension area. The DTS may be an absolute value or a relative value to the PTS. The transmitting side may also signal whether the DTS of the first sample is included.

[0232] Note that in both Method 1 and Method 2, the DTS of the following samples will be calculated assuming a fixed frame rate.

[0233] In addition to using the extended area to store the DTS for each sample in the packet header, another method is to store the DTS of the samples contained in the MMT packet in the 32-bit NTP timestamp field of the MMT packet header. If the DTS cannot be represented with the number of bits (32 bits) of a single packet header, the DTS may be represented using multiple packet headers. Furthermore, the DTS may be represented by a combination of the NTP timestamp field and the extended area of ​​the packet header. If no DTS information is included, a known value (e.g., ALL0) is used.

[0234] <Determination of PTS for the sample> The receiving device obtains the PTS of the first sample from the MPU timestamp descriptor for each asset included in the MPU. For subsequent sample PTSs, the receiving device calculates them based on parameters indicating the display order of samples such as POCs, assuming a fixed frame rate. Thus, in order to calculate DTS and PTS without using header information, transmission at a fixed frame rate is essential.

[0235] Also, when MF metadata is being transmitted, the receiving device can calculate the absolute values of DTS and PTS from the relative time information of DTS and PTS from the start sample indicated in the MF metadata and the absolute value of the time stamp of the MPU start sample indicated in the MPU time stamp descriptor.

[0236] Note that when calculating DTS and PTS by analyzing the encoded data, the receiving device may calculate using the SEI information included in the access unit.

[0237] <Initialization information (parameter set)> [In the case of video] In the case of video, the parameter set is stored in the sample data. Also, when MPU metadata and MF metadata are not transmitted, it is guaranteed that the parameter set necessary for decoding can be obtained by referring only to the sample data.

[0238] Also, as shown in (a) and (d) of FIG. 21, when MPU metadata is transmitted before the media data, it may be stipulated that the parameter set is not stored in the SampleEntry. In this case, the receiving device refers only to the parameter set in the sample without referring to the parameter set of the SampleEntry.

[0239] Also, when MPU metadata is transmitted before the media data, a parameter set common to the MPU or a default parameter set is stored in the SampleEntry, and the receiving device may refer to the parameter set of the SampleEntry and the parameter set in the sample. By storing the parameter set in the SampleEntry, it becomes possible to perform decoding even with a conventional receiving device that cannot play back if there is no parameter set in the SampleEntry.

[0240] [In the case of audio] In the case of audio, a LATM header is required for decoding, and in MP4, the LATM header must be included in the sample entry. However, if the header information is not transmitted, it is difficult for the receiving device to obtain the LATM header, so the LATM header is included separately in control information such as SI. The LATM header may also be included in the message, table, or descriptor. In addition, the LATM header may be included within the sample.

[0241] The receiving device obtains the LATM header from the SI or other source before starting decoding, and then begins decoding the audio. Alternatively, as shown in Figures 21(a) and 21(d), if the MPU metadata is transmitted before the media data, the receiving device can receive the LATM header before the media data. Therefore, if the MPU metadata is transmitted before the media data, decoding can be performed even using conventional receiving devices.

[0242] <Other> The transmission order and transmission order type may be notified as control information in the MMT packet header, payload header, or MPT or other tables, messages, descriptors, etc. The transmission order type here refers to, for example, the four types of transmission orders shown in Figure 21 (a) to (d), and it is sufficient that an identifier for each type is stored in a location that can be obtained before decryption begins.

[0243] Furthermore, different transmission order types may be used for audio and video, or a common type may be used for both audio and video. Specifically, for example, audio may be transmitted in the order of MPU metadata, MF metadata, and media data, as shown in Figure 21(a), while video may be transmitted in the order of MPU metadata, media data, and MF metadata, as shown in Figure 21(d).

[0244] As described above, the receiving device can decode without using header information. Furthermore, if the MPU metadata is transmitted before the media data (Figures 21(a) and 21(d)), conventional receiving devices can also decode the data.

[0245] In particular, by transmitting the MF metadata after the media data (Figure 21(d)), delays caused by encapsulation are avoided, and decoding becomes possible even with conventional receiving devices.

[0246] [Configuration and operation of the transmitting device] Next, the configuration and operation of the transmitting device will be described. Figure 23 is a block diagram of the transmitting device according to Embodiment 2, and Figure 24 is a flowchart of the transmitting method according to Embodiment 2.

[0247] As shown in Figure 23, the transmitting device 15 comprises an encoding unit 16, a multiplexing unit 17, and a transmitting unit 18.

[0248] The encoding unit 16 generates encoded data by encoding the video or audio to be encoded, for example, according to H.265 (S10).

[0249] The multiplexing unit 17 multiplexes (packets) the encoded data generated by the encoding unit 16 (S11). Specifically, the multiplexing unit 17 packets the sample data, MPU metadata, and MF metadata that constitute the MP4 format file. The sample data is data in which a video signal or audio signal has been encoded, the MPU metadata is an example of the first metadata, and the MF metadata is an example of the second metadata. Both the first metadata and the second metadata are metadata used for decoding the sample data, but the difference between them is that the second metadata includes data that can only be generated after the sample data has been generated.

[0250] Here, data that can only be generated after the generation of sample data is, for example, data other than the sample data stored in the mdat file in the MP4 format (data in the mdat header, i.e., type and length as shown in Figure 20). Here, the second metadata only needs to include at least a part of this data, namely length.

[0251] The transmitting unit 18 transmits a packetized MP4 format file (S12). The transmitting unit 18 transmits the MP4 format file in the manner shown in Figure 21(d), for example. That is, it transmits the packetized MPU metadata, packetized sample data, and packetized MF metadata in that order.

[0252] The encoding unit 16, the multiplexing unit 17, and the transmission unit 18 are each implemented by a microcomputer, processor, or dedicated circuitry.

[0253] [Configuration of the receiving device] Next, the configuration and operation of the receiving device will be described. Figure 25 is a block diagram of the receiving device according to Embodiment 2.

[0254] As shown in Figure 25, the receiving device 20 includes a packet filtering unit 21, a transmission order type determination unit 22, a random access unit 23, a control information acquisition unit 24, a data acquisition unit 25, a PTS / DTS calculation unit 26, an initialization information acquisition unit 27, a decoding command unit 28, a decoding unit 29, and a presentation unit 30.

[0255] [Receiver operation 1] First, we will describe the operation of the receiving device 20 to identify the MPU starting position and the NAL unit position when the media is video. Figure 26 is a flowchart of this operation of the receiving device 20. Here, we assume that the transmission order type of the MPU data is stored in the SI information by the transmitting device 15 (multiplexing unit 17).

[0256] First, the packet filtering unit 21 performs packet filtering on the received file. The transmission order type determination unit 22 analyzes the SI information obtained by packet filtering to acquire the transmission order type of the MPU data (S21).

[0257] Next, the transmission order type determination unit 22 determines (discriminates) whether the MPU header information (at least one of MPU metadata or MF metadata) is included in the data after packet filtering (S22). If the MPU header information is included (Yes in S22), the random access unit 23 identifies the MPU start sample by detecting that the fragment type of the MMT payload header switches to media data (S23).

[0258] On the other hand, if the MPU header information is not included (No in S22), the random access unit 23 identifies the MPU start sample based on the RAP_flag of the MMT packet header or the sample number of the MMT payload header (S24).

[0259] Also, the transmission order type determination unit 22 determines whether MF metadata is included in the packet-filtered data (S25). If it is determined that MF metadata is included (Yes in S25), the data acquisition unit 25 acquires the NAL unit by reading the NAL unit based on the samples, sub-sample offsets, and size information included in the MF metadata (S26). On the other hand, if it is determined that MF metadata is not included (No in S25), the data acquisition unit 25 acquires the NAL unit by sequentially reading data of the size of the NAL unit from the start NAL unit of the sample (S27).

[0260] Furthermore, even if the receiving device 20 determines in step S22 that MPU header information is included, it may use the processing in step S24 instead of step S23 to identify the first MPU sample. Alternatively, if it is determined that MPU header information is included, the processing in step S23 and the processing in step S24 may be used in combination.

[0261] Furthermore, even if the receiving device 20 determines in step S25 that MF metadata is included, it may acquire the NAL unit using the process in step S27 without using the process in step S26. Also, if it is determined that MF metadata is included, the processes in steps S23 and S24 may be used in combination.

[0262] Furthermore, it is assumed that in step S25, it is determined that MF metadata is included, and that the MF data is transmitted after the media data. In this case, the receiving device 20 may buffer the media data and wait until the MF metadata is acquired before proceeding with the processing in step S26, or the receiving device 20 may decide whether or not to proceed with the processing in step S27 without waiting for the MF metadata to be acquired.

[0263] For example, the receiving device 20 may decide whether to wait to acquire MF metadata based on whether it has a buffer with a buffer size capable of buffering media data. Alternatively, the receiving device 20 may decide whether to wait to acquire MF metadata based on whether the end-to-end delay is reduced. Furthermore, the receiving device 20 may primarily use the processing in step S26 to perform the decoding process, and use the processing in step S27 when a processing mode occurs, such as when packet loss occurs.

[0264] If the transmission order type is predetermined, steps S22 and S26 may be omitted. In this case, the receiving device 20 may determine how to identify the first sample of the MPU and how to identify the NAL unit, taking into consideration the buffer size and end-to-end delay.

[0265] Furthermore, if the transmission sequence type is known in advance, the transmission sequence type determination unit 22 in the receiving device 20 is unnecessary.

[0266] Furthermore, although not shown in Figure 26 above, the decoding command unit 28 outputs the data acquired by the data acquisition unit to the decoding unit 29 based on the PTS and DTS calculated by the PTS and DTS calculation unit 26 and the initialization information acquired by the initialization information acquisition unit 27. The decoding unit 29 decodes the data, and the presentation unit 30 presents the decoded data.

[0267] [Operation of the receiving device 2] Next, we will describe the operation in which the receiving device 20 acquires initialization information based on the transmission order type and decodes the media data based on the initialization information. Figure 27 is a flowchart of this operation.

[0268] First, the packet filtering unit 21 performs packet filtering on the received file. The transmission order type determination unit 22 analyzes the SI information obtained by packet filtering and obtains the transmission order type (S301).

[0269] Next, the transmission order type determination unit 22 determines whether or not MPU metadata has been transmitted (S302). If it is determined that MPU metadata has been transmitted (Yes in S302), the transmission order type determination unit 22 determines, based on the analysis in step S301, whether or not MPU metadata has been transmitted before media data (S303). If MPU metadata has been transmitted before media data (Yes in S303), the initialization information acquisition unit 27 decodes the media data based on the common initialization information included in the MPU metadata and the initialization information of the sample data (S304).

[0270] On the other hand, if it is determined that the MPU metadata is transmitted after the media data (No in S303), the data acquisition unit 25 buffers the media data until the MPU metadata is acquired (S305), and then performs the processing in step S304 after the MPU metadata has been acquired.

[0271] Furthermore, if it is determined in step S302 that MPU metadata has not been transmitted (No in S302), the initialization information acquisition unit 27 decodes the media data based only on the initialization information of the sample data (S306).

[0272] Furthermore, if the decoding of media data is guaranteed only when the initialization information of the sample data is used on the transmitting side, the processing based on the determinations in steps S302 and S303 is not performed, and the processing in step S306 is used instead.

[0273] Furthermore, the receiving device 20 may determine whether or not to buffer the media data before step S305. In this case, if the receiving device 20 determines to buffer the media data, it proceeds to the process of step S305; if it determines not to buffer the media data, it proceeds to the process of step S306. The determination of whether or not to buffer the media data may be based on the buffer size and occupancy of the receiving device 20, or it may be determined by considering the end-to-end delay, for example, by selecting the option with the smaller end-to-end delay.

[0274] [Operation of the receiving device 3] This section details the transmission and reception methods when MF metadata is transmitted after media data (Figures 21(c) and 21(d)). The following explanation uses Figure 21(d) as an example. Note that only the method shown in Figure 21(d) is used for transmission, and no transmission order type signaling is performed.

[0275] As mentioned above, when transmitting data in the order of MPU metadata, media data, and MF metadata, as shown in Figure 21(d), (D-1) The receiving device 20 decodes the media data after acquiring the MPU metadata and then the MF metadata. (D-2) After acquiring the MPU metadata, the receiving device 20 decodes the media data without using the MF metadata. Two decryption methods are possible.

[0276] Here, D-1 requires buffering of media data for MF metadata acquisition, but since decoding can be performed using MPU header information, it can be decoded by conventional MP4-compliant receivers. On the other hand, D-2 does not require buffering of media data for MF metadata acquisition, but since it cannot be decoded using MF metadata, special processing is required for decoding.

[0277] Furthermore, the method shown in Figure 21(d) has the advantage that, since MF metadata is transmitted after the media data, no delay occurs due to encapsulation, and end-to-end delay can be reduced.

[0278] The receiving device 20 can select one of the two decoding methods described above, depending on the capabilities of the receiving device 20 and the quality of service it provides.

[0279] The transmitting device 15 must ensure that the decoding operation in the receiving device 20 can be performed with reduced occurrence of buffer overflow and underflow. For example, the following parameters can be used as elements to define the decoder model when decoding using the D-1 method.

[0280] • Buffer size for reconfiguring the MPU (MPU buffer) For example, buffer size = maximum rate × maximum MPU time × α, where the maximum rate is the upper limit rate of the encoding data profile and level + the overhead of the MPU header. Also, the maximum MPU time is the maximum duration of a GOP (video) assuming 1 MPU = 1 GOP.

[0281] Here, the audio may be in the same GOP units as the video above, or it may be in a different unit. α is a margin to prevent overflow and may be multiplied or added to the maximum rate × maximum MPU time. If multiplied, α ≥ 1, and if added, α ≥ 0.

[0282] • The upper limit of the decoding delay time from when data is input to the MPU buffer until it is decoded. (TSTD_delay in MPEG-TS STD) For example, during transmission, the DTS is set so that the time of completion of MPU data acquisition at the receiver is less than or equal to the DTS, taking into account the maximum MPU time and the upper limit of the decoding delay time.

[0283] Furthermore, the transmitting device 15 may add DTS and PTS according to the decoder model when decoding using method D-1. This allows the transmitting device 15 to guarantee the decoding performed by the receiving device using method D-1, while simultaneously transmitting auxiliary information necessary when decoding is performed using method D-2.

[0284] For example, the transmitting device 15 can guarantee the operation of the receiving device decoding using the D-2 method by signaling the pre-buffering time in the decoder buffer when decoding using the D-2 method.

[0285] The pre-buffering time may be included in SI control information such as messages, tables, and descriptors, or in the headers of MMT packets and MMT payloads. The SEI in the encoded data may also be overwritten. The DTS and PTS for decoding using method D-1 are stored in the MPU timestamp descriptor and SampleEntry, while the DTS and PTS for decoding using method D-2, or the pre-buffering time, may be described in the SEI.

[0286] If the receiving device 20 supports only MP4-compliant decoding using the MPU header, it may select decoding method D-1. If it supports both D-1 and D-2, it may select either one.

[0287] The transmitting device 15 may provide DTS and PTS to ensure the decoding operation of one of the devices (D-1 in this description), and may also transmit auxiliary information to assist the decoding operation of the other device.

[0288] Furthermore, when the D-2 method is used, the end-to-end delay is likely to be larger compared to when the D-1 method is used, due to delays caused by pre-buffering of MF metadata. Therefore, the receiver 20 may select the D-2 method for decoding when it wants to reduce the end-to-end delay. For example, the receiver 20 may always use the D-2 method when it always wants to reduce the end-to-end delay. Alternatively, the receiver 20 may use the D-2 method only when operating in a low-latency presentation mode where it wants to present live content, channel selection, or zapping operations with low latency.

[0289] Figure 28 is a flowchart of this receiving method.

[0290] First, the receiving device 20 receives the MMT packet and acquires the MPU data (S401). Then, the receiving device 20 (transmission order type determination unit 22) determines whether to present the program in low-latency presentation mode (S402).

[0291] If the program is not presented in low-latency presentation mode (No in S402), the receiving device 20 (random access unit 23 and initialization information acquisition unit 27) acquires random access and initialization information using the header information (S405). The receiving device 20 (PTS, DTS calculation unit 26, decoding command unit 28, decoding unit 29, presentation unit 30) performs decoding and presentation processing based on the PTS and DTS assigned by the transmitting side (S406).

[0292] On the other hand, when the program is presented in low-latency presentation mode (Yes in S402), the receiving device 20 (random access unit 23 and initialization information acquisition unit 27) acquires random access and initialization information using a decoding method that does not use header information (S403). The receiving device 20 also performs decoding and presentation processing based on auxiliary information for decoding without using PTS, DTS, and header information provided by the transmitting side (S404). Note that in steps S403 and S404, processing may be performed using MPU metadata.

[0293] [Method of sending and receiving data using auxiliary data] The above describes the transmission and reception operations when MF metadata is transmitted after media data (in the cases of Figure 21(c) and Figure 21(d)). Next, we will describe a method in which the transmitting device 15 can start decoding earlier and reduce end-to-end delay by transmitting auxiliary data that has some of the functions of MF metadata. Here, we will describe an example in which auxiliary data is further transmitted based on the transmission method shown in Figure 21(d), but the method using auxiliary data is also applicable to the transmission methods shown in Figure 21(a) to (c).

[0294] Figure 29(a) shows an MMT packet transmitted using the method shown in Figure 21(d). In other words, the data is transmitted in the order of MPU metadata, media data, and MF metadata.

[0295] Here, Sample #1, Sample #2, Sample #3, and Sample #4 are samples included in the media data. While this example describes media data being stored in MMT packets on a sample basis, media data may also be stored in MMT packets on a NAL unit basis, or in units obtained by dividing a NAL unit. Furthermore, multiple NAL units may be aggregated and stored in an MMT packet.

[0296] As explained in D-1 above, in the case of the method shown in Figure 21(d), that is, when data is transmitted in the order of MPU metadata, media data, and MF metadata, there is a method in which the MPU metadata is acquired first, then the MF metadata is acquired, and then the media data is decoded. In this D-1 method, buffering of media data is required for acquiring the MF metadata, but since decoding is performed using the MPU header information, the D-1 method has the advantage that it can be applied to conventional MP4-compliant receivers as well. On the other hand, the receiver 20 has the disadvantage that it must wait to start decoding until the MF metadata is acquired.

[0297] In contrast, as shown in Figure 29(b), in methods that use auxiliary data, the auxiliary data is transmitted before the MF metadata.

[0298] MF metadata contains information indicating the DTS, PTS, offset, and size of all samples included in the movie fragment. In contrast, auxiliary data contains information indicating the DTS, PTS, offset, and size of some of the samples included in the movie fragment.

[0299] For example, MF metadata contains information for all samples (sample #1-sample #4), while auxiliary data contains information for only some samples (sample #1-#2).

[0300] In the case shown in Figure 29(b), the use of auxiliary data enables decoding of sample #1 and sample #2, thus reducing the end-to-entry delay compared to the D-1 transmission method. The auxiliary data may be combined and included in any way, and the auxiliary data may be transmitted repeatedly.

[0301] For example, in Figure 29(c), when the auxiliary information is transmitted at timing A, the transmitting device 15 includes the information of sample #1 in the auxiliary information; when the auxiliary information is transmitted at timing B, the auxiliary information includes the information of sample #1 and sample #2; and when the transmitting device 15 transmits the auxiliary information at timing C, the auxiliary information includes the information of sample #1, sample #2, and sample #3.

[0302] The MF metadata includes information on sample #1, sample #2, sample #3, and sample #4 (information on all samples within the movie fragment).

[0303] Auxiliary data does not necessarily need to be sent immediately after it is generated.

[0304] Furthermore, the headers of MMT packets and MMT payloads specify a type that indicates the presence of auxiliary data.

[0305] For example, if auxiliary data is stored in the MMT payload using MPU mode, the fragment_type field value (e.g., FT=3) specifies a data type indicating that it is auxiliary data. The auxiliary data may be based on the MOOF configuration or in other configurations.

[0306] If auxiliary data is stored in the MMT payload as control signals (descriptors, tables, messages), a descriptor tag, table ID, and message ID are specified to indicate that it is auxiliary data.

[0307] Additionally, PTS or DTS may be stored in the header of the MMT packet or MMT payload.

[0308] [Example of generating supplementary data] The following describes an example in which the transmitting device generates auxiliary data based on the MOOF configuration. Figure 30 is a diagram illustrating an example in which the transmitting device generates auxiliary data based on the MOOF configuration.

[0309] In a standard MP4 file, a MOO file is created for each movie fragment, as shown in Figure 20. The MOO file contains information indicating the DTS, PTS, offset, and size of the samples contained within the movie fragment.

[0310] Here, the transmitting device 15 uses only a portion of the sample data that make up the MPU to construct an MP4 (MP4 file) and generates auxiliary data.

[0311] For example, as shown in Figure 30(a), the transmitter 15 generates an MP4 using only sample #1 from the samples #1-#4 that make up the MPU, and uses the moof+mdat header as auxiliary data.

[0312] Next, as shown in Figure 30(b), the transmitter 15 generates an MP4 using sample #1 and sample #2 from the samples #1-#4 that constitute the MPU, and uses the moof+mdat header as the following auxiliary data.

[0313] Next, as shown in Figure 30(c), the transmitter 15 generates an MP4 using sample #1, sample #2, and sample #3 from the samples #1-#4 that constitute the MPU, and uses the moof+mdat header as the following auxiliary data.

[0314] Next, as shown in Figure 30(d), the transmitter 15 generates all MP4s from samples #1-#4 that make up the MPU, and the moof+mdat header of these MP4s becomes the movie fragment metadata.

[0315] In this example, the transmitter 15 generates auxiliary data for every sample, but it may also generate auxiliary data for every N samples. The value of N is any number; for example, if auxiliary data is transmitted M times when transmitting one MPU, then N = total samples / M.

[0316] Note that the information indicating the sample offset in moof may be the offset value after the sample entry area for the subsequent number of samples has been allocated as a NULL area.

[0317] Furthermore, auxiliary data may be generated in a configuration that fragments the MF metadata.

[0318] [Example of receiving operation using auxiliary data] The reception of the auxiliary data generated as explained in Figure 30 will now be described. Figure 31 is a diagram illustrating the reception of auxiliary data. In Figure 31(a), the number of samples constituting the MPU is 30, and auxiliary data is generated and transmitted every 10 samples.

[0319] In Figure 30(a), Auxiliary Data #1 contains sample information for samples #1-#10, Auxiliary Data #2 contains sample information for samples #1-#20, and MF metadata contains sample information for samples #1-#30.

[0320] Although samples #1-#10, #11-#20, and #21-#30 are stored in a single MMT payload, they may also be stored on a sample-by-sample or NAL-by-AL basis, or as fragments or aggregated units.

[0321] The receiving device 20 receives packets of MPU metadata, sample data, MF metadata, and auxiliary data, respectively.

[0322] The receiver 20 concatenates the sample data in the order it is received (from end to end), and updates the previous auxiliary data after receiving the latest auxiliary data. Furthermore, the receiver 20 can configure a complete MPU by finally replacing the auxiliary data with MF metadata.

[0323] When the receiving device 20 receives auxiliary data #1, it concatenates the data as shown in the upper part of Figure 31(b) to form an MP4. This allows the receiving device 20 to parse samples #1-#10 using the MPU metadata and information from auxiliary data #1, and to decode based on the PTS, DTS, offset, and size information contained in the auxiliary data.

[0324] Furthermore, upon receiving auxiliary data #2, the receiving device 20 concatenates the data as shown in the middle section of Figure 31(b) to form an MP4. This allows the receiving device 20 to parse samples #1-#20 using the MPU metadata and information from auxiliary data #2, and to perform decoding based on the PTS, DTS, offset, and size information contained in the auxiliary data.

[0325] Furthermore, upon receiving the MF metadata, the receiving device 20 concatenates the data as shown in the lower part of Figure 31(b) to form an MP4 file. This allows the receiving device 20 to parse samples #1-#30 using the MPU metadata and MF metadata, and to perform decoding based on the PTS, DTS, offset, and size information contained in the MF metadata.

[0326] In the absence of auxiliary data, the receiving device 20 could only acquire sample information after receiving the MF metadata, and therefore had to start decoding after receiving the MF metadata. However, by having the transmitting device 15 generate and transmit auxiliary data, the receiving device 20 can acquire sample information using the auxiliary data without waiting for the MF metadata to be received, thus shortening the decoding start time. Furthermore, by having the transmitting device 15 generate auxiliary data based on MOOF as explained using Figure 30, the receiving device 20 can use its conventional MP4 parser to parse the data.

[0327] Furthermore, the newly generated auxiliary data and MF metadata include sample information that overlaps with previously transmitted auxiliary data. Therefore, even if past auxiliary data cannot be obtained due to packet loss or other reasons, it is possible to reconstruct the MP4 file and obtain sample information (PTS, DTS, size, and offset) using the newly acquired auxiliary data and MF metadata.

[0328] Note that the auxiliary data does not necessarily need to include information from past sample data. For example, auxiliary data #1 may correspond to sample data #1-#10, and auxiliary data #2 may correspond to sample data #11-#20. For example, as shown in Figure 31(c), the transmitting device 15 may sequentially send out complete MF metadata as a data unit, and fragmented units of the data unit as auxiliary data.

[0329] Furthermore, the transmitting device 15 may repeatedly transmit auxiliary data or repeatedly transmit MF metadata in order to prevent packet loss.

[0330] Furthermore, the MMT packets and MMT payloads containing the auxiliary data include the MPU sequence number and asset ID, similar to the MPU metadata, MF metadata, and sample data.

[0331] The reception operation using the auxiliary data described above will be explained using the flowchart in Figure 32. Figure 32 is a flowchart of the reception operation using auxiliary data.

[0332] First, the receiving device 20 receives the MMT packet and analyzes the packet header and payload header (S501). Next, the receiving device 20 analyzes whether the fragment type is auxiliary data or MF metadata (S502). If the fragment type is auxiliary data, it overwrites and updates the past auxiliary data (S503). If there is no past auxiliary data for the same MPU, the receiving device 20 uses the received auxiliary data as new auxiliary data. Then, the receiving device 20 acquires a sample based on the MPU metadata, auxiliary data, and sample data and performs decoding (S507).

[0333] On the other hand, if the fragment type is MF metadata, the receiving device 20 overwrites the past auxiliary data with MF metadata in step S505 (S505). Then, the receiving device 20 acquires the sample in complete MPU form based on the MPU metadata, MF metadata, and sample data, and performs decoding (S506).

[0334] Although not shown in Figure 32, in step S502, the receiving device 20 stores the data in a buffer if the fragment type is MPU metadata, and stores the data concatenated after each sample in the buffer if it is sample data.

[0335] If auxiliary data cannot be obtained due to packet loss, the receiving device 20 can decode the sample by overwriting it with the latest auxiliary data or by using past auxiliary data.

[0336] The transmission cycle and number of transmissions of auxiliary data may be predetermined values. Information on the transmission cycle and number of transmissions (count, countdown) may be transmitted together with the data. For example, the data unit header may store the transmission cycle, number of transmissions, and timestamps such as initial_cpb_removal_delay.

[0337] By sending auxiliary data containing information about the MPU's initial sample at least once before initial_cpb_removal_delay, it becomes possible to follow the CPB buffer model. In this case, the MPU timestamp descriptor stores a value based on the picture timing SEI.

[0338] Furthermore, the transmission method used in receiving operations where such auxiliary data is used is not limited to the MMT method, but can also be applied to streaming transmission of packets composed of ISOBMFF file formats, such as MPEG-DASH.

[0339] [How to send multiple movie fragments when one MPU is composed of them] In the explanations from Figure 19 onwards, one MPU was assumed to consist of one movie fragment. However, here we will explain the case where one MPU is composed of multiple movie fragments. Figure 33 shows the configuration of an MPU composed of multiple movie fragments.

[0340] In Figure 33, the samples (#1-#6) stored in one MPU are divided and stored into two movie fragments. The first movie fragment is generated based on samples #1-#3, and the corresponding moof box is generated. The second movie fragment is generated based on samples #4-#6, and the corresponding moof box is generated.

[0341] The headers of the moof and mdat boxes in the first movie fragment are stored in the MMT payload and MMT packet as movie fragment metadata #1. On the other hand, the headers of the moof and mdat boxes in the second movie fragment are stored in the MMT payload and MMT packet as movie fragment metadata #2. Note that in Figure 33, the MMT payload containing the movie fragment metadata is hatched.

[0342] The number of samples that make up the MPU and the number of samples that make up the movie fragments are arbitrary. For example, the number of samples that make up the MPU may be the number of samples per GOP, and half the number of samples per GOP may be used as the number of movie fragments, resulting in two movie fragments.

[0343] Note that this example shows two movie fragments (moof box and mdat box) in one MPU, but one MPU may contain three or more movie fragments, not just two. Also, the samples stored in the movie fragments do not have to be divided equally; they may be divided into any number of samples.

[0344] In Figure 33, the MPU metadata unit and the MF metadata unit are stored in the MMT payload as data units. However, the transmitter 15 may store units such as ftyp, mmpu, moov, and moof as data units in the MMT payload, or it may store the data units in fragmented units in the MMT payload. Alternatively, the transmitter 15 may store the data units in aggregated units in the MMT payload.

[0345] Furthermore, in Figure 33, samples are stored in the MMT payload on a sample-by-sample basis. However, the transmitter 15 may configure data units not on a sample-by-sample basis, but on a NAL unit basis or on a unit formed by combining multiple NAL units, and store them in the MMT payload on a data unit basis. Alternatively, the transmitter 15 may store the data units in the MMT payload in fragmented units, or in aggregated units.

[0346] In Figure 33, the MPU is configured in the order of moof#1, mdat#1, moof#2, and mdat#2, and moof#1 is assigned an offset as if the corresponding mdat#1 is appended to it. However, the offset may also be assigned as if mdat#1 is appended to it before moof#1. In this case, however, it is not possible to generate movie fragment metadata in the form of moof+mdat, and the headers of moof and mdat are transmitted separately.

[0347] Next, we will explain the transmission order of MMT packets when the MPU configuration described in Figure 33 is transmitted. Figure 34 is a diagram illustrating the transmission order of MMT packets.

[0348] Figure 34(a) shows the transmission order when sending MMT packets in the configuration order of the MPU shown in Figure 33. Specifically, Figure 34(a) shows an example in which the following are transmitted in this order: MPU metadata, MF metadata #1, media data #1 (samples #1-#3), MF metadata #2, and media data #2 (samples #4-#6).

[0349] Figure 34(b) shows an example of sending MPU metadata, media data #1 (samples #1-#3), MF metadata #1, media data #2 (samples #4-#6), and MF metadata #2 in that order.

[0350] Figure 34(c) shows an example of sending media data #1 (samples #1-#3), MPU metadata, MF metadata #1, media data #2 (samples #4-#6), and MF metadata #2 in that order.

[0351] MF Meta #1 is generated using samples #1-#3, and MF Meta #2 is generated using samples #4-#6. Therefore, when the transmission method shown in Figure 34(a) is used, a delay occurs in the transmission of sample data due to encapsulation.

[0352] In contrast, when the transmission methods shown in Figure 34(b) and Figure 34(c) are used, samples can be transmitted without waiting for MF metadata to be generated, thus eliminating encapsulation delays and reducing end-to-end delays.

[0353] Furthermore, in the transmission sequence shown in Figure 34(a), one MPU is divided into multiple movie fragments, and the number of samples stored in the MF metadata is smaller than in Figure 19. Therefore, the delay due to encapsulation can be reduced compared to Figure 19.

[0354] In addition to the methods described herein, for example, the transmitter 15 may concatenate MF Meta #1 and MF Meta #2 and transmit them together at the end of the MPU. In this case, the MF Meta of different movie fragments may be aggregated and stored in a single MMT payload. Alternatively, the MF Meta of different MPUs may be aggregated and stored in the MMT payload.

[0355] [How to receive when a single MPU consists of multiple movie fragments] This section describes an example of the operation of the receiving device 20, which receives and decodes MMT packets transmitted in the transmission order described in Figure 34(b). Figures 35 and 36 are diagrams illustrating this example of operation.

[0356] The receiving device 20 receives MMT packets containing MPU metadata, samples, and MF metadata, transmitted in the order shown in Figure 35. The sample data is concatenated in the order it is received.

[0357] The receiver 20 concatenates the data at T1, the time when MF Meta #1 was received, as shown in (1) of Figure 36, to form an MP4. This allows the receiver 20 to acquire samples #1-#3 based on the MPU metadata and the information in MF Meta #1, and to perform decoding based on the PTS, DTS, offset, and size information contained in the MF Meta.

[0358] Furthermore, the receiver 20 concatenates the data at T2, the time when MF Meta #2 was received, as shown in (2) of Figure 36, to form an MP4. This allows the receiver 20 to acquire samples #4-#6 based on the MPU metadata and the information in MF Meta #2, and to perform decoding based on the PTS, DTS, offset, and size information of the MF Meta. Alternatively, the receiver 20 may acquire samples #1-#6 based on the information in MF Meta #1 and MF Meta #2 by concatenating the data as shown in (3) of Figure 36 to form an MP4.

[0359] By having a single MPU split the movie into multiple fragments, the time it takes for the MPU to acquire the initial MF metadata is reduced, thus speeding up the decoding start time. Additionally, the buffer size required to store samples before decoding can be reduced.

[0360] The transmitting device 15 may also set the movie fragment division unit such that the time from transmitting (or receiving) the first sample in the movie fragment to transmitting (or receiving) the MF meta corresponding to the movie fragment is shorter than the initial_cpb_removal_delay specified by the encoder. By setting it in this way, the receive buffer can follow the cpb buffer, enabling low-latency decoding. In this case, absolute times based on initial_cpb_removal_delay can be used for PTS and DTS.

[0361] Furthermore, the transmitting device 15 may divide the movie fragments at equal intervals, or divide subsequent movie fragments at shorter intervals than the previous movie fragments. This ensures that the receiving device 20 always receives MF metadata containing information about the sample before decoding the sample, enabling continuous decoding.

[0362] The absolute time for PTS and DTS can be calculated using the following two methods.

[0363] (1) The absolute time of PTS and DTS is determined based on the reception time (T1 or T2) of MF Meta #1 and MF Meta #2, and the relative time of PTS and DTS included in the MF Meta.

[0364] (2) The absolute time of the PTS and DTS is determined based on the absolute time signaled by the sender, such as the MPU timestamp descriptor, and the relative time of the PTS and DTS included in the MF metadata.

[0365] Furthermore, (2-A) the absolute time signaled by the transmitter 15 may be an absolute time calculated based on the initial_cpb_removal_delay specified by the encoder.

[0366] Furthermore, (2-B) the absolute time signaled by the transmitter 15 may be an absolute time calculated based on the predicted reception time of the MF meta.

[0367] Furthermore, MF Meta #1 and MF Meta #2 may be transmitted repeatedly. By repeatedly transmitting MF Meta #1 and MF Meta #2, the receiving device 20 can acquire the MF Meta again even if it was previously unable to acquire it due to packet loss or other reasons.

[0368] The payload header of an MFU containing samples that make up a movie fragment can store an identifier indicating the order of the movie fragment. On the other hand, the MMT payload does not include an identifier indicating the order of the MF meta that make up the movie fragment. Therefore, the receiving device 20 identifies the order of the MF meta using packet_sequence_number. Alternatively, the transmitting device 15 may signal by storing an identifier indicating which movie fragment the MF meta belongs to in the control information (message, table, descriptor), MMT header, MMT payload header, or data unit header.

[0369] The transmitting device 15 may transmit MPU metadata, MF metadata, and samples in a predetermined transmission order, and the receiving device 20 may perform reception processing based on the predetermined transmission order. Alternatively, the transmitting device 15 may signal the transmission order, and the receiving device 20 may select (determine) the reception processing based on the signaling information.

[0370] The above-described receiving method will be explained using Figure 37. Figure 37 is a flowchart of the operation of the receiving method described in Figures 35 and 36.

[0371] First, the receiving device 20 determines (identifies) whether the data contained in the MMT payload is MPU metadata, MF metadata, or sample data (MFU) based on the fragment type indicated in the MMT payload (S601, S602). If the data is sample data, the receiving device 20 buffers the sample and waits for the reception of the MF metadata corresponding to the sample and for the start of decoding (S603).

[0372] On the other hand, in step S602, if the data is MF metadata, the receiving device 20 obtains sample information (PTS, DTS, location information, and size) from the MF metadata, obtains a sample based on the obtained sample information, decodes the sample based on the PTS and DTS, and presents it (S604).

[0373] Although not shown in the diagram, if the data is MPU metadata, the MPU metadata contains initialization information necessary for decoding. Therefore, the receiving device 20 stores this information and uses it to decode the sample data in step S604.

[0374] Furthermore, when the receiving device 20 stores the received MPU data (MPU metadata, MF metadata, and sample data) in the storage device, it stores the data after rearranging it according to the MPU configuration as described in Figure 19 or Figure 33.

[0375] On the transmitting side, MMT packets are assigned a packet sequence number to packets that have the same packet ID. At this time, the packet sequence number may be assigned after the MMT packets, including MPU metadata, MF metadata, and sample data, have been sorted in the order of transmission, or it may be assigned in the order before sorting.

[0376] If packet sequence numbers are assigned in the order before sorting, the receiving device 20 can sort the data according to the MPU's configuration order based on the packet sequence numbers, making storage easier.

[0377] [Method for detecting the beginning of an access unit and the beginning of a slice segment] This section describes how to detect the beginning of an access unit or slice segment based on information from the MMT packet header and MMT payload header.

[0378] This section presents two examples: one where non-VCL NAL units (such as access unit delimiters, VPS, SPS, PPS, and SEI) are stored together as a data unit in the MMT payload, and another where each non-VCL NAL unit is treated as a data unit and these data units are aggregated to store in a single MMT payload.

[0379] Figure 38 shows a case where non-VCL NAL units are treated as individual data units and aggregated.

[0380] In the case of Figure 38, the beginning of the access unit is the first data of an MMT payload containing an MMT packet with a fragment_type value of MFU, an aggregation_flag value of 1, and an offset value of 0. In this case, the Fragmentation_indicator value is 0.

[0381] In addition, in the case of Figure 38, the beginning of the slice segment is an MMT packet with a fragment_type value of MFU, and is the first data of an MMT payload with an aggregation_flag value of 0 and a fragmentation_indicator value of 00 or 01.

[0382] Figure 39 shows the case where non-VCL NAL units are grouped together as a data unit. The field values ​​of the packet header are as shown in Figure 17 (or Figure 18).

[0383] In Figure 39, the beginning of the access unit is the first data of the payload in a packet with an Offset value of 0.

[0384] Furthermore, in the case of Figure 39, the beginning of a slice segment is the first data in the payload of a packet whose Offset value is a value other than 0 and whose fragmentation indicator value is 00 or 01.

[0385] [Reception process when packet loss occurs] Normally, when transmitting MP4 format data in an environment where packet loss occurs, the receiving device 20 restores the packets using ALFEC (Application Layer FEC) or packet retransmission control.

[0386] However, if packet loss occurs in streaming, such as broadcasting, where AL-FEC cannot be used, the packets cannot be recovered.

[0387] The receiving device 20 needs to resume decoding video and audio after data is lost due to packet loss. To do this, the receiving device 20 needs to detect the beginning of the access unit or NAL unit and start decoding from the beginning of the access unit or NAL unit.

[0388] However, since MP4 format NAL units do not have a start code at the beginning, the receiving device 20 cannot detect the beginning of the access unit or NAL unit even when analyzing the stream.

[0389] Figure 40 is a flowchart showing the operation of the receiving device 20 when packet loss occurs.

[0390] The receiving device 20 detects packet loss using the PacketSequence number, packet counter, fragment counter, etc., in the header of the MMT packet or MMT payload (S701), and determines which packet has been lost based on the context (S702).

[0391] If the receiving device 20 determines that no packet loss has occurred (No in S702), it constructs an MP4 file and decodes the access unit or NAL unit (S703).

[0392] If the receiving device 20 determines that packet loss has occurred (Yes in S702), it generates a NAL unit corresponding to the lost NAL unit using dummy data and constructs an MP4 file (S704). When the receiving device 20 puts dummy data into the NAL unit, it indicates that it is dummy data in the NAL unit type.

[0393] Furthermore, the receiving device 20 can detect the beginning of the next access unit or NAL unit based on the method described in Figures 17, 18, 38, and 39, and resume decoding by inputting the beginning data to the decoder (S705).

[0394] If packet loss occurs, the receiving device 20 may resume decoding from the beginning of the access unit and NAL unit based on information detected based on the packet header, or it may resume decoding from the beginning of the access unit and NAL unit based on the header information of the reconstructed MP4 file, including a NAL unit of dummy data.

[0395] When the receiving device 20 stores MP4 files (MPU), packet data (such as NAL units) lost due to packet loss may be separately acquired from broadcasts or communications and stored (replaced).

[0396] In this case, when the receiving device 20 retrieves the lost packet from the communication, it notifies the server of the information of the lost packet (such as the packet ID, MPU sequence number, packet sequence number, IP data flow number, and IP address) and retrieves the packet. The receiving device 20 may retrieve not only the lost packet but also the packets before and after the lost packet simultaneously.

[0397] [How to structure movie fragments] This section provides a detailed explanation of how to construct movie fragments.

[0398] As explained in Figure 33, the number of samples constituting a movie fragment and the number of movie fragments constituting one MPU are arbitrary. For example, the number of samples constituting a movie fragment and the number of movie fragments constituting one MPU may be a fixed predetermined number or may be determined dynamically.

[0399] Here, by configuring the movie fragments on the transmitting side (transmitting device 15) to satisfy the following conditions, low-latency decoding in the receiving device 20 can be guaranteed.

[0400] The conditions are as follows:

[0401] The transmitting device 15 generates and transmits MF metadata as movie fragments, dividing the sample data into units such that the receiving device 20 can always receive MF metadata containing information about any given sample (Sample(i)) before the decoding time (DTS(i)) of that sample.

[0402] Specifically, the transmitting device 15 constructs a movie fragment using samples (including the i-th sample) that have been encoded before DTS(i).

[0403] To ensure low-latency decoding, the following methods can be used to dynamically determine the number of samples that make up a movie fragment and the number of movie fragments that make up one MPU.

[0404] (1) At the start of decoding, the decoding time DTS(0) of the first sample Sample(0) in the GOP is the time based on initial_cpb_removal_delay. The transmitting device constructs the first movie fragment using the encoded samples at a time before DTS(0). The transmitting device 15 also generates MF metadata corresponding to the first movie fragment and transmits it at a time before DTS(0).

[0405] (2) The transmitting device 15 configures movie fragments in subsequent samples in such a way that the above conditions are met.

[0406] For example, if the first sample of a movie fragment is the k-th sample, the MF metadata of the movie fragment containing the k-th sample is transmitted by the decoding time DTS(k) of the k-th sample. If the encoding completion time of the l-th sample is before DTS(k) and the encoding completion time of the (l+1)-th sample is after DTS(k), the transmitting device 15 constructs a movie fragment using samples k through l.

[0407] The transmitting device 15 may also construct a movie fragment using samples from the kth sample down to the lth sample.

[0408] (3) After the encoding of the last sample of the MPU is completed, the transmitting device 15 constructs a movie fragment using the remaining samples, generates MF metadata corresponding to the movie fragment, and transmits it.

[0409] The transmitting device 15 may construct a movie fragment using only some of the encoded samples, rather than using all of the encoded samples.

[0410] In the above example, we showed how the number of samples constituting a movie fragment and the number of movie fragments constituting one MPU are dynamically determined based on the above conditions to ensure low-latency decoding. However, the method for determining the number of samples and movie fragments is not limited to this method. For example, the number of movie fragments constituting one MPU may be fixed to a predetermined value, and the number of samples may be determined to satisfy the above conditions. Alternatively, the number of movie fragments constituting one MPU and the time at which the movie fragments are divided (or the code amount of the movie fragments) may be fixed to predetermined values, and the number of samples may be determined to satisfy the above conditions.

[0411] Additionally, if the MPU is divided into multiple movie fragments, information indicating whether the MPU is divided into multiple movie fragments, the attributes of the divided movie fragments, or the attributes of the MF meta for the divided movie fragments may be transmitted.

[0412] Here, movie fragment attributes refer to information indicating whether a movie fragment is the first movie fragment in the MPU, the last movie fragment in the MPU, or any other movie fragment.

[0413] Furthermore, the attributes of an MF meta indicate whether the MF meta corresponds to the first movie fragment of the MPU, the last movie fragment of the MPU, or any other movie fragment.

[0414] The transmitting device 15 may also store and transmit the number of samples constituting a movie fragment and the number of movie fragments constituting one MPU as control information.

[0415] [Operation of the receiving device] The operation of the receiving device 20 based on the movie fragment configured as described above will now be explained.

[0416] The receiving device 20 determines the absolute time of the PTS and DTS based on the absolute time signaled from the transmitting side, such as the MPU timestamp descriptor, and the relative time of the PTS and DTS included in the MF metadata.

[0417] Based on information indicating whether the MPU is divided into multiple movie fragments, the receiving device 20 processes the data as follows, based on the attributes of the divided movie fragments, if the MPU is divided.

[0418] (1) If the movie fragment is the first movie fragment of the MPU, the receiving device 20 generates the absolute times of the PTS and DTS using the absolute time of the first sample PTS included in the MPU timestamp descriptor and the relative times of the PTS and DTS included in the MF metadata.

[0419] (2) If the movie fragment is not the first movie fragment of the MPU, the receiving device 20 generates the absolute times of the PTS and DTS using the relative times of the PTS and DTS included in the MF metadata, without using the information of the MPU timestamp descriptor.

[0420] (3) If the movie fragment is the last movie fragment of the MPU, the receiving device 20 calculates the absolute time of the PTS and DTS for all samples and then resets the PTS and DTS calculation process (relative time addition process). The reset process may also be performed for the first movie fragment of the MPU.

[0421] The receiving device 20 may determine whether the movie fragment is divided as shown below. The receiving device 20 may also obtain attribute information of the movie fragment as shown below.

[0422] For example, the receiving device 20 may determine whether the movie fragments are split based on the value of the movie_fragment_sequence_number field, an identifier indicating the order of the movie fragments shown in the MMTP (MMT Protocol) payload header.

[0423] Specifically, the receiving device 20 may determine that an MPU is divided into multiple movie fragments if the number of movie fragments contained in one MPU is 1, the value of the movie_fragment_sequence_number field is 1, and there are other values ​​of 2 or more for that field.

[0424] Furthermore, the receiving device 20 may determine that an MPU is divided into multiple movie fragments if the number of movie fragments contained in one MPU is 1, the value of the movie_fragment_sequence_number field is 0, and there is a value other than 0 for the field value.

[0425] Similarly, the attribute information of a movie fragment may also be determined based on the movie_fragment_sequence_number.

[0426] Alternatively, instead of using movie_freagment_sequence_number, it is also possible to determine whether a movie fragment is split and to identify the attribute information of the movie fragment by counting the transmissions of movie fragments and MF metadata included in a single MPU.

[0427] With the configuration of the transmitter 15 and receiver 20 described above, the receiver 20 can receive movie fragment metadata at shorter intervals than the MPU, enabling low-latency decoding. Furthermore, low-latency decoding can be performed using decoding processing based on the MP4 parsing method.

[0428] As explained above, the receiving operation when the MPU is divided into multiple movie fragments will now be explained using a flowchart. Figure 41 is a flowchart of the receiving operation when the MPU is divided into multiple movie fragments. This flowchart illustrates the operation of step S604 in Figure 37 in more detail.

[0429] First, the receiving device 20 acquires MF metadata based on the data type indicated in the MMTP payload header if the data type is MF metadata (S801).

[0430] Next, the receiving device 20 determines whether the MPU is divided into multiple movie fragments (S802). If the MPU is divided into multiple movie fragments (Yes in S802), it determines whether the received MF metadata is the metadata at the beginning of the MPU (S803). If the received MF metadata is the MF metadata at the beginning of the MPU (Yes in S803), the receiving device 20 calculates the absolute time of the PTS and DTS from the absolute time of the PTS indicated in the MPU timestamp descriptor and the relative time of the PTS and DTS indicated in the MF metadata (S804), and determines whether it is the metadata at the end of the MPU (S805).

[0431] On the other hand, if the received MF metadata is not the MF metadata at the beginning of the MPU (No in S803), the receiving device 20 does not use the information from the MPU timestamp descriptor, but instead calculates the absolute time of the PTS and DTS using the relative time of the PTS and DTS indicated in the MF metadata (S808), and proceeds to the processing in step S805.

[0432] In step S805, if it is determined that this is the last MF metadata for the MPU (Yes in S805), the receiving device 20 calculates the absolute time of the PTS and DTS for all samples and then resets the PTS and DTS calculation process. In step S805, if it is determined that this is not the last MF metadata for the MPU (No in S805), the receiving device 20 terminates the process.

[0433] Furthermore, if it is determined in step S802 that the MPU has not been divided into multiple movie fragments (No in S802), the receiving device 20 acquires sample data based on the MF metadata transmitted after the MPU and determines the PTS and DTS (S807).

[0434] Then, although not shown in the diagram, the receiving device 20 finally performs decoding and presentation processing based on the determined PTS and DTS.

[0435] [Challenges that arise when splitting movie fragments, and their solutions] So far, we have explained how to reduce end-to-end delay by splitting movie fragments. From here, we will explain the new challenges that arise when splitting movie fragments, and how to solve them.

[0436] First, as background information, we will explain the picture structure in encoded data. Figure 42 shows an example of the picture prediction structure for each TemporalId when achieving temporal scalability.

[0437] In encoding schemes such as MPEG-4 AVC and HEVC (High Efficiency Video Coding), temporal scalability can be achieved by using B-pictures (bidirectional reference prediction pictures) that can be referenced by other pictures.

[0438] The TemporalId shown in Figure 42(a) is an identifier for the hierarchy of the coding structure, where a larger TemporalId value indicates a deeper hierarchy. The square blocks represent pictures, with Ix representing an I-picture (in-screen prediction picture), Px representing a P-picture (forward-referencing prediction picture), and Bx and bx representing B-pictures (bidirectional-referencing prediction pictures). The x in Ix / Px / Bx indicates the display order, representing the order in which the pictures are displayed. The arrows between pictures indicate reference relationships; for example, picture B4 generates prediction images using I0 and B8 as reference images. Here, it is forbidden for one picture to use another picture with a TemporalId greater than its own as a reference image. The hierarchy is defined to provide temporal scalability. For example, in Figure 42, decoding all pictures yields a 120fps (frames per second) video, but decoding only the layers with TemporalId from 0 to 3 yields a 60fps video.

[0439] Figure 43 shows the relationship between the decryption time (DTS) and display time (PTS) for each picture in Figure 42. For example, picture I0 shown in Figure 43 is displayed after the decryption of B4 is complete, so that no gap occurs during decryption and display.

[0440] As shown in Figure 43, when the prediction structure includes a B-picture, the decoding order and the display order are different, so the receiving device 20 needs to perform a delay process on the picture and a reordering process on the picture after decoding it.

[0441] The above describes an example of a picture prediction structure in terms of time scalability. However, even when time scalability is not used, depending on the prediction structure, picture delay processing and reordering may be necessary. Figure 44 shows an example of a picture prediction structure that requires picture delay processing and reordering. The numbers in Figure 44 indicate the decoding order.

[0442] As shown in Figure 44, depending on the prediction structure, the first sample in the decoding order may differ from the first sample in the presentation order. In Figure 44, the first sample in the presentation order is the fourth sample in the decoding order. Note that Figure 44 is just one example of a prediction structure, and prediction structures are not limited to this one. In other prediction structures as well, the first sample in the decoding order may differ from the first sample in the presentation order.

[0443] Figure 45, similar to Figure 33, shows an example where an MPU composed of MP4 format is divided into multiple movie fragments and stored in an MMTP payload and MMTP packets. Note that the number of samples that make up the MPU and the number of samples that make up the movie fragments are arbitrary. For example, the number of samples that make up the MPU may be the number of samples per GOP, and half the number of samples per GOP may be used as movie fragments to form two movie fragments. One sample may be one movie fragment, or the samples that make up the MPU may not be divided at all.

[0444] Figure 45 shows an example where one MPU contains two movie fragments (a moof box and an mdat box), but an MPU does not have to contain two movie fragments. An MPU may contain three or more movie fragments, or it may be the number of samples contained in the MPU. Furthermore, the samples stored in the movie fragments do not have to be divided equally; they may be divided into any number of samples.

[0445] Movie fragment metadata (MF metadata) contains information about the PTS, DTS, offset, and size of the sample contained in the movie fragment. When decoding a sample, the receiving device 20 extracts the PTS and DTS from the MF metadata containing the sample information and determines the decoding timing and presentation timing.

[0446] From here on, for the sake of detailed explanation, we will denote the absolute value of the decoding time of sample i as DTS(i) and the absolute value of the presentation time as PTS(i).

[0447] In the MF metadata, the timestamp information stored within moof for the i-th sample specifically consists of the relative decoding times of the i-th sample and the (i+1)-th sample, and the relative decoding time of the i-th sample and its presentation time, which will be hereafter referred to as DT(i) and CT(i).

[0448] Movie fragment metadata #1 contains DT(i) and CT(i) for samples #1-#3, and movie fragment metadata #2 contains DT(i) and CT(i) for samples #4-#6.

[0449] Furthermore, the absolute value of the PTS of the access unit at the beginning of the MPU is stored in the MPU timestamp descriptor, and the receiving device 20 calculates the PTS and DTS based on the PTS_MPU of the access unit at the beginning of the MPU, the CT, and the DT.

[0450] Figure 46 is a diagram illustrating the calculation method and challenges of PTS and DTS when the MPU is configured using samples #1-#10.

[0451] Figure 46(a) shows an example where the MPU is not divided into movie fragments, Figure 46(b) shows an example where the MPU is divided into two movie fragments of 5 samples each, and Figure 46(c) shows an example where the MPU is divided into 10 movie fragments of sample units each.

[0452] As explained in Figure 45, when PTS and DTS are calculated using the MPU timestamp descriptor and the timestamp information (CT and DT) within the MP4, the sample that appears first in the presentation order in Figure 44 is the fourth sample in the decoding order. Therefore, the PTS stored in the MPU timestamp descriptor is the PTS (absolute value) of the fourth sample in the decoding order. Hereafter, this sample will be referred to as sample A. The first sample in the decoding order will be referred to as sample B.

[0453] Since the absolute time information related to the timestamp is only from the MPU timestamp descriptor, the receiving device 20 cannot calculate the PTS (absolute time) and DTS (absolute time) of the other samples until sample A arrives. The receiving device 20 also cannot calculate the PTS and DTS of sample B.

[0454] In the example shown in Figure 46(a), sample A is included in the same movie fragment as sample B and is stored in a single MF meta. Therefore, the receiving device 20 can immediately determine the DTS of sample B after receiving the MF meta.

[0455] In the example shown in Figure 46(b), sample A is included in the same movie fragment as sample B and is stored in a single MF meta. Therefore, the receiving device 20 can immediately determine the DTS of sample B after receiving the MF meta.

[0456] In the example shown in Figure 46(c), sample A is contained in a different movie fragment than sample B. Therefore, the receiving device 20 cannot determine the DTS of sample B until it has received the MF metadata, including the CT and DT, of the movie fragment containing sample A.

[0457] Therefore, in the example of Figure 46(c), the receiving device 20 cannot start decoding immediately after the arrival of the B sample.

[0458] Thus, if a movie fragment containing sample B does not contain sample A, the receiving device 20 cannot start decoding sample B until it has received the MF metadata relating to the movie fragment containing sample A.

[0459] This issue arises when the first sample presented and the first sample decoded do not match, and the movie fragment is split before sample A and sample B are stored in the same movie fragment. This issue also occurs regardless of whether the MF metadata is delayed or delayed.

[0460] Thus, when the first sample in the presentation order does not match the first sample in the decoding order, and when sample A and sample B are not stored in the same movie fragment, the DTS cannot be determined immediately after receiving sample B. Therefore, the transmitting device 15 separately transmits the DTS (absolute value) of sample B, or information that allows the receiving side to calculate the DTS (absolute value) of sample B. Such information may be transmitted using control information or a packet header.

[0461] The receiving device 20 uses this information to calculate the DTS (absolute value) of sample B. Figure 47 is a flowchart of the receiving operation when the DTS is calculated using this information.

[0462] The receiving device 20 receives the first movie fragment of the MPU (S901) and determines whether sample A and sample B are stored in the same movie fragment (S902). If they are stored in the same movie fragment (Yes in S902), the receiving device 20 calculates the DTS using only the MF metadata information, without using the DTS (absolute time) of sample B, and starts decoding (S904). In step S904, the receiving device 20 may also determine the DTS using the DTS of sample B.

[0463] On the other hand, if sample A and sample B are not stored in the same movie fragment in step S902 (No in S902), the receiving device 20 acquires the DTS (absolute time) of sample B, determines the DTS, and starts decoding (S903).

[0464] In the above explanation, we described an example of calculating the absolute value of the decoding time and the absolute value of the presentation time for each sample using the MF metadata (timestamp information stored in the moof file of the MP4 format) in the MMT standard. However, it goes without saying that the MF metadata can also be replaced with any control information that can be used to calculate the absolute value of the decoding time and the absolute value of the presentation time for each sample. Examples of such control information include control information in which the relative value CT(i) of the decoding times of the i-th sample and the (i+1)-th sample is replaced with the relative value of the presentation times of the i-th sample and the (i+1)-th sample, or control information that includes both the relative value CT(i) of the decoding times of the i-th sample and the (i+1)-th sample and the relative value of the presentation times of the i-th sample and the (i+1)-th sample.

[0465] (Embodiment 3) [overview] Embodiment 3 describes a method for transmitting content and a data structure when transmitting content such as video, audio, subtitles, and data broadcasting via broadcast. In other words, it describes a method for transmitting content and a data structure specifically for playback of broadcast streams.

[0466] In Embodiment 3, an example is described in which the MMT method (hereinafter also simply referred to as MMT) is used as the multiplexing method, but other multiplexing methods such as MPEG-DASH or RTP may also be used.

[0467] First, we will explain in detail how Data Units (DUs) are stored in the payload in MMT. Figure 48 is a diagram illustrating how Data Units are stored in the payload in MMT.

[0468] In MMT, the transmitting device stores a portion of the data constituting the MPU as a data unit in the MMTP payload and transmits it with a header. The header includes the MMTP payload header and the MMTP packet header. The data unit can be either a NAL unit or a sample.

[0469] Figure 48(a) shows an example in which a transmitting device aggregates multiple data units and stores them in a single payload. In the example in Figure 48(a), a Data Unit Header (DUH) and Data Unit Length (DUL) are added to the beginning of each of the multiple data units, and multiple data units with data unit headers and data unit lengths are stored together in the payload.

[0470] Figure 48(b) shows an example where one data unit is stored in one payload. In the example in Figure 48(b), a data unit header is added to the beginning of the data unit before it is stored in the payload. Figure 48(c) shows an example where one data unit is split, and each split data unit is given a data unit header before being stored in the payload.

[0471] Data units include timed-MFUs (media containing synchronization information such as video, audio, or subtitles), non-timed-MFUs (media not containing synchronization information such as files), MPU metadata, and MF metadata. A data unit header is defined according to the type of data unit. Note that MPU metadata and MF metadata do not have data unit headers.

[0472] Furthermore, while transmitting devices are generally not allowed to aggregate different types of data units, they may be specified to allow such aggregation. For example, if the size of MF metadata is small, such as when it is divided into movie fragments for each sample, the number of packets can be reduced by aggregating the MF metadata and media data, and furthermore, the transmission capacity can be reduced.

[0473] If the data unit is an MFU, some information about the MPU, such as information for configuring the MPU (MP4), is stored as a header.

[0474] For example, the header of a timed-MFU includes fields such as movie_fragment_sequence_number, sample_number, offset, priority, and dependency_counter, while the header of a non-timed-MFU includes item_iD. The meaning of each field is shown in standards such as ISO / IEC 23008-1 or ARIB STD-B60. The meaning of each field as defined in such standards will be explained below.

[0475] movie_fragment_sequence_number indicates the sequence number of the movie fragment to which the MFU belongs, and is also specified in ISO / IEC 14496-12.

[0476] The `sample_number` indicates the sample number to which the MFU belongs, and is also specified in ISO / IEC 14496-12.

[0477] The offset indicates the offset amount of the MFU in the sample to which the MFU belongs, in bytes.

[0478] The priority value indicates the relative importance of the MFU within the MPU to which it belongs. An MFU with a higher priority value is more important than an MFU with a lower priority value.

[0479] The dependency_counter indicates the number of MFUs whose decoding process depends on the MFU in question (i.e., the number of MFUs that cannot be decoded unless the MFU in question is decoded first). For example, if an MFU is HEVC and a B-picture or P-picture references an I-picture, then the B-picture or P-picture cannot be decoded unless the I-picture is decoded first.

[0480] Therefore, if the MFU is in sample units, the dependency_counter in the I-picture's MFU will show the number of pictures that reference that I-picture. If the MFU is in NAL unit units, the dependency_counter in the MFU belonging to the I-picture will show the number of NAL units belonging to the pictures that reference that I-picture. Furthermore, in the case of time-hierarchically encoded video signals, the MFU of the extension layer depends on the MFU of the base layer, so the dependency_counter in the base layer's MFU will show the number of MFUs in the extension layer. This field cannot be generated until the number of dependent MFUs has been determined.

[0481] The item_iD is an identifier that uniquely identifies the item.

[0482] [MP4 unsupported mode] As explained in Figures 19 and 21, the transmitting device can transmit the MPU in MMT in several ways: by sending the MPU metadata or MF metadata before or after the media data, or by sending only the media data. The receiving device can decode using an MP4-compliant receiving device or method, or by decoding without using a header.

[0483] One method of transmitting data specifically for broadcast stream playback is, for example, a transmission method that does not support MP4 reconstruction in the receiving device.

[0484] Transmission methods that do not support MP4 reconstruction in the receiving device include, for example, methods that do not transmit metadata (MPU metadata and MF metadata), as shown in Figure 21(b). In this case, the field value of the fragment type (information indicating the type of data unit) included in the MMTP packet is fixed at 2 (=MFU).

[0485] If metadata is not transmitted, as explained above, MP4-compliant receivers cannot decode the received data as MP4, but they can decode it without using metadata (headers).

[0486] Therefore, metadata is not necessarily essential information for broadcast stream decoding and playback. Similarly, the data unit header information in the timed-MFU, as explained in Figure 48, is information for reconstructing the MP4 in the receiving device. Since it is not necessary to reconstruct the MP4 during broadcast stream playback, the data unit header information in the timed-MFU (hereinafter also referred to as the timed-MFU header) is not necessarily essential information for broadcast stream playback.

[0487] The receiving device can easily reconstruct the MP4 file using metadata and information for reconstructing the MP4 in the data unit header (hereinafter also referred to as MP4 configuration information). However, the receiving device cannot reconstruct the MP4 file if only one of the metadata or the MP4 configuration information in the data unit header is transmitted. There is little benefit to transmitting only one of the metadata or the information for reconstructing the MP4, and generating and transmitting unnecessary information leads to increased processing load and decreased transmission efficiency.

[0488] Therefore, the transmitting device controls the data structure and transmission of MP4 configuration information using the following method. The transmitting device decides whether or not to include MP4 configuration information in the data unit header based on whether or not metadata is transmitted. Specifically, if metadata is transmitted, the transmitting device includes MP4 configuration information in the data unit header; if metadata is not transmitted, it does not include MP4 configuration information in the data unit header.

[0489] One way to avoid displaying MP4 configuration information in the data unit header is to use methods such as those described below.

[0490] 1. The transmitting device reserves the MP4 configuration information and does not use it. This reduces the processing load on the sending side (the processing load on the transmitting device) that generates the MP4 configuration information.

[0491] 2. The transmitting device removes MP4 configuration information and compresses the header. This reduces the processing load on the sending side that generates MP4 configuration information, and also reduces transmission capacity.

[0492] Furthermore, if the transmitting device removes MP4 configuration information and compresses the header, it may indicate a flag that the MP4 configuration information has been removed (compressed). The flag may be shown in the header (MMTP packet header, MMTP payload header, data unit header) or in the control information, etc.

[0493] Furthermore, information regarding whether metadata is transmitted may be predetermined, or it may be signaled separately in the header or control information and transmitted to the receiving device.

[0494] For example, the MFU header may contain information indicating whether metadata corresponding to the MFU has been transmitted.

[0495] On the other hand, the receiving device can determine whether MP4 configuration information is displayed based on whether metadata is being transmitted.

[0496] If the order of data transmission is predetermined (for example, MPU metadata, MF metadata, media data), the receiving device may determine whether the metadata was received before the media data.

[0497] If MP4 configuration information is provided, the receiving device can use the MP4 configuration information to reconstruct the MP4. Alternatively, the receiving device can use the MP4 configuration information to detect the beginning of other access units or NAL units, etc.

[0498] Note that the MP4 configuration information may consist of all or part of the timed-MFU header.

[0499] Furthermore, the transmitting device also determines whether metadata is being transmitted in the non-timed-MFU header, based on the same criteria. You may decide whether or not to include an ID.

[0500] The transmitting device may display MP4 configuration information in either the timed-MFU or the non-timed-MFU, but not both. If MP4 configuration information is displayed in only one of them, the transmitting device decides whether to display the MP4 configuration information based on whether metadata is transmitted, as well as whether it is a timed-MFU or a non-timed-MFU. The receiving device can determine whether metadata is transmitted and whether MP4 configuration information is displayed based on the timed / non-timed flag.

[0501] In the above explanation, the transmitting device determined whether to display MP4 configuration information based on whether metadata (both MPU metadata and MF metadata) was transmitted. However, the transmitting device may choose not to display MP4 configuration information if a portion of the metadata (either MPU metadata or MF metadata) is not transmitted.

[0502] Furthermore, the transmitting device may decide whether to display MP4 configuration information based on information other than metadata.

[0503] For example, modes such as MP4 support mode and MP4 non-support mode may be defined, and the transmitting device may indicate MP4 configuration information in the data unit header when in MP4 support mode, and not indicate MP4 configuration information in the data unit header when in MP4 non-support mode. Alternatively, the transmitting device may transmit metadata and indicate MP4 configuration information in the data unit header when in MP4 support mode, and not transmit metadata and not indicate MP4 configuration information in the data unit header when in MP4 non-support mode.

[0504] [Transmitter Operation Flow] Next, we will explain the operation flow of the transmitting device. Figure 49 shows the operation flow of the transmitting device.

[0505] The transmitting device first determines whether to transmit metadata (S1001). If the transmitting device determines to transmit metadata (Yes in S1002), it proceeds to step S1003, generates MP4 configuration information, stores it in the header, and transmits it (S1003). In this case, the transmitting device also generates and transmits metadata.

[0506] On the other hand, if the transmitting device determines that metadata should not be transmitted (No in S1002), it transmits the data without generating MP4 configuration information or storing it in the header (S1004). In this case, the transmitting device does not generate or transmit metadata.

[0507] In step S1001, whether or not metadata is transmitted may be predetermined, or it may be determined based on whether metadata is generated within the transmitting device or whether metadata is being transmitted within the transmitting device.

[0508] [Operation Flow of the Receiving Device] Next, we will explain the operation flow of the receiving device. Figure 50 shows the operation flow of the receiving device.

[0509] The receiving device first determines whether metadata is being transmitted (S1101). Whether metadata is being transmitted can be determined by monitoring the fragment type in the MMTP packet payload. Alternatively, whether or not metadata is being transmitted may be predetermined.

[0510] If the receiving device determines that metadata has been transmitted (Yes in S1102), it reconstructs the MP4 and performs decoding using the MP4 configuration information (S1103). On the other hand, if it determines that metadata has not been transmitted (No in S1102), it does not reconstruct the MP4 and performs decoding without using the MP4 configuration information (S1104).

[0511] Furthermore, the receiving device can use the methods described above to detect random access points, the beginning of access units, the beginning of NAL units, etc., without using MP4 configuration information, and can perform decoding, packet loss detection, and recovery from packet loss.

[0512] For example, the first access unit is the first data of an MMT payload with an aggregation_flag value of 1. In this case, the Fragmentation_indicator value is 0.

[0513] Furthermore, the beginning of a slice segment is the first data point of the MMT payload where the aggregation_flag value is 0 and the fragmentation_indicator value is 00 or 01.

[0514] Based on the information described above, the receiving device can detect the beginning of the access unit and the slice segment.

[0515] The receiving device may also analyze the NAL unit header in packets containing the beginning of a data unit with a fragmentation_indicator value of 00 or 01, and detect that the type of NAL unit is an AU delimiter and that the type of NAL unit is a slice segment.

[0516] [Simple Broadcast Mode] Up to this point, we have described a method of transmitting data specifically for broadcast stream playback that does not support MP4 configuration information in the receiving device. However, this is not the only method of transmitting data specifically for broadcast stream playback.

[0517] For example, the following methods may be used as data transmission methods specifically for broadcast stream playback.

[0518] • Transmitting equipment does not need to use AL-FEC in a fixed broadcast reception environment. If AL-FEC is not used, the FEC_type in the MMTP packet header is always fixed at 0.

[0519] • Transmitting devices may always use AL-FEC in mobile broadcasting reception environments and in UDP communication transmission mode. When AL-FEC is used, the FEC_type in the MMTP packet header is always 0 or 1.

[0520] • The transmitting device does not need to perform bulk transmission of assets. If bulk transmission of assets is not performed, location_infolocation, which indicates the number of transmission locations for the assets within MPT, may be fixed at 1.

[0521] • The transmitting device does not need to perform hybrid transmission of assets, programs, and messages.

[0522] Furthermore, for example, if a broadcast simple mode is defined, the transmitting device may set itself to an MP4 non-support mode when broadcast simple mode is enabled, or it may use the data transmission method specifically for broadcast stream playback as described above. Whether or not it is broadcast simple mode may be predetermined, or the transmitting device may store a flag indicating that it is broadcast simple mode as control information and transmit it to the receiving device.

[0523] Furthermore, the transmitting device may, based on whether it is in MP4 non-support mode (i.e., whether metadata is being transmitted), use the data transmission method specifically for broadcast stream playback as described above, in broadcast simple mode if it is in MP4 non-support mode, as explained in Figure 49.

[0524] When the receiving device is in broadcast simple mode, it can decode the MP4 without reconstructing it, treating it as an MP4 non-support mode.

[0525] Furthermore, when the receiving device is in broadcast simple mode, it can determine that it has functions specifically for broadcasting and perform reception processing specifically for broadcasting.

[0526] As a result, in broadcast-simple mode, by using only broadcast-specific functions, unnecessary processing for both the transmitting and receiving devices can be reduced, and transmission overhead can be reduced by not compressing and transmitting unnecessary information.

[0527] Furthermore, if an MP4-unsupported mode is used, hint information supporting storage methods other than MP4 configuration may be provided.

[0528] Other storage methods besides MP4 format include, for example, directly storing MMT packets or IP packets, or converting MMT packets into MPEG-2 TS packets.

[0529] In addition, in non-MP4 supported modes, formats that do not conform to the MP4 structure may be used.

[0530] For example, in the case of MP4 non-support mode, the data stored in the MFU may be in a format that includes a byte start code, rather than a format that includes the size of the NAL unit at the beginning of the NAL unit, which is in MP4 format.

[0531] In MMT, the asset type, which indicates the type of asset, is described by a 4CC registered in MP4REG (http: / / www.mp4ra.org). When HEVC is used as the video signal, 'HEV1' or 'HVC1' is used. 'HVC1' is a format in which the sample may contain a parameter set, while 'HEV1' is a format in which the sample does not contain a parameter set, but the parameter set is included in the sample entry in the MPU metadata.

[0532] In broadcast simple mode or MP4 non-support mode, if MPU metadata and MF metadata are not transmitted, it may be specified that the parameter set must always be included in the sample. Furthermore, it may be specified that the 'HVC1' format must always be used, regardless of whether 'HEV1' or 'HVC1' is indicated in the asset type.

[0533] [Supplement 1: Transmitter] As described above, if metadata is not transmitted, the MP4 configuration information will be set to reserved, and the transmitting device that is not in operation can be configured as shown in Figure 51. Figure 51 is a diagram showing a specific example of the configuration of the transmitting device.

[0534] The transmitting device 300 comprises an encoding unit 301, an assignment unit 302, and a transmitting unit 303. Each of the encoding unit 301, assignment unit 302, and transmitting unit 303 is implemented, for example, by a microcomputer, processor, or dedicated circuit.

[0535] The encoding unit 301 encodes a video signal or an audio signal to generate sample data. Specifically, the sample data is a data unit.

[0536] The assignment unit 302 assigns header information, including MP4 configuration information, to the sample data, which is data in which a video signal or audio signal has been encoded. The MP4 configuration information is information that allows the receiving side to reconstruct the sample data as an MP4 format file, and its content differs depending on whether or not the presentation time of the sample data is specified.

[0537] As described above, the assignment unit 302 includes MP4 configuration information such as movie_fragment_sequence_number, sample_number, offset, priority, and dependency_counter in the header (header information) of timed-MFU, which is an example of sample data (sample data containing information about synchronization) for which a presentation time has been determined.

[0538] On the other hand, the assignment unit 302 includes MP4 configuration information such as item_id in the header (header information) of timed-MFU, which is an example of sample data for which the presentation time is not specified (sample data that does not contain information about synchronization).

[0539] Then, if metadata corresponding to the sample data is not transmitted by the transmission unit 303 (for example, in the case shown in Figure 21(b)), the assignment unit 302 assigns header information that does not include MP4 configuration information to the sample data, depending on whether or not a presentation time for the sample data has been specified.

[0540] Specifically, the assignment unit 302 assigns header information that does not include the first MP4 configuration information to the sample data when a presentation time for the sample data is specified, and assigns header information that includes the second MP4 configuration information to the sample data when a presentation time for the sample data is not specified.

[0541] For example, as shown in step S1004 of Figure 49, if the transmission unit 303 does not transmit metadata corresponding to the sample data, the assignment unit 302 sets the MP4 configuration information to reserved (fixed value), thereby effectively not generating MP4 configuration information and not effectively storing it in the header (header information). The metadata includes MPU metadata and movie fragment metadata.

[0542] The transmitting unit 303 transmits sample data to which header information has been added. More specifically, the transmitting unit 303 packets the sample data to which header information has been added using the MMT method and transmits it.

[0543] As mentioned above, in transmission and reception methods specifically designed for broadcast stream playback, the receiving device does not need to reconstruct the data units into MP4 format. When the receiving device does not need to reconstruct into MP4, the processing load on the transmitting device is reduced by not generating unnecessary information such as MP4 configuration information.

[0544] On the other hand, while the transmitting device must send the necessary information, it also needs to maintain consistency with the standard so as not to have to send any unnecessary additional information separately.

[0545] With a configuration like that of the transmitter 300, by fixing the area where MP4 configuration information is stored, it is possible to transmit only the necessary information according to the standard without transmitting the MP4 configuration information, thus eliminating the need to transmit unnecessary additional information. In other words, the configuration of the transmitter and the processing load of the transmitter can be reduced. Furthermore, by not transmitting unnecessary data, transmission efficiency can be improved.

[0546] [Supplement 2: Receiving device] Furthermore, the receiving device corresponding to the transmitting device 300 may be configured as shown in Figure 52, for example. Figure 52 is a diagram showing another example of the configuration of the receiving device.

[0547] The receiving device 400 comprises a receiving unit 401 and a decoding unit 402. The receiving unit 401 and the decoding unit 402 are implemented, for example, by a microcomputer, a processor, or a dedicated circuit.

[0548] The receiving unit 401 receives sample data which is data in which a video signal or audio signal has been encoded, and which has header information attached that includes MP4 configuration information for reconstructing the sample data as an MP4 format file.

[0549] The decoding unit 402 decodes the sample data without using MP4 configuration information if the receiving unit has not received metadata corresponding to the sample data and the presentation time of the sample data has been determined.

[0550] For example, as shown in step S1104 of Figure 50, the decoding unit 402 performs the decoding process without using MP4 configuration information if the receiving unit 401 does not receive metadata corresponding to the sample data.

[0551] This makes it possible to reduce the configuration of the receiving device 400 and the processing load in the receiving device 400.

[0552] (Embodiment 4) [overview] Embodiment 4 describes a method for storing asynchronous (non-timed) media, such as files, that do not contain synchronization information, into the MPU, and a method for transmitting them using MMTP packets. Although Embodiment 4 uses the MPU in MMT as an example, it is also applicable to DASH, which is also MP4-based.

[0553] First, we will explain the details of how non-timed media (hereinafter also referred to as "asynchronous media data") is stored in the MPU using Figure 53. Figure 53 is a diagram showing how non-timed media is stored in the MPU and how it is transmitted using MMTP packets.

[0554] The MPU, which stores non-timed media, consists of boxes such as ftyp, mmpu, moov, and meta, which store information about the files to be stored in the MPU. A meta box can contain multiple idat boxes, and each idat box stores one file as an item.

[0555] Some of the ftyp, mmpu, moov, and meta boxes constitute a single data unit as MPU metadata, while the item or idat boxes constitute a data unit as an MFU.

[0556] After data units are aggregated or fragmented, they are transmitted as MMTP packets with a data unit header, MMTP payload header, and MMTP packet header added.

[0557] Figure 53 shows an example where File#1 and File#2 are stored in a single MPU. The MPU metadata is not fragmented, and the MFU is fragmented and stored in MMTP packets, but this is not the only option; aggregation or fragmentation may be used depending on the size of the data units. Furthermore, the MPU metadata does not have to be transmitted; in that case, only the MFU is transmitted.

[0558] Header information such as the data unit header includes the itemID (an identifier that uniquely identifies the item), while the MMTP payload header and MMTP packet header include the packet sequence number (a sequence number for each packet) and the MPU sequence number (a sequence number for the MPU, a unique number within the asset).

[0559] Note that the data structures of the MMTP payload header and the MMTP packet header other than the data unit header are the same as those of the timed media (hereinafter also referred to as "synchronized media data") described so far, and include an aggregation_flag, a fragmentation_indicator, a fragment_counter, etc.

[0560] Next, a specific example of the header information when a file (= Item = MFU) is split and packetized will be described using FIGS. 54 and 55.

[0561] FIGS. 54 and 55 are diagrams showing examples of packetizing and transmitting each of a plurality of split data obtained by splitting a file. FIGS. 54 and 55 specifically show information (packet sequence number, fragment counter, fragmentation indicator, MPU sequence number, item ID) included in any of the data unit header, MMTP payload header, and MMTP packet header, which is the header information for each split MMTP packet. Note that FIG. 54 shows an example in which File #1 is split into M (M <= 256) parts, and FIG. 55 shows an example in which File #2 is split into N (256 < N) parts.

[0562] The split data number indicates the index of the split data from the beginning of the file, and this information is not transmitted. That is, the split data number is not included in the header information. Also, the split data number is a number assigned to each packet corresponding to each of the plurality of split data obtained by splitting the file, and is a number assigned by adding 1 in ascending order from the first packet.

[0563] The packet sequence number is the sequence number of packets having the same packet ID. In FIGS. 54 and 55, assuming the split data at the beginning of the file is A, consecutive numbers are assigned up to the split data at the end of the file. The packet sequence number is a number assigned by adding 1 in ascending order from the split data at the beginning of the file, and is a number corresponding to the split data number.

[0564] The fragment counter indicates the number of fragments that occur after the fragment in question, out of the multiple fragments obtained when a single file is split. Furthermore, if the number of fragments (the number of multiple fragments obtained when splitting a single file) exceeds 256, the fragment counter displays the remainder when the number of fragments is divided by 256. In the example in Figure 54, since the number of fragments is 256 or less, the field value of the fragment counter is (M - fragment number). On the other hand, in the example in Figure 55, since the number of fragments exceeds 256, the value is ((N - fragment number)%256), which is obtained by dividing (N - fragment number) by 256.

[0565] The fragmentation indicator indicates the state of data fragmentation stored in the MMTP packet. It shows whether the data is the first fragment, the last fragment, any other fragment, or one or more unfragmented data units. Specifically, the fragmentation indicator is "01" for the first fragment, "11" for the last fragment, "10" for the remaining fragments, and "00" for unfragmented data units.

[0566] In this embodiment, when the number of divided data points exceeds 256, the remainder obtained by dividing the number of divided data points by 256 is used for explanation. However, the number of divided data points is not limited to 256 and may be any other number (a predetermined number).

[0567] As shown in Figures 54 and 55, when a file is split and conventional header information is added to each of the multiple split data obtained by splitting the file and then transmitted, the receiving device does not have information that allows it to determine which split data number the data stored in the received MMTP packet corresponds to in the original file (split data number), the number of split data in the file, or the split data number and the number of split data. Therefore, with conventional transmission methods, even if an MMTP packet is received, it is not possible to uniquely detect the split data number or the number of split data stored in the received MMTP packet.

[0568] For example, as shown in Figure 54, if the number of divided data is 256 or less, and it is known in advance that the number of divided data is 256 or less, it is possible to identify the divided data number and the number of divided data by referring to the fragment counter. However, if the number of divided data is 256 or more, it is not possible to identify the divided data number and the number of divided data.

[0569] Furthermore, if the number of data segments in a file is limited to 256 or less, and the data size that can be transmitted in one packet is x [bytes], then the maximum size of a file that can be transmitted is limited to x * 256 [bytes]. For example, in broadcasting, x = 4k [bytes] is assumed, in which case the maximum size of a file that can be transmitted is limited to 4k * 256 = 1M [bytes]. Therefore, if you want to transmit a file larger than 1 [Mbytes], you cannot limit the number of data segments in the file to 256 or less.

[0570] Furthermore, for example, by referring to the fragmentation indicator, it is possible to detect the fragmented data at the beginning or end of a file. Therefore, it is possible to count the number of MMTP packets until the MMTP packet containing the last fragmented data of the file is received, or, after receiving the MMTP packet containing the last fragmented data of the file, calculate the fragmented data number and the number of fragmented data by combining it with the packet sequence number. Thus, signaling the fragmented data number and the number of fragmented data by combining the fragmentation indicator and the packet sequence number is also possible. However, if reception starts with an MMTP packet containing fragmented data in the middle of the file (i.e., fragmented data that is neither the first nor the last fragmented data of the file), the fragmented data number and the number of fragmented data for that fragment cannot be determined. The fragmented data number and the number of fragmented data for that fragment can only be determined after receiving the MMTP packet containing the last fragmented data of the file.

[0571] To address the problem described in Figures 54 and 55, namely, to uniquely determine the segmentation data number and the number of segmented data when a packet containing segmented file data is received midway through the process, the following method is used.

[0572] First, let's explain the segmented data numbers.

[0573] For the segmented data number, the packet sequence number in the segmented data at the beginning of the file (item) is used as the signaling element.

[0574] As a signaling method, the data is stored in the control information that manages the file. Specifically, in Figures 54 and 55, the packet sequence number A of the first segmented data in the file is stored in the control information. The receiving device obtains the value of A from the control information and calculates the segmented data number from the packet sequence number shown in the packet header.

[0575] The segment data number of the segmented data is obtained by subtracting the packet sequence number A of the first segmented data from the packet sequence number of the segmented data in question.

[0576] One example of control information for managing files is the asset management table defined in ARIB STD-B60. The asset management table contains information such as file size and version for each file, and this information is stored and transmitted in the data transmission message. Figure 56 shows the syntax of the file-by-file loop in the asset management table.

[0577] If the existing asset management table's space cannot be expanded, signaling may be performed using a 32-bit area of ​​the item_info_byte field, which contains information about the item. A flag indicating whether the packet sequence number in the split data at the beginning of the file (item) is shown in a portion of the item_info_byte may be indicated in the reserved_future_use field of the control information.

[0578] When repeatedly transmitting files, such as in a data carousel, multiple packet sequence numbers may be indicated, or the packet sequence number of the first file to be transmitted immediately afterward may be indicated.

[0579] The information that links the file's split data number with the packet sequence number is not limited to the packet sequence number of the split data at the beginning of the file; any information that links the split data number with the packet sequence number is acceptable.

[0580] Next, I will explain the number of data segments.

[0581] The order of loops for each file included in the asset management table may be defined as the file transmission order. This allows us to determine the packet sequence number of the first two consecutive files in transmission order, and by subtracting the first packet sequence number of the previously transmitted file from the first packet sequence number of the later transmitted file, we can determine the number of data segments in the previously transmitted file. In other words, for example, if File#1 shown in Figure 54 and File#2 shown in Figure 55 are consecutive files in this order, then the last packet sequence number of File#1 and the first packet sequence number of File#2 are assigned consecutive numbers.

[0582] Alternatively, the method of splitting the file may be specified to allow for the determination of the number of data segments within a file. For example, if the number of data segments is N, the size of each segment from the 1st to the (N-1)th segment can be defined as L, and the size of the Nth segment can be defined as the remainder (item_size - L * (N-1)). In this case, the number of data segments can be calculated from the item_size shown in the asset management table. In this case, the number of data segments will be an integer value obtained by rounding up (item_size / L). Note that this is not the only method for splitting a file.

[0583] Alternatively, the number of divided data points may be stored directly in the asset management table.

[0584] The receiving device receives control information using the method described above and calculates the number of divided data based on the control information. It can also calculate the packet sequence number corresponding to the file's divided data number based on the control information. If the timing of receiving the divided data packets is earlier than the timing of receiving the control information, the divided data number and the number of divided data may be calculated at the time the control information is received.

[0585] Furthermore, when signaling the fragment data number or the number of fragment data using the method described above, the fragment data number or the number of fragment data is not identified based on the fragment counter, making the fragment counter unnecessary data. Therefore, in the transmission of asynchronous media, if information that can identify the fragment data number and the number of fragment data is signaled using the method described above, the fragment counter may be disabled or the header may be compressed. This can reduce the processing load on the transmitting and receiving devices and improve transmission efficiency. In other words, when transmitting asynchronous media, the fragment counter may be reserved (disabled). Specifically, the value of the fragment counter may be set to a fixed value, for example, "0". Also, when receiving asynchronous media, the fragment counter may be ignored.

[0586] When storing synchronized media such as video or audio, the transmission order of MMTP packets at the transmitting device matches the arrival order of MMTP packets at the receiving device, and packets are not retransmitted. In such cases, if there is no need to detect and reconstruct packet loss, the fragment counter may not be operated. In other words, in this case, the fragment counter may be reserved (disabled).

[0587] Furthermore, even without using a fragment counter, it is possible to detect random access points, the beginning of access units, and the beginning of NAL units, enabling decoding, packet loss detection, and recovery from packet loss.

[0588] Furthermore, in the transmission of real-time content such as live broadcasts, lower latency transmission is required, and it is necessary to sequentially packetize and send data as encoding is completed. However, in the transmission of real-time content, conventional fragment counters cannot determine the number of fragments when sending the first fragmented data. Therefore, the first fragmented data is sent only after all data unit encoding is complete and the number of fragments has been determined, resulting in a delay. Even in such cases, this delay can be reduced by not using a fragment counter using the method described above.

[0589] Figure 57 shows the operation flow for identifying the segmented data number in the receiving device.

[0590] The receiving device obtains control information containing file information (S1201). The receiving device determines whether the control information indicates the packet sequence number of the beginning of the file (S1202). If the control information indicates the packet sequence number of the beginning of the file (Yes in S1202), it calculates the packet sequence number corresponding to the segmented data number of the segmented data of the file (S1203). Then, after obtaining the MMTP packet containing the segmented data, the receiving device identifies the segmented data number of the file from the packet sequence number stored in the packet header of the obtained MMTP packet (S1204). On the other hand, if the control information does not indicate the packet sequence number of the beginning of the file (No in S1202), the receiving device obtains the MMTP packet containing the last segmented data of the file, and then identifies the segmented data number using the fragment indicator and packet sequence number stored in the packet header of the obtained MMTP packet (S1205).

[0591] Figure 58 shows the operation flow for determining the number of divided data segments in the receiving device.

[0592] The receiving device obtains control information containing file information (S1301). The receiving device determines whether the control information contains information that allows for the calculation of the number of divided data files (S1302). If it determines that the control information contains information that allows for the calculation of the number of divided data files (Yes in S1302), it calculates the number of divided data files based on the information contained in the control information (S1303). On the other hand, if the receiving device determines that it is not possible to calculate the number of divided data files (No in S1302), it obtains the MMTP packet containing the last divided data file, and then identifies the number of divided data files using the fragment indicator stored in the packet header of the obtained MMTP packet and the packet sequence number (S1304).

[0593] Figure 59 shows the operation flow for determining whether or not to operate a fragment counter in the transmitting device.

[0594] First, the transmitting device determines whether the medium to be transmitted (hereinafter also referred to as "media data") is a synchronous medium or an asynchronous medium (S1401).

[0595] If the result of the determination in step S1401 is that it is a synchronous medium (synchronous medium in S1402), the transmitting device determines whether the order of transmitted and received MMTP packets matches in the environment in which the synchronous medium is transmitted, and whether packet reconstruction is unnecessary in the event of packet loss (S1403). If the transmitting device determines that it is unnecessary (Yes in S1403), it does not operate the fragment counter (S1404). On the other hand, if the transmitting device determines that it is not unnecessary (No in S1403), it operates the fragment counter (S1405).

[0596] If the result of the determination in step S1401 is that it is an asynchronous medium (asynchronous medium in S1402), the transmitting device decides whether or not to operate the fragment counter based on whether the fragment data number and the number of fragment data are signaled using the method described above. Specifically, if the fragment data number and the number of fragment data are signaled (Yes in S1406), the transmitting device does not operate the fragment counter (S1404). On the other hand, if the fragment data number and the number of fragment data are not signaled (No in S1406), the transmitting device operates the fragment counter (S1405).

[0597] If the transmitting device does not operate a fragment counter, it may set the fragment counter value to "reserved" or compress the header.

[0598] Furthermore, the transmitting device may decide whether or not to signal the aforementioned fragment data number and the number of fragment data based on whether or not it operates a fragment counter.

[0599] Furthermore, if the synchronous media does not operate a fragment counter, the transmitting device may signal the fragment data number and the number of fragment data using the method described above in the asynchronous media. Conversely, the operation of the synchronous media may be determined based on whether or not the asynchronous media operates a fragment counter. In this case, the operation of fragmentation can be the same for both the synchronous and asynchronous media.

[0600] Next, we will explain how to identify the number of divided data and the number of divided data (when using a fragment counter). Figure 60 is a diagram illustrating how to identify the number of divided data and the number of divided data (when using a fragment counter).

[0601] As explained using Figure 54, if the number of divided data is 256 or less, and it is known in advance that the number of divided data is 256 or less, it is possible to identify the divided data number and the number of divided data by referring to the fragment counter.

[0602] If the number of data segments in a file is limited to 256 or less, and the data size that can be transmitted in one packet is x [bytes], then the maximum size of a file that can be transmitted is limited to x * 256 [bytes]. For example, in broadcasting, x = 4k [bytes] is assumed, in which case the maximum size of a file that can be transmitted is limited to 4k * 256 = 1M [bytes].

[0603] If the file size exceeds the maximum size of a file that can be transmitted, the file is pre-divided so that the size of each divided file is x*256 [bytes] or less. Each of the multiple divided files obtained by splitting the file is treated as a single file (item), and is further divided into 256 or less sections. The resulting divided data is then stored in MMTP packets and transmitted.

[0604] Furthermore, information indicating that an item is a split file, the number of split files, and the sequence number of the split files may be stored in the control information and sent to the receiving device. Alternatively, this information may be stored in the asset management table, or it may be indicated using a part of the existing field item_info_byte.

[0605] If an item is one of several split files obtained by splitting a single file, the receiving device can identify the other split files and reconstruct the original file. Furthermore, the receiving device can uniquely identify the number of split data items and the split data item number by using the number of split files, the index of the split files, and the fragment counter in the control information. It can also uniquely identify the number of split data items and the split data item number without using packet sequence numbers or similar information.

[0606] Here, it is desirable that the item_ids of each of the multiple split files obtained by splitting a single file are the same. If different item_ids are assigned, the item_id of the first split file may be used to uniquely refer to the file from other control information, etc.

[0607] Furthermore, multiple split files may always belong to the same MPU. When storing multiple files in an MPU, different types of files may not be stored, and all files stored may always be split versions of a single file. The receiving device can detect file updates by checking the version information for each MPU, without having to check the version information for each item.

[0608] Figure 61 shows the operation flow of the transmitting device when a fragment counter is used.

[0609] First, the transmitting device checks the size of the file to be transmitted (S1501). Next, the transmitting device determines whether the file size exceeds x*256 [bytes] (where x is the data size that can be transmitted in one packet, for example, the MTU size) (S1502). If the file size exceeds x*256 [bytes] (Yes in S1502), the device splits the file so that the size of each split file is less than x*256 [bytes] (S1503). Then, the split files are transmitted as items, and information about the split files (for example, that they are split files, the sequence number in the split files, etc.) is stored in the control information and transmitted (S1504). On the other hand, if the file size is less than x*256 [bytes] (No in S1502), the file is transmitted as an item as usual (S1505).

[0610] Figure 62 shows the operation flow of the receiving device when a fragment counter is used.

[0611] First, the receiving device acquires and analyzes control information related to file transmission, such as the asset management table (S1601). Next, the receiving device determines whether the desired item is a split file (S1602). If the receiving device determines that the desired file is a split file (Yes in S1602), it acquires information from the control information to reconstruct the file, such as the split file and the index of the split file (S1603). Then, the receiving device acquires the items that make up the split file and reconstructs the original file (S1604). On the other hand, if the receiving device determines that the desired file is not a split file (No in S1602), it acquires the file as usual (S1605).

[0612] In short, the transmitting device signals the packet sequence number of the fragmented data at the beginning of the file. It also signals information that allows for the identification of the number of fragmented data. Alternatively, the transmitting device defines fragmentation rules that allow for the identification of the number of fragmented data. Furthermore, the transmitting device either reserves or compresses the header without using a fragment counter.

[0613] If the receiving device has signaled the packet sequence number of the data at the beginning of the file, it will identify the segmented data number and the number of segmented data from the packet sequence number of the segmented data at the beginning of the file and the packet sequence number of the MMTP packet.

[0614] From another perspective, the transmitting device splits the file and transmits the data in separate files. It signals information linking the split files (sequence number, number of splits, etc.).

[0615] The receiving device identifies the segmented data number and the number of segmented data using the fragment counter and the sequence number of the segmented file.

[0616] This allows for the unique identification of segmented data numbers and segmented data. Furthermore, since the segmented data number of a segmented data is identified as soon as it is received, waiting time and memory usage can be reduced.

[0617] Furthermore, by not using a fragment counter, the configuration of the transceiver can reduce the processing load and improve transmission efficiency.

[0618] Figure 63 shows the service configuration when the same program is transmitted using multiple IP data flows. Here, a portion of the data (video and audio) of the program with service ID=2 is transmitted using an IP data flow with the MMT method, while data with the same service ID but different from that portion is transmitted using an IP data flow with the advanced BS data transmission method (in this example, the file transmission protocol is different but could be the same protocol).

[0619] The transmitting device performs IP data multiplexing to ensure that the receiving device receives data composed of multiple IP data flows by the time of decoding.

[0620] The receiving device can achieve guaranteed receiver operation by processing data composed of multiple IP data flows based on the decoding time.

[0621] [Note: Transmitting device and receiving device] As described above, a transmitting device that transmits data without operating a fragment counter can be configured as shown in Figure 64. Similarly, a receiving device that receives data without operating a fragment counter can be configured as shown in Figure 65. Figure 64 is a diagram showing a specific example of the configuration of a transmitting device. Figure 65 is a diagram showing a specific example of the configuration of a receiving device.

[0622] The transmitting device 500 comprises a divided section 501, a component section 502, and a transmitting section 503. Each of the divided section 501, component section 502, and transmitting section 503 is implemented by, for example, a microcomputer, a processor, or a dedicated circuit.

[0623] The receiving device 600 comprises a receiving unit 601, a determination unit 602, and a component unit 603. Each of the receiving unit 601, the determination unit 602, and the component unit 603 is implemented by, for example, a microcomputer, a processor, or a dedicated circuit.

[0624] Detailed descriptions of each component of the transmitting device 500 and the receiving device 600 will be provided in the descriptions of the transmitting method and the receiving method, respectively.

[0625] First, the transmission method will be explained using Figure 66. Figure 66 shows the operation flow (transmission method) by the transmitting device.

[0626] First, the splitting unit 501 of the transmitting device 500 divides the data into multiple split data (S1701).

[0627] Next, component 502 of the transmitting device 500 constructs multiple packets by adding header information to each of the multiple divided data and packaging them (S1702).

[0628] Then, the transmitting unit 503 of the transmitting device 500 transmits the configured plurality of packets (S1703). The transmitting unit 503 transmits the fragmentation data information and the value of the invalidated fragment counter. The fragmentation data information is information for identifying the fragmentation data number and the number of fragmentation data. The fragmentation data number is a number indicating which of the plurality of fragmentation data the fragmentation data is. The number of fragmentation data is the number of the plurality of fragmentation data.

[0629] This reduces the processing load on the transmitting device 500.

[0630] Next, the receiving method will be explained using Figure 67. Figure 67 shows the operation flow (receiving method) by the receiving device.

[0631] First, the receiving unit 601 of the receiving device 600 receives multiple packets (S1801).

[0632] Next, the determination unit 602 of the receiving device 600 determines whether or not segmented data information has been acquired from the multiple packets received (S1802).

[0633] Then, if the determination unit 602 determines that it has acquired fragmented data information, the component 603 of the receiving device 600 constructs data from the multiple received packets without using the value of the fragment counter included in the header information (S1803).

[0634] On the other hand, if the determination unit 602 determines that it has not acquired fragmented data information, the component 603 may use the value of the fragment counter included in the header information to construct data from the multiple received packets (S1804).

[0635] This reduces the processing load on the receiving device 600.

[0636] (Embodiment 5) [overview] Embodiment 5 describes a method for transmitting transmission packets (TLV packets) when NAL units are stored in the multiplexing layer in NAL size format.

[0637] As described in Embodiment 1, there are two formats for storing H.264 and H.265 NAL units in the multiplexing layer. One is a format called the byte stream format, which adds a start code consisting of a specific bit sequence immediately before the NAL unit header. The other is a format called the NAL size format, which adds a field indicating the size of the NAL unit. The byte stream format is used in MPEG-2 systems and RTP, while the NAL size format is used in MP4, or in DASH and MMT, which use MP4.

[0638] In byte stream format, the start code consists of 3 bytes, and any additional byte (a byte with a value of 0) can be added.

[0639] On the other hand, in the NAL size format of typical MP4 files, size information is represented by either 1 byte, 2 bytes, or 4 bytes. This size information is shown in the `lengthSizeMinusOne` field in the HEVC sample entry. A value of "0" in this field indicates 1 byte, "1" indicates 2 bytes, and "3" indicates 4 bytes.

[0640] In ARIB STD-B60, "Media Transport Method by MMT in Digital Broadcasting," standardized in July 2014, when storing NAL units in the multiplexing layer, if the output of the HEVC encoder is a byte stream, the byte start code is removed, and the size of the NAL unit in bytes, represented by a 32-bit (unsigned integer), is added as length information immediately before the NAL unit. Note that MPU metadata, including HEVC sample entries, is not transmitted, and the size information is fixed at 32 bits (4 bytes).

[0641] Furthermore, ARIB STD-B60 "Media Transport System by MMT in Digital Broadcasting" specifies that in the receive buffer model considered by the transmitting device during transmission to guarantee buffer operation in the receiving device, the pre-decoded buffer for the video signal is a CPB (Critical Point Buffer).

[0642] However, there are the following challenges. CPB in MPEG-2 systems and HRD in HEVC are defined on the assumption that the video signal is in byte stream format. For example, if rate control of transmission packets is performed on the assumption that it is in byte stream format with a 3-byte start code, a receiving device that receives a transmission packet in NAL size format with an added 4-byte size area may not be able to satisfy the receive buffer model in ARIB STD-B60. Furthermore, since the receive buffer model in ARIB STD-B60 does not specify the buffer size and extraction rate, it is difficult to guarantee the buffer operation in the receiving device.

[0643] Therefore, in order to solve the above problems, a receive buffer model to guarantee buffer operation in the receiver is defined as follows.

[0644] Figure 68 shows the receive buffer model based on the receive buffer model specified in ARIB STD B-60, specifically for the case where only broadcast transmission lines are used.

[0645] The receive buffer model includes a TLV packet buffer (first buffer), an IP packet buffer (second buffer), an MMTP buffer (third buffer), and a pre-decode buffer (fourth buffer). Note that in broadcast transmission lines, digitter buffers and buffers for FEC are not necessary and are therefore omitted.

[0646] The TLV packet buffer receives TLV packets (transmission packets) from the broadcast transmission path, converts the IP packets, which consist of a variable-length packet header (IP packet header, full header during IP packet compression, compressed header during IP packet compression) and a variable-length payload stored in the received TLV packets, into IP packets (first packets) with a fixed-length IP packet header after header decompression, and outputs the resulting IP packets at a constant bitrate.

[0647] The IP packet buffer converts IP packets into MMTP packets (second packets) that have a packet header and a variable-length payload, and outputs the resulting MMTP packets at a constant bitrate. The IP packet buffer may also be merged into the MMTP buffer.

[0648] The MMTP buffer converts the output MMTP packets into NAL units and outputs the resulting NAL units at a constant bitrate.

[0649] The pre-decryption buffer sequentially stores the output NAL units, generates access units from the multiple stored NAL units, and outputs the generated access units to the decoder at the timing corresponding to the decoding time of the access unit.

[0650] In the receive buffer model shown in Figure 68, a distinctive feature is that the MMTP buffer and pre-decode buffer, which are buffers other than the preceding TLV packet buffer and IP packet buffer, follow the receive buffer model in MPEG-2 TS.

[0651] For example, the MMTP buffer (MMTP B1) in video consists of buffers equivalent to the transport buffer (TB) and multiplexing buffer (MB) in MPEG-2 TS. Similarly, the MMTP buffer (MMTP Bn) in audio consists of a buffer equivalent to the transport buffer (TB) in MPEG-2 TS.

[0652] The transport buffer size is fixed, similar to that of MPEG-2 TS. For example, it can be n times the MTU size (where n can be a decimal or an integer, and is greater than or equal to 1).

[0653] Furthermore, the MMTP packet size is defined such that the overhead rate of the MMTP packet header is smaller than that of the PES packet header. This allows the same extraction rates RX1, RXn, and RXs as those used in MPEG-2 TS to be applied to the transport buffer.

[0654] Furthermore, the size of the multiplexing buffer and the extraction rate are, respectively, those of MPEG-2. The MB size in TS is specified as RBX1.

[0655] In addition to the above receive buffer model, the following constraints are imposed to solve the problem.

[0656] The HEVC HRD specification assumes a byte stream format, and MMT is a NAL-sized format that adds a 4-byte size area to the beginning of the NAL unit. Therefore, during encoding, rate control is performed to satisfy the HRD in the NAL-sized format.

[0657] In other words, the transmitting device controls the transmission packet rate based on the above-described receive buffer model and constraints.

[0658] The receiving device can perform decoding operations that prevent underflow or overflow by using the above-mentioned signals for reception processing.

[0659] Even if the size area at the beginning of the NAL unit is not 4 bytes, rate control is performed to satisfy HRD by taking the size area at the beginning of the NAL unit into consideration.

[0660] The TLV packet buffer extraction rate (the bitrate at which the TLV packet buffer outputs IP packets) should be set considering the transmission rate after IP header decompression.

[0661] In other words, it takes a TLV packet with variable data size as input, removes the TLV header, expands (restores) the IP header, and then considers the transmission rate of the output IP packet. To put it another way, it takes into account the increase or decrease in header size relative to the input transmission rate.

[0662] Specifically, because the data size is variable, packets with and without IP header compression coexist, and the IP header size differs depending on the packet type (IPv4, IPv6, etc.), the transmission rate of output IP packets is not unique. Therefore, the average packet length of the variable-length data is defined, and the transmission rate of IP packets output from TLV packets is determined.

[0663] Here, in order to define the maximum transmission speed after decompression of the IP header, the transmission rate is determined assuming that the IP header is always compressed.

[0664] Furthermore, when IPv4 and IPv6 packet types are mixed, or when packet types are defined without distinction, the transmission rate is determined assuming IPv6 packets, which have a larger header size and a larger increase rate after header expansion.

[0665] For example, if the average packet length of TLV packets input to the TLV packet buffer is S, and all IP packets stored in the TLV packets are IPv6 packets with header compression, then the maximum output transmission rate after removing the TLV header and decompressing the IP header is: Input rate × {S / (S + IPv6 header compression amount)} This is the result.

[0666] More specifically, the average packet length S of TLV packets is S = 0.75 × 1500 (1500 is assumed to be the maximum MTU size) Set based on this, IPv6 header compression amount = TLV header length - IPv6 header length - UDP header length =3-40-8 In that case, the maximum output transmission rate after removing the TLV header and decompressing the IP header is: Input rate × 1.0417 ≈ Input rate × 1.05 This is the result.

[0667] Figure 69 shows an example of aggregating multiple data units and storing them in a single payload.

[0668] In the MMT method, when data units are aggregated, the data unit length and data unit header are added before the data unit, as shown in Figure 69.

[0669] However, for example, when storing a video signal in NAL size format as a single data unit, as shown in Figure 70, there are two fields indicating the size for a single data unit, resulting in redundant information. Figure 70 is an example of aggregating multiple data units and storing them in a single payload, specifically showing an example where a video signal in NAL size format is treated as a single data unit. Specifically, both the size area at the beginning of the NAL size format (hereinafter referred to as the "size area") and the data unit length field located before the data unit header in the MMTP payload header are fields indicating size, and therefore the information is redundant. For example, if the length of the NAL unit is L bytes, the size area will show L bytes, and the data unit length field will show L bytes + "length of the size area" (bytes). The values ​​shown in the size area and the data unit length field do not perfectly match, but since the other value can be easily calculated from the other, it can be said that they are redundant.

[0670] Thus, when data containing size information is stored as a data unit, and multiple such data units are aggregated and stored in a single payload, there is a problem of high overhead and poor transmission efficiency due to the duplication of size information.

[0671] Therefore, in a transmission device, data containing size information is stored as a data unit, and when multiple such data units are aggregated and stored in a single payload, it is conceivable to store them as shown in Figures 71 and 72.

[0672] As shown in Figure 71, it is conceivable to store the NAL unit, including the size area, as a data unit, and not indicate the data unit length that is conventionally included in the MMTP payload header. Figure 71 is a diagram showing the payload structure of an MMTP packet in which the data unit length is not indicated.

[0673] Furthermore, as shown in Figure 72, a flag indicating whether the data unit length is indicated, and information indicating the length of the size area may be newly stored in the header. The location where the flag and information indicating the length of the size area are stored may be indicated on a data unit basis, such as in the data unit header, or it may be indicated on a unit of aggregation of multiple data units (packet unit). Figure 72 shows an example of this being indicated in the extend area assigned to a packet unit. Note that the storage location of the newly indicated information is not limited to this, and may also be in the MMTP payload header, MMTP packet header, or control information.

[0674] On the receiving end, if a flag indicating whether the data unit length is compressed indicates that the data unit length is compressed, the length information of the size area inside the data unit is obtained, and based on the length information of the size area, the size area is obtained, and the data unit length can be calculated using the obtained length information of the size area and the size area.

[0675] By using the above method, the amount of data can be reduced on the sending side, thereby improving transmission efficiency.

[0676] Alternatively, overhead can be reduced by reducing the size area instead of reducing the data unit length. If the size area is reduced, information indicating whether the size area has been reduced, and information indicating the length of the data unit length field, may be stored.

[0677] The MMTP payload header also includes length information.

[0678] When storing NAL units that include the size area as data units, the payload size area in the MMTP payload header may be reduced, regardless of whether aggregation is performed or not.

[0679] Furthermore, even when storing data that does not include the size area as a data unit, if it is aggregated and the data unit length is indicated, the payload size area in the MMTP payload header may be reduced.

[0680] When reducing the payload size area, a flag indicating whether or not it was reduced, or length information of the reduced size field, or length information of the size field that was not reduced, may be indicated, as described above.

[0681] Figure 73 shows the operation flow of the receiving device.

[0682] As described above, the sending device stores the NAL unit, including the size area, as a data unit, and the data unit length included in the MMTP payload header is not indicated in the MMTP packet.

[0683] The following sections will explain, using flags to indicate whether the data unit length is shown, and examples of cases where the length information of the size field is shown in the MMTP packet.

[0684] The receiving device determines, based on the information transmitted from the sending side, whether the data unit includes a size area and whether the data unit length has been reduced (S1901).

[0685] If it is determined that the data unit length has been reduced (Yes in S1902), the length information of the size area inside the data unit is obtained, and then the size area inside the data unit is analyzed to calculate the data unit length (S1903).

[0686] On the other hand, if it is determined that the data unit length has not been reduced (No in S1902), the data unit length is calculated as usual from either the data unit length or the size area inside the data unit (S1904).

[0687] Note that if the receiving device already knows the flag indicating whether the data unit length has been reduced, or the length information of the size area, it does not need to be transmitted. In this case, the receiving device performs the processing shown in Figure 73 based on the predetermined information.

[0688] [Note: Transmitting device and receiving device] As described above, a transmitting device that performs rate control to satisfy the specifications of the receive buffer model during encoding can also be configured as shown in Figure 74. Furthermore, a receiving device that receives and decodes transmission packets sent from the transmitting device can also be configured as shown in Figure 75. Figure 74 shows an example of a specific configuration of a transmitting device. Figure 75 shows an example of a specific configuration of a receiving device.

[0689] The transmitting device 700 comprises a generation unit 701 and a transmitting unit 702. Each of the generation unit 701 and the transmitting unit 702 is implemented, for example, by a microcomputer, a processor, or a dedicated circuit.

[0690] The receiving device 800 comprises a receiving unit 801, a first buffer 802, a second buffer 803, a third buffer 804, a fourth buffer 805, and a decoding unit 806. Each of the receiving unit 801, the first buffer 802, the second buffer 803, the third buffer 804, the fourth buffer 805, and the decoding unit (decoder) 806 is implemented by, for example, a microcomputer, a processor, or a dedicated circuit.

[0691] Detailed descriptions of each component of the transmitting device 700 and the receiving device 800 will be provided in the descriptions of the transmitting method and the receiving method, respectively.

[0692] First, the transmission method will be explained using Figure 76. Figure 76 shows the operation flow (transmission method) by the transmitting device.

[0693] First, the generation unit 701 of the transmitting device 700 generates an encoded stream by performing rate control to satisfy the specifications of a predetermined receive buffer model in order to guarantee the buffer operation of the receiving device (S2001).

[0694] Next, the transmitting unit 702 of the transmitting device 700 packets the generated encoded stream and transmits the transmission packets obtained by packetization (S2002).

[0695] The receiving buffer model used in the transmitting device 700 is configured to include the first to fourth buffers 802 to 805 of the receiving device 800, so its explanation is omitted.

[0696] This allows the transmitting device 700 to guarantee the buffer operation of the receiving device 800 when transmitting data using a method such as MMT.

[0697] Next, the receiving method will be explained using Figure 77. Figure 77 shows the operation flow (receiving method) by the receiving device.

[0698] First, the receiving unit 801 of the receiving device 800 receives a transmission packet consisting of a fixed-length packet header and a variable-length payload (S2101).

[0699] Next, the first buffer 802 of the receiving device 800 converts the packet, which consists of a variable-length packet header and a variable-length payload stored in the received transmission packet, into a first packet having a fixed-length packet header with the header expanded, and outputs the first packet obtained by the conversion at a constant bit rate (S2102).

[0700] Next, the second buffer 803 of the receiving device 800 converts the first packet obtained by the conversion into a second packet consisting of a packet header and a variable-length payload, and outputs the second packet obtained by the conversion at a constant bit rate (S2103).

[0701] Next, the third buffer 804 of the receiving device 800 converts the output second packet into a NAL unit and outputs the resulting NAL unit at a constant bit rate (S2104).

[0702] Next, the fourth buffer 805 of the receiving device 800 sequentially stores the output NAL units, generates an access unit from the multiple stored NAL units, and outputs the generated access unit to the decoder at the timing of the decoding time corresponding to the access unit (S2105).

[0703] Then, the decoding unit 806 of the receiving device 800 decodes the access unit output by the fourth buffer (S2106).

[0704] This allows the receiving device 800 to perform decoding operations without underflow or overflow.

[0705] (Other embodiments) Although the transmitting device, receiving device, transmitting method, and receiving method according to the embodiments have been described above, the present invention is not limited to these embodiments.

[0706] Furthermore, each processing unit included in the transmitting and receiving devices according to the above embodiment is typically implemented as an LSI (Large-Scale Integrated Circuit). These may be individually integrated into a single chip, or some or all of them may be integrated into a single chip.

[0707] Furthermore, integrated circuit implementation is not limited to LSIs; it may also be achieved using dedicated circuits or general-purpose processors. Field-Programmable Gate Arrays (FPGAs), which can be programmed after LSI manufacturing, or reconfigurable processors, which allow for the reconfiguration of the connections and settings of circuit cells within the LSI, may also be used.

[0708] In each of the above embodiments, each component may be implemented by dedicated hardware or by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0709] In other words, the transmitting device and the receiving device comprise a processing circuitry and a storage device electrically connected to the processing circuitry (accessible from the processing circuitry). The processing circuitry includes at least one of dedicated hardware and a program execution unit. The storage device also stores a software program executed by the program execution unit if the processing circuitry includes the program execution unit. The processing circuitry uses the storage device to execute the transmitting method or receiving method according to the above embodiment.

[0710] Furthermore, the present invention may be the software program described above, or a non-temporary computer-readable recording medium on which the program is recorded. It goes without saying that the program can also be distributed via a transmission medium such as the Internet.

[0711] Furthermore, the figures used above are all illustrative to specifically explain the present invention, and the present invention is not limited to the illustrated figures.

[0712] Furthermore, the division of functional blocks in the block diagram is just one example; multiple functional blocks can be implemented as a single functional block, a single functional block can be divided into multiple parts, or some functions can be moved to other functional blocks. In addition, the functions of multiple functional blocks with similar functions can be processed in parallel or time-sharing by a single piece of hardware or software.

[0713] Furthermore, the order in which the steps included in the above-described transmission or reception method are performed is illustrative for the purpose of specifically illustrating the present invention, and may be performed in a different order. Also, some of the above steps may be performed simultaneously (in parallel) with other steps.

[0714] Although a transmitting device, receiving device, transmitting method, and receiving method according to one or more embodiments of the present invention have been described above based on embodiments, the present invention is not limited to these embodiments. Without departing from the spirit of the present invention, various modifications that a person skilled in the art can conceive of may be applied to these embodiments, and forms constructed by combining components from different embodiments may also be included within the scope of one or more embodiments of the present invention. [Industrial applicability]

[0715] This invention can be applied to devices or equipment that perform media transport, such as video data and audio data. [Explanation of symbols]

[0716] 15, 100, 300, 500, 700 Transmitter 16, 101, 301 Encoding section 17, 102 Multiplexing section 18, 104 Transmitter 20, 200, 400, 600, 800 receiving devices 21 Packet Filtering Section 22 Transmission sequence type determination unit 23 Random Access Section 24, 212 Control Information Acquisition Unit 25 Data Acquisition Unit 26 Calculation Section 27 Initialization information acquisition unit 28, 206 Decoding instruction section 29, 204A, 204B, 204C, 204D, 402 Decoding section 30 Presentation section 201 Tuner 202 Demodulation Unit 203 Demultiplexer 205 Display section 211 Type discrimination unit 213 Slice Information Acquisition Unit 214 Decoded Data Generation Unit 302 Assignment section 303, 503, 702 Transmitting Units 401, 601, 801 Receiver 501 Split section 502, 603 Components 602 Judgment section 701 Generation part 802 First buffer 803 Second buffer 804 Third buffer 805 The fourth buffer 806 Decoding section

Claims

1. A receiving unit that receives a transmission packet in which an encoded stream has been packetized, the transmission packet comprising a fixed-length packet header and a variable-length payload containing a first IP packet having a variable-length packet header, A first buffer that converts the first IP packet stored in the received transmission packet, including the first IP packet, into a first packet having a fixed-length packet header with the header expanded, and outputs the first packet obtained by the conversion at a bit rate of 1.05 times the input rate to the first buffer, A second buffer that converts the output first packet into a second packet consisting of a packet header and a variable-length payload, and outputs the second packet obtained by the conversion at a constant bitrate, A third buffer that converts the output second packet into a NAL unit and outputs the resulting NAL unit at a constant bitrate, A fourth buffer sequentially stores the outputted NAL units, generates an access unit from the stored NAL units, and outputs the generated access unit at the timing of the decoding time corresponding to the access unit. The system comprises a decoder for decoding the access unit output by the fourth buffer, The second buffer and the third buffer are merged into a single buffer. The aforementioned NAL unit is stored in a multiplexing layer and has an NAL size format with a 4-byte size area appended to the beginning. Receiving device.

2. A receiving step in which an encoded stream is a packetized transmission packet, the transmission packet comprising a fixed-length packet header and a variable-length payload containing a first IP packet having a variable-length packet header, A first conversion step involves converting the first IP packet stored in the received transmission packet, which includes the first IP packet, into a first packet having a fixed-length packet header with the header expanded, and outputting the first packet obtained by the conversion at a bit rate of 1.05 times the input rate to the first buffer. A second conversion step involves converting the output first packet into a second packet consisting of a packet header and a variable-length payload, and outputting the second packet obtained by the conversion at a constant bitrate. A third conversion step involves converting the output second packet into a NAL unit and outputting the NAL unit obtained by the conversion at a constant bitrate. A generation step which involves sequentially accumulating the outputted NAL units, generating an access unit from the multiple accumulated NAL units, and outputting the generated access unit to the decoder at the timing of the decoding time corresponding to the access unit, The process includes a decoding step for decoding the access unit output by the generation step, The second buffer used in the second conversion step and the third buffer used in the third conversion step are merged into a single buffer. The aforementioned NAL unit is stored in a multiplexing layer and has an NAL size format with a 4-byte size area appended to the beginning. Reception method.

Images