Method for transmission

Abstract

To simplify subtitle display processing in a rate change regeneration mode on a reception side.SOLUTION: A video stream is generated, which is made of a video packet having coded image data in a payload. A subtitle stream is generated, which is made of a subtitle packet having subtitle information in a payload. A multiplexed stream including the video stream and the subtitle stream is generated and sent. When the multiplexed stream is generated, the subtitle packet is arranged in a random access position.SELECTED DRAWING: Figure 18

Description

[Technical Field]

[0001] The present technology relates to a transmitting device, a transmitting method, a receiving device, and a receiving method, and in particular to a transmitting device or the like that transmits subtitle information together with image data. [Background technology]

[0002] In the past, for example, in DVB (Digital Video Broadcasting) broadcasts, subtitle information was transmitted as bitmap data. Recently, however, it has been proposed to transmit subtitle information in text character code, i.e., text-based. In this case, the font is expanded according to the resolution on the receiving side.

[0003] Furthermore, when transmitting subtitle information in a text-based format, it has been proposed to include timing information in the text information. For example, the World Wide Web Consortium (W3C) has proposed the Timed Text Markup Language (TTML) as this text information (see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-169885 Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present technology is to simplify the subtitle display process in variable speed playback mode on the receiving side. [Means for solving the problem]

[0006] The concept of this technology is: a video encoding unit that generates a video stream composed of video packets having encoded image data in their payloads; a subtitle encoding unit that generates a subtitle stream composed of subtitle packets having subtitle information in their payloads; a multiplexed stream generator that generates a multiplexed stream including the video stream and the subtitle stream; a transmitter for transmitting the multiplexed stream, The multiplexed stream generation unit The above subtitle packets are placed in the random access position. Located in the transmitting device.

[0007] In this technology, a video encoding unit generates a video stream composed of video packets having encoded image data in their payloads. A subtitle encoding unit generates a subtitle stream composed of subtitle packets having subtitle information in their payloads. For example, the subtitle information may be subtitle text information in a predetermined format having display timing information. In this case, for example, the subtitle text information in the predetermined format may be in TTML or a format derived from this TTML.

[0008] The multiplexed stream generating unit generates a multiplexed stream including at least a video stream and a subtitle stream. In this case, the subtitle packets are arranged in a random access position. For example, the video packets and subtitle packets may be PES packets, and the multiplexed stream may be an MPEG2 transport stream. The transmitting unit transmits the multiplexed stream.

[0009] In this way, in this technology, when a video stream and a subtitle stream are multiplexed, subtitle packets are placed at random access positions, so that on the receiving side, when extracting video packets at random access positions from the multiplexed stream, the subtitle packets can also be extracted, simplifying the subtitle display process in variable speed playback mode.

[0010] In the present technology, for example, the subtitle encoding unit may insert, into the timestamp insertion position of the header of the subtitle packet, a timestamp whose value is the same as or close to the value of the timestamp inserted into the timestamp insertion position of the header of the video packet at the random access position. In this case, when packets are multiplexed into the multiplexed stream with reference to the timestamp, the subtitle packet will be placed at the random access position.

[0011] In addition, in the present technology, for example, the subtitle encoding unit may insert a unique timestamp indicating the subtitle display time into the header or payload of the subtitle packet, which allows the receiving side to easily control the subtitle display timing based on the unique timestamp.

[0012] Furthermore, in the present technology, for example, the multiplexed stream generation unit may insert identification information into the multiplexed stream indicating that a unique timestamp has been inserted into a subtitle packet. In this case, the receiving side can easily recognize that a unique timestamp has been inserted based on this identification information. In this case, for example, information indicating the insertion position may be added to the identification information. In this case, the receiving side can easily obtain the unique timestamp based on the information indicating the insertion position.

[0013] Another concept of the present technology is a receiving unit for receiving a multiplexed stream including a video stream constituted by video packets having coded image data in their payloads and a subtitle stream constituted by subtitle packets having subtitle information in their payloads; In the multiplexed stream, the subtitle packets are arranged at random access positions, The apparatus further includes a processing unit that extracts the subtitle packet from the multiplexed stream together with the video packet at the random access position and performs a subtitle display process. It is in the receiving device.

[0014] In this technology, a receiving unit receives a multiplexed stream including a video stream composed of video packets having coded image data in their payloads and a subtitle stream composed of subtitle packets having subtitle information in their payloads. In the multiplexed stream, the subtitle packets are arranged at random access positions. A processing unit extracts the subtitle packets together with the video packets at the random access positions from the multiplexed stream and performs subtitle display processing.

[0015] In this way, with this technology, subtitle packets are placed at random access positions in a multiplexed stream, and the subtitle packets are extracted from this multiplexed stream together with the video packets at the random access positions for subtitle display processing, which simplifies the subtitle display processing in variable speed playback mode.

[0016] In the present technology, for example, a first timestamp having a value equal to or close to the value of the timestamp inserted at the timestamp insertion position of the header of the video packet at the random access position may be inserted at the timestamp insertion position of the header of the subtitle packet, a second timestamp indicating the display time of the subtitle may be inserted in the header or payload of the subtitle packet, and the processing unit may perform the subtitle display process by referring to the second timestamp in normal playback mode. In this case, the display timing of the subtitle may be easily controlled.

[0017] In this case, for example, identification information indicating that a second timestamp has been inserted into the subtitle packet may be inserted into the multiplexed stream, and the processing unit may extract and use the second timestamp from the subtitle packet based on this identification information. [Effects of the Invention]

[0018] According to the present technology, it is possible to simplify the subtitle display process in variable speed playback mode on the receiving side. Note that the effects described in this specification are merely examples and are not limiting, and additional effects may also be provided. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a block diagram showing an example of the configuration of a transmission / reception system according to an embodiment; [Figure 2] 1 is a block diagram showing an example of the configuration of a stream generation unit of a broadcast transmission system. [Figure 3] FIG. 1 is a diagram for explaining a TTML structure. [Figure 4] FIG. 10 is a diagram showing an example of the structure of each element of metadata, styling, and layout present in the header of the TTML structure. [Figure 5] FIG. 10 is a diagram showing an example of the structure of a body of a TTML structure. [Figure 6] FIG. 2 is a diagram illustrating an example of the configuration of a timing management unit in a control unit. [Figure 7] FIG. 10 is a diagram illustrating a detailed configuration of a subtitle display timing management unit. [Figure 8] FIG. 10 is a diagram illustrating an example of the configuration of a PES packet. [Figure 9] A figure showing an example (1 / 2) of the configuration of the PES extension "PES extension". [Figure 10] A figure showing an example (2 / 2) of the configuration of the PES extension "PES extension". [Figure 11] FIG. 10 is a diagram showing an example of the structure of a PES payload (PES_payload) placed in the "PES packet data bytes" area. [Figure 12] The contents of new elements in the example configuration of the PES extension and PES payload are shown. [Figure 13] FIG. 10 is a diagram showing an example structure of a subtitle lap descriptor and the contents of the main information in the example structure. [Figure 14] FIG. 1 is a diagram illustrating an example of the configuration of a transport stream TS. [Figure 15] FIG. 1 is a diagram illustrating an example of the configuration of a television receiver. [Figure 16] FIG. 10 is a diagram illustrating the timing of decoding and output processes for a subtitle PES packet read at a RAP (Random Access Point) position. [Figure 17] FIG. 1 is a diagram for explaining conventional multiplexing. [Figure 18] FIG. 1 is a diagram for explaining multiplexing of the present technology. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, modes for carrying out the invention (hereinafter referred to as "embodiments") will be described. The description will be made in the following order. 1. Embodiment 2. Variations

[0021] <1. Embodiment> [Example of transmission and reception system configuration] FIG. 1 shows an example of the configuration of a transmission / reception system 10 according to an embodiment. This transmission / reception system 10 is made up of a broadcast transmission system 100 and a television receiver 200. The broadcast transmission system 100 transmits a transport stream TS, which is a multiplexed stream, over broadcast waves. The transport stream TS has a video stream and a subtitle stream. The video stream is made up of video PES packets having image data in their payloads. The subtitle stream is made up of subtitle PES packets having subtitle (subtitle) information in their payloads.

[0022] A timestamp with a value equal to or close to the timestamp value inserted at the timestamp insertion position of the header of the video PES packet at the random access position is inserted at the timestamp insertion position of the header of the subtitle PES packet. Also, a unique timestamp indicating the display time of the subtitle is inserted into the header or payload of the subtitle PES packet.

[0023] When a subtitle stream is multiplexed with a video stream, etc., a TS packet containing a subtitle PES packet (hereinafter simply referred to as a PES packet) is placed at a random access position. In this case, for example, as described above, the timestamp inserted at the timestamp insertion position of the header is referenced, and the subtitle PES packet is placed at the random access position, i.e., at a position close to the position of the video PES packet at the random access position.

[0024] Identification information indicating that a unique timestamp has been inserted into the subtitle PES packet is inserted into the transport stream TS. This identification information is inserted, for example, under the program map table. As mentioned above, a unique timestamp is inserted into the header or payload of the subtitle PES packet. Information indicating the insertion position is added to this identification information.

[0025] The television receiver 200 receives a transport stream TS transmitted by broadcast waves from the broadcast transmission system 100. As described above, this transport stream TS has at least a video stream and a subtitle stream. The transport stream TS may also include an audio stream. The video stream is made up of video PES packets whose payloads contain image data. The subtitle stream is made up of subtitle PES packets whose payloads contain subtitle information. When this transport stream TS is multiplexed, the subtitle PES packets are arranged at random access positions.

[0026] The television receiver 200 extracts the subtitle PES packet from the transport stream TS together with the video PES packet at the random access position, and performs display processing of the subtitles (captions). For example, in variable speed playback mode, the television receiver 200 immediately processes the subtitle PES packet extracted from the random access position of the transport stream TS to display the subtitles.

[0027] In normal playback mode, for example, a subtitle PES packet extracted from a random access position of the transport stream TS is processed based on a unique timestamp inserted in the header or payload of the subtitle packet, which indicates the display time of the subtitle, and the subtitle is displayed at the display time. In this case, the television receiver 200 extracts and uses the unique timestamp from the subtitle PES packet based on the identification information inserted in the transport stream TS.

[0028] [Example of the configuration of the stream generation part of a broadcast transmission system] 2 shows an example of the configuration of the stream generation unit 110 of the broadcast transmission system 100. This stream generation unit 110 has a control unit 111, a video encoder 112, an audio encoder 113, a text format conversion unit 114, a subtitle encoder 115, and a TS formatter (multiplexer) 116.

[0029] The control unit 111 is configured to include, for example, a CPU (Central Processing Unit), and controls the operation of each unit in the stream generation unit 110. The video encoder 112 inputs image data DV, encodes the image data DV, and generates a video stream (PES stream) made up of video PES packets having encoded image data in their payloads. The audio encoder 113 inputs audio data DA, encodes the audio data DA, and generates an audio stream (PES stream) made up of audio PES packets having encoded audio data.

[0030] The text format conversion unit 114 receives text data (character code) DT as subtitle information and obtains subtitle text information in a predetermined format having display timing information. This text information may be, for example, TTML or a format derived from TTML, but in this embodiment, it is assumed to be TTML.

[0031] Figure 3 shows the TTML structure. TTML is written in XML format. The header contains the elements of metadata, styling, and layout. Figure 4(a) shows an example of the structure of metadata (TTM: TTML Metadata). This metadata includes information on the metadata title and copyright information.

[0032] Figure 4(b) shows an example of the structure of styling (TTS: TTML Styling). In addition to an identifier (id), this styling includes information such as color (color), font (fontFamily), size (fontSize), and alignment (textAlign). Figure 4(c) shows an example of the structure of layout (region: TTML layout). In addition to the identifier (id) of the region where the subtitle will be placed, this layout includes information such as extent (extent), offset (padding), background color (backgroundColor), and alignment (displayAlign).

[0033] Fig. 5 shows an example of the structure of a body. In the example shown, information on three subtitles, subtitle 1, subtitle 2, and subtitle 3, is included. For each subtitle, the display start timing and display end timing are recorded, along with text data. For example, for subtitle 1, the display start timing is "0.76 seconds," the display end timing is "3.45 seconds," and the text data is "It seems a paradox, dose it not."

[0034] Returning to Figure 2, the subtitle encoder 115 converts the TTML obtained by the text format conversion unit 114 into various segments and generates a subtitle stream (PES stream) consisting of subtitle PES packets in which these segments (subtitle information) are placed in the payload.

[0035] Under the control of the control unit 111, the subtitle encoder 115 inserts a timestamp (first timestamp) at the timestamp insertion position in the header of the subtitle PES packet, the timestamp value being the same as or close to the timestamp value to be inserted at the timestamp insertion position in the header of the video PES packet at the random access position, i.e., the video PES packet targeted for the RAP (Random Access Point).

[0036] Here, the leading portion of a video PES packet to be subjected to RAP means, for example, the leading portion of a video PES packet having coded image data of an intra-picture (I-picture) in its payload. In so-called RAP playback, only the coded image data of the intra-picture contained in the video PES packet to be subjected to RAP is decoded from the transport stream TS and displayed as an image.

[0037] Furthermore, under the control of the control unit 111, the subtitle encoder 115 inserts a unique timestamp (second timestamp) indicating the display time (display start time, display end time) of the subtitle into the header or payload of the subtitle PES packet.

[0038] 6 shows a schematic configuration example of the timing management unit 120 in the control unit 111. This timing management unit 120 has a video timing management unit 121, an audio timing management unit 122, a RAP timing management unit 123, and a subtitle display timing management unit 124.

[0039] The video timing management unit 121 manages the timestamps (timestamps of video access units) to be inserted at the timestamp insertion positions of each video PES packet. Based on the management information of this video timing management unit 121, the video encoder 112 inserts the timestamps at the timestamp insertion positions of each video PES packet.

[0040] The audio timing management unit 122 manages the timestamps (timestamps of audio access units) to be inserted at the timestamp insertion positions of each audio PES packet. Based on the management information of this audio timing management unit 122, the audio encoder 113 inserts the timestamps at the timestamp insertion positions of each audio PES packet.

[0041] The RAP timing management unit 123 manages the timestamps of the video PES packets that are the RAP targets, and also manages the audio PES packets that are the RAP targets among the audio PES packets. In this case, the audio PES packets that have timestamps close to the timestamp of the video PES packets that are the RAP targets are treated as the audio PES packets to be played back during the RAP.

[0042] Based on the management information of the audio PES packet to be RAPed, the TS formatter 116 places the audio PES packet to be RAPed at a random access position, i.e., a position close to the position of the video PES packet to be RAPed, as described below.

[0043] The subtitle display timing management unit 124 manages the first timestamp (the timestamp inserted in the timestamp insertion area of ​​the header) and the second timestamp (a unique timestamp that indicates the display time of the subtitle inserted in the header or payload) of the subtitle PES packet. The subtitle encoder 115 inserts the first timestamp and the second timestamp into the subtitle PES packet based on the management information of the subtitle display timing management unit 124.

[0044] 7 shows a detailed configuration of the subtitle display timing management unit 124. The subtitle display timing management unit 124 obtains a first timestamp (timestamp of the subtitle stream) based on the RAP timing information from the RAP timing management unit 123. The subtitle display timing management unit 124 also performs timestamp precision conversion on the relative start time "begin" and relative end time "end" included in the TTML body to obtain second timestamps (display start timestamp, display end timestamp).

[0045] As shown in Figure 5, if the display timing of multiple regions is different, the relative start time "begin" and relative end time "end" of the region with the earliest timing are converted to timestamp precision to obtain the display start timestamp and display end timestamp as the second timestamp.

[0046] In this case, the receiving side uses the display start timestamp and display end timestamp as the second timestamp for display control of the earliest region. Also, for display control of subsequent regions, display timestamps (display start timestamp, display end timestamp) converted to timestamp accuracy based on the difference from the relative time of the earliest region supplied in the TTML body are generated and used.

[0047] "Insertion position of the second timestamp (unique timestamp)" Here, we will explain a specific example of the insertion position of the second timestamp (unique timestamp). Figure 8 shows an example of the structure (Syntax) of a PES packet. The first timestamp is inserted into the 33-bit area, indicated by arrow P, which is the timestamp insertion area of ​​the header. This timestamp insertion area exists in the "optional fields" below the "optional PES HEADER" that makes up the header.

[0048] As described above, the second timestamp is inserted into the header or payload. When inserted into the header, it is inserted, for example, into the "PES extension fields data" area indicated by arrow Q. This "PES extension fields data" exists in the "optional fields" below the "PES extension." Note that the "PES extension" exists in the "optional fields" below the "optional PES HEADER" that constitutes the header. On the other hand, when inserted into the payload, it is inserted into the "PES packet data bytes" area indicated by arrow R.

[0049] Figures 9 and 10 show examples of the syntax of the PES extension "PES extension." Figure 11 shows examples of the syntax of the PES payload (PES_payload) placed in the "PES packet data bytes" area. Figure 12 shows the semantics of new elements in each example.

[0050] First, an example of the configuration of the PES extension shown in Figures 9 and 10 will be described. As flag information, a new flag, "PES_extension_negflag_new", is defined. This flag indicates that newly defined data is to be placed in the extension area of ​​the PES header. "0" indicates that newly defined data is to be placed in the extension area of ​​the PES header. "1" indicates that newly defined data is not to be placed in the extension area of ​​the PES header.

[0051] When "PES_extension_negflag_new" is "0", an 8-bit field "extension_type" exists. This field indicates the type of newly defined data. "1" indicates that it is a timestamp indicating the start and end of subtitle display. When "extension_type" is "1", a 33-bit field "subtitle_start_time" and a 33-bit field "subtitle_end_time" exist.

[0052] The 33-bit field "subtitle_start_time" indicates the value (display start timestamp) that represents the start time of subtitle display with system clock accuracy. The 33-bit field "subtitle_end_time" indicates the value (display end timestamp) that represents the end time of subtitle display with system clock accuracy.

[0053] Next, we will explain an example of the configuration of the PES payload (PES_payload) shown in Figure 11. A subtitle PES packet is defined as a private PES packet. In the case of a private PES packet, the "PES packet data bytes" can be freely defined, and a timestamp indicating the start and end of display is inserted into part of it.

[0054] The 8-bit field of "private_type" indicates the type of private PES packet. "EC" indicates that it is a subtitle stream with a timestamp. The flag information of "timestamp_insertion_flag" indicates that a timestamp is placed at the beginning of the packet payload. "1" indicates that a timestamp is placed. "0" indicates that a timestamp is not placed. When "timestamp_insertion_flag" is "1", a 33-bit field of "subtitle_start_time" and a 33-bit field of "subtitle_end_time" exist.

[0055] Returning to Figure 2, the TS formatter 116 transport packets and multiplexes the video stream generated by the video encoder 112, the audio stream generated by the audio encoder 113, and the subtitle stream generated by the subtitle encoder 115, to obtain a transport stream TS as a multiplexed stream.

[0056] When multiplexing the streams in this way, the TS formatter 116 places the subtitle PES packet at a random access position, i.e., at a position close to the position of the video PES packet to be RAPed, based on the first timestamp (the timestamp inserted at the timestamp insertion position in the header.) At this time, the TS formatter 116 also places the audio PES packet to be RAPed at a random access position, i.e., at a position close to the position of the video PES packet to be RAPed, based on the management information of the audio PES packet to be RAPed.

[0057] The TS formatter 116 also inserts into the transport stream TS identification information indicating that a second timestamp (a unique timestamp indicating the display time of the subtitle) has been inserted into the subtitle PES packet. In this embodiment, the TS formatter 116 inserts a subtitle wrap descriptor (Subtitle_rap_descriptor) into a subtitle elementary stream loop corresponding to a subtitle stream under the program map table (PMT).

[0058] Figure 13(a) shows an example structure (Syntax) of a subtitle wrap descriptor. Figure 13(b) shows the main information content (Semantics) of this example structure. The 8-bit field "descriptor_tag" indicates the descriptor type, which in this case indicates that it is a subtitle wrap descriptor. The 8-bit field "descriptor_length" indicates the length (size) of the descriptor, and indicates the number of subsequent bytes as the length of the descriptor.

[0059] The flag information of "subtitle_presentation_time_flag" indicates whether subtitles are displayed at a timing different from the conventional timestamp (first timestamp), mainly at a delayed timing, i.e., whether a display timestamp unique to the subtitle is arranged. "1" indicates that delayed display (arrangement of a display timestamp unique to the subtitle) is performed. "0" indicates that delayed display (arrangement of a display timestamp unique to the subtitle) is not performed.

[0060] When "subtitle_presentation_time_flag" is "1", a 2-bit field "time_insertion_type" exists. This field indicates the insertion type of the subtitle-specific presentation timestamp, that is, where the timestamp is inserted. "01" indicates that it is inserted in the PES header extension section. "10" indicates that it is inserted in the PES private stream payload.

[0061] 14 shows an example of the structure of a transport stream TS. In this example, there is a video PES packet "Video PES" which is a PES packet of a video stream identified by PID1. In addition, there is an audio PES packet "Audio PES" which is a PES packet of an audio stream identified by PID2. In addition, there is a subtitle PES packet "Subtitle PES" which is a PES packet of a subtitle stream identified by PID3.

[0062] A PES packet consists of a PES header and a PES payload. In a video PES packet, a DTS / PTS timestamp is inserted in the PES header, and a video coded stream (coded image data) is inserted in the PES payload. In an audio PES packet, a PTS timestamp is inserted in the PES header, and an audio coded stream (coded audio data) is inserted in the PES payload.

[0063] In addition, in a subtitle PES packet, a PTS timestamp (first timestamp) is inserted in the PES header, and a subtitle encoded stream (multiple subtitle segments as subtitle information) is inserted in the PES payload. Also, the PES header or PES payload contains a display start timestamp "subtitle_start_time" and a display end timestamp "subtitle_end_time" as second timestamps (unique timestamps).

[0064] The transport stream TS also includes a PMT (Program Map Table) as PSI (Program Specific Information). This PSI is information that indicates to which program each elementary stream included in the transport stream TS belongs. The transport stream TS also includes an EIT (Event Information Table) as SI (Serviced Information) that manages events. This EIT describes metadata for each program.

[0065] The PMT contains a program descriptor, which describes information related to the entire program. The PMT also contains elementary stream loops, which contain information related to each elementary stream. In this example configuration, there is a video elementary stream loop (Video ES loop), an audio elementary stream loop (Audio ES loop), and a subtitle elementary stream loop (Subtitle ES loop).

[0066] Each loop contains information such as a packet identifier (PID), as well as a descriptor that describes information related to that elementary stream. The subtitle elementary stream loop contains the subtitle rap descriptor (Subtitle_rap_descriptor) described above as one of its descriptors.

[0067] The operation of the stream generation unit 110 shown in Fig. 2 will now be briefly described. Image data DV is supplied to a video encoder 112. The video encoder 112 encodes this image data DV to generate a video stream (PES stream) made up of video PES packets whose payloads contain encoded image data. This video stream is supplied to a TS formatter 116.

[0068] The audio data DA is also supplied to an audio encoder 113. The audio encoder 113 encodes the audio data DA to generate an audio stream (PES stream) made up of audio PES packets containing encoded audio data. This audio stream is supplied to a TS formatter 116.

[0069] Furthermore, text data (character code) DT as subtitle information is supplied to a text format conversion unit 114. This text format conversion unit 114 obtains subtitle text information in a predetermined format having display timing information, in this case TTML. This TTML is supplied to a subtitle encoder 115.

[0070] The subtitle encoder 115 converts the TTML into various segments and generates a subtitle stream consisting of subtitle PES packets with these segments arranged in the payload. This subtitle stream is supplied to the TS formatter 116.

[0071] When generating a subtitle stream, the subtitle encoder 115, under the control of the control unit 111, inserts a timestamp (first timestamp) at the timestamp insertion position in the header of the subtitle PES packet, the timestamp being the same as or close to the timestamp value to be inserted at the timestamp insertion position in the header of the video PES packet to be RAPed.

[0072] In addition, when generating a subtitle stream, the subtitle encoder 115, under the control of the control unit 111, inserts a unique timestamp (second timestamp) indicating the display time of the subtitle (display start time, display end time) into the header or payload of the subtitle PES packet.

[0073] In the TS formatter 116, the video stream generated by the video encoder 112, the audio stream generated by the audio encoder 113, and the subtitle stream generated by the subtitle encoder 115 are transport packetized and multiplexed to generate a transport stream TS as a multiplexed stream.

[0074] When each stream is multiplexed in this manner, the subtitle PES packet is placed at a random access position, i.e., a position close to the position of the video PES packet to be RAPed, based on the first timestamp (the timestamp inserted at the timestamp insertion position in the header) under the control of the control unit 111.

[0075] Also, at this time, under the control of the control unit 111, based on the management information of the audio PES packet to be RAPed, the audio PES packet to be RAPed is placed at a random access position, i.e., a position close to the position of the video PES packet to be RAPed.

[0076] The TS formatter 116 also inserts identification information into the transport stream TS. This identification information indicates that a second timestamp (a unique timestamp indicating the display time of the subtitle) has been inserted into the subtitle PES packet. In other words, the TS formatter 116 places a subtitle rap descriptor (Subtitle_rap_descriptor) in the subtitle elementary stream loop corresponding to the subtitle stream under the program map table (PMT).

[0077] [Example of a TV receiver configuration] 15 shows an example configuration of a television receiver 200. The television receiver 200 has a receiving unit 201, a TS analysis unit (demultiplexer) 202, a video decoder 203, a video superimposition unit 204, a panel drive circuit 205, and a display panel 206. The television receiver 200 also has an audio decoder 207, an audio output circuit 208, a speaker 209, and a subtitle decoder 210. The television receiver 200 also has a CPU 221, a flash ROM 222, a DRAM 223, an internal bus 224, a remote control receiving unit 225, and a remote control transmitter 226.

[0078] The CPU 221 controls the operation of each unit of the television receiver 200. The flash ROM 222 stores control software and data. The DRAM 223 constitutes a work area for the CPU 221. The CPU 221 loads software and data read from the flash ROM 222 onto the DRAM 223, starts up the software, and controls each unit of the television receiver 200.

[0079] The remote control receiving unit 225 receives a remote control signal (remote control code) transmitted from the remote control transmitter 226 and supplies it to the CPU 221. The CPU 221 controls each unit of the television receiver 200 based on the remote control code. The CPU 221, flash ROM 222, and DRAM 223 are connected to an internal bus 224.

[0080] The receiving unit 201 receives a transport stream TS transmitted via broadcast waves from the broadcast transmission system 100. As described above, this transport stream TS includes a video stream, an audio stream, and a subtitle stream. The TS analyzing unit 202 extracts PES packets for each of the video, audio, and subtitle streams from the transport stream TS.

[0081] Here, the transport stream TS received by the receiving unit 201 is temporarily stored in a storage medium (buffer or storage) 201a, and a portion corresponding to the playback mode is extracted and sent to the TS analysis unit 202. For example, in normal playback mode, all of the stored data is sent to the TS analysis unit 202. In contrast, in variable speed playback mode, an intermittent portion corresponding to the multiplication speed, including a portion at a RAP position (random access position), is sent to the TS analysis unit 202. In particular, in RAP playback mode, only the portion at the RAP position (random access position) is sent to the TS analysis unit 202.

[0082] The TS analyzer 202 also extracts various pieces of information inserted into the transport stream TS and sends it to the CPU 221. This information includes information about the subtitle rap descriptor (Subtitle_rap_descriptor). This allows the CPU 221 to easily recognize that a second timestamp (unique timestamp) has been inserted into the subtitle PES packet and its insertion location, which can be used effectively to control the subtitle decoder 210.

[0083] The audio decoder 207 decodes the audio PES packets extracted by the TS analysis unit 202 to obtain audio data. The audio output circuit 208 performs necessary processing such as D / A conversion and amplification on the audio data, and supplies the data to a speaker 209. The video decoder 203 decodes the video PES packets extracted by the TS analysis unit 202 to obtain image data.

[0084] In normal playback mode, the decoding and output process for each PES packet in the audio decoder 207 and video decoder 203 is controlled by the timestamp inserted in the header, but in variable speed playback mode, it is not controlled by the timestamp and is performed immediately after being supplied from the TS analysis unit 202.

[0085] The subtitle decoder 210 decodes the subtitle PES packets extracted by the TS analyzer 202 to obtain bitmap data for each region to be superimposed on the image data. In normal playback mode, the decoding and output process for the subtitle PES packets in the subtitle decoder 210 is controlled by a second timestamp (unique timestamp) inserted in the header or payload, but in variable-speed playback mode, it is not controlled by the second timestamp and is performed immediately after being supplied from the TS analyzer 202.

[0086] 16 shows, for example, the timing of decoding and output of a subtitle PES packet read at a RAP position. In normal playback mode, decoding is completed by the time indicated by the display start timestamp, and output of the decoded bitmap data begins at that time and continues until the time indicated by the display end timestamp. In this case, the subtitle display period is from the time indicated by the display start timestamp to the time indicated by the display end timestamp.

[0087] In contrast, in variable-speed playback mode, decoding is performed immediately after reading from the storage medium 201a, and output of the decoded bitmap data begins. The duration of this output is the subtitle display period. This output duration is determined by the variable-speed playback speed and the subtitle display maintenance function.

[0088] 15, the video superimposing unit 204 superimposes the bitmap data of each region obtained from the subtitle decoder 210 on the image data obtained by the video decoder 203. The panel driving circuit 205 drives the display panel 206 based on the display image data obtained by the video superimposing unit 204. The display panel 206 is configured, for example, by an LCD (Liquid Crystal Display), an organic EL display (organic electroluminescence display), or the like.

[0089] The operation of the television receiver 200 shown in Fig. 15 will now be briefly described. The receiving unit 201 receives a transport stream TS transmitted on broadcast waves from the broadcast transmission system 100. This transport stream TS includes a video stream, an audio stream, and a subtitle stream.

[0090] This transport stream TS is supplied to the TS analysis unit 202 via the storage medium 201a. In this case, a portion corresponding to the playback mode is extracted and sent to the TS analysis unit 202. For example, in normal playback mode, all of the stored data is sent to the TS analysis unit 202. In contrast, in variable speed playback mode, an intermittent portion corresponding to the multiplication speed, including a portion at the RAP position (random access position), is sent to the TS analysis unit 202. In particular, in RAP playback mode, only the portion at the RAP position (random access position) is sent to the TS analysis unit 202.

[0091] The video PES packets extracted by the TS analyzer 202 are supplied to the video decoder 203. The video decoder 203 decodes the video PES packets extracted by the TS analyzer 202 to obtain image data. The decoding and output process for each video PES packet is controlled by a timestamp inserted in the header in normal playback mode, but is not controlled by the timestamp in variable-speed playback mode, and is performed immediately after being supplied from the TS analyzer 202. This image data is supplied to the video superimposing unit 204.

[0092] The subtitle stream (PES stream) extracted by the TS analyzer 202 is supplied to the subtitle decoder 210. The subtitle decoder 210 processes the segment data of each region to obtain bitmap data for each region to be superimposed on the image data. In normal playback mode, the decoding and output process for each subtitle PES packet is controlled by a second timestamp (unique timestamp) inserted in the header or payload, but in variable-speed playback mode, it is not controlled by the second timestamp and is performed immediately after being supplied from the TS analyzer 202 (see FIG. 16).

[0093] The bitmap data for each region output from the subtitle decoder 210 is supplied to the video superimposing unit 204. The video superimposing unit 204 superimposes the bitmap data for each region output from the subtitle decoder 210 on the image data obtained by the video decoder 203.

[0094] The image data for display obtained by the video superimposing unit 204 is supplied to a panel driving circuit 205. The panel driving circuit 205 drives a display panel 206 based on the video data for display. As a result, an image with a subtitle superimposed thereon is displayed on the display panel 206. In this case, in normal playback mode, the image is played back at a normal speed, and in variable speed playback mode, the image is played back at a variable speed corresponding to the double speed.

[0095] The audio PES packets extracted by the TS analyzer 202 are supplied to the audio decoder 207. The audio decoder 207 decodes the audio PES packets to obtain audio data. The decoding and output process for each audio PES packet is controlled by a timestamp inserted in the header in normal playback mode, but is not controlled by the timestamp in variable-speed playback mode, and is performed immediately after being supplied from the TS analyzer 202.

[0096] This audio data is supplied to an audio output circuit 208. The audio output circuit 208 performs necessary processing such as D / A conversion and amplification on the audio data. The processed audio data is then supplied to a speaker 209. As a result, an audio output corresponding to the image displayed on the display panel 206 is obtained from the speaker 209.

[0097] As described above, in the transmission / reception system 10 shown in Fig. 1, when the transmission side generates a transport stream TS as a multiplexed stream, the subtitle PES packet is placed at a random access position. Therefore, on the reception side, when extracting the video PES packet at the random access position from the multiplexed stream, the subtitle PES packet can also be extracted, simplifying the subtitle display process in variable speed playback mode.

[0098] For example, Figure 17 shows conventional multiplexing. Figure 17(a) shows a sequence of video PES packets that make up a video stream. "V_TS00", "V_TS10", "V_TS20", "V_TS30", etc. indicate timestamps inserted in the header, and "V0", "V10", "V20", "V30", etc. indicate coded image data placed in the payload.

[0099] Figure 17(b) shows a sequence of audio PES packets that make up an audio stream. "A_TS00", "A_TS10", "A_TS20", "A_TS30", etc. indicate timestamps inserted in the header, and "A0", "A10", "A20", "A30", etc. indicate coded audio data stored in the payload.

[0100] Figure 17(c) shows a sequence of subtitle PES packets that make up a subtitle stream. "S_TS15", "S_TS19", "S_TS35", "A_TS39", etc. indicate timestamps inserted in the header, and "S15", "S19", "S35", "S39", etc. indicate subtitle information placed in the payload. A timestamp is inserted in the header of a subtitle PES packet, regardless of the timestamps inserted in the headers of the video PES packet and audio PES packet.

[0101] 17(d) shows an example of the arrangement of each PES packet in a multiplexed stream (transport stream TS). During multiplexing, the arrangement of each PES packet is determined by referring to the timestamp inserted in the header of each PES packet. In this case, the subtitle PES packet is arranged slightly before the display period, regardless of the RAP position (random access position).

[0102] In RAP playback, for example, as shown in playback method 1 in Fig. 17(e), a certain length of PES packet is read for each RAP position, decoded, and image display and audio output are performed. In this case, subtitle PES packets are not necessarily placed at the RAP positions, so subtitles are not displayed.

[0103] Even with this RAP playback, to display subtitles, it is possible to read the subtitle PES packets in addition to reading the RAP positions, as shown in playback method 2 in Fig. 17(f). In this case, in addition to the jump for reading the RAP positions, it is necessary to repeat jumps for reading the subtitle PES packets, which complicates the playback process. Also, image display and audio output must wait until the subtitle PES packets are read, which causes delays in image display and audio output.

[0104] Figure 18 shows the multiplexing of this technology. 18(a) shows a sequence of video PES packets that make up a video stream. "V_TS00", "V_TS10", "V_TS20", "V_TS30", etc. indicate timestamps inserted in the header, and "V0", "V10", "V20", "V30", etc. indicate coded image data stored in the payload.

[0105] 18(b) shows a sequence of audio PES packets that make up an audio stream. "A_TS00", "A_TS10", "A_TS20", "A_TS30", etc. indicate timestamps inserted in the header, and "A0", "A10", "A20", "A30", etc. indicate coded image data placed in the payload.

[0106] Figure 18(c) shows a sequence of subtitle PES packets that make up a subtitle stream. "S_TS10", "S_TS30", etc. indicate timestamps inserted in the header, and "S10", "S130", etc. indicate subtitle information placed in the payload. A timestamp (first timestamp) that is the same as or close to the timestamp inserted in the header of the video PES packet that is the target of RAP is inserted in the header of the subtitle PES packet.

[0107] 18(d) shows an example of the arrangement of each PES packet in a multiplexed stream (transport stream TS). When multiplexing, the arrangement of each PES packet is determined by referring to the timestamp inserted in the header of each PES packet. In this case, the subtitle PES packet is arranged at the RAP position (random access position) in the same way as the video PES packet and audio PES packet that are the target of the RAP.

[0108] In RAP playback, for example, as shown in playback method 3 in Fig. 18(e), PES packets are read out for each RAP position over a certain length and decoded. In this case, subtitle PES packets are read out and decoded together with video and audio PES packets, so that subtitles can be displayed well along with image and audio output.

[0109] By performing multiplexing in this manner as in the present technology, it is not necessary to repeatedly jump to read the subtitle PES packet in addition to the jump to read the RAP position, as in playback method 2 shown in Figure 17(f), and this simplifies the variable speed playback process on the receiving side to enable subtitle display.

[0110] 1, a second timestamp (unique timestamp) indicating the subtitle display time is inserted into the header or payload of the subtitle PES packet, which allows the receiving side to easily control the subtitle display timing based on this unique timestamp.

[0111] 1, identification information indicating that a unique timestamp has been inserted into a subtitle PES packet is inserted into the transport stream TS. Therefore, the receiving side can easily recognize that a unique timestamp has been inserted based on this identification information, and can efficiently extract the unique timestamp.

[0112] <2. Modifications> In the above-described embodiment, an example has been shown in which TTML is used as subtitle text information in a predetermined format having display timing information. However, the present technology is not limited to this, and other timed text information having information equivalent to TTML may also be used. For example, a format derived from TTML may be used. Furthermore, it goes without saying that the present technology can also be applied to conventional bitmap formats as subtitle formats in the same manner.

[0113] Furthermore, in the above-described embodiment, the transmission / reception system 10 including the broadcast transmission system 100 and the television receiver 200 is shown, but the configuration of the transmission / reception system to which the present technology can be applied is not limited to this. For example, the television receiver 200 may be configured as a set-top box and a monitor connected via a digital interface such as HDMI (High-Definition Multimedia Interface). Note that "HDMI" is a registered trademark.

[0114] The present technology can also be configured as follows. (1) a video encoding unit that generates a video stream composed of video packets having encoded image data in their payloads; a subtitle encoding unit that generates a subtitle stream composed of subtitle packets having subtitle information in their payloads; a multiplexed stream generator that generates a multiplexed stream including the video stream and the subtitle stream; a transmitter for transmitting the multiplexed stream, The multiplexed stream generation unit The above subtitle packets are placed in the random access position. Transmitting device. (2) The subtitle encoding section is A timestamp having a value equal to or close to the value of the timestamp inserted at the timestamp insertion position of the header of the video packet at the random access position is inserted at the timestamp insertion position of the header of the subtitle packet. The transmitting device according to (1) above. (3) The subtitle encoding section is Insert a unique timestamp indicating the subtitle display time into the header or payload of the subtitle packet. The transmitting device according to (2) above. (4) The multiplexed stream generating unit Inserting identification information into the multiplexed stream, the identification information indicating that the unique timestamp is inserted into the subtitle packet. The transmitting device according to (3) above. (5) The above identification information includes information indicating the insertion position. The transmitting device according to (4) above. (6) The subtitle information is text information of a subtitle in a predetermined format having display timing information. The transmitting device according to any one of (1) to (5). (7) The text information of the subtitles in the above specified format is TTML or a format derived from the TTML. The transmitting device according to (6) above. (8) the video packets and the subtitle packets are PES packets; The multiplexed stream is an MPEG2 transport stream. The transmitting device according to any one of (1) to (7). (9) a video encoding step of generating a video stream composed of video packets having encoded image data in their payloads; a subtitle stream generating step of generating a subtitle stream composed of subtitle packets having subtitle information in their payloads; a multiplexed stream generating step of generating a multiplexed stream including the video stream and the subtitle stream; a transmitting step of transmitting the multiplexed stream by a transmitting unit; In the multiplexed stream generating step, The above subtitle packets are placed in the random access position. Sending method. (10) A receiver for receiving a multiplexed stream including a video stream composed of video packets having coded image data in their payloads and a subtitle stream composed of subtitle packets having subtitle information in their payloads, In the multiplexed stream, the subtitle packets are arranged at random access positions, The apparatus further includes a processing unit that extracts the subtitle packet from the multiplexed stream together with the video packet at the random access position and performs a subtitle display process. Receiving device. (11) A first timestamp having a value equal to or close to the value of a timestamp inserted at the timestamp insertion position of the header of the video packet at the random access position is inserted at the timestamp insertion position of the header of the subtitle packet; a second timestamp indicating a subtitle display time is inserted into the header or payload of the subtitle packet; The processing unit In normal playback mode, the subtitle display process is performed by referring to the second timestamp. The receiving device according to (10) above. (12) Identification information indicating that the second timestamp is inserted into the subtitle packet is inserted into the multiplexed stream; The processing unit extracts and uses the second timestamp from the subtitle packet based on the identification information. The receiving device according to (11) above. (13) A receiving step of receiving, by a receiving unit, a multiplexed stream including a video stream constituted by video packets having coded image data in their payloads and a subtitle stream constituted by subtitle packets having subtitle information in their payloads, In the multiplexed stream, the subtitle packets are arranged at random access positions, The method further includes a processing step of extracting the subtitle packets from the multiplexed stream together with the video packets at the random access position and performing a subtitle display process. Receiving method.

[0115] The main feature of this technology is that when generating a transport stream TS as a multiplexed stream, subtitle PES packets are placed at random access positions, thereby simplifying the subtitle display process in variable speed playback mode on the receiving side (see Figure 18). [Explanation of symbols]

[0116] 10. Transmitting and receiving system 100···Broadcast transmission system 110 Stream generation unit 111 Control unit 112...Video Encoder 113 Audio Encoder 114 Text format conversion section 115···Subtitle Encoder 116···TS formatter (multiplexer) 120 Timing control section 121 Video Timing Control Unit 122 Audio timing control unit 123 RAP Timing Control Unit 124 Subtitle display timing control unit 200...TV receiver 201 Receiving unit 201a...Storage medium 202...TS analysis department 203 Video decoder 204 Video overlay unit 205 Panel drive circuit 206···Display panel 207 Audio Decoder 208 Audio output circuit 209···Speaker 210 Subtitle Decoder 221 CPU

Claims

1. a video stream generating unit that generates a video stream composed of video packets having encoded image data in their payloads; a subtitle stream generator for generating a subtitle stream composed of subtitle packets each having subtitle information in its payload; a multiplexed stream generator that generates a multiplexed stream including the video stream and the subtitle stream; a transmitter for transmitting the multiplexed stream, The subtitle information is subtitle text information in TTML or a format derived from the TTML, which has display timing information, In the multiplexed stream, the subtitle packets are arranged at random access positions. Transmitting device.

2. The above subtitle encoding section is A timestamp having a value equal to or close to the value of the timestamp inserted at the timestamp insertion position of the header of the video packet at the random access position is inserted at the timestamp insertion position of the header of the subtitle packet. The transmitting device according to claim 1 .

3. The above subtitle encoding section is Insert a unique timestamp indicating the subtitle display time into the header or payload of the subtitle packet. The transmitting device according to claim 2 .

4. The multiplexed stream generation unit Inserting identification information into the multiplexed stream, the identification information indicating that the unique timestamp is inserted into the subtitle packet. The transmitting device according to claim 3 .

5. The identification information includes information indicating whether the unique timestamp is inserted into the header or payload of the subtitle packet. The transmitting device according to claim 4.

6. the video packets and the subtitle packets are PES packets, The multiplexed stream is an MPEG2 transport stream. The transmitting device according to claim 1 .

7. a video stream generating step of generating a video stream composed of video packets having encoded image data in their payloads; a subtitle stream generating step of generating a subtitle stream composed of subtitle packets having subtitle information in their payloads; a multiplexed stream generating step of generating a multiplexed stream including the video stream and the subtitle stream; a transmitting step of transmitting the multiplexed stream by a transmitting unit; The subtitle information is subtitle text information in TTML or a format derived from the TTML, which has display timing information, In the multiplexed stream, the subtitle packets are arranged at random access positions. Sending method.

8. a receiving unit for receiving a multiplexed stream including a video stream constituted by video packets having coded image data in their payloads and a subtitle stream constituted by subtitle packets having subtitle information in their payloads; The subtitle information is subtitle text information in TTML or a format derived from the TTML, which has display timing information, In the multiplexed stream, the subtitle packets are arranged at random access positions, The apparatus further includes a processing unit that extracts the subtitle packet from the multiplexed stream together with the video packet at the random access position and performs a subtitle display process. Receiving device.

9. a first timestamp having a value equal to or close to the value of a timestamp inserted at the timestamp insertion position of the header of the video packet at the random access position is inserted at the timestamp insertion position of the header of the subtitle packet; a second timestamp indicating a display time of the subtitle is inserted into a header or a payload of the subtitle packet; The processing unit In normal playback mode, the subtitle display process is performed by referring to the second timestamp.

9. The receiving device according to claim 8.

10. Identification information indicating that the second timestamp is inserted into the subtitle packet is inserted into the multiplexed stream, The processing unit extracts and uses the second timestamp from the subtitle packet based on the identification information.

10. The receiving device according to claim 9.

11. a receiving step of receiving, by a receiving unit, a multiplexed stream including a video stream constituted by video packets having coded image data in their payloads and a subtitle stream constituted by subtitle packets having subtitle information in their payloads; The subtitle information is subtitle text information in TTML or a format derived from the TTML, which has display timing information, In the multiplexed stream, the subtitle packets are arranged at random access positions, The method further includes a processing step of extracting the subtitle packets from the multiplexed stream together with the video packets at the random access position and performing a subtitle display process. Receiving method.

Images