Transmission method and transmission device
By encoding subtitle streams with display timing and abstract information, the method addresses the increased processing load in DVB by allowing efficient subtitle display control without scanning TTML files, thereby reducing processing overhead.
Patent Information
- Application Number
- JP2024146276
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2015-07-16
- Filing Date
- 2024-08-28
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2036-07-05
AI Technical Summary
Existing methods for transmitting subtitle information in digital video broadcasting (DVB) require the receiving side to scan an entire file of TTML markup language to obtain important parameters, leading to increased processing load.
A video encoder generates a video stream with encoded video data, and a subtitle encoder generates a subtitle stream that includes text information with display timing and abstract information, which is transmitted in a container format, allowing the receiving side to control subtitle display timing and state without scanning the text information.
This approach reduces the processing load for subtitle display by enabling efficient extraction and utilization of abstract information, facilitating quicker and more manageable subtitle processing on the receiving side.
Smart Images

Figure 2024161147000001_ABST
Abstract
Description
[Technical field]
[0001] The present technology relates to a transmitting device, a transmitting method, a receiving device and a receiving method, and in particular to a transmitting device or the like that transmits text information together with video information. [Background technology]
[0002] Conventionally, for example, in DVB (Digital Video Broadcasting) broadcasts, subtitle information has been transmitted as bitmap data. Recently, it has been proposed to transmit subtitle information in text character code, that is, text-based. In this case, font expansion according to the resolution is performed on the receiving side.
[0003] Also, when transmitting subtitle information on a text basis, it has been proposed to provide timing information to the text information. For example, the World Wide Web Consortium (W3C) has proposed the Timed Text Markup Language (TTML) as this text information (see Patent Document 1). [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2012-169885 A Summary of the Invention [Problem to be solved by the invention]
[0005] The text information of subtitles expressed in TTML is handled as a file in the form of a markup language. In this case, since there is no restriction on the order in which each parameter can be transmitted, the receiver must scan the entire file to obtain important parameters.
[0006] An object of the present technology is to reduce the processing load for displaying subtitles on the receiving side. [Means for solving the problem]
[0007] The concept of this technology is as follows: a video encoding unit for generating a video stream including encoded video data; a subtitle encoding unit that generates a subtitle stream including subtitle text information having display timing information and abstract information having information corresponding to a part of the plurality of pieces of information indicated by the text information; a transmitting unit for transmitting a container in a predetermined format including the video stream and the subtitle stream; Located in the transmitting device.
[0008] In the present technology, a video encoding unit generates a video stream including encoded video data. A subtitle encoder generates a subtitle stream including subtitle text information having display timing information and abstract information having information corresponding to a part of the multiple pieces of information indicated by the text information. For example, the subtitle text information may be in TTML or a format derived from this TTML. Then, a container in a predetermined format including the video stream and the subtitle stream is transmitted by a transmitting unit.
[0009] For example, the abstract information may include subtitle display timing information, which allows the receiving side to control the subtitle display timing based on the subtitle display timing information included in the abstract information without scanning the subtitle text information.
[0010] In this case, for example, the display timing information of the subtitle may have information on the display start timing and the display period. In this case, the subtitle stream may be configured with PES packets each consisting of a PES header and a PES payload, the subtitle text information and abstract information may be placed in the PES payload, and the display start timing may be indicated by a display offset from a PTS (Presentation Time Stamp) inserted in the PES header.
[0011] Also, for example, the abstract information may include display control information for controlling the display state of the subtitles, so that the receiving side can control the display state of the subtitles based on the display control information included in the abstract information without scanning the text information of the subtitles.
[0012] In this case, for example, the display control information may include at least any one of information on the display position, color gamut, and dynamic range of the subtitle, and may further include information on the target video.
[0013] Also, for example, the abstract information may include notification information notifying that an element of the subtitle text information has changed, so that the receiving side can easily recognize that an element of the subtitle text information has changed and can efficiently scan the element of the subtitle text information.
[0014] Also, for example, the subtitle encoding unit may be configured to segment the subtitle text information and abstract information and generate a subtitle stream having a predetermined number of segments, in which case the receiving side can easily obtain the abstract information by extracting the segment including the abstract information from the subtitle stream.
[0015] In this case, for example, the subtitle stream may have a segment of abstract information arranged first, followed by a segment of subtitle text information, such that the receiving side can easily and efficiently extract the segment of abstract information from the subtitle stream.
[0016] In this way, in the present technology, the subtitle stream contains not only the subtitle text information but also abstract information corresponding to the text information, so that the receiving side can perform processing for displaying the subtitles using the abstract information, thereby reducing the processing load.
[0017] Another concept of the present technology is A receiving unit receives a container in a predetermined format including a video stream and a subtitle stream, the video stream comprises encoded video data; the subtitle stream includes subtitle text information having display timing information and abstract information having information corresponding to a part of the plurality of pieces of information indicated by the text information; The video decoding apparatus further includes a control unit for controlling a video decoding process for decoding the video stream to obtain video data, a subtitle decoding process for decoding the subtitle stream to obtain subtitle bitmap data and extracting the abstract information, a video overlay process for overlaying the subtitle bitmap data on the video data to obtain video data for display, and a bitmap data process for processing the subtitle bitmap data overlaid on the video data based on the abstract information. in the receiving device.
[0018] In this technology, a container in a predetermined format including a video stream and a subtitle stream is received. The video stream includes encoded video data. The subtitle stream includes subtitle text information having display timing information, and abstract information having information corresponding to a part of the information indicated by the text information.
[0019] The control unit controls the video decoding process, the subtitle decoding process, the video overlay process, and the bitmap data process. In the video decoding process, the video stream is decoded to obtain video data. In the subtitle decoding process, the subtitle stream is decoded to obtain bitmap data of the subtitle and extract abstract information.
[0020] In the video overlay process, the subtitle bitmap data is overlaid on the video data to obtain video data for display, whereas in the bitmap data process, the subtitle bitmap data to be overlaid on the video data is processed based on the abstract information.
[0021] For example, the abstract information may include subtitle display timing information, and in the bitmap data processing, the timing of superimposing the subtitle bitmap data on the video data may be controlled based on the subtitle display timing information.
[0022] Also, for example, the abstract information may include display control information that controls the display state of the subtitle, and in the bitmap data processing, the state of the bitmap of the subtitle superimposed on the video data may be controlled based on the display control information.
[0023] In this way, in this technology, the bitmap data of the subtitles superimposed on the video data is processed based on the abstract information extracted from the subtitle stream, which makes it possible to reduce the processing load for displaying the subtitles.
[0024] Another concept of the present technology is a video encoding unit for generating a video stream including encoded video data; a subtitle encoding unit that generates one or more segments in which elements of subtitle text information having display timing information are arranged, and generates a subtitle stream including the one or more segments; a transmitting unit for transmitting a container in a predetermined format including the video stream and the subtitle stream; Located in the transmitting device.
[0025] In the present technology, a video encoding unit generates a video stream including encoded video data. A subtitle encoding unit generates one or more segments in which elements of subtitle text information having display timing information are arranged, and generates a subtitle stream including the one or more segments. For example, the subtitle text information may be in TTML or a format derived from the TTML. A transmitting unit transmits a container in a predetermined format including the video stream and the subtitle stream.
[0026] In this way, in the present technology, subtitle text information having display timing information is segmented and transmitted in the subtitle stream, which enables the receiving side to receive each element of the subtitle text information satisfactorily.
[0027] In addition, in the present technology, for example, when generating one segment in which all elements of subtitle text information are arranged, the subtitle encoding unit may insert information regarding the transmission order and / or whether or not the subtitle text information has been updated into the layer of the segment or the layer of the elements arranged therein. By inserting information regarding the transmission order of the subtitle text information, the receiving side can recognize the transmission order of the subtitle text information, and therefore it becomes possible to perform the decoding process efficiently. In addition, by inserting information regarding whether or not the subtitle text information has been updated, the receiving side can easily know whether or not the subtitle text information has been updated. Effect of the Invention
[0028] According to the present technology, it is possible to reduce the processing load for displaying subtitles on the receiving side. Note that the effects described in this specification are merely examples and are not limiting, and additional effects may also be provided. [Brief description of the drawings]
[0029] [Figure 1] 1 is a block diagram showing an example of the configuration of a transmission / reception system according to an embodiment; [Diagram 2] FIG. 2 is a block diagram showing a configuration example of a transmission device. [Diagram 3] FIG. 11 is a diagram illustrating an example of photoelectric conversion characteristics. [Figure 4] 13 is a diagram showing an example of the structure of a dynamic range SEI message and the contents of main information in the example structure. FIG. [Diagram 5] FIG. 1 is a diagram showing the TTML structure. [Figure 6] FIG. 1 is a diagram showing the TTML structure. [Figure 7] The following shows an example of the structure of metadata (TTM: TTML Metadata) present in the header (head) of the TTML structure. [Figure 8]FIG. 2 is a diagram showing an example of the structure of styling (TTS: TTML Styling) present in the header (head) of the TTML structure. [Figure 9] FIG. 1 is a diagram showing an example of the structure of a styling extension (TTSE: TTML Styling Extension) present in the header (head) of the TTML structure. [Figure 10] FIG. 13 is a diagram showing an example of the structure of a layout (TTL: TTML layout) present in the header (head) of the TTML structure. [Figure 11] FIG. 2 is a diagram showing an example of the body structure of the TTML structure. [Figure 12] 1 is a diagram illustrating an example of the configuration of a PES packet. [Figure 13] FIG. 13 shows the segment interfaces inside the PES. [Figure 14] A diagram showing an example of the structure of "TimedTextSubtitling_segments()" placed in the PES data payload. [Figure 15] A figure showing another example structure of "TimedTextSubtitling_segments()" placed in the PES data payload. [Figure 16] A figure showing an example of the structure of a THMS (text_header_metadata_segment) in which metadata (TTM) is placed. [Figure 17] A figure showing an example of the structure of THSS (text_header_styling_segment) in which styling (TTS) is placed. [Figure 18] A diagram showing an example of the structure of THSES (text_header_styling_extension_segment) where styling extensions (TTML) are placed. [Figure 19] FIG. 13 is a diagram showing an example of the structure of a THLS (text_header_layout_segment) in which a layout (TTL) is placed. [Figure 20]FIG. 2 is a diagram showing an example of the structure of a TBS (text_body_segment) in which a body of a TTML structure is placed. [Figure 21] A figure showing an example of the structure of THAS (text_header_all_segment) in which the header (head) of the TTML structure is placed. [Figure 22] FIG. 1 is a diagram showing an example of the structure of a TWS (text whole segment) in which the entire TTML structure is placed. [Figure 23] A figure showing another example structure of a TWS (text whole segment) in which the entire TTML structure is placed. [Figure 24] FIG. 1 / 2 shows an example of the structure of an APTS (abstract_parameter_TimedText_segment) in which abstract information is placed. [Diagram 25] FIG. 2 is a diagram (2 / 2) showing an example of the structure of an APTS (abstract_parameter_TimedText_segment) in which abstract information is placed. [Figure 26] FIG. 1 / 2 shows the contents of the main information in an example of the APTS structure. [Figure 27] FIG. 2 is a diagram (2 / 2) showing the contents of the main information in an example of an APTS structure. [Figure 28] FIG. 13 is a diagram for explaining the settings of "PTS", "start_time_offset", and "end_time_offset" when converting TTML into segments. [Figure 29] FIG. 2 is a block diagram showing a configuration example of a receiving device. [Diagram 30] FIG. 11 is a block diagram showing an example of the configuration of a subtitle decoder. [Diagram 31] 4 is a diagram illustrating an example of the configuration of a color gamut / brightness conversion unit. [Diagram 32] 11 is a diagram showing an example of the configuration of a component related to a luminance signal Y included in a luminance conversion unit. FIG. [Diagram 33] 5A and 5B are diagrams illustrating the operation of a luminance conversion unit. [Diagram 34] 11 is a diagram for explaining position conversion in a position / size conversion unit. FIG. [Diagram 35] 11 is a diagram for explaining size conversion in a position / size conversion unit. FIG. [Diagram 36] FIG. 11 is a diagram illustrating an example of display control of subtitles in chronological order. [Figure 37] FIG. 11 is a diagram illustrating an example of display control of subtitles in chronological order. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0030] Hereinafter, modes for carrying out the invention (hereinafter, referred to as "embodiments") will be described. The description will be made in the following order. 1. Embodiment 2. Variations
[0031] <1. Preferred embodiment> [Example of transmission and reception system configuration] 1 shows an example of the configuration of a transmission / reception system 10 according to an embodiment of the present invention. The transmission / reception system 10 includes a transmission device 100 and a reception device 200.
[0032] The transmitting device 100 generates an MPEG2 transport stream TS as a container and transmits this transport stream TS via broadcast waves or network packets. This transport stream TS includes a video stream having encoded video data.
[0033] This transport stream TS also includes a subtitle stream. This subtitle stream includes text information of subtitles (captions) having display timing information, and abstract information having information corresponding to a part of the multiple pieces of information indicated by this text information. In this embodiment, the text information is, for example, TTML (Timed Text Markup Language) proposed by W3C (World Wide Web Consortium).
[0034] In this embodiment, the abstract information includes subtitle display timing information. This display timing information has information on the display start timing and display period. Here, the subtitle stream is composed of PES packets consisting of a PES header and a PES payload, and the subtitle text information and display timing information are placed in the PES payload. For example, the display start timing is indicated by a display offset from the PTS inserted in the PES header.
[0035] In this embodiment, the abstract information includes display control information for controlling the display state of the subtitle. In this embodiment, the display control information includes information on the display position, color gamut, and dynamic range of the subtitle. In this embodiment, the abstract information includes information on the target video.
[0036] The receiving device 200 receives the transport stream TS transmitted by broadcast waves from the transmitting device 100. As described above, this transport stream TS has a video stream including encoded video data, and a subtitle stream including subtitle text information and abstract information.
[0037] The receiving device 200 obtains video data from the video stream, obtains subtitle bitmap data from the subtitle stream, and extracts abstract information. The receiving device 200 superimposes the subtitle bitmap data on the video data to obtain video data for display. The television receiver 200 processes the subtitle bitmap data superimposed on the video data based on the abstract information.
[0038] In this embodiment, the abstract information includes display timing information of the subtitle, and the receiving device 200 controls the timing of superimposing the bitmap data of the subtitle on the video data based on the display timing information. Also, in this embodiment, the abstract information includes display control information for controlling the display state of the subtitle (display position, color gamut, dynamic range, etc.), and the receiving device 200 controls the state of the bitmap of the subtitle based on the display control information.
[0039] "Example of transmitting device configuration" 2 shows an example of the configuration of the transmitting device 100. This transmitting device 100 has a control unit 101, a camera 102, a video photoelectric conversion unit 103, an RGB / YCbCr conversion unit 104, a video encoder 105, a subtitle generation unit 106, a text format conversion unit 107, a subtitle encoder 108, a system encoder 109, and a transmitting unit 110.
[0040] The control unit 101 is configured with a CPU (Central Processing Unit) and controls the operation of each part of the transmission device 100 based on a control program. The camera 102 captures an image of a subject and outputs HDR (High Dynamic Range) or SDR (Standard Dynamic Range) video data (image data). An HDR image has a contrast ratio of 0 to 100%*N (N is a number greater than 1), for example 0 to 1000%, which exceeds the white peak brightness of an SDR image. Here, a level of 100% corresponds to a white luminance value of 100 cd / m2, for example.
[0041] The video photoelectric conversion unit 103 performs photoelectric conversion on the video data obtained by the camera 102 to obtain transmission video data V1. In this case, if the video data is SDR video data, the video data is photoelectrically converted by applying SDR photoelectric conversion characteristics to obtain SDR transmission video data (transmission video data having SDR photoelectric conversion characteristics). On the other hand, if the video data is HDR video data, the video data is photoelectrically converted by applying HDR photoelectric conversion characteristics to obtain HDR transmission video data (transmission video data having HDR photoelectric conversion characteristics).
[0042] The RGB / YCbCr conversion unit 104 converts the transmission video data from the RGB domain to the YCbCr (luminance / chrominance) domain. The video encoder 105 performs encoding, such as MPEG4-AVC or HEVC, on the transmission video data V1 converted to the YCbCr domain, and generates a video stream (PES stream) VS including the encoded video data.
[0043] At this time, the video encoder 105 inserts meta information such as information indicating the electro-optical conversion characteristics (transfer function) corresponding to the electro-optical conversion characteristics of the transmitted video data V1, information indicating the color gamut of the transmitted video data V1, and information indicating the reference level into the VUI (video usability information) area of the SPS NAL unit of the access unit (AU).
[0044] In addition, the video encoder 105 inserts a newly defined dynamic range SEI message (Dynamic_range SEI message) into the “SEIs” portion of the access unit (AU). The dynamic range SEI message has meta information such as information (transfer function) indicating the electro-optical conversion characteristics corresponding to the electro-optical conversion characteristics of the transmission video data V1 and information on the reference level.
[0045] Here, the reason why the dynamic range SEI message has information indicating the electro-optical conversion characteristics is that even if the transmission video data V1 is HDR transmission video data, if the HDR electro-optical conversion characteristics are compatible with the SDR electro-optical conversion characteristics, information indicating the electro-optical conversion characteristics (gamma characteristics) corresponding to the SDR electro-optical conversion characteristics is inserted into the VUI of the SPS NAL unit, and therefore information indicating the electro-optical conversion characteristics corresponding to the HDR electro-optical conversion characteristics is required in a location other than the VUI.
[0046] FIG. 3 shows an example of photoelectric conversion characteristics. In this figure, the horizontal axis indicates the input luminance level, and the vertical axis indicates the transmission code value. Curve a shows an example of SDR photoelectric conversion characteristics. Curve b1 shows an example of HDR photoelectric conversion characteristics (not compatible with SDR photoelectric conversion characteristics). Curve b2 shows an example of HDR photoelectric conversion characteristics (compatible with SDR photoelectric conversion characteristics). In this example, the input luminance level matches the SDR photoelectric conversion characteristics up to the compatibility limit value. When the input luminance level is at the compatibility limit value, the transmission code value is at a compatible level.
[0047] In addition, the dynamic range SEI message contains reference level information because, when the transmission video data V1 is SDR transmission video data, information indicating the electro-optical conversion characteristics (gamma characteristics) corresponding to the SDR electro-optical conversion characteristics is inserted into the VUI of the SPS NAL unit, but the standard does not specify the insertion of a reference level.
[0048] Figure 4(a) shows an example of the structure (Syntax) of the dynamic range SEI message. Figure 4(b) shows the main information content (Semantics) in the example structure. The 1-bit flag information "Dynamic_range_cancel_flag" indicates whether to refresh the "Dynamic_range" message. "0" indicates that the message is refreshed, and "1" indicates that the message is not refreshed, i.e., the previous message is maintained as is.
[0049] When "Dynamic_range_cancel_flag" is "0", the following fields exist. The 8-bit field "coded_data_bit_depth" indicates the number of encoded pixel bits. The 8-bit field "reference_level" indicates the reference luminance level value as the reference level. The 1-bit field "modify_tf_flag" indicates whether to modify the Transfer Function (TF) indicated in the VUI (video usability information). "0" indicates that the TF indicated in the VUI is the target, and "1" indicates that the TF of the VUI is modified by the TF specified by "transfer_function" of this SEI. The 8-bit field "transfer_function" indicates the electrical-to-optical conversion characteristics corresponding to the optical-to-electrical conversion characteristics of the transmitted video data V1.
[0050] 2, a subtitle generation unit 106 generates text data (character code) DT as subtitle information. A text format conversion unit 107 inputs the text data DT and obtains subtitle text information in a predetermined format, which in this embodiment is TTML (Timed Text Markup Language).
[0051] Figure 5 shows an example of a TTML (Timed Text Markup Language) structure. TTML is written in XML format. Figure 6(a) also shows an example of a TTML structure. As in this example, it is also possible to specify a subtitle area at the root container position using "tts:extent". Figure 6(b) shows a subtitle area of 1920 pixels horizontally and 1080 pixels vertically, specified by "tts:extent="1920px 1080px"".
[0052] TTML consists of a header and a body. The header contains elements such as metadata, styling, styling extensions, and layout. Figure 7 shows an example of the structure of metadata (TTM: TTML Metadata). This metadata contains information on the metadata title and copyright information.
[0053] Fig. 8(a) shows an example of the structure of styling (TTS: TTML Styling). In addition to an identifier (id), this styling includes information such as the region position, size, color, font (fontFamily), font size (fontSize), and text alignment (textAlign).
[0054] "tts:origin" specifies the start position of the region, which is the display area for the subtitle, in pixels. In this example, it is "tts:origin“480px 600px”", which indicates that the start position (see arrow P) is (480,600) as shown in FIG. 8(b). "tts:extent" specifies the end position of the region in terms of the number of horizontal and vertical offset pixels from the start position. In this example, it is "tts:extent“560px 350px”", which indicates that the end position (see arrow Q) is (480+560,600+350) as shown in FIG. 8(b). Here, the number of offset pixels corresponds to the horizontal and vertical size of the region.
[0055] "tts:opacity="1.0"" indicates the mixing ratio of subtitles (subtitles) and background video. For example, "1.0" indicates that the subtitles are 100% and the background video is 0%, and "0.1" indicates that the subtitles (subtitles) are 0% and the background video is 100%. In the illustrated example, it is set to "1.0".
[0056] FIG. 9 shows an example of the structure of a styling extension (TTML Styling Extension). In addition to an identifier (id), this styling extension includes information on color space and dynamic range. The color space information specifies the color space expected by the subtitle. In the example shown, it is indicated that it is "ITUR2020". The dynamic range information specifies whether the dynamic range expected by the subtitle is SDR or HDR. In the example shown, it is indicated that it is SDR.
[0057] Figure 10 shows an example of the layout (region: TTML layout) structure. In addition to the identifier (id) of the region where the subtitle is placed, this layout includes information such as offset (padding), background color (backgroundColor), and alignment (displayAlign).
[0058] 11 shows an example of the structure of a body. In the example shown, information on three subtitles, subtitle 1, subtitle 2, and subtitle 3, is included. For each subtitle, the display start timing and display end timing are recorded, along with text data. For example, for subtitle 1, the display start timing is "T1", the display end timing is "T3", and the text data is "ABC".
[0059] Returning to FIG. 2, the subtitle encoder 108 converts the TTML obtained by the text format conversion unit 107 into various segments, and generates a subtitle stream SS made up of PES packets in which these segments are arranged in the payload.
[0060] Figure 12 shows an example of the structure of a PES packet. The PES header includes a Presentation Time Stamp (PTS). The PES data payload includes the following segments: abstract_parameter_TimedText_segment (APTS), text_header_metadata_segment (THMS), text header styling segment (THSS), text_header_styling_extension_segment (THSES), text_header_layout_segment (THLS), and text_body_segment (TBS).
[0061] The PES data payload may also include the APTS (abstract_parameter_TimedText_segment), THAS (text_header_all_segment), and TBS (text_body_segment) segments. The PES data payload may also include the APTS (abstract_parameter_TimedText_segment), and TWS (text_whole_segment) segments.
[0062] Figure 13 shows the segment interface inside PES. "PES_data_field" indicates the container part of the PES data payload of the PES packet. The 8-bit field of "data_identifier" indicates an ID that identifies the type of data transmitted in the container part mentioned above. Conventional subtitles (in the case of bitmap) are indicated by "0x20", so in the case of text, it is also possible to identify it by a new value, for example, "0x21".
[0063] The 8-bit field of "subtitle_stream_id" indicates an ID that identifies the type of subtitle stream. A subtitle stream that transmits text information can be identified by a new value, for example, "0x01", which can be distinguished from the conventional subtitle stream "0x00" that transmits a bitmap.
[0064] A group of segments is placed in the field of "TimedTextSubtitling_segments()". Figure 14 shows an example of the structure of "TimedTextSubtitling_segments()" when each segment of APTS (abstract_parameter_TimedText_segment), THMS (text_header_metadata_segment), THSS (text header styling segment), THSES (text_header_styling_extension_segment), THLS (text_header_layout_segment), and TBS (text_body_segment) is placed in the PES data payload.
[0065] Fig. 15(a) shows an example of the structure of "TimedTextSubtitling_segments()" when each segment of APTS (abstract_parameter_TimedText_segment), THAS (text_header_all_segment), and TBS (text_body_segment) is placed in the PES data payload. Fig. 15(b) shows an example of the structure of "TimedTextSubtitling_segments()" when each segment of APTS (abstract_parameter_TimedText_segment) and TWS (text_whole_segment) is placed in the PES data payload.
[0066] Incidentally, whether or not each segment is inserted into the subtitle stream is flexible; for example, if there are no changes other than the displayed subtitle, the stream will consist of only two segments, APTS (abstract_parameter_TimedText_segment) and TBS (text_body_segment). In either case, the PES data payload will have the APTS segment with abstract information placed first, followed by the other segments. This arrangement allows the receiving side to easily and efficiently extract the abstract information segment from the subtitle stream.
[0067] FIG. 16(a) shows an example of the structure (syntax) of THMS (text_header_metadata_segment). This structure includes the information of "sync_byte", "segment_type", "page_id", "segment_length", "thm_version_number", and "segment_payload()". "segment_type" is 8-bit data indicating the segment type, and is set to, for example, "0x20" indicating THMS here. "segment_length" is 8-bit data indicating the length (size) of the segment. Metadata as shown in FIG. 16(b) is placed as XML information in "segment_payload()". This metadata is the same as the metadata elements present in the TTML header (head) (see FIG. 7).
[0068] FIG. 17(a) shows an example of the structure (syntax) of THSS (text_header_styling_segment). This structure includes the information of "sync_byte", "segment_type", "page_id", "segment_length", "ths_version_number", and "segment_payload()". "segment_type" is 8-bit data indicating the segment type, and is set to, for example, "0x21" indicating THSS here. "segment_length" is 8-bit data indicating the length (size) of the segment. Metadata as shown in FIG. 17(b) is placed as XML information in "segment_payload()". This metadata is the same as the styling element present in the TTML header (see FIG. 8(a)).
[0069] FIG. 18(a) shows an example of the structure (syntax) of THSES (text_header_styling_extension_segment). This structure includes the information of "sync_byte", "segment_type", "page_id", "segment_length", "thse_version_number", and "segment_payload()". "segment_type" is 8-bit data indicating the segment type, and is set to, for example, "0x22" indicating THSES here. "segment_length" is 8-bit data indicating the length (size) of the segment. Metadata as shown in FIG. 18(b) is placed as XML information in "segment_payload()". This metadata is the same as the styling extension (styling_extension) element present in the TTML header (head) (see FIG. 9(a)).
[0070] FIG. 19(a) shows an example of the structure (syntax) of THLS (text_header_layout_segment). This structure includes the information of "sync_byte", "segment_type", "page_id", "segment_length", "thl_version_number", and "segment_payload()". "segment_type" is 8-bit data indicating the segment type, and is set to, for example, "0x23" indicating THLS here. "segment_length" is 8-bit data indicating the length (size) of the segment. Metadata as shown in FIG. 19(b) is placed as XML information in "segment_payload()". This metadata is the same as the layout element present in the TTML header (see FIG. 10).
[0071] FIG. 20(a) shows an example of the structure (syntax) of TBS (text_body_segment). This structure includes the following information: "sync_byte", "segment_type", "page_id", "segment_length", "tb_version_number", and "segment_payload()". "segment_type" is 8-bit data indicating the segment type, and is set to, for example, "0x24" in this example, indicating TBS. Metadata as shown in FIG. 20(b) is placed in "segment_payload()" as XML information. This metadata is the same as the body of TTML (see FIG. 11).
[0072] FIG. 21(a) shows an example of the structure (syntax) of THAS (text_header_all_segment). This structure includes the information of "sync_byte", "segment_type", "page_id", "segment_length", "tha_version_number", and "segment_payload()". "segment_type" is 8-bit data indicating the segment type, and is set to, for example, "0x25" indicating THAS here. "segment_length" is 8-bit data indicating the length (size) of the segment. Metadata as shown in FIG. 21(b) is placed in "segment_payload()" as XML information. This metadata is the entire header (head).
[0073] FIG. 22(a) shows an example of the structure (syntax) of TWS (text whole segment). This structure includes the information of "sync_byte", "segment_type", "page_id", "segment_length", "tw_version_number", and "segment_payload()". "segment_type" is 8-bit data indicating the segment type, and is set to, for example, "0x26" indicating TWS here. "segment_length" is 8-bit data indicating the length (size) of the segment. Metadata as shown in FIG. 22(b) is placed as XML information in "segment_payload()". This metadata is the entire TTML (see FIG. 5). This structure is a structure for maintaining compatibility across the entire TTML, and places the entire TTML in one segment.
[0074] When all TTML elements are placed in one segment and sent in this way, two new elements, "ttnew:sequentialinorder" and "ttnew:partialupdate", are inserted into the element layer as shown in Figure 22(b). Note that these do not have to be inserted at the same time.
[0075] "ttnew:sequentialinorder" configures information about the transmission order of TTML. This "ttnew:sequentialinorder" is placed before . "ttnew:sequentialinorder=true (=1)" indicates that there is a restriction on the transmission order. In that case, the inside of <metadata> 、 <styling>、<styling extension>、 <layout>are arranged in the order of ", followed by the contents of " 、 text " follows. Note that<styling extension> If does not exist, <metadata> 、 <styling> 、 <layout>On the other hand, "ttnew:sequentialinorder=false (=0)" indicates that there is no such constraint.
[0076] By inserting the "ttnew:sequentialinorder" element in this way, the receiving side can recognize the transmission order of the TTML, and even if all TTML elements are sent together, it can recognize that the TTML transmission order is in a specified order, simplifying the decoding process and enabling the decoding process to be performed efficiently.
[0077] Additionally, "ttnew:partialupdate" constitutes information regarding whether or not TTML has been updated. This "ttnew:partialupdate" is placed before . "ttnew:partialupdate=true (=1)" indicates that an update to one of the elements of will occur before . On the other hand, "ttnew:partialupdate=false (=0)" indicates that there will be no such update. By inserting the "ttnew:sequentialinorder" element in this way, the receiving side can easily determine whether or not TTML has been updated.
[0078] In the above, an example was shown in which two new elements, "ttnew:sequentialinorder" and "ttnew:partialupdate", are inserted into the element layer. However, an example can also be considered in which these new elements are inserted into the segment layer, as shown in Figure 23(a). Figure 23(b) shows the metadata (XML information) placed in "segment_payload()" in that case.
[0079] "APTS (abstract_parameter_TimedText_segment) segment" Here, we will explain the APTS (abstract_parameter_TimedText_segment) segment. This APTS segment contains abstract information. This abstract information has information that corresponds to some of the multiple pieces of information indicated by TTML.
[0080] Figures 24 and 25 show an example structure (syntax) of APTS (abstract_parameter_TimedText_segment). Figures 26 and 27 show the main information contents (semantics) in this example structure. Like other segments, this structure contains the information "sync_byte", "segment_type", "page_id", and "segment_length". "segment_type" is 8-bit data indicating the segment type, and in this case is set to, for example, "0x19" indicating APTS. "segment_length" is 8-bit data indicating the length (size) of the segment.
[0081] The 4-bit field "APT_version_number" indicates whether there is a change from the content previously sent in the APTS (abstract_parameter_TimedText_segment) element, and if there is a change, the value is increased by 1. The 4-bit field "TTM_version_number" indicates whether there is a change from the content previously sent in the THMS (text_header_metadata_segment) element, and if there is a change, the value is increased by 1. The 4-bit field "TTS_version_number" indicates whether there is a change from the content previously sent in the THSS (text_header_styling_segment) element, and if there is a change, the value is increased by 1.
[0082] The 4-bit field "TTSE_version_number" indicates whether there is a change from the content previously sent in the THSES (text_header_styling_extension_segment) element, and if there is a change, the value is increased by 1. The 4-bit field "TTL_version_number" indicates whether there is a change from the content previously sent in the THLS (text_header_layout_segment) element, and if there is a change, the value is increased by 1.
[0083] The 4-bit field "TTHA_version_number" indicates whether there is a change from the content previously sent in the THAS (text_header_all_segment) element, and if there is a change, the value is increased by 1. The 4-bit field "TW_version_number" indicates whether there is a change from the content previously sent in the TWS (text whole segment) element, and if there is a change, the value is increased by 1.
[0084] The 4-bit field of "subtitle_display_area" specifies the subtitle display area. For example, "0x1" specifies 640h*480v, "0x2" specifies 720h*480v, "0x3" specifies 720h*576v, "0x4" specifies 1280h*720v, "0x5" specifies 1920h*1080v, "0x6" specifies 3840h*2160v, and "0x7" specifies 7680h*4320v.
[0085] The 4-bit field "subtitle_color_gamut_info" specifies the color gamut expected for the subtitle. The 4-bit field "subtitle_dynamic_range_info" specifies the dynamic range expected for the subtitle. For example, "0x1" indicates SDR, and "0x2" indicates HDR. When HDR is specified for the subtitle, it indicates that the luminance of the subtitle is expected to be suppressed below the standard white level of the video.
[0086] The 4-bit field "target_video_resolution" specifies the expected video resolution, for example "0x1" specifies 640h*480v, "0x2" specifies 720h*480v, "0x3" specifies 720h*576v, "0x4" specifies 1280h*720v, "0x5" specifies 1920h*1080v, "0x6" specifies 3840h*2160v, and "0x7" specifies 7680h*4320v.
[0087] The 4-bit field "target_video_color_gamut_info" specifies the expected video color gamut. For example, "0x1" indicates "BT.709", and "0x2" indicates "BT.2020". The 4-bit field "target_video_dynamic_range_info" specifies the expected video dynamic range. For example, "0x1" indicates "BT.709", "0x2" indicates "BT.202x", and "0x3" indicates "Smpte 2084".
[0088] The 4-bit field "number_of_regions" specifies the number of regions. The following fields are repeated for each region. The 16-bit field "region_id" indicates the region ID.
[0089] The 8-bit field "start_time_offset" indicates the subtitle display start time as an offset value from the PTS. The offset value of "start_time_offset" is signed, and a negative value indicates that the display starts earlier than the PTS. When the offset value of "start_time_offset" is 0, it means that the display starts at the timing of the PTS. The precision of the value in 8-bit representation is up to one decimal place, calculated by dividing the sign value by 10.
[0090] The 8-bit field "end_time_offset" indicates the end time of the subtitle display as an offset value from "start_time_offset". This offset value indicates the display period. When the offset value of "start_time_offset" mentioned above is 0, it indicates that the display will end at the timing of the PTS plus the offset value of this "end_time_offset". The precision of the value in 8-bit representation is up to one decimal place, calculated by dividing the sign value by 10.
[0091] It is also possible to transmit "start_time_offset" and "end_time_offset" with the same 90 kHz precision as the PTS. In that case, a 32-bit space is reserved for each of the "start_time_offset" and "end_time_offset" fields.
[0092] As shown in Fig. 28, when converting TTML into segments, the subtitle encoder 108 sets the "PTS", "start_time_offset", and "end_time_offset" of each subtitle by referring to the system time information (PCR, video / audio synchronization time) based on the description of the display start timing (begin) and display end timing (end) of each subtitle included in the TTML body. At this time, the subtitle encoder 108 may set the "PTS", "start_time_offset", and "end_time_offset" using a decoder buffer model while verifying that the operation on the receiving side is performed correctly.
[0093] The 16-bit field "region_start_horizontal" indicates the horizontal pixel position of the upper left corner of the region in the subtitle display area specified by "subtitle_display_area" above (see point P in Figure 8(b)). The 16-bit field "region_start_vertical" indicates the vertical pixel position of the upper left corner of the region in the subtitle display area. The 16-bit field "region_end_horizontal" indicates the horizontal pixel position of the lower right corner of the region in the subtitle display area (see point Q in Figure 8(b)). The 16-bit field "region_end_vertical" indicates the vertical pixel position of the lower right corner of the region in the subtitle display area.
[0094] Returning to Fig. 2, the system encoder 109 generates a transport stream TS including the video stream VS generated by the video encoder 105 and the subtitle stream SS generated by the subtitle encoder 108. The transmitter 110 transmits this transport stream TS to the receiving device 200 via broadcast waves or network packets.
[0095] A brief description will be given of the operation of the transmitting device 100 shown in Fig. 2. Video data (image data) captured by the camera 102 is supplied to a video photoelectric conversion unit 103. In the video photoelectric conversion unit 103, photoelectric conversion is performed on the video data captured by the camera 102, and transmission video data V1 is obtained.
[0096] In this case, if the video data is SDR video data, the SDR photoelectric conversion characteristics are applied to perform photoelectric conversion, and SDR transmission video data (transmission video data having the SDR photoelectric conversion characteristics) is obtained. On the other hand, if the video data is HDR video data, the HDR photoelectric conversion characteristics are applied to perform photoelectric conversion, and HDR transmission video data (transmission video data having the HDR photoelectric conversion characteristics) is obtained.
[0097] The transmission video data V1 obtained by the video photoelectric conversion unit 103 is converted from the RGB domain to the YCbCr (luminance / chrominance) domain by the RGB / YCbCr conversion unit 104, and then supplied to the video encoder 105. The video encoder 105 encodes the transmission video data V1 using, for example, MPEG4-AVC or HEVC, and generates a video stream (PES stream) VS including the encoded video data.
[0098] In addition, in the video encoder 105, meta information such as information indicating the electrical-optical conversion characteristics corresponding to the electrical-optical conversion characteristics of the transmission video data V1 (transfer function), information indicating the color gamut of the transmission video data V1, and information indicating the reference level are inserted into the VUI area of the SPS NAL unit of the access unit (AU).
[0099] In addition, in the video encoder 105, a newly defined dynamic range SEI message (see FIG. 4) having meta information such as information (transfer function) indicating the electro-optical conversion characteristics corresponding to the electro-optical conversion characteristics of the transmission video data V1 and information on the reference level is inserted into the “SEIs” portion of the access unit (AU).
[0100] The subtitle generation unit 106 generates text data (character code) DT as subtitle information. This text data DT is supplied to a text format conversion unit 107. The text format conversion unit 107 converts the text data DT into subtitle text information having display timing information, that is, TTML (see FIGS. 3 and 4). This TTML is supplied to a subtitle encoder 108.
[0101] In the subtitle encoder 108, the TTML obtained in the text format conversion unit 107 is converted into various segments, and a subtitle stream SS consisting of PES packets with these segments arranged in the payload is generated. In this case, the payload of the PES packet is arranged first with an APTS segment having abstract information (see Figures 24 to 27), followed by a segment having subtitle text information (see Figure 12).
[0102] The video stream VS generated by the video encoder 105 is supplied to the system encoder 109. The subtitle stream SS generated by the subtitle encoder 108 is supplied to the system encoder 109. The system encoder 109 generates a transport stream TS including the video stream VS and the subtitle stream SS. This transport stream TS is transmitted to the receiving device 200 by the transmitting unit 110 via broadcast waves or net packets.
[0103] "Example of receiving device configuration" 29 shows an example of the configuration of a receiving device 200. This receiving device 200 has a control unit 201, a user operation unit 202, a receiving unit 203, a system decoder 204, a video decoder 205, a subtitle decoder 206, a color gamut / brightness conversion unit 207, and a position / size conversion unit 208. The receiving device 200 also has a video superimposition unit 209, a YCbCr / RGB conversion unit 210, an electro-optical conversion unit 211, a display mapping unit 212, and a CE monitor 213.
[0104] The control unit 201 is configured with a CPU (Central Processing Unit) and controls the operation of each unit of the receiving device 200 based on a control program. The user operation unit 202 is a switch, a touch panel, a remote control transmission unit, etc. that allows a user such as a viewer to perform various operations. The receiving unit 203 receives a transport stream TS transmitted from the transmitting device 100 via broadcast waves or net packets.
[0105] The system decoder 204 extracts the video stream VS and the subtitle stream SS from the transport stream TS. The system decoder 204 also extracts various pieces of information inserted in the transport stream TS (container) and sends them to the control unit 201.
[0106] The video decoder 205 performs a decoding process on the video stream VS extracted by the system decoder 204, and outputs the transmission video data V1. The video decoder 205 also extracts parameter sets and SEI messages inserted in each access unit constituting the video stream VS, and sends them to the control unit 201.
[0107] The VUI area of the SPS NAL unit contains information (transfer function) indicating electrical-optical conversion characteristics corresponding to the electrical-electrical conversion characteristics of the transmission video data V1, information indicating the color gamut of the transmission video data V1, information indicating a reference level, etc. This SEI message also contains a dynamic range SEI message (see FIG. 4) having information (transfer function) indicating electrical-optical conversion characteristics corresponding to the electrical-electrical conversion characteristics of the transmission video data V1, information on the reference level, etc.
[0108] The subtitle decoder 206 processes segment data of each region included in the subtitle stream SS, and outputs bitmap data of each region to be superimposed on the video data. The subtitle decoder 206 also extracts abstract information included in the APTS segments, and sends it to the control unit 201.
[0109] This abstract information includes subtitle display timing information, subtitle display control information (subtitle display position, color gamut, and dynamic range information), and target video information (resolution, color gamut, and dynamic range information).
[0110] Here, the subtitle display timing information and display control information are included in the XML information placed in "segment_payload()" of segments other than APTS, so it is possible to obtain them by scanning that XML information, but they can also be easily obtained by simply extracting abstract information from APTS segments. Note that the target video information (resolution, color gamut, dynamic range information) can be obtained from the video stream VS system, but they can also be easily obtained by simply extracting abstract information from APTS segments.
[0111] 30 shows an example of the configuration of the subtitle decoder 206. This subtitle decoder 206 has a coded buffer 261, a subtitle segment decoder 262, a font development unit 263, and a bitmap buffer 264.
[0112] The coded buffer 261 temporarily holds the subtitle stream SS. The subtitle segment decoder 262 decodes the segment data of each region held in the coded buffer 261 at a predetermined timing to obtain text data and control codes of each region.
[0113] The font expansion unit 263 expands the font based on the text data and control codes of each region obtained by the subtitle segment decoder 262, and obtains bitmap data of the subtitle of each region. In this case, the font expansion unit 263 uses, for example, position information included in the abstract information as the position information of each region ("region_start_horizontal", "region_start_vertical", "region_end_horizontal", "region_end_vertical").
[0114] The subtitle bitmap data is obtained in the RGB domain, the color gamut of the subtitle bitmap data matches the color gamut indicated by the subtitle color gamut information contained in the abstract information, and the dynamic range of the subtitle bitmap data matches the dynamic range indicated by the subtitle dynamic range information contained in the abstract information.
[0115] For example, when the dynamic range information is "SDR", the subtitle bitmap data has a dynamic range of SDR and has been photoelectrically converted by applying SDR photoelectric conversion characteristics. Also, when the dynamic range information is "HDR", the subtitle bitmap data has a dynamic range of HDR and has been photoelectrically converted by applying HDR photoelectric conversion characteristics. In this case, the luminance range is limited to the HDR reference level, assuming that HDR will be superimposed on the video.
[0116] The bitmap buffer 264 temporarily holds the bitmap data of each region obtained by the font development unit 263. The bitmap data of each region held in the bitmap buffer 264 is read out from the display start timing and superimposed on the image data, and this continues for the display period.
[0117] Here, the subtitle segment decoder 262 extracts the PTS from the PES header of the PES packet. The subtitle segment decoder 262 also extracts abstract information from the APTS segment. This information is sent to the control unit 201. The control unit 201 controls the timing of reading out the bitmap data of each region from the bitmap buffer 264 based on the PTS and the information "start_time_offset" and "end_time_offset" included in the abstract information.
[0118] Returning to Fig. 29, the color gamut / brightness conversion unit 207, under the control of the control unit 201, matches the color gamut of the subtitle bitmap data to the color gamut of the video data, based on the color gamut information of the subtitle bitmap data ("subtitle_color_gamut_info") and the color gamut information of the video data ("target_video_color_gamut_info"). Also, under the control of the control unit 201, the color gamut / brightness conversion unit 207 adjusts the maximum luminance level of the subtitle bitmap data to be equal to or lower than the reference level of luminance of the video data, based on the dynamic range information of the subtitle bitmap data ("subtitle_dynamic_range_info") and the dynamic range information of the video data ("target_video_dynamic_range_info").
[0119] 31 shows an example of the configuration of the color gamut / luminance converter 207. This color gamut / luminance converter 210 has an electric-to-optical converter 221, a color gamut converter 222, an optical-to-electrical converter 223, an RGB / YCbCr converter 224, and a luminance converter 225.
[0120] The electric-to-optical converter 221 performs photoelectric conversion on the input subtitle bitmap data. Here, when the dynamic range of the subtitle bitmap data is SDR, the electric-to-optical converter 221 applies SDR electric-to-optical conversion characteristics to perform electric-to-optical conversion to make it linear. Also, when the dynamic range of the subtitle bitmap data is HDR, the electric-to-optical converter 221 applies HDR electric-to-optical conversion characteristics to perform electric-to-optical conversion to make it linear. Note that it is also possible that the input subtitle bitmap data is in a linear state without having been subjected to photoelectric conversion. In that case, the electric-to-optical converter 221 is unnecessary.
[0121] The color gamut conversion unit 222 matches the color gamut of the subtitle bitmap data output from the electro-optical conversion unit 221 to the color gamut of the video data. For example, when the color gamut of the subtitle bitmap data is "BT.709" and the color gamut of the video data is "BT.2020", the color gamut of the subtitle bitmap data is converted from "BT.709" to "BT.2020". Note that when the color gamut of the subtitle bitmap data is the same as the color gamut of the video data, the color gamut conversion unit 222 does nothing substantially and outputs the input subtitle bitmap data as is.
[0122] The photoelectric conversion unit 223 performs photoelectric conversion by applying the same photoelectric conversion characteristics as those applied to the video data to the subtitle bitmap data output from the color gamut conversion unit 222. The RGB / YCbCr conversion unit 224 converts the subtitle bitmap data output from the photoelectric conversion unit 223 from the RGB domain to the YCbCr (luminance / chrominance) domain.
[0123] The luminance conversion unit 225 obtains output bitmap data by adjusting the subtitle bitmap data output from the RGB / YCbCr conversion unit 224 so that the maximum luminance level of the subtitle bitmap data is equal to or lower than the reference level of luminance of the video data or the reference white level. In this case, if the luminance adjustment of the subtitle bitmap data has already been performed in consideration of rendering to HDR video, when the video data is HDR, the input subtitle bitmap data is output as is without actually doing anything.
[0124] 32 shows an example of the configuration of a configuration section 225Y related to the luminance signal Y included in the luminance conversion section 225. This configuration section 225Y has a coding pixel bit number adjustment section 231 and a level adjustment section 232.
[0125] The coding pixel bit number adjustment unit 231 matches the coding pixel bit number of the luminance signal Ys of the subtitle bitmap data to the coding pixel bit number of the video data. For example, when the coding pixel bit number of the luminance signal Ys is "8 bits" and the coding pixel bit number of the video data is "10 bits", the coding pixel bit number of the luminance signal Ys is converted from "8 bits" to "10 bits". The level adjustment unit 232 adjusts the maximum level of the luminance signal Ys with the matching coding pixel bit number so that it is equal to or lower than the luminance reference level of the video data or the reference white level, and outputs the result as the output luminance signal Ys'.
[0126] Fig. 33 is a schematic diagram of the operation of the component 225Y shown in Fig. 32. The illustrated example shows a case where the video data is HDR. The reference level corresponds to the boundary between the non-shiny part and the shiny part.
[0127] The reference level exists between the maximum level (sc_high) and the minimum level (sc_low) of the luminance signal Ys after the number of encoded pixel bits has been adjusted. In this case, the maximum level (sc_high) is adjusted to be equal to or lower than the reference level. In this case, the clip method would result in a whiteout-like state, so a linear scale-down method, for example, is used.
[0128] By adjusting the level of the luminance signal Ys in this manner, when the subtitle bitmap data is superimposed on the video data, the subtitle is prevented from being displayed shining brightly against the background video, making it possible to maintain high picture quality.
[0129] In the above, the component 225Y (see FIG. 32) related to the luminance signal Ys included in the luminance conversion unit 225 has been described. In the luminance conversion unit 225, only the process of adjusting the number of encoded pixel bits to the number of encoded pixel bits of the video data is performed for the color difference signals Cb, Cr. For example, conversion from an 8-bit space to a 10-bit space is performed such that the entire range represented by the bit width is set to 100%, the median value is used as the reference, and the conversion is performed so that the fluctuation range is 50% in the positive direction and 50% in the negative direction from the reference value.
[0130] Returning to Fig. 29, the position / size conversion unit 208 performs position conversion processing on the subtitle bitmap data obtained by the color gamut / brightness conversion unit 207 under the control of the control unit 201. When the supported resolution of the subtitle (indicated by the information in "subtitle_display_area") differs from the resolution of the video ("target_video_resolution"), the position / size conversion unit 208 converts the position of the subtitle so that the subtitle is displayed at an appropriate position in the background video.
[0131] For example, a subtitle supports HD resolution and a video supports UHD resolution, where UHD resolution is higher than HD resolution and includes 4K resolution or 8K resolution.
[0132] FIG. 34(a) shows an example where the video is in UHD resolution and the subtitles are compatible with HD resolution. The display area for the subtitles is represented by the "subtitle area" in the figure. The positional relationship between the "subtitle area" and the video is represented by sharing the reference position of both, that is, the top left (left-top). The pixel position of the start point of the region is (a,b), and the pixel position of its end point is (c,d). In this case, because the resolution of the background video is higher than the compatible resolution of the subtitles, the display position of the subtitles on the background video is biased to the upper right, rather than the position intended by the production side.
[0133] Figure 34(b) shows an example of the case where position conversion processing has been performed. The pixel position of the start point of the region, which is the display area of the subtitle, is (a',b'), and the pixel position of its end point is (c',d'). In this case, the position coordinates of the region before position conversion are coordinates of the HD display area, so they are converted to the coordinates of the UHD display area according to their relationship with the video picture frame, and therefore based on the ratio of the UHD resolution to the HD resolution. Note that in this example, size conversion processing of the subtitle is also performed at the same time as the position conversion.
[0134] In addition, the position / size conversion unit 208 performs size conversion processing on the subtitle bitmap data obtained by the color gamut / brightness conversion unit 207 under the control of the control unit 201, for example, in response to an operation by a user such as a viewer, or automatically based on the relationship between the video resolution and the corresponding resolution of the subtitle.
[0135] As shown in Fig. 35(a), the distance from the display center (dc) of the display area to the center of the region, i.e., the point dividing the region in half horizontally and vertically (region center: rc), is determined in proportion to the video resolution. For example, if the video resolution is assumed to be HD and the region center rc is defined from the subtitle display area center dc, when the video resolution is 4K (=3840x2160), the position is controlled so that the distance from dc to rc is doubled in number of pixels.
[0136] As shown in Figure 35(b), when the size of a region (Region) is changed from r_org (Region 00) to r_mod (Region 01), the start position (rsx1, rsy1) and end position (rex1, rey1) are modified to the start position (rsx2, rsy2) and end position (rex2, rey2), respectively, so as to satisfy Ratio = (r_mod / r_org).
[0137] In other words, the ratio of the distance from rc to (rsx2,rsy2) to the distance from rc to (rsx1,rsy1), and the ratio of the distance from rc to (rex2,rey2) to the distance from rc to (rex1,rey1) are made to match Ratio. By doing this, the center position rc of the region remains the same even when the size is converted, and it is possible to convert the size of the subtitle (region) while maintaining a constant relative positional relationship of the entire display area.
[0138] 29 , the video superimposing unit 209 superimposes the subtitle bitmap data output from the position / size conversion unit 208 onto the transmission video data V1 output from the video decoder 205. In this case, the video superimposing unit 209 mixes the subtitle bitmap data at a mixing ratio indicated by the mixing ratio information (Mixing data) obtained by the subtitle decoder 206.
[0139] The YCbCr / RGB conversion unit 210 converts the transmission video data V1' on which the subtitle bitmap data is superimposed from the YCbCr (luminance / chrominance) domain to the RGB domain. In this case, the YCbCr / RGB conversion unit 210 performs the conversion using a conversion formula corresponding to the color gamut based on the color gamut information.
[0140] The electro-optical conversion unit 211 performs electro-optical conversion on the transmission video data V1' converted into the RGB domain by applying electro-optical conversion characteristics corresponding to the photoelectric conversion characteristics applied thereto, thereby obtaining display video data for displaying an image. The display mapping unit 212 performs display luminance adjustment on the display video data in accordance with the maximum luminance display capability of the CE monitor 213, etc. The CE monitor 213 displays an image based on the display video data that has been subjected to the display luminance adjustment. The CE monitor 213 is composed of, for example, an LCD (Liquid Crystal Display), an organic EL display (organic electroluminescence display), etc.
[0141] The operation of the receiving device 200 shown in Fig. 29 will be briefly described. The receiving unit 203 receives the transport stream TS transmitted from the transmitting device 100 via broadcast waves or network packets. This transport stream TS is supplied to the system decoder 204. The system decoder 204 extracts a video stream VS and a subtitle stream SS from this transport stream TS. The system decoder 204 also extracts various pieces of information inserted in the transport stream TS (container) and sends the information to the control unit 201.
[0142] The video stream VS extracted by the system decoder 204 is supplied to the video decoder 205. The video decoder 205 performs a decoding process on the video stream VS to obtain transmission video data V1. The video decoder 205 also extracts parameter sets and SEI messages inserted in each access unit constituting the video stream VS and sends them to the control unit 201.
[0143] The VUI area of the SPS NAL unit contains information (transfer function) indicating electrical-optical conversion characteristics corresponding to the electrical-electrical conversion characteristics of the transmission video data V1, information indicating the color gamut of the transmission video data V1, information indicating a reference level, etc. This SEI message also contains a dynamic range SEI message (see FIG. 4) having information (transfer function) indicating electrical-optical conversion characteristics corresponding to the electrical-electrical conversion characteristics of the transmission video data V1, information on the reference level, etc.
[0144] The subtitle stream SS extracted by the system decoder 204 is supplied to the subtitle decoder 206. The subtitle decoder 206 decodes the segment data of each region included in the subtitle stream SS to obtain bitmap data of the subtitle of each region to be superimposed on the video data.
[0145] Furthermore, the subtitle decoder 206 extracts abstract information contained in the APTS segments (see Figs. 24 and 25) and sends it to the control unit 201. This abstract information includes subtitle display timing information, subtitle display control information (information on the subtitle display position, color gamut, and dynamic range), and further target video information (information on resolution, color gamut, and dynamic range).
[0146] Under the control of the control unit 201, the subtitle decoder 206 controls the output timing of the subtitle bitmap data for each region based on, for example, subtitle display timing information ("start_time_offset", "end_time_offset") included in the abstract information.
[0147] The subtitle bitmap data for each region obtained by the subtitle decoder 206 is supplied to a color gamut / luminance conversion unit 207. Under the control of the control unit 201, the color gamut / luminance conversion unit 207 matches the color gamut of the subtitle bitmap data to the color gamut of the video data, for example, based on the color gamut information ("subtitle_color_gamut_info", "target_video_color_gamut_info") included in the abstract information.
[0148] Furthermore, under the control of the control unit 201, the color gamut / brightness conversion unit 207 adjusts the maximum brightness level of the subtitle bitmap data so that it is below the reference brightness level of the video data, for example, based on the dynamic range information contained in the abstract information ("subtitle_dynamic_range_info", "target_video_dynamic_range_info").
[0149] The subtitle bitmap data for each region obtained by the color gamut / brightness conversion unit 207 is supplied to a position / size conversion unit 208. Under the control of the control unit 201, the position / size conversion unit 208 performs position conversion processing on the subtitle bitmap data for each region based on, for example, resolution information ("subtitle_display_area", "target_video_resolution") included in the abstract information.
[0150] Furthermore, in the position / size conversion unit 208, under the control of the control unit 201, the subtitle size conversion process is performed on the subtitle bitmap data obtained in the color gamut / brightness conversion unit 207, for example, in response to an operation by a user such as a viewer, or automatically based on the relationship between the video resolution and the corresponding resolution of the subtitle.
[0151] The transmission video data V1 obtained by the video decoder 204 is supplied to a video superimposing unit 209. Also, the subtitle bitmap data of each region obtained by the position / size conversion unit 208 is supplied to the video superimposing unit 209. The video superimposing unit 209 superimposes the subtitle bitmap data of each region on the transmission video data V1. In this case, the subtitle bitmap data is mixed at a mixing ratio indicated by mixing ratio information (Mixing data).
[0152] The transmission video data V1' on which the subtitle bitmap data of each region is superimposed, obtained by the video superimposition unit 209, is converted from the YCbCr (luminance / chrominance) domain to the RGB domain by a YCbCr / RGB conversion unit 210, and is supplied to an electric-to-optical conversion unit 211. In the electric-to-optical conversion unit 211, electric-to-optical conversion is performed by applying an electric-to-optical conversion characteristic corresponding to the applied optoelectric conversion characteristic to the transmission video data V1', and display video data for displaying an image is obtained.
[0153] The display video data is supplied to a display mapping unit 212. In this display mapping unit 212, the display luminance adjustment is performed on the display video data according to the maximum luminance display capability of the CE monitor 213. The display video data that has undergone such display luminance adjustment is supplied to the CE monitor 213. An image is displayed on the CE monitor 213 based on this display video data.
[0154] As described above, in the transmission / reception system 10 shown in Fig. 1, the subtitle stream contains not only the subtitle text information but also abstract information corresponding to the text information. Therefore, the receiving side can perform processing for displaying the subtitles using the abstract information, which can reduce the processing load.
[0155] In this case, the processing load on the receiving side is reduced, and it becomes easy to handle chronological display control in which the subtitle display changes relatively quickly. For example, consider the case in which the subtitle display changes as shown in Figure 36(a)-(f).
[0156] In this case, first, a subtitle stream SS is transmitted including PES packets in which APTS (abstract_parameter_TimedText_segment) and TBS (text body segment) segments are arranged in the PES data payload, as shown in FIG. 37(a). On the receiving side, subtitle bitmap data is generated for displaying the wording "ABC" at the position of the region "region r1" based on the TBS segment data and the region position information (Region_position) included in the APTS segment.
[0157] Then, on the receiving side, this bitmap data is output from display start timing T1 to display end timing T3 based on PTS1 and the display timing information (STS1, ETS1) included in the APTS segments. As a result, on the receiving side, the wording "ABC" is continuously displayed on the screen from T1 to T3, as shown in FIG.
[0158] Next, for example, as shown in FIG. 37(b), a subtitle stream SS is transmitted including PES packets in which APTS (abstract_parameter_TimedText_segment) and TBS (text body segment) segments are arranged in the PES data payload. On the receiving side, subtitle bitmap data is generated for displaying the wording "DEF" at the position of the region "region r2" based on the TBS segment data and the region position information (Region_position) included in the APTS segment.
[0159] Then, on the receiving side, this bitmap data is output from display start timing T2 to display end timing T5 based on PTS2 and the display timing information (STS2, ETS2) included in the APTS segments. As a result, on the receiving side, the wording "DEF" is continuously displayed on the screen from T2 to T5, as shown in FIG.
[0160] Next, for example, as shown in FIG. 37(c), a subtitle stream SS is transmitted including PES packets in which APTS (abstract_parameter_TimedText_segment) and TBS (text body segment) segments are arranged in the PES data payload. On the receiving side, subtitle bitmap data is generated for displaying the wording "GHI" at the position of the region "region r3" based on the TBS segment data and the region position information (Region_position) included in the APTS segment.
[0161] Then, on the receiving side, this bitmap data is output from display start timing T4 to display end timing T6 based on PTS3 and the display timing information (STS3, ETS3) included in the APTS segments. As a result, on the receiving side, the message "GHI" is continuously displayed on the screen from T4 to T6, as shown in FIG.
[0162] <2. Modifications> In the above embodiment, an example has been shown in which TTML is used as text information for subtitles in a predetermined format having display timing information. However, the present technology is not limited to this, and other timed text information having information equivalent to TTML may be used. For example, a derived format of TTML may be used.
[0163] In the above embodiment, the container is a transport stream (MPEG-2 TS). However, the present technology is not limited to MPEG-2 TS containers, and can be realized in the same way with containers of other formats, such as MMT or ISOBMFF.
[0164] In the above embodiment, an example has been shown in which TTML and abstract information are placed in a segment and then placed in the PES data payload of a PES packet. However, the present technology also contemplates placing TTML and abstract information directly in the PES data payload.
[0165] In the above embodiment, the transmission / reception system 10 including the transmission device 100 and the reception device 200 is shown, but the configuration of the transmission / reception system to which the present technology can be applied is not limited to this. For example, the reception device 200 may be a set-top box and a monitor connected by a digital interface such as HDMI (High-Definition Multimedia Interface). Note that "HDMI" is a registered trademark.
[0166] The present technology can also be configured as follows. (1) a video encoding unit for generating a video stream including encoded video data; a subtitle encoding unit that generates a subtitle stream including subtitle text information having display timing information and abstract information having information corresponding to a part of the plurality of pieces of information indicated by the text information; a transmitting unit for transmitting a container including the video stream and the subtitle stream; Transmitting device. (2) The above abstract information includes subtitle display timing information. The transmitting device according to (1) above. (3) The display timing information of the subtitles includes information on the display start timing and the display period. The transmitting device according to (2) above. (4) The subtitle stream is composed of PES packets each having a PES header and a PES payload, The subtitle text information and the abstract information are placed in the PES payload, The display start timing is indicated by a display offset from the PTS inserted in the PES header. The transmitting device according to (3) above. (5) The abstract information includes display control information for controlling the display state of the subtitles. A transmitting device according to any one of (1) to (4). (6) The display control information includes at least one of information on a display position of a subtitle, a color gamut, and a dynamic range. The transmitting device according to (5) above. (7) The display control information further includes information about the target video. The transmitting device according to (6) above. (8) The abstract information includes notification information that notifies the user that there is a change in an element of the text information of the subtitle. A transmitting device according to any one of (1) to (7). (9) The subtitle encoding unit is Segmenting the subtitle text information and the abstract information to generate the subtitle stream having a predetermined number of segments. A transmitting device according to any one of (1) to (8). (10) The above subtitle stream includes The abstract information segment is placed first, followed by the subtitle text information segment. The transmitting device according to (9) above. (11) The text information of the subtitles is in TTML or a format derived from TTML. A transmitting device according to any one of (1) to (10). (12) a video encoding step of generating a video stream including encoded video data; a subtitle encoding step of generating a subtitle stream including subtitle text information having display timing information and abstract information having information corresponding to a part of the plurality of pieces of information indicated by the text information; A transmitting step of transmitting a container including the video stream and the subtitle stream by a transmitting unit. Transmission method. (13) A receiving unit for receiving a container in a predetermined format including a video stream and a subtitle stream, the video stream comprises encoded video data; the subtitle stream includes subtitle text information having display timing information and abstract information having information corresponding to a part of the plurality of pieces of information indicated by the text information; a video decoding unit for decoding the video stream to obtain video data; a subtitle decoding unit for decoding the subtitle stream to obtain subtitle bitmap data and extracting the abstract information; a video overlay unit overlaying the subtitle bitmap data on the video data to obtain video data for display; a control unit for controlling bitmap data of a subtitle superimposed on the video data based on the abstract information. Receiving device. (14) The above abstract information includes subtitle display timing information, The control unit is The timing of superimposing the bitmap data of the subtitle on the video data is controlled based on display timing information of the subtitle. The receiving device according to (13) above. (15) The abstract information includes display control information for controlling a display state of the subtitle, The control unit is A state of a bitmap of the subtitle superimposed on the video data is controlled based on the display control information. The receiving device according to (13) or (14). (16) A receiving step of receiving a container in a predetermined format including a video stream and a subtitle stream by a receiving unit, the video stream comprises encoded video data; the subtitle stream includes subtitle text information having display timing information and abstract information having information corresponding to a part of the plurality of pieces of information indicated by the text information; a video decoding step of decoding the video stream to obtain video data; a subtitle decoding step of decoding the subtitle stream to obtain subtitle bitmap data and extracting the abstract information; a video overlaying step of overlaying the subtitle bitmap data on the video data to obtain video data for display; The method further includes a control step of controlling bitmap data of a subtitle superimposed on the video data based on the abstract information. Receiving method. (17) a video encoding unit for generating a video stream including encoded video data; a subtitle encoding unit that generates one or more segments in which elements of subtitle text information having display timing information are arranged, and generates a subtitle stream including the one or more segments; a transmitting unit for transmitting a container in a predetermined format including the video stream and the subtitle stream; Transmitting device. (18) The subtitle encoding unit is To generate one segment containing all the elements of the subtitle text information above, Insert information regarding the transmission order and / or update of the text information of the subtitle into the layer of the segment or the layer of the element. The transmitting device according to (17) above. (19) The text information of the subtitle is in TTML or a format derived from said TTML. The transmitting device according to (17) or (18). (20) a video encoding step of generating a video stream including encoded video data; a subtitle encoding step of generating one or more segments in which elements of subtitle text information having display timing information are arranged, and generating a subtitle stream including the one or more segments; A transmitting step of transmitting a container in a predetermined format including the video stream and the subtitle stream by a transmitting unit. Transmission method.
[0167] The main feature of this technology is that it reduces the processing load for displaying subtitles on the receiving side by including abstract information corresponding to the subtitle text information in the subtitle stream (see Fig. 12). [Explanation of symbols]
[0168] 10. Transmitting and receiving system 100 Transmitting device 101...Control section 102 Camera 103 Video photoelectric conversion section 104 RGB / YCbCr conversion section 105...Video Encoder 106 Subtitle generation section 107 Text format conversion section 108...Subtitle Encoder 109 System Encoder 110 Transmitter 200 Receiving device 201... Control unit 202 User operation section 203 Receiving unit 204 System Decoder 205...Video decoder 206...Subtitle Decoder 207 Color gamut and brightness conversion section 208 Position and size conversion section 209 Video overlay unit 210 YCbCr / RGB conversion section 211...Electro-optical conversion unit 212 Display mapping section 213...CE Monitor 221...Electro-optical conversion unit 222...Color gamut conversion unit 223... Photoelectric conversion unit 224···RGB / YCbCr conversion section 225...Luminance converter 225Y...Component 231: Encoding pixel bit number adjustment unit 232 Level adjustment section 261 Coded Buffer 262 Subtitle Segment Decoder 263 Font expansion section 264 Bitmap Buffer< / layout> < / styling> < / metadata> < / layout> < / styling> < / metadata>
Claims
1. a video encoding unit for generating a video stream including encoded video data; a subtitle encoding unit that generates a subtitle stream including subtitle text information having display timing information and abstract information having information corresponding to a part of the plurality of pieces of information indicated by the text information; a transmitting unit for transmitting a container in a predetermined format including the video stream and the subtitle stream; Transmitting device.
2. The abstract information includes subtitle display timing information. The transmitting device according to claim 1 .
3. The display timing information of the subtitles includes information on the display start timing and the display period. The transmitting device according to claim 2.
4. The subtitle stream is composed of PES packets each having a PES header and a PES payload. The subtitle text information and the abstract information are placed in a PES payload, The display start timing is indicated by a display offset from the PTS inserted in the PES header. The transmitting device according to claim 3.
5. The abstract information includes display control information for controlling the display state of the subtitle. The transmitting device according to claim 1 .
6. The display control information includes at least one of information on a display position of a subtitle, a color gamut, and a dynamic range. The transmitting device according to claim 5.
7. The display control information further includes information about the target video. The transmitting device according to claim 6.
8. The abstract information includes notification information that notifies that an element of the text information of the subtitle has changed. The transmitting device according to claim 1 .
9. The above subtitle encoding section is Segmenting the subtitle text information and the abstract information to generate the subtitle stream having a predetermined number of segments. The transmitting device according to claim 1 .
10. The above subtitle stream includes: The abstract information segment is placed first, followed by the subtitle text information segment. The transmitting device according to claim 9.
11. The subtitle text information is in TTML format or a format derived from TTML. The transmitting device according to claim 1 .
12. a video encoding step for generating a video stream including encoded video data; a subtitle encoding step of generating a subtitle stream including subtitle text information having display timing information and abstract information having information corresponding to a part of the plurality of pieces of information indicated by the text information; A transmitting step of transmitting a container in a predetermined format including the video stream and the subtitle stream by a transmitting unit. Transmission method.
13. A receiving unit receives a container in a predetermined format including a video stream and a subtitle stream, the video stream comprises encoded video data; the subtitle stream includes subtitle text information having display timing information and abstract information having information corresponding to a part of the plurality of pieces of information indicated by the text information; The video decoding apparatus further includes a control unit for controlling a video decoding process for decoding the video stream to obtain video data, a subtitle decoding process for decoding the subtitle stream to obtain subtitle bitmap data and extracting the abstract information, a video overlay process for overlaying the subtitle bitmap data on the video data to obtain video data for display, and a bitmap data process for processing the subtitle bitmap data overlaid on the video data based on the abstract information. Receiving device.
14. The abstract information includes subtitle display timing information. In the bitmap data processing, The timing of superimposing the bitmap data of the subtitle on the video data is controlled based on display timing information of the subtitle.
14. A receiving device according to claim 13.
15. the abstract information includes display control information for controlling a display state of the subtitle; In the bitmap data processing, A state of a bitmap of the subtitle superimposed on the video data is controlled based on the display control information.
14. A receiving device according to claim 13.
16. A receiving step of receiving a container in a predetermined format including a video stream and a subtitle stream by a receiving unit, the video stream comprises encoded video data; the subtitle stream includes subtitle text information having display timing information and abstract information having information corresponding to a part of the plurality of pieces of information indicated by the text information; a video decoding step of decoding the video stream to obtain video data; a subtitle decoding step of decoding the subtitle stream to obtain subtitle bitmap data and extracting the abstract information; a video overlaying step of overlaying the subtitle bitmap data on the video data to obtain video data for display; The method further includes a control step of controlling bitmap data of a subtitle superimposed on the video data based on the abstract information. Receiving method.
17. a video encoding unit for generating a video stream including encoded video data; a subtitle encoding unit that generates one or more segments in which elements of subtitle text information having display timing information are arranged, and generates a subtitle stream including the one or more segments; a transmitting unit for transmitting a container in a predetermined format including the video stream and the subtitle stream; Transmitting device.
18. The above subtitle encoding section is To generate one segment containing all the elements of the subtitle text information, Insert information regarding the transmission order and / or update of the text information of the subtitle into the layer of the segment or the layer of the element.
18. A transmitting device according to claim 17.
19. The subtitle text information is in TTML format or a format derived from TTML.
18. A transmitting device according to claim 17.
20. a video encoding step for generating a video stream including encoded video data; a subtitle encoding step of generating one or more segments in which elements of subtitle text information having display timing information are arranged, and generating a subtitle stream including the one or more segments; A transmitting step of transmitting a container in a predetermined format including the video stream and the subtitle stream by a transmitting unit. Transmission method.
Citation Information
Patent Citations
Display control method, recording medium and display control unit
JP2012169885A
Receiving device and data processing method
JP5713142B1
Transmission method and transmission device
JP7259901B2
Methods and apparatus to measure exposure to streaming media
WO2012177874A2
Streaming distribution device and method, streaming receiving device and method, streaming system, program, and recording medium
WO2013018517A1
