Transmission method and transmission device

Abstract

To easily recognize a high quality format corresponding to encoded image data possessed by an extended video stream at a receiving side.SOLUTION: Two video streams of a basic video stream having encoded image data of basic format image data and an extended video stream having encoded image data of image data of a high quality format of one kind selected from a plurality of kinds are generated. A container of a predetermined format including the basic video stream and the extended video stream is transmitted. Information representing the high quality format corresponding to the encoded image data possessed by the extended video stream is inserted into the extended video stream and / or the container.SELECTED DRAWING: Figure 20

Description

[Technical field]

[0001] This technology transmits Method and send Device Regarding. [Background technology]

[0002] Conventionally, it is known to transmit high-quality format image data together with basic format image data, and selectively use the basic format image data or the high-quality format image data on the receiving side. For example, Patent Document 1 describes that media coding is performed in a scalable manner to generate a base layer stream for a low-resolution video service and an enhancement layer stream for a high-resolution video service, and a broadcast signal including these is transmitted. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2008-543142 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present technology is to enable the receiving side to easily recognize the high-quality format to which the encoded image data contained in the extended video stream corresponds. [Means for solving the problem]

[0005] The concept of this technology is as follows: an image encoding unit that generates two video streams, a basic video stream having encoded image data of basic format image data and an extended video stream having encoded image data of one type of high quality format image data selected from a plurality of types; a transmitting unit for transmitting a container in a predetermined format including the basic video stream and the extended video stream; an information inserting unit that inserts information indicating a high-quality format to which encoded image data contained in the extended video stream corresponds, into the extended video stream and / or the container; Located in the transmitting device.

[0006] In this technology, an image encoding unit generates two video streams, a basic video stream having encoded image data of basic format image data, and an extended video stream having encoded image data of one type of high-quality format image data selected from multiple types. Then, a transmission unit transmits a container in a predetermined format including the basic video stream and the extended video stream.

[0007] For example, the image encoding unit may be configured to, with respect to basic format image data, perform predictive encoding processing within the basic format image data to obtain encoded image data, and with respect to high quality format image data, selectively perform predictive encoding processing within the high quality format image data or predictive encoding processing between the high quality format image data and the basic format image data to obtain encoded image data.

[0008] In this case, for example, the basic format image data may be image data with normal dynamic range and low frame rate, the high quality format image data may be either image data with high dynamic range and high frame rate, image data with high dynamic range and low frame rate, or image data with normal dynamic range and high frame rate, and the encoded image data of the high quality format image data may have an encoded component of high dynamic range image data with difference information relative to the normal dynamic range image data and / or an encoded component of high frame rate image data with difference information relative to the low frame rate image data.

[0009] For example, when obtaining difference information for the normal dynamic range image data, the image encoding unit may perform dynamic range conversion on the normal dynamic range image data to reduce the difference value. In this case, for example, the image encoding unit may perform dynamic range conversion on the normal dynamic range image data based on conversion information for converting values ​​of data converted by normal dynamic range photoelectric conversion characteristics into values ​​of data converted by high dynamic range photoelectric conversion characteristics.

[0010] The information inserting unit inserts information (high-quality format information) representing a high-quality format to which the encoded image data of the extended video stream corresponds, into the extended video stream and / or the container. For example, the extended video stream may have a NAL unit structure, and the information inserting unit may insert the high-quality format information into a header of the NAL unit.

[0011] Also, for example, the extended video stream may have a NAL unit structure, and the information inserting unit may insert the high quality format information into an area of ​​the SEI NAL unit. Also, for example, the container may be MPEG2-TS, and the information inserting unit may insert the high quality format information into a video elementary stream loop corresponding to the extended video stream existing under the program map table.

[0012] For example, the information inserting unit may further insert the above conversion information into the extended video stream and / or the container, so that the receiving side can appropriately perform dynamic range conversion processing to be performed on the normal dynamic range image data to obtain the high dynamic range image data based on the conversion information.

[0013] In this way, in the present technology, information (high-quality format information) indicating the high-quality format corresponding to the encoded image data of the extended video stream is inserted into the extended video stream and / or the container. Therefore, the receiving side can easily recognize the high-quality format of the high-quality format image data. Then, based on this information and the display capability information, the receiving side can obtain image data corresponding to the display capability from the basic video stream and the extended video stream as display image data.

[0014] In addition, in the present technology, for example, the image encoding unit may set the time indicated by the decode time stamp added to the encoded image data of each picture included in the extended video stream to be the same as or intermediate to the time indicated by the decode time stamp added to the encoded image data of each picture included in the basic video stream, set the intervals between the times indicated by the decode time stamps added to the encoded image data of each picture included in the basic video stream to be equal, and set the intervals between the times indicated by the decode time stamps added to the encoded image data of each picture included in the extended video stream to be equal. By setting the intervals between the decode time stamps to be equal in this way, the decoding capability of the receiving side can be used efficiently.

[0015] Another concept of the present technology is a receiving unit for receiving a container in a predetermined format including two video streams, a basic video stream having encoded image data of basic format image data, and an extended video stream having encoded image data of image data in one type of high quality format selected from a plurality of types; information indicating a high-quality format corresponding to encoded image data of the extended video stream is inserted into the extended video stream and / or the container; an information extractor for extracting said information from said enhanced video stream and / or container; a processing unit for obtaining image data corresponding to a display capability from the basic video stream and the extended video stream as image data for display based on the extracted information and display capability information. in the receiving device.

[0016] In this technology, a receiving unit receives a container of a predetermined format including two video streams, a basic video stream having encoded image data of basic format image data, and an extended video stream having encoded image data of image data of one type of high-quality format selected from a plurality of types. For example, the encoded image data included in the basic video stream may be generated by performing a predictive encoding process within the basic format image data on the basic format image data, and the encoded image data included in the extended video stream may be generated by selectively performing a predictive encoding process within the high-quality format image data or a predictive encoding process between the high-quality format image data and the basic format image data on the high-quality format image data.

[0017] Information indicating a high-quality format to which the encoded image data in the extended video stream corresponds is inserted into the extended video stream and / or the container. The information extraction unit extracts this information from the extended video stream and / or the container. The processing unit obtains image data corresponding to the display capability from the basic video stream and the extended video stream as display image data based on the extracted information and the display capability information.

[0018] In this way, in the present technology, image data corresponding to the display capability is obtained as display image data from the basic video stream and the extended video stream based on the information indicating the high-quality format corresponding to the encoded image data in the extended video stream and the display capability information. Therefore, it is possible to efficiently obtain image data corresponding to the display capability as display image data. Effect of the Invention

[0019] According to the present technology, it is possible to easily recognize the high-quality format corresponding to the encoded image data of the extended video stream on the receiving side. Note that the effects described here are not necessarily limited to those described herein, and may be any of the effects described in the present disclosure. [Brief description of the drawings]

[0020] [Figure 1] 1 is a block diagram showing an example of the configuration of a transmission / reception system according to an embodiment; [Diagram 2] FIG. 2 is a block diagram showing a configuration example of a transmission device. [Diagram 3] 11 is a block diagram showing an example of the configuration of an image data generating unit that generates basic format image data Vb and high quality format image data Ve. FIG. [Figure 4] 2 is a block diagram showing an example of the configuration of a main part of an encoding unit. FIG. [Diagram 5] FIG. 13 is a diagram for explaining level adjustment (dynamic range conversion) in the case of dynamic range expansion. [Figure 6] A diagram showing an example of the structure of a NAL unit header and the contents of main parameters in the example structure. [Figure 7] FIG. 2 is a diagram showing the first access unit of a GOP when the encoding method is HEVC. [Figure 8] FIG. 2 is a diagram showing an access unit other than the first access unit of a GOP when the encoding method is HEVC. [Figure 9] A diagram showing an example of the structure of a scalable linkage SEI message. [Figure 10] A diagram showing the contents of main information in an example structure of a scalable linkage SEI message. [Figure 11] FIG. 11 is a diagram showing an example configuration (HDR+HFR) of coded image data of a basic video stream BS and coded image data of an extended video stream ES. [Figure 12]FIG. 13 is a diagram for explaining management of decode / display timing (decode / display timestamp) in encoded image data. [Figure 13] FIG. 11 is a diagram showing an example configuration (HDR+LFR) of coded image data of a basic video stream BS and coded image data of an extended video stream ES. [Figure 14] 11 is a diagram showing an example configuration (SDR+HFR) of coded image data of a basic video stream BS and coded image data of an extended video stream ES. FIG. [Figure 15] FIG. 13 is a diagram showing an example of the structure of coded image data of a basic video stream BS and coded image data of an extended video stream ES ("SDR+HFR" to "HDR+HFR"). [Figure 16] FIG. 13 is a diagram showing an example of the structure of coded image data of a basic video stream BS and coded image data of an extended video stream ES ("SDR+HFR" to "HDR+LFR"). [Figure 17] FIG. 13 is a diagram showing an example of the structure of coded image data of a basic video stream BS and coded image data of an extended video stream ES ("HDR+HFR" to "HDR+LFR"). [Figure 18] FIG. 1 is a diagram showing an example of a scalable linkage descriptor structure and the contents of main information in the example structure. [Figure 19] 13 is a diagram showing an example of values ​​of each field when the value of "nuh_layer_id" in the header of the NAL unit is fixed depending on the coding component and when the value of "nuh_layer_id" in the header of the NAL unit is flexibly assigned. [Figure 20] FIG. 2 is a diagram illustrating an example of the configuration of a transport stream TS. [Figure 21] FIG. 2 is a block diagram showing a configuration example of a receiving device. [Figure 22] 4 is a block diagram showing an example of the configuration of a main part of a decoding unit; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, modes for carrying out the invention (hereinafter, referred to as "embodiments") will be described in the following order. 1. Embodiment 2. Variations

[0022] <1. Preferred embodiment> [Transmitting and receiving system] 1 shows an example of the configuration of a transmission / reception system 10 according to an embodiment. The transmission / reception system 10 includes a transmission device 100 and a reception device 200. The transmission device 100 transmits a transport stream TS as a container via broadcast waves or network packets.

[0023] This transport stream TS includes two video streams, a basic video stream and an extended video stream. The basic video stream has encoded image data of basic format image data. This basic format image data is image data with a normal dynamic range and a low frame rate. This encoded image data of basic format image data is generated by applying predictive encoding processing such as H.264 / AVC and H.265 / HEVC to the basic format image data. In this case, predictive encoding processing is performed within the basic format image data to obtain encoded image data.

[0024] The extended video stream has encoded image data of one type of high quality format image data selected from a plurality of types. In this embodiment, the high quality format image data is either image data with a high dynamic range and a high frame rate, image data with a high dynamic range and a low frame rate, or image data with a normal dynamic range and a high frame rate.

[0025] The encoded image data of this extended video stream is generated by performing predictive encoding processing such as H.264 / AVC, H.265 / HEVC, etc. on high-quality format image data. In this case, predictive encoding processing within the high-quality format image data or predictive encoding processing between the high-quality format image data and basic format image data is selectively performed to obtain encoded image data.

[0026] In this case, the encoded image data of the high quality format image data has encoded components of high dynamic range image data based on difference information relative to normal dynamic range image data and / or encoded components of high frame rate image data based on difference information relative to low frame rate image data.

[0027] The transmitting device 100 inserts information indicating the high-quality format to which the encoded image data of the extended video stream corresponds (hereinafter, appropriately referred to as "high-quality format information") into the extended video stream and / or the transport stream TS as a container. This information is inserted into the header of the NAL unit or the area of ​​the SEI NAL unit. In addition, this information is inserted into the video elementary stream loop corresponding to the extended video stream existing under the program map table.

[0028] The receiving device 200 receives the transport stream TS transmitted by broadcast waves or network packets from the transmitting device 100. As described above, this transport stream TS has a basic video stream having coded image data of basic format image data, and an extended video stream having coded image data of high quality format image data.

[0029] As described above, high-quality format information is inserted into the extended video stream and / or the transport stream TS as a container. Based on this information and the display capability information, the receiving device 200 obtains image data corresponding to the display capability from the basic video stream and the extended video stream as display image data.

[0030] "Transmitter Configuration" 2 shows a configuration example of the transmission device 100. This transmission device 100 handles basic format image data Vb and high-quality format image data Ve as transmission image data. Here, the basic format image data Vb is SDR (Standard Dynamic Range) image data with a frame frequency of 50 Hz (LFR: Low Frame Rate).

[0031] The high-quality format image data Ve is one type of high-quality format image data selected from three types: (a) HDR image data with a frame frequency of 100Hz (HFR: high frame rate), (b) HDR image data with a frame frequency of 50Hz (LFR), or (c) SDR image data with a frame frequency of 100Hz (HFR). If the brightness of the white peak of a conventional LDR image is 100%, the HDR image data has a brightness range of 0 to 100%*N, for example, 0 to 1000% or more.

[0032] 3 shows an example of the configuration of an image data generating unit 150 that generates basic format image data Vb and high quality format image data Ve. The image data generating unit 150 includes a camera 151 and a conversion unit 152.

[0033] The camera 151 captures an image of a subject and outputs high-quality format image data Ve. When the high-quality format image data Ve output by the camera 151 is HDR image data with a frame frequency of 100 Hz, the conversion unit 152 converts the frame rate and dynamic range and outputs basic format image data Vb, which is SDR image data with a frame frequency of 50 Hz.

[0034] Furthermore, if the high-quality format image data Ve output by the camera 151 is HDR image data with a frame frequency of 50 Hz, the conversion unit 152 performs only dynamic range conversion and outputs basic format image data Vb that is SDR image data with a frame frequency of 50 Hz. Furthermore, if the high-quality format image data Ve output by the camera 151 is SDR image data with a frame frequency of 100 Hz, the conversion unit 152 performs only frame rate conversion and outputs basic format image data Vb that is SDR image data with a frame frequency of 50 Hz.

[0035] 2, the transmitting device 100 has a control unit 101, a photoelectric conversion unit 102, an RGB / YCbCr conversion unit 103, a photoelectric conversion unit 104, an RGB / YCbCr conversion unit 105, a video encoder 106, a system encoder 107, and a transmitting unit 108. The control unit 101 is configured with a CPU (Central Processing Unit), and controls the operation of each unit of the transmitting device 100 based on a control program.

[0036] The photoelectric conversion unit 102 applies SDR photoelectric conversion characteristics (SDR OETF curve) to the basic format image data Vb to obtain basic format image data Vb' for transmission. The RGB / YCbCr conversion unit 103 converts this basic format image data V' from the RGB domain to the YCbCr (luminance / chrominance) domain.

[0037] The photoelectric conversion unit 104 applies HDR photoelectric conversion characteristics (HDR OETF curve) or SDR photoelectric conversion characteristics (SDR OETF curve) to the high-quality format image data Ve to obtain high-quality format image data Ve' for transmission. The RGB / YCbCr conversion unit 105 converts this high-quality format image data Ve' from the RGB domain to the YCbCr (luminance / chrominance) domain.

[0038] The video encoder 106 includes an encoding unit 106b and an encoding unit 106e. The encoding unit 106b obtains encoded image data by performing predictive encoding processing such as H.264 / AVC or H.265 / HEVC on basic format image data Vb′ for transmission, and generates a basic video stream (video elementary stream) BS having the encoded image data.

[0039] The encoding unit 106e obtains encoded image data by performing predictive encoding processing such as H.264 / AVC or H.265 / HEVC on the high-quality format image data Ve' for transmission, and generates an extended video stream (video elementary stream) ES having this encoded image data. In this case, the encoding unit 106e selectively performs prediction within the image data Ve' or prediction between the image data Vb' for each encoding block in order to reduce the prediction residual.

[0040] 4 shows an example of the configuration of the main parts of the encoding unit 106e. The encoding unit 106e has an intra-layer prediction unit 161, an inter-layer prediction unit 162, a prediction adjustment unit 163, a selection unit 164, and an encoding function unit 165.

[0041] The intra-layer prediction unit 161 performs prediction (intra-layer prediction) on the image data V1 to be coded to obtain prediction residual data. The inter-layer prediction unit 162 performs prediction (inter-layer prediction) on the image data V1 to be coded with reference image data V2 to obtain prediction residual data.

[0042] In order to efficiently perform inter-layer prediction in the inter-layer prediction unit 162, the prediction adjustment unit 163 performs the following processing according to the type of scalable extension of the image data V1 to the image data V2. That is, in the case of dynamic range extension, level adjustment (dynamic range conversion) is performed to convert from SDR to HDR. In the case of frame rate extension, bypassing is performed.

[0043] With reference to Fig. 5, the level adjustment (dynamic range conversion) in the case of dynamic range expansion will be further described. The solid line a shows an example of an SDR OETF curve showing SDR photoelectric conversion characteristics. The solid line b shows an example of an HDR OETF curve showing HDR photoelectric conversion characteristics. The horizontal axis shows the input luminance level, P1 shows the input luminance level corresponding to the SDR maximum level, and P2 shows the input luminance level corresponding to the HDR maximum level.

[0044] The vertical axis indicates the relative value of the transmission code value or the normalized encoding level. The relative maximum level M indicates the HDR maximum level and the SDR maximum level. The reference level G indicates the transmission level of the HDR OETF at the input luminance level P1 corresponding to the SDR maximum level, and means the so-called reference white level. The branch level B indicates the level at which the SDR OETF curve and the HDR OETF curve branch off from the same orbit. Pf indicates the input luminance level corresponding to the branch level. This branch level B can be any value equal to or greater than 0.

[0045] In the level adjustment (dynamic range conversion) in the prediction adjustment unit 163, the branch level B to the relative maximum level M of the basic format image data Vb' are converted to have values ​​of converted data according to the HDR photoelectric conversion characteristics. In this case, the relative maximum level M of SDR is made to match the reference level G. Note that input data below the branch level B is used as output data as is.

[0046] Here, the conversion information is given as a conversion table or a conversion coefficient. When it is given as a conversion table, the prediction adjustment unit 163 performs the conversion by referring to this conversion table. On the other hand, when it is given as a conversion coefficient, the prediction adjustment unit 163 performs the conversion by a calculation using this conversion coefficient. For example, when the conversion coefficient is C, the input data from the branch level B to the relative maximum level M is converted by the following formula (1). Output data = branch level B + (input data - branch level B) * C (1)

[0047] The selection unit 164 selectively extracts the prediction residual data obtained by the intra-layer prediction unit 161 or the prediction residual data obtained by the inter-layer prediction unit 162 for each coding block, and sends it to the encoding function unit 165. In this case, the selection unit 164 extracts, for example, the prediction residual having a smaller value. The encoding function unit 165 performs encoding processes such as transform coding, quantization, and entropy coding on the prediction residual data extracted from the selection unit 164 to obtain encoded image data CV.

[0048] The encoded image data included in the extended video stream includes encoded components of a type corresponding to the high-quality format image data Ve. That is, when the image data Ve is (a) HFR (100 Hz) and HDR image data, the encoded components are HDR and HFR. When the image data Ve is (b) LFR (50 Hz) and HDR image data, the encoded components are HDR and LFR. When the image data Ve is (c) HFR (100 Hz) and SDR image data, the encoded components are SDR and HFR.

[0049] 2, the encoding unit 106e inserts information (high-quality format information) indicating the high-quality format to which the encoded image data of the extended video stream corresponds, into the extended video stream ES. The encoding unit 106e inserts this high-quality format information, for example, into the header of the NAL unit or the area of ​​the SEI NAL unit.

[0050] Figure 6(a) shows an example structure (Syntax) of a NAL unit header, and Figure 6(b) shows the contents (Semantics) of the main parameters in that example structure. The 1-bit field "forbidden_zero_bit" must be 0. The 6-bit field "nal_unit_type" indicates the NAL unit type. The 6-bit field "nuh_layer_id" indicates the ID of the enhancement layer. The 3-bit field "nuh_temporal_id_plus1" indicates temporal_id (0-6), and takes on a value of 1 plus 1 (1-7).

[0051] In the basic video stream, the value of "nuh_layer_id" is set to "0." When high-quality format information is inserted into the header of a NAL unit, the value of "nuh_layer_id" in the extended video stream is fixed according to the type of coding component included in the coded image data. In other words, the value is set to a fixed value according to the type of high-quality format to which the coded image data corresponds.

[0052] In this case, "nuh_layer_id" constitutes high quality format information. For example, when the coding components included in the coded image data are HDR and HFR coding components, "nuh_layer_id" is set to "6". When the coding components included in the coded image data are HDR and LFR coding components, "nuh_layer_id" is set to "5". When the coding components included in the coded image data are SDR and HFR coding components, "nuh_layer_id" is set to "0".

[0053] On the other hand, in the case of flexibly assigning the value of "nuh_layer_id" in this extended video stream, the high-quality format information is inserted into the SEI NAL unit. In this case, "nuh_layer_id" does not have the function of directly indicating the high-quality format corresponding to the coded image data of the extended video stream.

[0054] At this time, the encoding unit 106e inserts a newly defined scalable linkage SEI message (Scalable_linkage SEI message) having high quality format information into the "SEIs" portion of the access unit (AU).

[0055] FIG. 7 shows the first access unit of a GOP (Group Of Pictures) when the coding method is HEVC. FIG. 8 shows an access unit other than the first access unit of a GOP when the coding method is HEVC. In the case of the HEVC coding method, a group of SEI messages for decoding, "Prefix_SEIs", is placed before slices in which pixel data is coded, and a group of SEI messages for display, "Suffix_SEIs", is placed after these slices. The scalable linkage SEI message is placed, for example, as a group of SEI messages, "Suffix_SEIs" or "Prefix SEIs", as shown in the figure.

[0056] Figure 9 shows an example of the structure (Syntax) of a Scalable Linkage SEI message. Figure 10 shows the main information content (Semantics) of this example structure. The 1-bit flag information of "Scalable_linkage_cancel_flag" indicates whether to refresh the "Scalable_linkage" SEI message. "0" indicates that this SEI message is refreshed. "1" indicates that this SEI message is not refreshed, that is, the previous message is maintained as is.

[0057] When "Scalable_linkage_cancel_flag" is "0", the following fields are present. The 16-bit field of "scalable_index" is an index indicating the type of the scalable extension stream. In other words, this field constitutes high-quality format information.

[0058] For example, "0x8" indicates an extension component to HDR, i.e., the encoded component included in the encoded image data is an HDR and LFR encoded component. "0x10" indicates an extension component to HFR, i.e., the encoded component included in the encoded image data is an SDR and HFR encoded component. "0x18" indicates an extension component to HDR and HFR, i.e., the encoded component included in the encoded image data is an HDR and HFR encoded component.

[0059] The 6-bit field "nuh_layer_id" indicates the layer ID included in the extended video stream. The 15-bit field "reference_level" indicates the reference brightness level, that is, the reference level G (see Figure 5). The 1-bit flag information "conversion_table_flag" indicates that it is based on a conversion table, that is, that conversion table information exists.

[0060] When "conversion_table_flag" is "1", an 8-bit field "table_size" is present. This field indicates the number of inputs in the conversion table. There are as many 16-bit fields "predctrl_y[i]", "predctrl_cb[i]", and "predctrl_cr[i]" as there are inputs. The "predctrl_y[i]" field indicates the value of the prediction adjustment transform for luminance. The "predctrl_cb[i]" field indicates the value of the prediction adjustment transform for chrominance Cb. The "predctrl_cr[i]" field indicates the value of the prediction adjustment transform for chrominance Cr.

[0061] 11 shows an example of the configuration of the coded image data of the basic video stream BS obtained by the encoding unit 106b and the coded image data of the extended video stream ES obtained by the encoding unit 106e. In this example, the coded image data of the basic video stream BS contains coded components of SDR and LFR, and the coded image data of the extended video stream ES contains coded components of HDR and HFR. In this example, the value of "nuh_layer_id" is fixed depending on the type of coded components contained in the coded image data, and this "nuh_layer_id" constitutes high-quality format information.

[0062] The horizontal axis indicates the picture order of composition (POC), with the left side indicating earlier display time and the right side indicating later display time. The vertical axis indicates the hierarchy. Each rectangular box indicates a picture, and the arrows show an example of the picture reference relationship in predictive coding processing. For both inter-layer and intra-layer prediction, the target picture changes for each block, and the prediction direction and number of references are not limited to the example shown. The numbers in the rectangular boxes indicate the order in which the pictures are coded, that is, the encoding order (decoding order on the receiving side). Several sub-groups of pictures form a GOP (Group Of Pictures).

[0063] The coded image data of the basic video stream is hierarchically coded in two layers, 0 and 1. The temporal_id (layer identification information) placed in the headers of the NAL units (nal_unit) constituting the coded image data of pictures in layers 0 and 1 is set to 0 and 1, respectively. In addition, the value of "nuh_layer_id" in the header of each NAL unit is set to "0".

[0064] Also, the coded image data of the extended video stream is hierarchically coded in three layers: 0, 1, and 2. 0, 1, and 2 are set as temporal_id (layer identification information) placed in the headers of NAL units (nal_unit) constituting the coded image data of pictures in layers 0, 1, and 2, respectively. Also, the value of "nuh_layer_id" in the header of each NAL unit is set to "6", indicating that the coding components included in the coded image data are HDR and HFR coding components.

[0065] Figure 12 shows a schematic diagram of management of decode / display timing (decode / display timestamp) in the case of hierarchical coding in Figure 11. Each number corresponds to the number in each rectangular frame showing a picture in Figure 11.

[0066] "B" indicates a picture that constitutes the basic frame frequency. "L" indicates the basic frame frequency, and the dynamic range indicates the picture component that is the predicted difference with the basic video stream BS. "U" is a frame that is located at the middle timing between frames of the basic frame frequency, and becomes a high frame frequency when temporally combined with an "L" or "B" frame. "U" can also have a difference component with "B" or "L" in the dynamic range, and when combined with these, it becomes a high dynamic range.

[0067] In managing the decode timing (decode timestamp), the encode buffer is managed (HRD management) so that "B" and "L" are decoded at the same time, and "U" is decoded exactly halfway between the decode times of the previous and next "B" or "L." Similarly, in managing the display timing (display timestamp), the encode buffer is managed (HRD management) so that "B" and "L" are displayed using predictive synthesis at the same time, and "U" is displayed exactly halfway between the display times of the previous and next basic frame frequency images.

[0068] Although detailed description is omitted, in the other configuration examples described below, management of the decode / display timing (timestamp of the decode / display) is performed in a similar manner.

[0069] 13 also shows a configuration example of the coded image data of the basic video stream BS obtained by the encoding unit 106b and the coded image data of the extended video stream ES obtained by the encoding unit 106e. In this example, the coded image data of the basic video stream BS contains coded components of SDR and LFR, and the coded image data of the extended video stream ES contains coded components of HDR and LFR. In this example, the value of "nuh_layer_id" is fixed depending on the type of coded components contained in the coded image data, and this "nuh_layer_id" constitutes high-quality format information.

[0070] The coded image data of the basic video stream is hierarchically coded in two layers, 0 and 1. The temporal_id (layer identification information) placed in the headers of the NAL units (nal_unit) constituting the coded image data of pictures in layers 0 and 1 is set to 0 and 1, respectively. In addition, the value of "nuh_layer_id" in the header of each NAL unit is set to "0".

[0071] In addition, the coded image data of the extended video stream is hierarchically coded in two layers, 0 and 1, in the sub-picture group before the switch, and the temporal_id (layer identification information) placed in the header of the NAL unit (nal_unit) that constitutes the coded image data of the pictures in layers 0 and 1 is set to 0 and 1, respectively.

[0072] The sub-picture group after the switch is hierarchically coded in two layers, 0 and 2, and the temporal_id (layer identification information) placed in the headers of the NAL units (nal_unit) constituting the coded image data of pictures in layers 0 and 2 is set to 0 and 2, respectively. In addition, the value of "nuh_layer_id" in the header of each NAL unit is set to "5", indicating that the coding components included in the coded image data are HDR and LFR coding components.

[0073] 14 also shows an example of the configuration of the coded image data of the basic video stream BS obtained by the encoding unit 106b and the coded image data of the extended video stream ES obtained by the encoding unit 106e. In this example, the coded image data of the basic video stream BS contains coded components of SDR and LFR, and the coded image data of the extended video stream ES contains coded components of SDR and HFR. In this example, the value of "nuh_layer_id" is fixed depending on the type of coded components contained in the coded image data, and this "nuh_layer_id" constitutes high-quality format information.

[0074] The coded image data of the basic video stream is hierarchically coded in two layers, 0 and 1. The temporal_id (layer identification information) placed in the headers of the NAL units (nal_unit) constituting the coded image data of pictures in layers 0 and 1 is set to 0 and 1, respectively. In addition, the value of "nuh_layer_id" in the header of each NAL unit is set to "0".

[0075] Also, the coded image data of the extended video stream is hierarchically coded in three layers, 0, 1, and 2, in the sub-picture group before switching, and 0, 1, and 2 are set as the temporal_id (layer identification information) placed in the header of the NAL unit (nal_unit) constituting the coded image data of pictures in layers 0, 1, and 2, respectively. Here, pictures "2", "5", "8", and "11" are copies of pictures "1", "4", "7", and "10", respectively, and picture "3" refers to picture "2", and picture "6" refers to picture "5".

[0076] The sub-picture group after the switch is hierarchically coded at one layer, 0, and 0 is set as the temporal_id (layer identification information) placed in the header of the NAL unit (nal_unit) that constitutes the coded image data of the picture at layer 0. Note that, as indicated by the dashed lines, one layer of layer 1 or one layer of layer 2 can also be used. In addition, the value of "nuh_layer_id" in the header of each NAL unit is set to "0", indicating that the coded components included in the coded image data are SDR and HFR coded components.

[0077] 15 also shows an example of the configuration of the coded image data of the basic video stream BS obtained by the encoding unit 106b and the coded image data of the extended video stream ES obtained by the encoding unit 106e. In this example, the coded image data of the basic video stream BS includes SDR and LFR coded components, and the coded image data of the extended video stream ES includes SDR and HFR coded components before switching and includes HDR and HFR coded components after switching.

[0078] This example is an example in which the value of "nuh_layer_id" is flexibly assigned and this "nuh_layer_id" does not constitute high-quality format information, that is, the "scalable_index" of the scalable linkage SEI message constitutes high-quality format information.

[0079] The coded image data of the basic video stream is hierarchically coded in two layers, 0 and 1. The temporal_id (layer identification information) placed in the headers of the NAL units (nal_unit) constituting the coded image data of pictures in layers 0 and 1 is set to 0 and 1, respectively. In addition, the value of "nuh_layer_id" in the header of each NAL unit is set to "0".

[0080] Also, the coded image data of the extended video stream is hierarchically coded in three layers, 0, 1, and 2, in the sub-picture group before switching, and 0, 1, and 2 are set as the temporal_id (layer identification information) placed in the header of the NAL unit (nal_unit) constituting the coded image data of pictures in layers 0, 1, and 2, respectively. Here, pictures "2", "5", "8", and "11" are copies of pictures "1", "4", "7", and "10", respectively, and picture "3" refers to picture "2", and picture "6" refers to picture "5".

[0081] The sub-picture group after switching is hierarchically coded in three layers, 0, 1, and 2, and 0, 1, and 2 are set as the temporal_id (layer identification information) placed in the header of the NAL unit (nal_unit) constituting the coded image data of pictures in layers 0, 1, and 2. Here, pictures "14", "17", "20", and "23" are the HDR difference pictures of pictures "13", "16", "19", and "22", respectively.

[0082] The value of "nuh_layer_id" in the header of each NAL unit is "6" both before and after the switch, and this "nuh_layer_id" does not indicate the type of coding components contained in the coded image data of the extended video stream. In this case, "scalable_index" is "0x10" before the switch, indicating that it is an SDR and HFR coding component, and "scalable_index" is "0x18" after the switch, indicating that it is an HDR and HFR coding component.

[0083] 16 also shows an example of the configuration of the coded image data of the basic video stream BS obtained by the encoding unit 106b and the coded image data of the extended video stream ES obtained by the encoding unit 106e. In this example, the coded image data of the basic video stream BS contains SDR and LFR coded components, and the coded image data of the extended video stream ES contains SDR and HFR coded components before switching and contains HDR and LFR coded components after switching.

[0084] This example is an example in which the value of "nuh_layer_id" is flexibly assigned and this "nuh_layer_id" does not constitute high-quality format information, that is, the "scalable_index" of the scalable linkage SEI message constitutes high-quality format information.

[0085] The coded image data of the basic video stream is hierarchically coded in two layers, 0 and 1. The temporal_id (layer identification information) placed in the headers of the NAL units (nal_unit) constituting the coded image data of pictures in layers 0 and 1 is set to 0 and 1, respectively. In addition, the value of "nuh_layer_id" in the header of each NAL unit is set to "0".

[0086] Also, the coded image data of the extended video stream is hierarchically coded in three layers, 0, 1, and 2, in the sub-picture group before switching, and 0, 1, and 2 are set as the temporal_id (layer identification information) placed in the header of the NAL unit (nal_unit) constituting the coded image data of pictures in layers 0, 1, and 2, respectively. Here, pictures "2", "5", "8", and "11" are copies of pictures "1", "4", "7", and "10", respectively, and picture "3" refers to picture "2", and picture "6" refers to picture "5".

[0087] The sub-picture group after switching is hierarchically coded in three layers, 0, 1, and 2, and 0, 1, and 2 are set as the temporal_id (layer identification information) placed in the header of the NAL unit (nal_unit) that constitutes the coded image data of pictures in layers 0, 1, and 2. Here, pictures "14", "16", "18", and "20" are the HDR difference pictures of pictures "13", "15", "17", and "19", respectively.

[0088] The value of "nuh_layer_id" in the header of each NAL unit is "6" both before and after the switch, and this "nuh_layer_id" does not indicate the type of coding components contained in the coded image data of the extended video stream. In this case, "scalable_index" is "0x10" before the switch, indicating that it is an SDR and HFR coding component, and "scalable_index" is "0x08" after the switch, indicating that it is an HDR and LFR coding component.

[0089] 17 also shows an example of the configuration of the coded image data of the basic video stream BS obtained by the encoding unit 106b and the coded image data of the extended video stream ES obtained by the encoding unit 106e. In this example, the coded image data of the basic video stream BS includes SDR and LFR coded components, and the coded image data of the extended video stream ES includes HDR and HFR coded components before switching and includes HDR and LFR coded components after switching.

[0090] This example is an example in which the value of "nuh_layer_id" is flexibly assigned and this "nuh_layer_id" does not constitute high-quality format information, that is, the "scalable_index" of the scalable linkage SEI message constitutes high-quality format information.

[0091] The coded image data of the basic video stream is hierarchically coded in two layers, 0 and 1. The temporal_id (layer identification information) placed in the headers of the NAL units (nal_unit) constituting the coded image data of pictures in layers 0 and 1 is set to 0 and 1, respectively. In addition, the value of "nuh_layer_id" in the header of each NAL unit is set to "0".

[0092] Also, the coded image data of the extended video stream is hierarchically coded in three layers, 0, 1, and 2, in the sub-picture group before switching, and 0, 1, and 2 are set as the temporal_id (layer identification information) arranged in the header of the NAL unit (nal_unit) constituting the coded image data of pictures in layers 0, 1, and 2, respectively. Here, pictures "2", "5", "8", and "11" are pictures of the HDR differences of pictures "1", "4", "7", and "10", respectively.

[0093] The sub-picture group after switching is hierarchically coded in three layers, 0, 1, and 2, and 0, 1, and 2 are set as the temporal_id (layer identification information) placed in the header of the NAL unit (nal_unit) that constitutes the coded image data of pictures in layers 0, 1, and 2. Here, pictures "14", "16", "18", and "20" are the HDR difference pictures of pictures "13", "15", "17", and "19", respectively.

[0094] The value of "nuh_layer_id" in the header of each NAL unit is "6" both before and after the switch, and this "nuh_layer_id" does not indicate the type of coding components contained in the coded image data of the extended video stream. In this case, "scalable_index" is "0x18" before the switch, indicating that it is an HFR coding component in HDR, and "scalable_index" is "0x08" after the switch, indicating that it is an LFR coding component in HDR.

[0095] 2, the system encoder 107 generates a transport stream TS by packetizing the basic video stream BS and the extended video stream ES generated by the video encoder 106 into PES packets and TS packets. Then, the transmitter 108 transmits this transport stream TS to the receiving device 200 via broadcast waves or network packets.

[0096] The system encoder 107 inserts high-quality format information into the transport stream TS as a container. In this embodiment, a newly defined scalable linkage descriptor (Scalable_linkage descriptor) is inserted into a video elementary stream loop corresponding to an extended video stream existing under a PMT (Program Map Table).

[0097] Figure 18(a) shows an example of the structure (Syntax) of a scalable linkage descriptor. Figure 18(b) shows the main information content (Semantics) of the example structure. The 8-bit field "descriptor_tag" indicates the descriptor type, which indicates that it is a scalable linkage descriptor in this case. The 8-bit field "descriptor_length" indicates the length (size) of the descriptor, and indicates the number of subsequent bytes as the length of the descriptor.

[0098] The 16-bit field of "scalable_index" is an index indicating the type of the scalable extension stream. This "scalable_index" field is set to the same value as the "scalable_index" field of the scalable linkage SEI message (see Figure 9) described above. Therefore, when the value of "nuh_layer_id" in the header of the NAL unit is flexibly assigned and the "scalable_index" field of the scalable linkage SEI message constitutes high-quality format information, the "scalable_index" field of this scalable linkage descriptor also constitutes similar information.

[0099] The 6-bit field of "nuh_layer_id" indicates the layer ID included in the extended video stream. This "nuh_layer_id" field is set to the same value as the "nuh_layer_id" field in the header of the NAL unit described above. Therefore, when the value of "nuh_layer_id" in the header of the NAL unit is fixed depending on the type of coding component included in the coded image data of the extended video stream and the "nuh_layer_id" field constitutes high-quality format information, the "nuh_layer_id" field in this scalable linkage descriptor also constitutes similar information.

[0100] FIG. 19(a) shows an example of the values ​​of each field when the value of "nuh_layer_id" in the NAL unit header is fixed depending on the coding component. In this case, the "nuh_layer_id" field in the NAL unit header constitutes high-quality format information. In this case, the "nuh_layer_id" field in the scalable linkage descriptor also constitutes high-quality format information. In this case, the value of "scalable_index" in the scalable linkage descriptor does not represent the high-quality format corresponding to the coded image data of the extended video stream.

[0101] FIG. 19(b) shows an example of the values ​​of each field when the value of "nuh_layer_id" in the NAL unit header is flexibly assigned. In this case, the "nuh_layer_id" field in the NAL unit header does not constitute high-quality format information. In this case, the "scalable_index" field in the scalable linkage SEI message constitutes high-quality format information. In this case, the "scalable_index" field in the scalable linkage descriptor also constitutes high-quality format information. Although not shown, it is also possible to include predictive conversion information (dynamic range conversion information) in this scalable linkage descriptor, similar to the above-mentioned scalable linkage SEI message (see FIG. 9).

[0102] [Transport Stream TS Structure] 20 shows an example of the structure of a transport stream TS. This transport stream TS includes a basic video stream BS and an extended video stream ES. In this example, a PES packet "video PES" exists for each video stream.

[0103] The packet identifier (PID) of the basic video stream BS is, for example, PID1. This basic video stream BS includes coded image data of each picture of the basic format image data. This coded image data includes NAL units such as AUD, VPS, SPS, PPS, PSEI, SLICE, SSEI, and EOS.

[0104] The packet identifier (PID) of the extended video stream ES is, for example, PID2. The extended video stream ES includes coded image data of each picture of the high-quality format image data. The coded image data includes NAL units such as AUD, SPS, PPS, PSEI, SLICE, SSEI, and EOS.

[0105] For example, when the value of "nuh_layer_id" in the header of a NAL unit is fixed by the coding component, the "nuh_layer_id" field constitutes high-quality format information. On the other hand, when the value of "nuh_layer_id" in the header of a NAL unit is flexibly assigned, a scalable linkage SEI message (see Figure 9) having a "scalable_index" field constituting high-quality format information is inserted in the "SEIs" part of an access unit (AU).

[0106] The transport stream TS also includes a program map table (PMT) as program specific information (PSI). This PSI is information that indicates which program each elementary stream included in the transport stream belongs to.

[0107] The PMT contains a Program loop that describes information related to the entire program, and an Elementary Stream loop that describes information related to each elementary stream.

[0108] In this configuration example, there are video elementary stream loops (video ES loops) corresponding to the basic video stream BS and the extended video stream ES, respectively. In the video elementary stream loop corresponding to the basic video stream BS, information such as a stream type (ST0) and a packet identifier (PID1) is placed, and a descriptor describing information related to this extended video stream ES is also placed.

[0109] In addition, in the video elementary stream loop corresponding to the extended video stream ES, information such as the stream type (ST1) and packet identifier (PID2) is placed, and a descriptor describing information related to this extended video stream ES is also placed. One of these descriptors is the above-mentioned scalable linkage descriptor (see FIG. 18).

[0110] The operation of the transmitting device 100 shown in Fig. 2 will be briefly described. Basic format image data Vb, which is SDR image data with a frame frequency of 50 Hz, is supplied to an opto-electrical conversion unit 102. In this opto-electrical conversion unit 102, SDR opto-electrical conversion characteristics (SDR OETF curve) are applied to the basic format image data Vb to obtain basic format image data Vb' for transmission. This basic format image data Vb' is converted from the RGB domain to the YCbCr (luminance / chrominance) domain by an RGB / YCbCr conversion unit 103, and then supplied to encoding units 106b and 106e of a video encoder 106.

[0111] Moreover, the high-quality format image data Ve is supplied to the photoelectric conversion unit 104. The high-quality format image data Ve is either (a) HDR image data with a frame frequency of 100 Hz, (b) HDR image data with a frame frequency of 50 Hz, or (c) SDR image data with a frame frequency of 100 Hz.

[0112] In the photoelectric conversion unit 104, HDR photoelectric conversion characteristics (HDR OETF curve) or SDR photoelectric conversion characteristics (SDR OETF curve) are applied to the high-quality format image data Ve to obtain high-quality format image data Ve' for transmission. This high-quality format image data Ve' is converted from the RGB domain to the YCbCr (luminance / chrominance) domain by the RGB / YCbCr conversion unit 105, and then supplied to the encoding unit 106e of the video encoder 106.

[0113] In the encoding unit 106b, a predictive encoding process such as H.264 / AVC or H.265 / HEVC is performed on the basic format image data Vb' for transmission to obtain encoded image data, and a basic video stream (video elementary stream) BS including this encoded video data is generated. This basic video stream BS is supplied to the system encoder 107.

[0114] In the encoding unit 106e, a predictive encoding process such as H.264 / AVC or H.265 / HEVC is performed on the high-quality format image data Ve' for transmission to obtain encoded image data, and an extended video stream (video elementary stream) ES including this encoded video data is generated. In this case, in the encoding unit 106e, in order to reduce the prediction residual, a prediction within the image data Ve' or a prediction between the image data Vb' is selectively performed for each encoding block.

[0115] In the encoding unit 106e, high-quality format information is inserted into the extended video stream ES. For example, this information is inserted into the header of the NAL unit. In this case, the value of "nuh_layer_id" (see FIG. 6) is fixed depending on the type of coding component included in the coded image data. Also, for example, this information is inserted into the area of ​​the SEI NAL unit. In this case, a newly defined scalable linkage SEI message (see FIG. 9) is inserted into the "SEIs" part of the access unit (AU).

[0116] The system encoder 107 performs PES packetization and TS packetization on the basic video stream BS and extended video stream ES generated by the video encoder 106 to generate a transport stream TS. At this time, the system encoder 107 inserts high quality format information into the transport stream TS as a container. That is, a newly defined scalable linkage descriptor (see FIG. 18) is placed in a video elementary stream loop corresponding to the extended video stream existing under the PMT.

[0117] The transport stream TS generated by the system encoder 107 is sent to the transmitting unit 108. In the transmitting unit 108, this transport stream TS is transmitted to the receiving device 200 via broadcast waves or net packets.

[0118] "Configuration of receiving device" Fig. 21 shows an example of the configuration of a receiving device 200. This receiving device 200 corresponds to the example of the configuration of the transmitting device 100 in Fig. 2. This receiving device 200 has a control unit 201, a receiving unit 202, a system decoder 203, a video decoder 204, a YCbCr / RGB conversion unit 105, an electric-to-optical conversion unit 206, a YCbCr / RGB conversion unit 207, an electric-to-optical conversion unit 208, and a display unit 209.

[0119] The control unit 201 includes a CPU (Central Processing Unit) and controls the operation of each unit of the receiving device 200 based on a control program. The receiving unit 202 receives a transport stream TS transmitted from the transmitting device 100 via broadcast waves or network packets. The system decoder 203 extracts a basic video stream BS and an extended video stream ES from this transport stream TS.

[0120] The system decoder 203 also extracts various pieces of information inserted into the transport stream TS as a container, and sends them to the control unit 201. This information also includes the above-mentioned scalable linkage descriptor (see FIG. 18). The control unit 201 recognizes the type of coded components included in the coded image data of the extended video stream ES based on the "nuh_layer_id" field or the "scalable_index" field of this descriptor. That is, the control unit 201 recognizes whether coded components of HFR in HDR, LFR in HDR, or HFR in SDR are included, and selects the decoding target according to the configuration of the receiving and display system.

[0121] The video decoder 204 includes a decoding unit 204b and a decoding unit 204e. The decoding unit 204b performs a decoding process on the basic video stream BS to generate basic format image data Vb'. In this case, the decoding unit 204b performs prediction compensation within the image data Vb'.

[0122] The decoding unit 204e performs a decoding process on the extended video stream ES to generate high-quality format image data Ve'. In this case, the decoding unit 204e performs prediction compensation within the image data Ve' or prediction compensation between the image data Vb' for each coding block in correspondence with the prediction at the time of coding.

[0123] Fig. 22 shows a configuration example of the main part of the decoding unit 204e. This decoding unit 204e performs processing that is the reverse of the processing of the encoding unit 106e in Fig. 4. This decoding unit 204e has a decoding function unit 241, an intra-layer prediction compensation unit 242, an inter-layer prediction compensation unit 243, a prediction adjustment unit 244, and a selection unit 245.

[0124] The decode function unit 241 performs a decoding process other than prediction compensation on the encoded image data CV to obtain prediction residual data. The intra-layer prediction compensation unit 242 performs prediction compensation (intra-layer prediction compensation) on the prediction residual data within the image data V1 to obtain image data V1. The inter-layer prediction compensation unit 243 performs prediction compensation (inter-layer prediction compensation) on the prediction residual data between the image data V2 to be referenced and obtains image data V1.

[0125] Although detailed description is omitted, the prediction adjustment unit 244 performs processing according to the type of scalable extension of the image data V1 to the image data V2, similar to the prediction adjustment unit 163 of the encoding unit 106e in Fig. 4. The selection unit 245 selectively extracts the image data V1 obtained by the intra-layer prediction compensation unit 242 or the image data V1 obtained by the inter-layer prediction compensation unit 243 for each coding block in correspondence with the prediction at the time of coding, and outputs it.

[0126] 21, the decoding unit 204e extracts parameter sets and SEI messages inserted in each access unit constituting the extended video stream ES, and sends them to the control unit 201. The control unit 201 recognizes the high-quality format to which the coded image data of the extended video stream ES corresponds from the "nuh_layer_id" field in the NAL unit header or the "scalable_index" field in the scalable linkage SEI message.

[0127] As described above, the control unit 201 recognizes the type of coding components included in the coded image data of the extended video stream ES from the "nuh_layer_id" field or the "scalable_index" field of the scalable linkage descriptor. However, container-level identification information such as a descriptor cannot keep up with dynamic changes in video frames. By recognizing the type of coding components included in the coded image data of the extended video stream ES from the "nuh_layer_id" field of the NAL unit header or the "scalable_index" field of the scalable linkage SEI message, it is possible to keep up with dynamic changes in frames.

[0128] Furthermore, the control unit 201 recognizes conversion table information for prediction adjustment from the "scalable_index" field of the scalable linkage SEI message. The control unit 201 sets the conversion table in the prediction adjustment unit 244. This makes it possible to reliably cause the prediction adjustment unit 244 to perform level adjustment (dynamic range conversion) similar to that performed by the prediction adjustment unit 163 on the transmitting side.

[0129] The YCbCr / RGB conversion unit 105 converts the basic format image data Vb' obtained by the decoding unit 204b from the YCbCr (luminance / chrominance) domain to the RGB domain. The electro-optic conversion unit 206 performs electro-optic conversion with the reverse characteristics to those of the photoelectric conversion unit 102 in the transmission device 100 described above on the basic format image data Vb' converted to the RGB domain, thereby obtaining basic format image data Vb. This basic format image data is SDR image data with a frame frequency of 50 Hz.

[0130] The YCbCr / RGB conversion unit 207 converts the high-quality format image data Ve' obtained by the decoding unit 204e from the YCbCr (luminance / chrominance) domain to the RGB domain. The electro-optic conversion unit 208 performs electro-optic conversion with the reverse characteristics to those of the photoelectric conversion unit 104 in the transmission device 100 described above on the high-quality format image data Ve' converted to the RGB domain, thereby obtaining the high-quality format image data Ve.

[0131] This high quality format image data is either (a) HDR image data with a frame frequency of 100 Hz (HFR: high frame rate), (b) HDR image data with a frame frequency of 50 Hz (LFR), or (c) SDR image data with a frame frequency of 100 Hz (HFR).

[0132] The display unit 209 is configured with, for example, an LCD (Liquid Crystal Display), an organic EL (Organic Electro-Luminescence) panel, etc. The display unit 209 displays an image based on either the basic format image data Vb or the high-quality format image data Ve, depending on the display capability.

[0133] In this case, the control unit 201 controls image data to be supplied to the display unit 209, that is, image data obtained as display image data. This control is performed based on information on the high-quality format to which the encoded image data of the extended video stream ES corresponds, that is, information on the high-quality format of the high-quality format image data Ve, and display capability information of the display unit 209.

[0134] That is, when the display unit 209 cannot display an image based on the high-quality format image data Ve, the control unit 208 controls so that the basic format image data Vb is supplied to the display unit 209. On the other hand, when the display unit 209 can display an image based on the high-quality format image data Ve, the control unit 208 controls so that the high-quality format image data Ve is supplied to the display unit 209.

[0135] The operation of the receiving device 200 shown in Fig. 21 will be briefly described. The receiving unit 202 receives the transport stream TS transmitted from the transmitting device 100 via broadcast waves or network packets. This transport stream TS is supplied to a system decoder 203. The system decoder 203 extracts a basic video stream BS and an extended video stream ES from this transport stream TS. The basic video stream BS is supplied to a decoding unit 204b. The extended video stream ES is supplied to a decoding unit 204e.

[0136] Furthermore, the system decoder 203 extracts various pieces of information inserted into the transport stream TS as a container, and sends the extracted information to the control unit 201. This information also includes a scalable linkage descriptor (see FIG. 18). The control unit 201 recognizes the high-quality format to which the encoded image data of the extended video stream ES corresponds, based on the "nuh_layer_id" field or the "scalable_index" field of this descriptor.

[0137] In the decoding unit 204b, a decoding process is performed on the basic video stream BS to generate basic format image data Vb'. In this case, in the decoding unit 204b, prediction compensation is performed within the image data Vb'. In the decoding unit 204e, a decoding process is performed on the extended video stream ES to generate high quality format image data Ve'. In this case, in the decoding unit 204e, prediction compensation is performed within the image data Ve' or prediction compensation between the image data Vb' for each coding block in correspondence with the prediction at the time of coding.

[0138] Furthermore, the decoding unit 204e extracts parameter sets and SEI messages inserted in each access unit constituting the extended video stream ES, and sends them to the control unit 201. The control unit 201 recognizes the high-quality format to which the coded image data of the extended video stream ES corresponds on a frame-by-frame basis from the "nuh_layer_id" field in the header of the NAL unit or the "scalable_index" field in the scalable linkage SEI message.

[0139] The basic format image data Vb' obtained by the decoding unit 204b is converted from the YCbCr (luminance / chrominance) domain to the RGB domain by a YCbCr / RGB conversion unit 205, and is supplied to an electric-to-optical conversion unit 206. In the electric-to-optical conversion unit 206, the basic format image data Vb' converted into the RGB domain is subjected to electric-to-optical conversion with reverse characteristics to the optical-to-electrical conversion in the transmission device 100, and the basic format image data Vb is obtained.

[0140] The high-quality format image data Ve' obtained by the decoding unit 204e is converted from the YCbCr (luminance / chrominance) domain to the RGB domain by a YCbCr / RGB conversion unit 207, and is supplied to an electric-to-optical conversion unit 208. In the electric-to-optical conversion unit 208, the high-quality format image data Ve' converted to the RGB domain is subjected to electric-to-optical conversion with reverse characteristics to the optical-to-electrical conversion in the transmission device 100, and the high-quality format image data Ve is obtained.

[0141] The control unit 201 controls image data (display image data) to be supplied to the display unit 209. This control is performed based on information on the high-quality format to which the encoded image data of the extended video stream ES corresponds, that is, information on the high-quality format of the high-quality format image data Ve, and display capability information of the display unit 209.

[0142] When the display unit 209 is unable to display an image using the high-quality format image data Ve, the basic format image data Vb is supplied to the display unit 209. As a result, an image in the basic format (50H, normal dynamic range) is displayed on the display unit 209. On the other hand, when the display unit 209 is able to display an image using the high-quality format image data Ve, the high-quality format image data Ve is supplied to the display unit 209. As a result, an image in the high-quality format is displayed on the display unit 209.

[0143] As described above, in the transmission / reception system 10 shown in Fig. 1, information (high-quality format information) indicating the high-quality format corresponding to the encoded image data of the extended video stream is inserted into the extended video stream and / or the container. Therefore, the receiving side can easily recognize the high-quality format of the high-quality format image data. Then, based on this information and the display capability information, the receiving side can select the basic format image data Vb or the high-quality format image data Ve as the image data for display, and can easily perform image display suited to the display capability.

[0144] <2. Modifications> In the above embodiment, the transmission / reception system 10 including the transmission device 100 and the reception device 200 is shown, but the configuration of the transmission / reception system to which the present technology can be applied is not limited to this. For example, the reception device 200 may be configured as a set-top box and a monitor connected by a digital interface such as HDMI (High-Definition Multimedia Interface). In this case, the set-top box can obtain display capability information by obtaining EDID (Extended display identification data) from the monitor. Note that "HDMI" is a registered trademark.

[0145] In the above embodiment, an example in which the container is a transport stream (MPEG-2 TS) has been shown. However, the present technology can be similarly applied to a system configured to distribute to a receiving terminal using a network such as the Internet. In Internet distribution, MP4 or other format containers are often used. In other words, containers of various formats such as the transport stream (MPEG-2 TS) adopted in the digital broadcasting standard, MMT (MPEG Media Transport) of the next generation transport, and MP4 used in Internet distribution are applicable.

[0146] The present technology can also be configured as follows. (1) an image encoding unit that generates two video streams, a basic video stream having encoded image data of basic format image data and an extended video stream having encoded image data of one type of high quality format image data selected from a plurality of types; a transmitting unit for transmitting a container in a predetermined format including the basic video stream and the extended video stream; an information inserting unit that inserts information indicating a high-quality format to which encoded image data of the video stream corresponds, into the extended video stream and / or the container; Transmitting device. (2) The image encoding unit is With respect to the basic format image data, a predictive coding process is performed within the basic format image data to obtain coded image data; With respect to the high-quality format image data, a predictive coding process is selectively performed within the high-quality format image data or a predictive coding process between the high-quality format image data and the basic format image data to obtain coded image data. The transmitting device according to (1) above. (3) The basic format image data is image data with a normal dynamic range and a low frame rate, the high quality format image data being either high dynamic range and high frame rate image data, high dynamic range and low frame rate image data, or normal dynamic range and high frame rate image data; The encoded image data of the high quality format image data has an encoded component of high dynamic range image data by difference information for normal dynamic range image data and / or an encoded component of high frame rate image data by difference information for low frame rate image data. The transmitting device according to (2) above. (4) The image encoding unit includes: When obtaining difference information for the normal dynamic range image data, dynamic range conversion is performed on the normal dynamic range image data to reduce the difference value. The transmitting device according to (3) above. (5) The image encoding unit includes: The dynamic range conversion is performed on the normal dynamic range image data based on conversion information for converting values ​​of data converted using normal dynamic range photoelectric conversion characteristics into values ​​of data converted using high dynamic range photoelectric conversion characteristics. The transmitting device according to (4) above. (6) The information insertion unit is and further inserting said transformation information into said extended video stream and / or container. The transmitting device according to (5) above. (7) The image encoding unit includes: a time indicated by a decode time stamp added to the coded image data of each picture included in the extended video stream is set to be the same as or intermediate to a time indicated by a decode time stamp added to the coded image data of each picture included in the basic video stream; The time intervals indicated by the decode time stamps added to the coded image data of each picture included in the basic video stream are made equal; The time intervals indicated by the decode time stamps added to the encoded image data of each picture included in the extended video stream are made equal. A transmitting device according to any one of (1) to (6). (8) The extended video stream has a NAL unit structure, The information inserting unit inserts information indicating a high-quality format to which the encoded image data of the extended video stream corresponds into a header of the NAL unit. A transmitting device according to any one of (1) to (7). (9) The extended video stream has a NAL unit structure, The information inserting unit inserts information indicating a high-quality format corresponding to the encoded image data of the extended video stream into an area of ​​the SEI NAL unit. A transmitting device according to any one of (1) to (7). (10) The container is MPEG2-TS, The information inserting unit inserts information indicating a high quality format to which the encoded image data of the extended video stream corresponds into a video elementary stream loop corresponding to the extended video stream existing under a program map table. A transmitting device according to any one of (1) to (9). (11) an image encoding step of generating two video streams, a basic video stream having encoded image data of basic format image data and an extended video stream having encoded image data of image data of one high quality format selected from a plurality of types; a transmitting step of transmitting a container in a predetermined format including the basic video stream and the extended video stream by a transmitting unit; An information inserting step of inserting information representing a high quality format to which encoded image data of the high quality format image data corresponds, into the extended video stream and / or the container. Transmission method. (12) A receiving unit for receiving a container of a predetermined format including two video streams, a basic video stream having encoded image data of basic format image data, and an extended video stream having encoded image data of image data of one high quality format selected from a plurality of types, information indicating a high-quality format to which encoded image data of the high-quality format image data corresponds is inserted into the extended video stream and / or the container; an information extractor for extracting said information from said enhanced video stream and / or container; a processing unit for obtaining image data corresponding to a display capability from the basic video stream and the extended video stream as image data for display based on the extracted information and display capability information. Receiving device. (13) The encoded image data included in the basic video stream is generated by performing a predictive encoding process on the basic format image data, The encoded image data included in the extended video stream is generated by selectively performing a predictive encoding process within the high-quality format image data or a predictive encoding process between the high-quality format image data and the basic format image data on the high-quality format image data. The receiving device according to (12) above. (14) A receiving step of receiving, by a receiving unit, a container of a predetermined format including two video streams, a basic video stream having encoded image data of basic format image data, and an extended video stream having encoded image data of image data of one high quality format selected from a plurality of types, information indicating a high-quality format to which encoded image data of the high-quality format image data corresponds is inserted into the extended video stream and / or the container; an information extraction step for extracting said information from said enhanced video stream and / or from a container; The method further includes a processing step of obtaining image data corresponding to the display capability from the basic video stream and the extended video stream as image data for display based on the extracted information and display capability information. Receiving method.

[0147] The main feature of this technology is that it enables the receiving side to easily recognize the high-quality format that the encoded image data in the extended video stream corresponds to by inserting information indicating the high-quality format that the encoded image data in the extended video stream corresponds to into the extended video stream and / or container (see Figure 20).

[0148] 10. Transmitting and receiving system 100 Transmitting device 101...Control section 102, 104: Photoelectric conversion unit 103, 105 RGB / YCbCR conversion section 106 Video Encoder 106b, 106e Encoding section 107 System Encoder 108 Transmitter 150 Image data generating unit 151···Camera 152...Conversion unit 161... Intra-layer prediction unit 162... Inter-layer prediction unit 163 Forecast Adjustment Section 164...Selection section 165 Encoding function section 200 Receiving device 201...Control section 202 Receiving unit 203 System Decoder 204: Video decoder 204b, 204e: Decoder 205, 207...YCbCr / RGB conversion section 206, 208...Electro-optical conversion unit 209...Display section 241 Decoding function unit 242... Intra-layer prediction compensation unit 243... Inter-layer prediction compensation unit 244... Forecast Adjustment Section 245...Selection section

Claims

1. an image encoding step of generating two video streams, a basic video stream having encoded image data of basic format image data and an enhanced video stream having encoded image data of high frame rate image data; a transmitting step of transmitting the basic video stream and the extended video stream; In the image encoding step, a time interval indicated by a decode time stamp added to the coded image data of each picture included in the basic video stream is set to an interval corresponding to a frame rate of the basic video stream; The time indicated by the decode time stamp added to the coded image data of each picture included in the extended video stream is set to the middle of the time indicated by the decode time stamp added to the coded image data of each picture included in the basic video stream. Transmission method.

2. an image encoding unit that generates two video streams, a basic video stream having encoded image data of basic format image data and an extended video stream having encoded image data of high frame rate image data; a transmitter for transmitting the basic video stream and the extended video stream, The image encoding unit is a time interval indicated by a decode time stamp added to the coded image data of each picture included in the basic video stream is set to an interval corresponding to a frame rate of the basic video stream; The time indicated by the decode time stamp added to the coded image data of each picture included in the extended video stream is set to the middle of the time indicated by the decode time stamp added to the coded image data of each picture included in the basic video stream. Transmitting device.

Images