Apparatus and method for dynamic range conversion of an image

Abstract

An image processing apparatus comprises a receiver (201) for receiving an image signal, the image signal comprising at least an encoded image and a target display reference. The target display reference indicates the dynamic range of a target display for which the encoded image is encoded. A dynamic range processor (203) generates an output image by applying a dynamic range transform to the encoded image in response to the target display reference. The output (205) then outputs an output image signal comprising the output image, for example, to a suitable display. Furthermore, the dynamic range transform may be performed in response to a display dynamic range indication received from the display. The present invention may be used to generate an improved high dynamic range (HDR) image from, for example, a low dynamic range (LDR) image, or vice versa.

Description

Technical Field

[0001] The present invention relates to dynamic range conversion for images, and more particularly, but not exclusively, to image processing for generating a high dynamic range image from a low dynamic range image or generating a low dynamic range image from a high dynamic range image. Background Art

[0002] Digital coding of various source signals has become increasingly important over the past few decades as digital signal representation and communication increasingly replaces analog representation and communication. Continuous research and development is underway to improve the quality that can be obtained from coded image and video sequences while maintaining the data rate at an acceptable level.

[0003] An important factor in perceived image quality is the dynamic range that can be reproduced when displaying an image. Conventionally, the dynamic range of reproduced images tends to be significantly reduced relative to normal vision. In fact, luminance levels encountered in the real world span a dynamic range of up to 14 orders of magnitude, from a moonless night to looking directly at the sun. On a clear day or at night, the instantaneous luminance dynamic range and the corresponding human visual system response can fall between 10,000:1 and 100,000:1 (bright reflections versus dark shadow regions). Traditionally, the dynamic range of displays has been limited to approximately 2-3 orders of magnitude, and sensors also have limited ranges, such as <10,000:1, depending on noise tolerance. Therefore, it has traditionally been possible to store and transmit images in an 8-bit gamma-encoded format without introducing noticeable artifacts on conventional reproduction devices. However, in the effort to record more accurate and vivid images, novel high dynamic range (HDR) image sensors have been developed that are capable of recording dynamic ranges exceeding 6 orders of magnitude. Furthermore, most special effects, computer graphics enhancements, and other post-production work are already routinely performed at higher bit depths and with higher dynamic ranges.

[0004] Furthermore, the contrast ratio and peak luminance of state-of-the-art display systems continue to increase. Recently, new prototype displays have been proposed with luminances as high as 3000 Cd / m 2 HDR images are often rendered in 8-bit RGB, but the HDR image quality is often more complex than 8-bit images. The ...

[0005] As a result, there is a growing demand for new methods that allow consumers to fully benefit from the capabilities of the latest (and future) sensor and display systems. Preferably, the representation of such additional information is backwards compatible, so that older equipment can still receive the ordinary video stream, while new HDR-enabled devices can take full advantage of the additional information conveyed by the new format. Therefore, it is desirable that the encoded video data not only represent HDR images, but also allow the encoding of corresponding traditional low dynamic range (LDR) images that can be displayed on conventional equipment.

[0006] In order to successfully introduce HDR systems and fully exploit the promise of HDR, it is important that the approach taken provides backward compatibility and allows optimization or at least adaptation to HDR displays. However, this inherently involves a conflict between optimization for HDR and optimization for traditional LDR.

[0007] For example, typically, image content such as video clips will be processed (color grading and tone mapping) in the studio to best appear on a specific display. Traditionally, such optimization has been performed for LDR displays. For example, during the production of standard LDR displays, color grading experts will balance many picture quality aspects to create the desired "look" for the storyline. This can involve balancing blocks and local contrast, and sometimes even intentionally shearing pixels. For example, on a display with relatively low peak brightness, explosions or bright highlights are often severely sheared to convey the impression of high brightness to the viewer (the same thing happens for dark shadow details on a display with a poor black level). This operation will typically be performed by assuming a nominal LDR display, and traditionally, displays deviate from such nominal LDR displays relatively rarely, because in fact almost all consumer displays are LDR displays.

[0008] However, if the film is adapted for an HDR target display, the results will be very different. In fact, color experts will perform optimizations that will result in very different code mappings. For example, not only may highlight and shadow detail be better preserved on an HDR display, but they may also be optimized to have a different distribution across mid-gray tones. Therefore, an optimal HDR image is not achieved by simply scaling the LDR image by a value corresponding to the difference in white point luminance (maximum achievable brightness).

[0009] Ideally, separate color grading and tone mapping would be performed for each possible dynamic range of the display. For example, a video sequence would be used for 500 Cd / m 2 The maximum white point illuminance is 1000Cd / m 2 , one for 1500Cd / m 2, and so on, up to the maximum possible brightness. A given display can then simply select the video sequence that corresponds to its brightness. However, such an approach is impractical because it requires the generation of a large number of video sequences, thereby increasing the resources required to generate these different video sequences. In addition, the required storage and distribution capacity would be greatly increased. Furthermore, this approach limits the possible maximum display brightness level to discrete levels, thereby providing suboptimal performance for displays with maximum display brightness levels between the levels for which the video sequences are provided. Furthermore, such an approach would not allow future displays to be developed with maximum brightness levels higher than the highest brightness level video sequence to be utilized.

[0010] Accordingly, it is desirable to create only a limited number of video sequences on the content provider side, and to apply automatic dynamic range conversion to such video sequences at a later point in the distribution chain in order to generate a video sequence suitable for the specific display on which the video sequence is to be reproduced. However, in such an approach, the resulting image quality is highly dependent on the automatic dynamic range conversion.

[0011] Hence, an improved method of supporting different dynamic ranges for images, and preferably for supporting different dynamic range images, would be advantageous. Summary of the Invention

[0012] Accordingly, the Invention seeks to preferably mitigate, alleviate or eliminate one or more of the above mentioned disadvantages singly or in any combination.

[0013] According to one aspect of the present invention, there is provided an image processing apparatus comprising: a receiver for receiving an image signal, the image signal comprising at least a first encoded image and a first target display reference, the first target display reference indicating a dynamic range of a first target display for which the first encoded image is encoded; a dynamic range processor arranged to generate an output image by applying a dynamic range transform to the first encoded image in response to the first target display reference; and an output for outputting an output image signal comprising the output image.

[0014] The present invention can allow a system to support different dynamic range images and / or displays. In particular, the method can allow for improved dynamic range conversion that can be adapted to the specific characteristics of image reproduction. In many scenarios, an improved dynamic range conversion from LDR to HDR images or from HDR to LDR can be achieved.

[0015] In some embodiments, the dynamic range transform increases the dynamic range of the output video signal relative to the first encoded picture. In some embodiments, the dynamic range transform decreases the dynamic range of the output video signal relative to the first encoded picture.

[0016] The dynamic range corresponds to the range of reproduced luminances, i.e. the range from the minimum to the maximum light output for the reproduced image. Therefore, the dynamic range is not just a ratio of maximum to minimum values ​​or a quantitative measure (e.g. the number of bits), but corresponds to the actual luminance range for image reproduction. Thus, the dynamic range can be expressed, for example, in candelas per square meter (cd / m²), also known as nits. 2 ) measured using a luminance value. The dynamic range is therefore the range of luminance values ​​from the light output (brightness) corresponding to the lowest luminance value (often assumed to be absolute black, i.e., no light output) to the light output (brightness) corresponding to the highest luminance value. The dynamic range can be characterized in particular by the highest light output value, also known as the white point, white point luminance, white luminance, or maximum luminance. For LDR images and LDR displays, the white point is typically 500 nits or less.

[0017] In particular, the output image signal may be fed to a display having a certain dynamic range, and thus the dynamic range transform may convert the encoded image from the dynamic range dictated by the target display reference to the dynamic range of the display on which the image is to be reproduced.

[0018] The image may be an image of a motion picture sequence such as a frame, or an image of a video sequence. As another example, the image may be a permanent background, or an overlay image such as a graphic or the like.

[0019] The first encoded image may in particular be an LDR image and the output image may be an HDR image.The first encoded image may in particular be an HDR image and the output image may be an LDR image.

[0020] In accordance with an optional feature of the invention, the first target display reference comprises a white point luminance of the first target display.

[0021] This may provide advantageous operation in many embodiments. In particular, it may allow for low complexity and / or low overhead while providing sufficient information to allow an improved dynamic range transform to be performed.

[0022] In accordance with an optional feature of the invention, the first target display reference comprises an indication of an electro-optical transfer function for the first target display.

[0023] This can provide advantageous operation in many embodiments. In particular, it can allow for low complexity and / or low overhead while providing sufficient information to allow for an improved dynamic range transform. In particular, the approach can allow for the dynamic range transform to also be adapted for specific characteristics such as mid-range illuminance. For example, it can allow for the dynamic range transform to account for differences in gamma between the target display and the end-user display.

[0024] In accordance with an optional feature of the invention, the first target display reference comprises a tone mapping indication representing a tone mapping used to generate the first encoded image for the first target display.

[0025] This may in many scenarios allow an improved dynamic range transform to be performed, and in particular may allow the dynamic range transform to compensate for specific characteristics of the tone mapping performed on the content creation side.

[0026] In some scenarios, the image processing device can therefore take into account both the characteristics of the display for which the encoded image is optimized and the characteristics of the particular tone mapping. This can, for example, allow subjective and, for example, artistic, tone mapping decisions to be taken into account when converting an image from one dynamic range to another.

[0027] In accordance with an optional feature of the invention, the image signal further comprises a data field comprising dynamic range transform control data; and the dynamic range processor is further arranged to perform a dynamic range transform in response to the dynamic range transform control data.

[0028] This may provide improved performance and / or functionality in many systems. In particular, it may allow local and targeted adaptation to a particular dynamic range display, while still allowing the content provider side to retain some control over the resulting image.

[0029] The dynamic range transform control data may comprise data specifying characteristics of a dynamic range transform that must and / or may be applied, and / or it may specify recommended characteristics of the dynamic range transform.

[0030] According to an optional feature of the invention, the dynamic range transform control data comprises different dynamic range transform parameters for different display maximum illumination levels.

[0031] This may provide improved control and / or adaptation in many embodiments.In particular, it may allow the image processing device 103 to select appropriate control data and apply that data to the particular dynamic range for which the output image is generated.

[0032] According to an optional feature of the invention, the dynamic range transform control data includes different tone mapping parameters for different display maximum illuminance levels, and the dynamic range processor is arranged to determine the tone mapping parameters for the dynamic range transform in response to the maximum illuminance for the output image signal and the different tone mapping parameters.

[0033] This can provide improved control and / or adaptation in many embodiments. In particular, it can allow the image processing device 103 to select appropriate control data and apply that data to the specific dynamic range for which the output image is generated. The tone mapping parameters can in particular provide parameters that must, can, or are recommended for dynamic range transformation.

[0034] In accordance with an optional feature of the invention, the dynamic range transform control data comprises data defining a set of transform parameters that must be applied by the dynamic range transform.

[0035] This allows the content provider to maintain control over the images rendered on the displays supported by the image processing device. This can ensure uniformity between different rendering scenarios. This approach can, for example, allow the content provider to ensure that the artistic impression of the image remains relatively unchanged when rendered on different displays.

[0036] In accordance with an optional feature of the invention, the dynamic range transform control data comprises data defining limits for transform parameters applied by the dynamic range transform.

[0037] This can provide improved operation and an improved user experience in many embodiments. In particular, it can allow, in many scenarios, an improved compromise between the desire of the content provider to retain control over the rendering of his / her content while allowing the end user to customize it according to his / her preferences.

[0038] According to an optional feature of the invention, the dynamic range transform control data comprises different transform control data for different image classes.

[0039] This can provide improved transformed images in many scenarios. In particular, it can allow the dynamic range transform to be optimized for the characteristics of different images. For example, different dynamic range transforms can be applied to images corresponding to the main image, images corresponding to graphics, images corresponding to the background, and so on.

[0040] According to an optional feature of the invention, the maximum luminance of the dynamic range of the first target display is no less than 1000 nits.

[0041] The image to be transformed may be an HDR image. The dynamic range transformation may transform such an HDR image into another HDR image having a different dynamic range (which is associated with a display having a dynamic range of no less than 1000 nits). Thus, improved image quality may be achieved by converting an HDR image of one dynamic range into another HDR image of another dynamic range (which may have a higher or lower white point luminance).

[0042] According to an optional feature of the invention, the image signal comprises a second encoded image and a second target display reference, the second target display reference indicating a dynamic range of a second target display for which the second encoded image is encoded, the dynamic range of the second target display being different from the dynamic range of the first target display; the dynamic range processor being arranged to apply a dynamic range transform to the second encoded image in response to the second target display reference.

[0043] This can allow for improved output quality in many scenarios. In particular, different transforms can be applied to the first encoded image and to the second encoded image depending on the differences in the associated target displays (and typically depending on how each of them relates to the desired dynamic range of the output image).

[0044] In accordance with an optional feature of the invention, the image dynamic range processor is arranged to generate the output image by combining the first encoded image and the second encoded image.

[0045] This may provide improved image quality in many embodiments and scenarios.In some scenarios, the combination may be a selection combination, where the combination is performed simply by selecting one of the images.

[0046] In accordance with an optional feature of the invention, the image processing apparatus further comprises: a receiver for receiving a data signal from a display, the data signal comprising a data field comprising a display dynamic range indication of the display, the display dynamic range indication comprising at least one luminance specification; and the dynamic range processor being arranged to apply a dynamic range transform to the first encoded image in response to the display dynamic range indication.

[0047] This may allow for improved image reproduction in many embodiments.

[0048] In accordance with an optional feature of the invention, the dynamic range processor is arranged to select between generating the output image as a first encoded image responsive to a first target display reference and generating the output image as a transformed image of the first encoded image.

[0049] This can allow for improved image reproduction and / or reduced computational load in many embodiments. For example, if the end-user display has a dynamic range very close to the dynamic range for which the encoded image was generated, improved quality of the reproduced image will typically be achieved by using the received image directly. However, if the dynamic range is sufficiently different, improved quality can be achieved by processing the image to adapt it to the different dynamic range. In some embodiments, the dynamic range transform can simply be adapted to switch between a no-op (using the first encoded image directly) and applying a predetermined and fixed dynamic range transform if the target display reference is sufficiently different from the end-user display.

[0050] According to an optional feature of the invention, the dynamic range transform comprises a color gamut transform.

[0051] This can allow for the generation of improved output images in many embodiments and scenarios. In particular, it can allow for perceived improved color reproduction and can, for example, compensate for changes in color perception caused by changes in the brightness of image regions. In some embodiments, the dynamic range transform can consist in a color gamut transform.

[0052] According to an optional feature of the invention, the image processing apparatus further comprises a control data transmitter for transmitting dynamic range control data to a source of the image signal.

[0053] This may allow the source to adapt the image signal in response to dynamic range control data. The dynamic range control data may in particular include an indication of a preferred dynamic range for the image and / or an indication of the dynamic range for the end-user display (e.g. white point luminance and optionally EOTF or gamma function).

[0054] According to one aspect of the present invention, an image signal source device is provided, comprising: a receiver for receiving an encoded image; a generator for generating an image signal, the image signal comprising the encoded image and a target display reference, the target display reference indicating a dynamic range of a target display for which the encoded image is encoded; and a transmitter for transmitting the image signal.

[0055] According to one aspect of the present invention, there is provided an image processing method, comprising:

[0056] receiving an image signal comprising at least a first encoded image and a first target display reference indicating a dynamic range of a first target display for which the first encoded image is encoded;

[0057] generating an output image by applying a dynamic range transform to the first encoded image in response to a first target display reference; and

[0058] An output image signal including an output image is output.

[0059] According to one aspect of the present invention, a method for sending an image signal is provided, the method comprising: receiving an encoded image; generating an image signal comprising the encoded image and a target display reference, the target display reference indicating a dynamic range of a target display for which the encoded image is encoded; and sending the image signal.

[0060] According to one aspect of the present invention, there is provided an image signal comprising at least a first encoded image and a first target display reference indicating a dynamic range of a first target display for which the first encoded image is encoded.

[0061] These and other aspects, features and advantages of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings, in which

[0063] Figure 1 is a diagram of an example of elements of an image reproduction system according to some embodiments of the present invention;

[0064] Figure 2 is a diagram of an example of elements of an image processing device;

[0065] Figure 3 An example of mapping for an image processing device is shown;

[0066] Figure 4 Figure 1 illustrates an example of an electro-optical transfer function (EOTF) for a display;

[0067] Figure 5 Blu-ray TM An instance of a model used to render a plane in standard HDMV-2D mode;

[0068] Figure 6 Figure 1 illustrates an example of dynamic range processing for HDR and LDR images;

[0069] Figure 7 An example of mapping for an image processing device is shown;

[0070] Figure 8-10 illustrates examples of images utilizing different dynamic range transforms when presented on the same display;

[0071] Figure 11illustrates an example of the relationship between possible mappings and luminance values ​​for an image processing device;

[0072] Figure 12 An example of mapping for an image processing device is shown;

[0073] Figure 13 An example of mapping for an image processing device is shown;

[0074] Figure 14 The diagram shows the blue light TM Standard graphics stream structure;

[0075] Figure 15 illustrates an example of dynamic range processing for an image and associated overlay graphic image;

[0076] Figure 16 Figure 1 illustrates an example of dynamic range processing for images and graphics;

[0077] Figure 17 is a diagram of an example of elements of an image processing device;

[0078] Figure 18 An example of mapping for an image processing device is shown;

[0079] Figure 19 is a diagram of an example of elements of an image processing device;

[0080] Figure 20 An example of mapping for an image processing device is shown;

[0081] Figure 21 is a diagram illustrating examples of elements of a display according to some embodiments of the present invention;

[0082] Figure 22 is a diagram of an example of elements of an image processing apparatus; and

[0083] Figure 23 The generation of an 8-bit image for encoding an HDR image by means of an encoding device is schematically illustrated. DETAILED DESCRIPTION

[0084] Figure 1The figure shows an example of an audiovisual distribution path. In this example, a content provider device 101 generates an audiovisual content signal for an audiovisual content item such as a movie, a television program, or the like. In particular, the content provider device 101 can encode the audiovisual content according to an appropriate encoding format and color representation. In particular, the content provider device 101 can encode the images of the video sequence of the audiovisual content item according to an appropriate representation such as YCrCb. The content provider device 101 can be considered to represent the production and distribution studio that creates and broadcasts the content.

[0085] The audiovisual content signal is then distributed to the image processing device 103 via a distribution path 105. The image processing device 103 may be, for example, a set-top box residing at a specific consumer of the content item, such as, for example, a personal video recorder, a Blu-ray player, or a similar device. TM players, network (e.g. Internet) streaming devices, satellite or terrestrial TV receivers, etc.

[0086] The audiovisual content is encoded and distributed via a medium, which may for example comprise an encapsulated medium or a communication medium, from the content provider apparatus 101. It then reaches a source device in the form of an image processing device 103, which comprises functionality for decoding and playing back the content.

[0087] It should be understood that the distribution path 105 can be any distribution path and can be via any medium or use any suitable communication standard. In addition, the distribution path need not be real-time, but can include permanent or temporary storage devices. For example, the distribution path can include the Internet, satellite, cable or terrestrial broadcast, mobile or fixed communication networks, etc., or a DVD or Blu-ray disc. TM Or storage on physical distribution media such as memory cards, etc.

[0088] The image processing device 103 is coupled to a display 107 via a communication path 109. The image processing device 103 generates a display signal representing the audiovisual content item. Thus, the source device streams the decoded content to a sink device, which may be a television or another device that converts a digital signal into a physical representation.

[0089] The image processing device 103 may perform, for example, image enhancement or signal processing algorithms on the data and may in particular decode and re-encode the (processed) audiovisual signal. The re-encoding may in particular be a different encoding or representation format than the received signal.

[0090] Figure 1The system is arranged in some embodiments to provide high dynamic range (HDR) video information to display 107, and in other embodiments or scenarios to provide low dynamic range (LDR) images to display 107. Furthermore, to provide, for example, improved backward compatibility, in some scenarios it may be capable of providing both LDR and HDR images, depending on the display on which they are displayed. In particular, the system is capable of transmitting / distributing image signals relating to both LDR and HDR images.

[0091] Conventional displays typically use LDR representations. Typically, such LDR representations are provided via a three-component 8-bit representation associated with defined primary colors. For example, an RGB color representation can be provided via three 8-bit samples that specifically reference the red, green, and blue primaries. Another representation uses a luma component and two chroma components (e.g., YCrCb). These LDR representations correspond to a given brightness or illuminance range.

[0092] HDR in particular allows for the appropriate presentation of much brighter images (or image regions) on an HDR display. In fact, an HDR image displayed on an HDR display can provide a much brighter white than a corresponding LDR image presented on an LDR display. In fact, an HDR display can allow for a white that is typically at least four times brighter than an LDR display. Brightness can in particular be measured relative to the darkest black that can be represented, or can be measured relative to a given grayscale or black level.

[0093] The LDR image may in particular correspond to specific display parameters such as a fixed bit resolution associated with a specific set of primary colors and / or a specific white point. For example, a fixed bit resolution may be used for a given set of RGB primary colors and, for example, 500 Cd / m 2 HDR images are images that include data that should be reproduced beyond these limitations. In particular, the brightness can be four times brighter than this white point (e.g., 2000 Cd / m²) or more.

[0094] High dynamic range pixel values ​​have a luminance contrast range (the brightest luminance in a pixel set divided by the darkest luminance) that is (much) larger than the range that can be faithfully displayed on displays standardized during the NTSC and MPEG-2 eras (with their typical RGB primaries and D65 white, with a maximum drive level of [255, 255, 255] and a reference luminance of, for example, 500 nits or less). Typically, for such reference displays, 8 bits are sufficient to display all grayscale values ​​between approximately 500 nits and approximately 0.5 nits in visually small steps (i.e., with a contrast range of 1000:1 or less), while HDR images are encoded with higher bit words, such as 10 bits (and are also captured by cameras with larger well depths and DACs (e.g., 14 bits)). In particular, HDR images typically contain many pixel values ​​(of bright image objects) above scene white. In particular, several pixels are brighter than twice the scene white. This scene white can typically be equivalent to the white of an NTSC / MPEG-2 reference display.

[0095] The number of bits, X, used for an HDR image may typically be greater than or equal to the number of bits, Y, used for an LDR image (X may typically be, for example, 10 or 12 or 14 bits (per color channel, if several channels are used), and Y may be, for example, 8 or 10 bits). A transformation / mapping may be required to make the pixels fit into a smaller range, e.g., compression scaling. Typically, non-linear transformations may be involved, e.g., logarithmic encoding can encode a much larger range of luminances (like brightness) with X-bit words than a linear encoding, even if the steps in luminance difference from one value to the next are then not equidistant, but this is not required for the human visual system.

[0096] It should be noted that the difference between LDR and HDR images isn't simply that a larger number of bits are used for HDR images compared to LDR images. Rather, HDR images cover a wider luminance range than LDR images and typically have a higher maximum luminance value, i.e., a higher white point. In fact, while LDR images have a maximum luminance (white) point corresponding to no more than 500 nits, HDR images have a maximum luminance (white) point corresponding to over 500 nits and typically no less than 1000 nits, 2000 nits, or even 4000 nits or higher. Therefore, HDR images don't just use more bits due to higher granularity or improved quantization, but rather due to a larger actual luminance range. Consequently, the brightest possible pixel value typically corresponds to a luminance / light output that is higher for HDR images than for LDR images. In fact, HDR and LDR images can use the same number of bits, but HDR image values ​​are referenced to a larger luminance dynamic range / brighter maximum luminance than LDR image values ​​(and, therefore, HDR images are represented with a coarser quantization on the luminance scale).

[0097] Ideally, the content provided by the content provider apparatus 101 would be captured and encoded with reference to a luminance range that matches the luminance range of the display 107. However, in a practical system, the content may be rendered on many different displays having many different characteristics and / or may be encoded according to a standard based on a luminance range that is different from the luminance range of the particular display 107. Furthermore, the content may not have been originally captured by a capture device or method that exactly matches the luminance range of the display.

[0098] Accordingly, supporting HDR in a content system typically requires some kind of transformation or conversion between different illuminance ranges. For example, if an LDR image is received and should be presented on an HDR display, a conversion from LDR to HDR should be performed. If an HDR image is received and should be presented on an LDR display, a conversion from HDR to LDR should be performed. Such a conversion is typically quite complex and is not simply equivalent to a simple scaling of the illuminance range, as such scaling would result in an image that is perceived as looking unnatural. Typically, quite complex transformations are used, and these transformations often involve the use of the term tone mapping.

[0099] In principle, such illumination transformations can be performed at three different places in the content distribution system.

[0100] One option is to perform it at the content provider device 101. Typically, this can allow the same luminance transformation operation to be distributed to multiple displays, allowing a single transformation to be used for many users. This can allow, for example, complex, manual, and resource-intensive tone mapping to be performed and justified by a skilled tone mapping expert. In fact, this can provide an image that is subjectively optimized for a given luminance range, often referred to as artistic tone mapping. However, this approach is very resource-intensive and is not feasible for application to many displays. Furthermore, a separate image stream is required for each supported luminance range, resulting in very high communication resource requirements that are impractical for many systems.

[0101] Another option is to perform the illumination transformation in the image processing device 103. However, since the average user is not proficient in illumination transformations and since the effort required makes manual adaptation impractical (particularly for moving images such as video clips, movies, etc.), the transformation should preferably be automatic. However, such transformations conventionally do not provide an optimal image. In particular, the optimal transformation may depend on the specific type of content, the expected characteristics of the image (e.g., different transformations may be appropriate for a scene that is expected to be dark and menacing and for a scene that is only expected to be dark to indicate a night scene). Furthermore, content creators may be concerned about the potential impact of such automatic transformations and may not want to lose control over how the content may be presented in different scenarios. Furthermore, the optimal transformation typically depends on the exact characteristics of the display 107, and transformations based on an assumed, nominal, or standard display will typically result in a suboptimal transformation.

[0102] Possibly, this transformation can also be performed in the display 107 .

[0103] exist Figure 1 In a system, the image processing device 103 includes functionality for performing a luminance dynamic range transform on an image (or a set of images, such as a video sequence) received from the content processing device 103 to increase its dynamic range. Specifically, the image processing device 103 receives an image from the content provider apparatus 101 and then processes the image to generate an image with a higher dynamic range. In particular, the received image may be an LDR image, which is converted into an HDR image by applying a luminance dynamic range transform to increase the dynamic range. The transformed image may then be output to the display 107, which is an HDR display, thereby converting the originally received LDR image into a reproduced HDR image. The dynamic range transform may map luminance values ​​of (at least a portion of) an input image associated with one dynamic range to luminance values ​​for (at least a portion of) an output image associated with a different dynamic range.

[0104] In another scenario, the image processing device 103 may receive an image from the content provider apparatus 101 and then process the image to generate an image with a lower dynamic range. In particular, the received image may be an HDR image that is converted into an LDR image by applying a luminance dynamic range transform to reduce the dynamic range. The transformed image may then be output to the display 107, which is an LDR display, resulting in the originally received HDR image being converted into a reproduced LDR image.

[0105] exist Figure 1 In the system, the dynamic range transform is adaptively adjusted based on information received from the content provider apparatus 101 and / or the display 107. Therefore, in the system, the dynamic range transform is not just an operation performed locally in the image processing device 103, but may also depend on characteristics, properties or information from the content provider apparatus 101 and / or the display 107.

[0106] First, the description will be made with reference to a case where the dynamic range transform is based on information supplied from the content provider apparatus 101 to the image processing device 103. Figure 1 system.

[0107] Figure 2 The diagram shows Figure 1 An example of an element of the image processing device 103.

[0108] The image processing device 103 includes a receiver 201 that receives an image signal from the content provider apparatus 101. The image signal includes one or more encoded images. In many scenarios, the image signal may be a video signal comprising an encoded video sequence (i.e., a sequence of images). It should be understood that any suitable encoding of the image may be used, including, for example, JPEG image encoding, MPEG video encoding, etc. The encoded image is represented by pixel values, which, for each pixel of the image, represent the corresponding light output for that pixel (or for an individual color channel sub-pixel). These pixel values ​​may be provided in accordance with any suitable color representation, such as, for example, RGB, YUV, etc.

[0109] In addition, the image signal includes a target display reference that indicates the dynamic range of the target display for which the first encoded image is encoded. Thus, the target display reference provides a reference for the encoded image that reflects the dynamic range for which the received image is constructed. The target display reference may indicate the luminance for which the tone mapping at the content provider device 101 is designed and, in particular, optimized.

[0110] Therefore, the content provider device 101 is arranged to generate such an image signal, which includes not only the encoded image itself, but also a target display reference representing the dynamic range of the display for which the encoded signal is generated. In particular, the content provider device 101 can receive the encoded image from an internal or external source. For example, the image can be provided as a result of artificial tone grading, which optimizes the encoded image for a specific display. In addition, the content provider device 101 can obtain information about the display that has been used for optimization, for example, via display information automatically transmitted to the content provider device 101 from a specific display (for example, the content provider device 101 can also include the functionality required to support artificial tone mapping and can be connected to the target / reference display for the tone mapping). As another example, the encoded tone-mapped image can be received on a medium on which the properties of the associated display are also stored. As yet another example, the content provider device 101 can receive information about the characteristics of the target display through manual user input.

[0111] In response to such information, the content provider apparatus 101 may generate an image signal comprising an encoded image and a target display reference indicating the dynamic range of the target display for tone mapping. For example, a data value corresponding to an identification of the white point luminance and, optionally, an electro-optical transfer function corresponding to an identification of the target display may be included in the image signal by the content provider apparatus 101.

[0112] Furthermore, the image processing device 103 includes a dynamic range processor 203 that applies a dynamic range transform to the received encoded image to generate an output image with a higher dynamic range, i.e., corresponding to a larger output luminance range when the image is reproduced. Specifically, the input encoded image may be an image encoded for an LDR display with a maximum luminance white point of 500 nits, and it may be transformed into an HDR output image with a maximum luminance white point of, for example, 1000 or 2000 nits. Typically, the dynamic range transform also increases the number of bits used to represent each value, but it should be understood that this is not essential, and in some embodiments, the same number of bits (or indeed, even fewer bits) may be used for the output image compared to the input image. As another example, the input encoded image may be an image encoded for an HDR display with a maximum white point luminance of 2000 nits, and it may be transformed into an LDR output image with a maximum white point luminance of, for example, 500 nits. Such a dynamic range reduction transform may also include a reduction in the number of bits used for pixel values.

[0113] The dynamic range transform is performed in response to the target display reference and can therefore be adapted to take into account not only the desired output luminance range, but also the luminance range for which the received image was encoded. For example, the system can adapt the dynamic range transform so that the transform for generating an output image for 1000 nits will be different depending on whether the input image was generated for 300 nits or 500 nits. This can result in a significantly improved output image.

[0114] In fact, in some embodiments, the input image itself may be an HDR image, such as, for example, a 1000 nit image. The optimal conversion of such an image to a 2000 nit image and a 5000 nit image, respectively, will typically be different, and the provision of a target display reference may allow the image processing device 103 to optimize the dynamic range conversion for this specific situation, thereby providing a much improved image for the specific display characteristics. In fact, if the display is a 500 nit display, then the dynamic range conversion should perform dynamic range compression rather than expansion.

[0115] These methods can be particularly advantageous in heterogeneous content distribution systems, such as are increasingly being considered for future television systems. Indeed, the (peak) brightness of HDR LCD / LED TVs is currently increasing rapidly, and in the near future, displays with a wide variety of (peak) brightness are expected to coexist on the market. Brighter images look better on TV screens, and brighter TVs sell better in stores. On the other hand, "low-end" displays in laptops, tablets, and smartphones are also becoming very popular and are also used to reproduce, for example, television content.

[0116] Since the display brightness (and typically the electro-optical transfer function that dictates how the display converts input pixel (color) drive values ​​into light values ​​(which then provide a specific psycho-visual impression to the viewer)) is no longer known on the content generation side (and is moreover typically different from the reference monitor for which the content was intended / graded), providing the best / optimal picture quality on the display becomes challenging. Furthermore, while there may have been some variation in display brightness in the past, this variation was relatively minor, and the known assumption of fixed brightness did not introduce significant degradation (and could often be manually compensated for by the user, for example, by setting the brightness and / or contrast of the display).

[0117] However, due to the dramatic increase in the variety of displays (smartphones, tablets, laptops, PC monitors, CRT displays, conventional LCD TV displays, and bright HDR displays), the characteristics of displays used for reproduction (especially brightness and contrast) show a huge variation. For example, the contrast and peak luminance of state-of-the-art high-end display systems are constantly increasing, and displays with brightness levels as high as 5000 cd / m² have been developed. 2 On the other hand, displays used in, for example, smartphones and tablet computers are becoming increasingly popular, but have relatively low performance characteristics.

[0118] As mentioned previously, content such as video for movies and the like is processed on the content creation side in order to provide the desired reproduced image. For example, when a video is produced for general distribution (e.g., via DVD or Blu-ray) TM When releasing a film, producers / studios typically adapt and customize the image to look best on a specific display. This process is often referred to as color grading and tone mapping. Tone mapping can be thought of as a nonlinear mapping of the luminance value of an input pixel to the luminance value of an output pixel. Tone mapping is performed to match the video to the characteristics of the display, viewing conditions, and subjective preferences. In the case of local tone mapping, the process varies depending on the location of the pixel within the image. In the case of global tone mapping, the same process is applied to all pixels.

[0119] For example, when converting content suitable for general consumer distribution, tone mapping is often performed to provide the desired output on a standard LDR display. This can be performed manually by a color grading expert who balances many aspects of picture quality to create the desired "look" for the storyline. This may involve balancing block-by-block and local contrast, and sometimes even intentionally shearing pixels. Therefore, tone mapping at this stage is typically not just a simple automated conversion, but is typically a manual, subjective, and often artistic conversion.

[0120] If content is graded for an HDR target display rather than an LDR target display, the tone mapping results will typically be very different. Consequently, when video content encoded for an LDR display is reproduced solely on an HDR display, the resulting image will be significantly different from the optimal image. Similarly, if an HDR-optimized image is reproduced solely on an LDR display, significant degradation in perceived image quality may occur.

[0121] This problem is Figure 1In the system of FIG107 , the dynamic range transform is solved by a dynamic range transform that is performed in the image processing device 103 but is based on information preferably received from both the content provider apparatus 101 and the display 107. In this way, the dynamic range transform (in particular the tone mapping algorithm) can be adapted to take into account the characteristics of the tone mapping performed in the content provider apparatus 101 and to the specific luminance range of the display 107. In particular, the tone mapping performed at the image processing device 103 can depend on the target display for which the tone mapping is performed on the content generation side.

[0122] The content provider apparatus 101 provides a target display reference to the image processing device 103 (either separately from the encoded image or integrated therewith, ie the image signal may consist of two separate data communications). The target display reference may specifically comprise or be the white point luminance of the target display.

[0123] For example, for a relatively low-complexity system, the content provider apparatus 101 can simply send an indication of the white point luminance of the target display for each encoded image (video) that has been encoded. For example, data indicating the number of nits available at the target display can be transmitted. The dynamic range transform can then adaptively adjust the transform based on the number of nits. For example, if the image processing device 103 performs a dynamic range transform to generate an output image for a 2000 nit display, then knowledge of whether the input image is tone mapped to a 500 nit display or a 1000 nit display can be used to optimize the dynamic range transform performed at the image processing device 103. In both scenarios, the dynamic range transform can apply a nonlinear transform, but the nonlinear transform can have different characteristics for the two scenarios, namely depending on the white point of the target display used for tone mapping on the content provider side.

[0124] For example, the following mapping between received LDR image pixels tone mapped for a 500 nit target display and output HDR image pixels for a 2000 nit end-user display may be performed:

[0125] 0-200 nits → 0-200 nits

[0126] 200–300 nits → 200-600 nits

[0127] 300–400 nits → 600-1000 nits

[0128] 400–500 nits → 1000-2000 nits

[0129] However, for a target display of 1000 nits, the following mapping can be performed instead:

[0130] 0-200 nits → 0-200 nits

[0131] 200–700 nits → 200-1000 nits

[0132] 700–1000 nits → 1000-2000 nits

[0133] Therefore, in terms of relative values ​​(percentage of complete mapping), these two different mappings can be described as follows: Figure 3 , which shows the relationship between the percentage of white level for the input image on the x-axis and the percentage of white level for the output image on the y-axis for a 500 nit target display 301 and a 1000 nit target display, respectively. In this example, two very different nonlinear tone mappings are applied to the same user display, depending on the target reference display used / assumed on the content provider side.

[0134] It will be appreciated that the same mapping can be used to map from a 2000 nit optimized image to a 500 or 1000 nit optimized image by interchanging the axes (corresponding to applying the inverse of the mapping described above). It will also be appreciated that the mapping to, for example, a 500 nit optimized image can be adapted depending on whether the input image is a 1000, 2000, or 4000 nit optimized image.

[0135] In some embodiments, the target display reference may alternatively or additionally include an electro-optical transfer function indication for the target display. For example, a gamma indication for the target display may be included.

[0136] A display's electro-optical transfer function (EOTF) describes the relationship between the input (drive) luminance value (Y') and the output luminance (Y) for that display. This transfer function depends on many characteristics of the display. Furthermore, user settings like brightness and contrast can have a significant impact on this function. Figure 4 The figure shows a typical example of an EOTF for an 8-bit (256-level) input value.

[0137] The transmission of the EOTF of the target display can provide an advantageous representation of the target or reference display used to generate the encoded image or video. This representation can then be used at the image processing device 103 to adapt the dynamic range transform to the differences between the characteristics of the target display and the end-user display. For example, the dynamic range transform can include compensation for the inversion of the ratio of the EOTFs of the target / reference display and the end-user display.

[0138] It should be understood that there are many ways to characterize the EOTF. One possibility is to provide a set of sample values ​​of the EOTF. The image processing device 103 can then, for example, use simple linear interpolation to interpolate between these sample points. Another possibility is to provide a specific model of the grayscale / contrast behavior of a display at least on the portion of the display range. As another example, the content provider apparatus 101 can transmit a specific mathematical function that characterizes the EOTF. In some scenarios, the associated parameters of the model / function stored locally in the image processing device 103 can be used to predefine a target display set. In this case, the content provider apparatus 101 can simply transmit the identification code of the target display to the image processing device 103.

[0139] As another example, an underlying mathematical function may be predetermined, and the target display instructions may include parameters for adapting the predetermined function to describe the EOTF of a specific target display. For example, the EOTF may be characterized by a gamma function for conventional displays, and the target display instructions may provide a specific gamma for the target display.

[0140] In many systems, the target display indication may include or be present in the maximum luminance and gamma of the target display. Therefore, in particular, the characterization of the EOTF may be provided by two values, namely gamma and white point / maximum luminance, and the following description will focus on such a scenario.

[0141] The description will also focus on the distribution system according to Blu-ray TM Standard Example. Blu-ray TM BD-ROM is a family of audio / video / data distribution formats based on optical disc technology. TM Blu-ray Disc is an acronym for the read-only Blu-ray Disc format, which is primarily used for the distribution of high-definition video (2D and 3D) and high-quality audio.

[0142] BD-ROM TM The player features two operating modes: HDMV and BD-J. At any point in time, the player is in either HDMV mode or BD-J mode. Profile 5 Blu-ray TM The player is characterized by reproducing 3D stereoscopic video / graphics close to standard 2D video / graphics reproduction. For example, Figure 5 A model for rendering a plane in HDMV-2D mode is shown.

[0143] As Figure 1 In a specific example of a system, the image signal may be in a BDROM TM The video signal is encoded on the image processing device 103, and thus the image processing device 103 can be a Blu-ray TMThe encoded video may be primary or optionally secondary video content on a disc. The primary video is typically an actual movie in 2D or possibly 3D stereo format.

[0144] In order to TM To achieve the best picture quality in the system, Figure 1 The system uses a BDROM that allows the transfer of target display parameters TM This data, along with hypothetical or actual information on the end-user display, is then stored by the BDROM. TM Players are used to perform dynamic range conversion. In particular, BDROM TM The player (image processing device 103 ) may perform additional video tone mapping or other processing according to the characteristics of the target display and / or the end-user display.

[0145] One option for transmitting information about the parameters of the target display is to embed data indicating the values ​​of these parameters into the BDROM on the disc. TM The extended data structure in the playlist file (xxxxx.mpls) can be used for this. The extended data structure will have a unique and novel identifier. Incompatible old BDROM TM The player will not know about this new data structure and will simply ignore it. This will ensure backward compatibility. One possible implementation of the syntax and semantics of such a Target_Display_descriptor is shown below.

[0146] syntax Number of bits Mnemonics Target_Display_Descriptor () { Abs_Max_Luminance 8 uimsbf Gamma (or the grayscale behavior model of the display, such as EOTF) 8 uimsbf }

[0147] In this example, Abs_Max_Luminance is a parameter with a value, for example, between 0 and 255, which indicates the absolute maximum luminance / white point of the target display according to the following formula:

[0148] Absolute maximum illuminance in cd / m2 = Abs_Max_Luminance[bit0-4] x 10 Abs _Max_Luminance[bit5-7] .

[0149] It will be appreciated that other amounts of bits for the mantissa or exponent may of course be used.

[0150] Gamma is a parameter having a value between 0 and 255, for example, which indicates the gamma of the target display according to the following formula:

[0151] Gamma of target display EOTF = Gamma / 25.

[0152] Therefore, in this example, the target display reference is composed of BDROM TM The gamma value and absolute maximum illuminance of the target display for which the video signal is generated are provided to the image processing device 103. The image processing device 103 then uses this information when performing automatic dynamic range conversion to increase or decrease the dynamic range of the video signal for higher / lower illuminance end-user displays.

[0153] It should be understood that many different dynamic range transforms are possible and many different ways of adapting such a dynamic range transform based on a target display reference may be used. In the following, various examples are provided, but it should be understood that other methods may be used in other embodiments.

[0154] First, the difference in the best mapping of a given original image to LDR and HDR images can be given by Figure 6 The figure shows that Figure 6 Examples of different tone mappings that can be used for an LDR display (lower part of the figure) and an HDR display (upper part of the figure) are shown. The original image is the same for both LDR and HDR. The histogram of this image is shown in Figure 6 It shows that most pixels have brightness values ​​in the low to medium range. The histogram also shows a second small peak at high brightness values ​​(such as car headlights or flash lights).

[0155] In this example, tone mapping is represented by three consecutive processing steps:

[0156] Clipping: Maps luma values ​​in the low and high ranges to a limited number of output luma values.

[0157] Expand: Adapts the dynamic range to the desired luminance dynamic range.

[0158] Brightness: Make the average illuminance level suitable for optimal brightness.

[0159] In the LDR case, the luminance range is mapped to the illuminance range of the LDR display. The dynamic range of the original image is much larger, and therefore the original image is severely clipped in order to fit into the limited dynamic range of the display.

[0160] In the HDR case (upper part of the figure), clipping can be less severe because the dynamic range of the display is an order of magnitude larger than that of an LDR display.

[0161] Figure 6The histograms after each processing step are shown, along with histograms of the image displayed on LDR and HDR displays, respectively. Specifically, the rightmost histogram plots an LDR tone-mapped image when displayed on an HDR display, and the reverse. In the first case, the image will be too bright, with too much detail lost in the low and high range luminance values. In the second case, the image will be too dark, with too much detail and contrast lost in the mid-range luminance values.

[0162] As can be seen, simply presenting (a luminance-scaled version of) an LDR-optimized image on an HDR display (or vice versa) can significantly degrade image quality, so the image processing device 103 may perform a dynamic range transform to improve image quality. Furthermore, since the optimization performed at the studio strongly depends on the characteristics of the display for which the optimization is performed, the optimal dynamic range transform to be performed by the image processing device 103 depends not only on the end-user display, but also on a reference display. Accordingly, providing the target display reference to the image processing device 103 allows the image processing device 103 to perform the desired dynamic range transform based not only on the assumed or known characteristics of the end-user display, but also on the actual display used by the content provider. In fact, the provision of the target display reference can be considered to allow the image processing device 103 to partially or completely reverse some of the tone mapping performed at the studio, thereby allowing the characteristics of the original image to be estimated. Based on this estimation, the image processing device 103 can then apply the desired tone mapping optimized for the specific dynamic range characteristics of the end-user HDR display.

[0163] It should be understood that the image processing device 103 typically does not seek to perform a specific inverse tone mapping to recreate the original signal, followed by a tone mapping appropriate for a specific end-user display. In fact, a dynamic range transform typically does not provide sufficient information to perform such an inverse tone mapping, and the tone mapping performed by the content provider may often be partially irreversible. However, the image processing device 103 can perform a dynamic range transform that seeks to adapt the received image via the dynamic range transform to provide a (possibly very crude) approximation of the more theoretical operation of generating an inverse tone mapping of the original image, followed by an optimized tone mapping of the original image to a specific desired dynamic range. Thus, the image processing device 103 can simply apply, for example, a simple mapping from the luminance values ​​at the input of the dynamic range transform to appropriate luminance values ​​at the output of the transform. However, this mapping not only reflects the desired tone mapping of the original image for a given end-user display, but also depends on the actual tone mapping already performed at the content provider apparatus 101. Thus, the image processing device 103 can use the dynamic range transform to adapt the applied transform to account for and adapt to the tone mapping already performed.

[0164] For example, the image processing device 103 may be arranged to provide an output image for display on an HDR image having a predetermined maximum illuminance (e.g. 4000 nits). The received image / video may be tone mapped for an LDR display of 500 nits. The tone mapping thus optimizes the image for a given maximum illuminance and gamma. As a specific example, the gamma function may be Figure 7 curve 701, and the resulting image when presented on a 500 nit display might look like Figure 8 Like that.

[0165] When this image is to be presented on a 4000 nit HDR display, for example, it is often desired that the light output for dark areas does not vary substantially, while the light output for bright areas should be increased substantially. Therefore, a very different relationship between (linear) luminance values ​​and actual drive values ​​is required. In particular, if using Figure 7 , i.e., if a higher gamma is applied at the content side tone mapping, a much improved image will be generated for the HDR image. However, this higher mapping will result in the following on a 500 nit display: Figure 9 The image shown in appears too dark.

[0166] In this system, the image processing device 103 is informed of the gamma value of the target display for the content, and can therefore derive curve 701. Furthermore, the desired curve 703 is known, as it depends on the dynamic range of the display for which the output image is generated (which, for example, may be provided to the image processing device 103 from the display 107, or may be assumed / predetermined). Therefore, the image processing device 103 can apply a transformation corresponding to the conversion from curve 701 to curve 703 to each pixel luminance value. In this manner, the image processing device 103 can continue to apply a dynamic range transformation that converts the generated output signal from a signal suitable for an LDR display to a signal suitable for an HDR display, using the target display reference provided by the content provider apparatus 101.

[0167] It will be appreciated that the same considerations may apply when performing a dynamic range transform that reduces the dynamic range. For example, if the received content is to be displayed on a low quality, low illumination display such as a mobile phone display, then the preferred gamma for the mapping curve may be as follows: Figure 7 As shown by the curve 705, a gamma of less than 1 may be preferred. When presented on a normal 500 nit LDR, the corresponding image will be as follows Figure 10 The image shown looks too bright and has too little contrast, and in fact the situation is even worse for an HDR display.

[0168] Therefore, if the image processing device 103 generates images for such a low brightness display, it may proceed to perform a dynamic range transform that reduces the dynamic range by adjusting the luminance values ​​for the gamma difference between curves 701 and 705 .

[0169] As another example, if the content provider apparatus 101 provides images intended for a low brightness / dynamic range display and are accordingly encoded according to curve 705, the image processing device 103 may use knowledge of this gamma provided by the dynamic range transform to transform the received values ​​into values ​​suitable for a 500 nit display by adapting to the difference between curves 705 and 701, or to transform the received values ​​into values ​​suitable for a 4000 nit display by adapting to the difference between curves 705 and 703.

[0170] Thus, provision of a dynamic range transform indicating the gamma values ​​and maximum / white point luminance assumed for the target display allows the image processing device 103 to convert received images to gamma values ​​appropriate for the particular brightness luminance values ​​of the display on which the image is to be rendered.

[0171] In some systems, the target display reference may include a tone mapping indication representing a tone mapping used to generate the first encoded video stream for the first target display.

[0172] In some systems, the target display reference can directly provide information specific to the tone mapping performed by the content provider. For example, the target display reference can include information defining the white point, luminance, and gamma of the display for which the LDR (or HDR) image is generated, i.e., for which tone mapping is performed. However, the target display reference can also provide specific information, such as information that is lost in the tone mapping performed by the content provider.

[0173] For example, in Figure 6In an example, an LDR tone-mapped image corresponding to a clipped image may be received by the image processing device 103. The image processing device 103 may apply a dynamic range transform to map this to an appropriate dynamic range, as well as a nonlinear relationship based on information about the target display gamma and white point. However, to provide improved adaptation, severe clipping for the LDR image should preferably be converted to less severe clipping (or indeed to no clipping in some scenarios). Accordingly, the content provider apparatus 101 may provide additional information identifying the specific clipping that has been performed by the content provider on the LDR image, thereby allowing the clipping to be partially or completely reversed. For example, the dynamic range transform may define the range of the clipping, and the image processing device 103 may accordingly distribute the clipped values ​​over this range according to an appropriate algorithm (which, for example, identifies a region of clipped values ​​(e.g., a blowup) and generates a brightness that increases towards the center of the region).

[0174] Alternatively or in addition, the dynamic range transform can provide information that defines additional tone mapping performed on the content provider side. For example, a relatively standard tone mapping may be performed on most images of a movie or other video sequence. The image processing device 103 can convert such tone-mapped images to images of a desired (higher or lower) dynamic range using a dynamic range transform that assumes a standard tone mapping on the content provider side based on gamma and white point luminance. However, for some images, the content provider may perform a dedicated and subjective tone mapping. For example, a color grader may desire a specific artistic effect or quality for some images, such as, for example, a fine gradation or color cast for dark images for tense situations (such as in a horror movie) or special effects for dreamlike scenes. This tone mapping can be characterized by data in the target display reference, allowing the image processing device 103 to adapt the dynamic range transform to the specific tone mapping that has been applied.

[0175] Therefore, in particular, in some scenarios, additional / modified tone mapping is performed on the content provider side in order to generate a specific appearance, so that the image is modified relative to the image expected by fixed adaptation to the naked electro-optical behavior of the target display. The data provided by the content provider device 101 can specify the desired appearance compared to the reference display, and given all factors, this can be used by the image processing device 103 to actually generate the desired optical behavior (for example, however, blind coding in the input signal may accidentally end up under the reflected ambient light, so that it can no longer be compensated according to the coded content provider side behavior).

[0176] For example, if it is known that the gamma of the target display is low for darker values, it may be possible to fine-tune the appearance of, for example, a horror scene for such a (reference) display. For example, the image can be compensated by an additional luminance boost so that the image still appears dimmed, but at least certain target structures are still visible.

[0177] For example, along with the gamma and white point luminance of the reference target, a color grader on the content provider's side can provide some (additional) information about the artistic impression of specific areas and / or images. For example, for a given EOTF, the content provider may indicate that they want certain areas to have increased brightness for better visibility, or reduced contrast to provide a hazy appearance, etc. Thus, along with the EOTF (e.g., represented by gamma and white point luminance), the target display reference can indicate the boundaries of the local / partial display luminance range and provide dynamic range transform data that provides more precise information about the preferred distribution of its grayscale levels.

[0178] In some embodiments, the dynamic range processor (203) may be arranged to select between generating the output image as a transformed image of the first encoded image in response to a target display reference and generating the output image as the received encoded image.

[0179] In particular, if the white point luminance indicated by the target display reference is sufficiently close to the white point luminance of the end-user display, then the dynamic range transform can simply consist of not performing any processing on the received encoded image; that is, the input image can simply be used as the output image. However, if the white point luminance indicated by the target display reference differs from the white point luminance of the end-user display, then the dynamic range transform can modify the received image according to an appropriate mapping of received image pixels to output image pixels. In such cases, the mapping can be adaptively adjusted based on the target display reference. In other examples, one or more predetermined mappings can be used.

[0180] For example, the image processing device 103 may include a predetermined first mapping that is determined to provide an appropriate output image for a doubling of the white point luminance level, and a predetermined second mapping that is determined to provide an appropriate output image for a halving of the white point luminance level. In such an example, the image processing device 103 may select between the first mapping, the second mapping, and the unit mapping based on the white point luminance of the target display reference and the white point of the end-user display. In particular, the image processing device 103 may select the mapping that most closely corresponds to the ratio between the target display reference white point luminance and the end-user display white point luminance.

[0181] For example, if an input image is received with a target display reference indicating that it is optimized for a 500 nit display, and the end-user display is a 1000 nit display, the image processing device 103 will select the first mapping. If, on the other hand, the target display reference indicates that the input image is optimized for a 1000 nit display, the image processing device 103 will select the unit mapping (i.e., use the input image directly). If the target display reference indicates that it is optimized for a 2000 nit display, the image processing device 103 will select the second mapping.

[0182] If intermediate values ​​of the white point luminances of the target display are received, the image processing device 103 may select a mapping that is closest to the ratio between these white point luminances, or may, for example, interpolate between these mappings.

[0183] In some embodiments, dynamic range conversion can include color gamut conversion or be in color gamut conversion. Therefore, in some embodiments, dynamic range processor 203 can modify the chroma of the reproduced image according to the target display benchmark. For example, when the received HDR image is reproduced on an LDR display, compression may cause each image object to have less variation and a more flat image with grading. Dynamic range conversion can compensate for such reduction by increasing chroma variation. For example, when the image of an apple with light is optimized for reproduction on an HDR display, reproduction on an LDR display with a reduced dynamic range typically makes the apple look less prominent and less clear and more matte. This can be compensated by dynamic range conversion by making the color of the apple more saturated. As another example, texture variation may become less noticeable perceptually due to the reduced illumination variation, and this can be compensated by increasing the chroma variation of the texture.

[0184] In some systems, the video signal may include a data field containing dynamic range transform control data, and the dynamic range processor 203 may adapt the dynamic range transform in response to this control data. This may be used by content owners / providers to maintain at least some input or control over the rendering of the provided content.

[0185] The control data may, for example, define the operations or parameters of the dynamic range transform that must be applied, may be applied, or is recommended to be applied. Furthermore, the control data may be differentiated for different end-user displays. For example, separate control data may be provided for a variety of possible end-user displays, such as one set of data for a 500 nit display, another for a 1000 nit display, another for a 2000 nit display, and yet another for a 4000 nit display.

[0186] For example, content creators can Figure 11 The end-user display characteristics shown in dictate which tone mapping should be performed by the dynamic range processor 203. In this example, the control data may specify a mapping for each of three regions corresponding to given values ​​of the maximum illuminance of the display (x-axis) and ambient light incident on the display (and thus reflection from the display - y-axis).

[0187] Therefore, in this particular example, Mapping 1 is used for low-brightness displays in low ambient light environments. Mapping 1 can be a simple unit mapping, meaning the received LDR image can be used directly. For high-maximum-illuminance (HDR) displays in relatively dark ambient environments (low screen reflections), Mapping 2 can be used. Mapping 2 can perform a mapping that further extends the bright illuminance of the LDR image while substantially maintaining the intensity of the darker segments. For high-maximum-illuminance (HDR) displays in relatively bright ambient environments (substantial screen reflections), Mapping 3 can be used. Mapping 3 can perform a more aggressive mapping that not only extends the bright illuminance of the LDR image but also brightens darker image areas and increases the contrast in these areas.

[0188] In some scenarios, the control data may specify boundaries between these mappings and predetermined (e.g., standardized or known both on the content provider side and on the renderer side) mappings. In some scenarios, the control data may further define the elements of the different mappings, or may specify the mappings very precisely, for example using gamma values, or specify a specific transfer function.

[0189] In some embodiments, the dynamic range transform control data may directly and explicitly specify the dynamic range transform that should be performed to transform the received image into an image having a different dynamic range. For example, the control data may specify a direct mapping from input image values ​​to output image values ​​for a range of target output display white points. This mapping may be provided as a simple parameter, allowing the dynamic range processor 203 to implement the appropriate transform, or detailed data may be provided, such as a specific lookup table or mathematical function.

[0190] As a low-complexity example, a dynamic range transform can simply apply a piecewise linear function to the input values ​​of an LDR image in order to generate improved HDR values. In fact, in many scenarios, it is possible to use a dynamic range transform such as Figure 12 , which consists of two linear relationships. The mapping shows a direct mapping between input pixel values ​​and output pixel values ​​(or in some scenarios, the mapping may reflect a (possibly continuous) mapping between input pixel luminances and output pixel luminances). It will be appreciated that the same mapping can be used to map from an input HDR image to an output LDR image.

[0191] In particular, for mapping from LDR to HDR, the method provides a dynamic range transform that maintains the dark areas of the image as dark, while allowing a greatly increased dynamic range to provide a much brighter reproduction of the bright areas, and a truly improved and more vivid looking mid-range. For mapping from HDR to LDR, the method provides a dynamic range transform that maintains the dark areas of the image, but compresses the brighter areas to reflect the reduced brightness range of the display.

[0192] However, the exact transformation depends on the target display for which the image is generated and the display on which the image is to be reproduced. For example, when reproducing an image intended for a 500 nit display on a 1000 nit display, a relatively modest transformation is required, and the stretching of bright areas is relatively limited. However, if the same image is to be displayed on a 5000 nit display, a much more extreme transformation is required to fully utilize the available brightness without brightening the dark areas too much.

[0193] Likewise, the mapping may depend on the target display for which the original image was generated. For example, if an input image optimized for 1000 nits is to be reproduced on a 2000 nit display, a relatively modest transformation is required, and the stretching of bright areas is relatively limited. However, if the image has been optimized for a 500 nit display and is to be displayed on a 2000 nit display, a much more extreme transformation is required to fully utilize the available brightness without brightening the dark areas too much. Figure 13 The figure shows how two different mappings may be used for a 1000 nit input image (curve 1301 , maximum value 255 corresponds to 1000 nits) and a 500 nit input image (curve 1303 , maximum value 255 corresponds to 500 nits), respectively, for display on a 2000 nit LDR input image (maximum value 255 corresponds to 2000 nits).

[0194] One advantage of this simple relationship is that the desired tone mapping can be communicated with very low overhead. In fact, the control data can specify the inflection point of the curve, that is, the transition point between two straight line segments. Thus, a simple two-component data value can specify the desired tone mapping performed by the image processing device 103 for different displays. The image processing device 103 can further determine appropriate values ​​for other maximum illuminance values ​​by interpolating between the provided values.

[0195] In some implementations, more points may be provided, for example, to define a curve that is still piecewise linear but has more linear intervals. This may allow for more accurate tone mapping and improve the resulting image quality while introducing only relatively minor overhead.

[0196] In many implementations, the control data may not specify a specific tone mapping that should be performed, but rather provide data defining boundaries within which the dynamic range transform / tone mapping may be freely adapted by the image processing device 103 .

[0197] For example, rather than specifying the maximum brightness level for a transition point, the control data may define limits for the transition point (possibly providing different limits for different maximum brightness levels). Figure 12 and Figure 13 . Thus, the image processing device 103 can individually determine the desired parameters for the dynamic range transform so that it can be set to provide a preferred transition for a specific display, taking into account, for example, specific user preferences. However, at the same time, the content provider can ensure that this freedom is limited to an acceptable range, allowing the content provider to retain a certain control over how the content is reproduced.

[0198] Thus, the dynamic range transform control data may include data defining the transform parameters that must be applied for the dynamic range transform performed by the dynamic range processor 203 and / or defining limits for these transform parameters. The control data may provide such information for a range of maximum brightness levels, thereby allowing the dynamic range transform to be adapted to different end-user displays. Furthermore, for maximum brightness levels not explicitly included in the control data, appropriate data values ​​may be generated from the available data values, for example, by interpolation. For example, if the inflection point between two straight line segments for 2000 nit and 4000 nit end-user displays is specified, then the appropriate value for a 3000 nit display may be found by simple interpolation (e.g., in this particular example, by simple averaging).

[0199] It will be appreciated that many different and varying approaches for dynamic range conversion and for how this is limited, adapted and controlled from the content provider side via additional control data may be used in different systems, depending on the specific preferences and requirements of the individual application.

[0200] In fact, many different commands or parameter values ​​may be provided in the control data in order to generate the tone map in accordance with the content provider's preferences.

[0201] For example, in a low-complexity system, a simple dynamic range transform can be applied, and the content provider device 101 can simply provide the white and black levels for the target display, which are then used by the dynamic range processor 203 to determine the tone mapping to be applied. In some systems, a tone mapping function (gamma or other) can be mandatory for mapping at least one range of the input image. For example, the control data can specify that the darker and / or middle ranges must be reproduced according to a given mapping, while allowing the image processing device 103 to freely map the brighter ranges.

[0202] In some scenarios, the control data may only provide suggestions for suitable mappings that can be applied, for example, in the mid-range region. In such cases, the content provider can therefore assist the image processing device 103 in providing suggested dynamic range transform parameters found (e.g., through manual optimization by the content provider) to provide high image quality when viewed on a given HDR display. The image processing device 103 can use this to advantage, but is free to modify the mapping, for example, to suit individual user preferences.

[0203] In many scenarios, mapping is performed at least in part based on control data, representing a relatively low-complexity functional relationship, such as a gamma mapping, an S-curve, a combined mapping defined by partial specification for various ranges, etc. However, in some scenarios, more complex mappings may of course be used.

[0204] It should also be understood that dynamic range transformations often involve increasing or decreasing the number of bits used to represent values. For example, an 8-bit image can be transformed into a 12- or 14-bit image. In such cases, control data from the content provider apparatus 101 can be provided independently of the changed quantization. For example, an 8-bit to 8-bit jointly encoded tone mapping (the "shape" of the grayscale sub-distribution) can be defined by the content provider apparatus 101, and the image processing device 103 can scale this mapping to the specific display white brightness by taking into account the transformation of more bits.

[0205] In other embodiments or scenarios, the dynamic range conversion may include a reduction in the number of bits used to represent a value. For example, a 12-bit image may be converted to an 8-bit image. Such scenarios may often occur when the dynamic range conversion provides a reduction in dynamic range, such as when converting a 12-bit HDR image to be reproduced on an 8-bit input value LDR display.

[0206] As mentioned, the control data may provide mandatory or voluntary control data. Indeed, the received data may include one or more fields indicating whether the provided tone mapping parameters are mandatory, allowed or recommended.

[0207] For example, a suggested tone mapping function may be provided along with an indication of how much deviation therefrom is acceptable. The image processing device 103 in a standard configuration may then automatically apply the suggested mapping. However, the transformation may be modified, for example, to reflect the user's personal preferences. For example, user input may change the settings of the image processing device 103, such as causing dark areas of an image to be reproduced brighter than the content provider considers ideal. For example, the user may simply press a button for increasing brightness, and the tone mapping may change accordingly (e.g., move up Figure 12 and Figure 13 The user can thus introduce fine-tuning into the tone mapping. However, the control data can include information on how much fine-tuning is acceptable to the content provider, thereby limiting the dynamic range transform to produce an output image that is still deemed by the content provider to maintain the integrity of the provided image. The control data can also specify the effects of user interaction, such as limiting or restricting the change in brightness that occurs with each button press by the user.

[0208] Accordingly, the dynamic range transform provides a dynamic range transform that is intended to provide an image suitable for a specific end-user display 107 while taking into account the display characteristics of the display for which the input image was generated. Thus, the image processing device 103 generates an output signal associated with a given maximum luminance / brightness value, i.e., intended for reproduction on a display having that white point / maximum luminance value. In some systems, the display's white point luminance may not be precisely known to the image processing device 103, and therefore the output signal may be generated for an assumed white point luminance (e.g., manually input by a user). In other applications (as will be described later), the display may provide information about the white point luminance, and the image processing device 103 may adaptively adjust the dynamic range transform based on this information.

[0209] If the white point luminance for which the output signal is generated corresponds exactly or sufficiently closely to the white point luminance of one of the received images (according to any appropriate criteria, such as a white point luminance difference below a threshold), the image processing device 103 can proceed to use that image directly in the output image; i.e., the dynamic range transform can simply be a unit mapping. Furthermore, if the output white point luminance does not directly correspond to the white point luminance of the received image, but does match the white point luminance of an end-user display for which explicit dynamic range transform control data is provided, then that control data can be used directly to adapt the dynamic range transform. If the output white point luminance does not directly correspond to the white point luminance of the received image or to the white point luminance for which dynamic range transform control data is provided, then the tone mapping parameters provided by the control data for different white point luminances can be used to adapt the dynamic range transform based on the output white point luminance. In particular, the dynamic range processor 203 can interpolate between tone mapping parameters for other white point luminance values ​​to a particular output white point luminance. In many embodiments, simple linear interpolation will be sufficient, but it will be appreciated that many other approaches may be used.

[0210] In fact, the control data may also, for example, provide information on how tone mapping parameters provided for different display white point luminances should be processed in order to generate tone mapping parameters for a specific output white point luminance. For example, the control data may indicate a non-linear interpolation function that must be used to generate appropriate tone mapping parameters.

[0211] It should also be understood that the dynamic range transform is not necessarily constant for different images or even for the same image.

[0212] In fact, in many systems, the dynamic range transform control data can be continuously updated, allowing the dynamic range transform performed by the dynamic range processor 203 to adapt to current characteristics. This can allow different tone mapping to be used for dark images / scenes compared to bright images / scenes. This can provide improved performance. In fact, a time-varying dynamic range transform controlled in response to dynamically updated dynamic range transform control data can be used to provide additional control to content providers. For example, the reproduction of a dark scene on an HDR display may be different depending on whether the scene is intended to provide an unsettling, tense scene or whether the scene is dark simply to correspond to a nighttime scenario (in the first case, the dark scene may be reproduced on the HDR display as dark as on the LDR display, and in the second case, the dark scene may be reproduced slightly brighter, thereby utilizing the additional dynamic range to allow for improved visually perceptible distinction in dark areas).

[0213] Similar considerations can be applied within an image. For example, a scene might have bright sky above dark, shadowed ground (e.g., bright sky in the upper half of the image and forest in the lower half). When mapping from LDR to HDR, these two regions can advantageously be mapped differently, and the dynamic range transform control data can specify these differences in mapping. Thus, the dynamic range transform control data can include tone mapping parameters that vary from image to image and / or depend on the location within the image.

[0214] As a specific example, at least some of the control data may be associated with a given image region, illumination range, and / or image range.

[0215] The dynamic range transform control data may be provided to the image processing device 103 in accordance with any suitable communication method or standard.

[0216] In this particular example, the communication between the content provider apparatus 101 and the image processing device 103 uses a blue light TM The transmission of control commands for dynamic range conversion can be achieved by embedding these parameter values ​​into the BDROM data on the disc. An extended data structure in the playlist file (xxxxx.mpls) can be used for this. This extended data structure will have a unique and new identifier. Legacy BDROM players will not understand this new data structure and will simply ignore it. This will ensure backward compatibility. One possible implementation of the syntax and semantics of such an LHDR_descriptor is shown below.

[0217] syntax Number of bits Mnemonics LHDR_Descriptor() { Video_Process_descriptor 8 uimsbf DR_Process_descriptor 8 uimsbf Level_Process_descriptor 8 uimsbf Dynamic_range }

[0218] In this example, the LHDR_descriptor contains three processing descriptors. These parameters specify additional processing of the video if the target display class is different from the end-user display class. For example, these parameters can have the following values.

[0219] Video_Process_descriptor (video processing descriptor):

[0220] value Video / graphics processing when target display = LDR, end-user display = HDR Video / graphics processing when target display = HDR, end-user display = LDR 0x00 No additional processing No additional processing 0x01 Allows limited additional processing according to DR_Process_descriptor and Level_Process_descriptor Allows limited additional processing according to DR_Process_descriptor and Level_Process_descriptor 0x02 No restrictions on additional processing No restrictions on additional processing 0x03-0xFF reserve reserve

[0221] DR_Process_descriptor (DR processing descriptor):

[0222] value Video / graphics processing when target display = LDR, end-user display = HDR Video / graphics processing when target display = HDR, end-user display = LDR 0x00 Allows to increase dynamic range up to 125% Allows dynamic range to be reduced to 80% 0x01 Allows to increase dynamic range up to 150% Allows dynamic range to be reduced to 70% 0x02 Allows to increase dynamic range up to 200% Allows dynamic range to be reduced to 50% 0x03-0xFF reserve reserve

[0223] Level_Process_descriptor (Level Processing Descriptor):

[0224] value Video / graphics processing when target display = LDR, end-user display = HDR Video / graphics processing when target display = HDR, end-user display = LDR 0x00 Allows adaptive adjustment of the horizontal range to 80-125% Allows adaptive adjustment of the horizontal range to 80-125% 0x01 Allows to increase the horizontal range to 70-150% Allows to increase the horizontal range to 70-150% 0x02 Allows increasing the horizontal range to 50-200% Allows increasing the horizontal range to 50-200% 0x03-0xFF reserve reserve

[0225] The previous examples have focused on instances where the signal received from the content provider apparatus 101 comprises only one version of an image / video sequence and in particular where the signal comprises only LDR image / video sequences.

[0226] However, in some systems and implementations, the content provider device 101 may generate an image signal that includes more than one image version. In such a scenario, one image may be tone mapped for one target display, and another image may correspond to the same original image but be tone mapped for a different target display. In particular, one image may be an LDR image generated for, for example, a 500 nit display, and the other image may be an HDR image generated for, for example, a 2000 nit display.

[0227] In such an example, the image signal may further include a second target display reference, i.e., a target display reference may be provided for each of the images, thereby indicating the display characteristics for which tone mapping at the encoder side is optimized for the individual image. In particular, maximum brightness and gamma parameters may be provided for each image / video sequence.

[0228] In such a system, the image processing device 103 may be arranged to apply the dynamic range transform in response to the second target display reference and in particular by taking into account the first and second target display references.

[0229] The dynamic range transform can not only adapt the specific mapping or operation performed on the image, but also select which image to use as the basis for the transform depending on the target display reference. As a low-complexity example, the dynamic range processor 203 can choose between using a first image and a second image based on how closely the associated target display reference matches the white point luminance for which the output signal is generated. In particular, the image associated with the white point luminance that is closest to the desired output white point luminance can be selected. Thus, if an LDR output image is generated, the dynamic range transform can be performed on the encoded LDR image. However, if an HDR image is generated that has a higher maximum brightness than the encoded HDR image, the dynamic range transform can be performed on the encoded HDR image.

[0230] If an image is to be generated for a maximum brightness between the white point luminances of the encoded images (e.g., for a 1000 nit display), then the dynamic range transform can be based on both images. In particular, an interpolation between these images can be performed. This interpolation can be linear or nonlinear and can be performed directly on the encoded images before the transform, or applied to the images after the transform is applied. The weighting of the images can typically depend on how close they are to the desired maximum output brightness.

[0231] For example, a first transformed image can be generated by applying a dynamic range transform to a first encoded image (LDR image), and a second transformed image can be generated by applying a dynamic range transform to the second transformed image. The first and second transformed images are then combined (e.g., summed) to generate an output image. The weights of the first and second transformed images are determined by how closely their respective target display references match the desired maximum output brightness.

[0232] For example, for a 700 nit display, the first transformed image can be weighted much higher than the second transformed image, and for a 3000 nit display, the second transformed image can be weighted much higher than the first transformed image. For a 2000 nit display, the two transformed images can potentially be weighted equally, and the output value can be generated by averaging the values ​​for each image.

[0233] As another example, the transformation may be performed selectively based on the first or the second image for different image regions, eg depending on image characteristics.

[0234] For example, for relatively dark areas, a dynamic range transform may be applied to the LDR image to generate pixel values ​​that are suitable for a 1000 nit display, yet take advantage of the finer resolution that may be used for the dark areas of the LDR image compared to the HDR image (e.g., if the same number of bits is used for both images). However, for brighter areas, pixel values ​​may be generated by applying a dynamic range transform to the HDR image, taking advantage of the fact that the image typically has more information in the high brightness range (in particular, the loss of information due to clipping is typically much smaller for HDR images relative to LDR images).

[0235] Thus, when more than one image is received from the content provider apparatus 101, the image processing device 103 may generate an output image based on one of these images, or may combine them when generating the output image. The selection and / or combination of these encoded images is based on the target display reference provided for each image and the maximum brightness for which the output signal is generated.

[0236] It should be understood that, in addition to combining and / or selecting the coded images, each dynamic range transform can also be adjusted and adapted in response to the dynamic range transform. For example, the previously described methods can be applied separately to each dynamic range transform. Similarly, dynamic range transform control data can be received that can be used to adapt and control each dynamic range transform as previously described. In addition, the dynamic range transform control data can include information defining mandatory, optional, or preferred / recommended parameters for the combination of the processing of the first and second coded images.

[0237] In some systems, the dynamic range transform control data comprises different transform control data for different image categories. In particular, different types of images / content may be processed differently when performing a dynamic range transform.

[0238] For example, different tone mappings may be defined or suggested for different types of video content. For example, different dynamic range transforms may be defined for cartoons, horror movies, football games, etc. In such cases, the received video signal may provide metadata describing the content type (or content analysis may be applied locally in the image processing device 103), and an appropriate dynamic range transform may be applied for the specific content.

[0239] As another example, the rendered image may be generated as a combination of overlay images using different transforms provided for different images. TM In the , many different presentation planes are defined (such as Figure 5 ), and different dynamic range transforms can be applied to these different rendering planes.

[0240] The characteristics of each of these presentation planes are optimized by the content provider for a specific target display. The end-user's viewing experience can be optimized by adapting the characteristics of the presentation plane to the end-user display. Typically, the optimal adaptation will be different for different presentation planes.

[0241] Regarding tone mapping, the current situation in BDROM systems is as follows:

[0242] - Video tone mapping (global and / or local) is performed in the studio using studio monitors.

[0243] - Graphics tone mapping (usually different from video tone mapping) is performed in the studio using studio monitors.

[0244] - OSD tone mapping is performed in BDROM players.

[0245] - Global and / or local tone mapping is performed in the display on the combined video & graphics signal. This processing cannot be controlled by the end user.

[0246] - Global tone mapping is performed in the display on the combined video & graphics signal. This process depends, among other things, on the brightness and contrast values ​​set by the end user.

[0247] Improved picture quality is achieved when:

[0248] 1. Video tone mapping is optimized for end-user displays.

[0249] 2. Graphics tone mapping is optimized for end-user displays.

[0250] 3. The system allows graphics tone mapping to be different from video tone mapping.

[0251] 4. The system allows different graphics tone mapping for different graphics components

[0252] 5. The system allows video & graphics tone mapping depending on the video characteristics.

[0253] It should also be noted that in case both LDR and HDR versions of the video are present on disc, the additional tone mapping will depend on two sets of parameters for the target display: one set for the LDR version of the video, and one set for the HDR version of the video.

[0254] In another enhanced implementation, the video and / or graphics tone mapping varies over time, for example, depending on the video content in the scene. The content provider can send tone mapping instructions to the player based on the characteristics of the video and graphics content. In another implementation, the player autonomously extracts the video characteristics from the video signal and adaptively adjusts the video and graphics tone mapping based on these characteristics.

[0255] For example, subtitles can be dimmed for a certain timespan, or a specific gamma change can be achieved for a certain amount of time (and the two can be coordinated).

[0256] In the following, an example of how to provide control commands for graphics tone mapping for a BDROM is described.

[0257] A BDROM graphics stream consists of segments embedded in PES packets which are embedded in the transport stream. Figure 14 The diagram shows the appropriate data structure.

[0258] Synchronization with the main video is done at the elementary stream level using the PTS value in the PES packet. A BDROM graphics segment consists of a segment descriptor and segment data. The segment descriptor contains the segment type and length.

[0259] The following table shows some of the segment types defined in the Blu-ray Disc standard:

[0260] value part 0x00 – 0x13 reserve 0x14 Palette definition section 0x15 Object definition section 0x16 Presentation component segments 0x17 Window definition section 0x18 Interactive composition segment 0x19 - 0x7F reserve 0x80 End of display settings segment 0x81 – 0x82 HDMV text subtitle stream uses 0x83 LHDR_Processing_Definition_Segment 0x84 - 0xFF reserve

[0261] In the existing specification, the values ​​0x83-0xFF are reserved. Therefore, a new segment type is defined using, for example, the value 0x83 to indicate a segment containing the LHDR_Processing_definition segment. Typically, the LHDR_Processing_definition segment defines how the graphics decoder processes graphics when the target display is different from the end-user display.

[0262] The following table shows an example of a possible structure for the LHDR_Processing_definition section:

[0263]

[0264] In this example, the LHDR_Processing_definition segment contains two processing descriptors: Pop-up_process_descriptor and Subtitle_process_descriptor. This segment can also contain a palette to use if the target display class is different from the end-user display class. The LHDR palette contains the same number of entries as the original palette, but these entries are optimized for the other display class.

[0265] The parameter Pop-up_process_descriptor specifies additional processing of pop-up graphics in case the target display class is different from the end-user display class.

[0266] For example, this parameter can have the following values.

[0267] - Pop-up_process_descriptor=0x00: No additional processing.

[0268] - Pop-up_process_descriptor=0x01-0x03: Set the minimum transparency value.

[0269] - Pop-up_process_descriptor=0x04: The graphics processor uses the palette defined in the LHDR_Processing_definition segment.

[0270] - Pop-up_process_descriptor=0x05: No restrictions on additional processing.

[0271] The parameter Subtitle_process_descriptor specifies additional processing of subtitle graphics in case the target display class is different from the end-user display class.

[0272] For example, this parameter can have the following values.

[0273] - Subtitle_process_descriptor=0x00: No additional processing.

[0274] - Pop-up_process_descriptor = 0x01-0x03: Adaptively adjust the brightness value.

[0275] - Subtitle_process_descriptor=0x04: The graphics processor uses the palette defined in the LHDR_Processing_definition segment.

[0276] - Subtitle_process_descriptor=0x05: No restrictions on additional processing.

[0277] Specific examples of syntax for Pop-up_process_descriptor and Subtitle_process_descriptor are provided in the following table:

[0278] value Graphics processing when target display = LDR and end-user display = HDR Graphics processing when target display = HDR and end-user display = LDR 0x00 No additional processing No additional processing 0x01 Set T_value >= 128 No additional processing 0x02 Set T_value >= 192 No additional processing 0x03 Set T_value >= 222 No additional processing 0x04 Using the LHDR palette Using the LHDR palette 0x05 No restrictions No restrictions 0x06-0xFF reserve reserve

[0279] value Graphics processing when target display = LDR and end-user display = HDR Graphics processing when target display = HDR and end-user display = LDR 0x00 No special treatment No special treatment 0x01 Luma:=Luma / 5 Luma:=Luma*5 0x02 Luma:=Luma / 3 Luma:=Luma*3 0x03 Luma:=Luma / 2 Luma:=Luma*2 0x04 Using the LHDR palette Using the LHDR palette 0x05 No restrictions No restrictions 0x06-0xFF reserve reserve

[0280] Figure 15 and Figure 16 The diagrams in Figure 3 show specific examples of different tone mapping depending on the display characteristics. In these examples, the original content features HDR video content and subtitles. Tone mapping for video is different from Figure 6 The same instance.

[0281] These graphics feature white subtitle characters with black borders. The original histogram shows one peak in the low luminance range and another peak in the high luminance range. This histogram is well suited for subtitle content on an LDR display, as it will result in bright, clear text on the display. However, on an HDR display, these characters will be too bright, causing distraction, halos, and glare. For this reason, Figure 16 Adapt the tone mapping for subtitle graphics as described in

[15] .

[0282] In the previous example, the image processing device 103 generates an output image corresponding to a desired maximum brightness, i.e., intended for presentation on a display having a given dynamic range / white point luminance. The output signal may specifically be generated to correspond to a user setting indicating a desired maximum / white point luminance, or may simply assume a given dynamic range for the display 107.

[0283] In some systems, the image processing device 103 may comprise a dynamic range processor 203 arranged to adapt its processing in accordance with data received from the display 107 indicative of illumination characteristics of the display 107 .

[0284] Figure 17 An example of such an image processing device 103 is shown in FIG. Figure 1 103 corresponds to an image processing device, but in this example, the image processing device 103 further includes a display receiver 1701 that receives a data signal from the display 107. The data signal includes a data field that includes a display dynamic range indication for the display 107. The display dynamic range indication includes at least one illuminance specification indicating the illuminance properties of the display. In particular, the illuminance specification may include a specification for the maximum brightness of the display, i.e., the maximum / white point illuminance. In particular, the display dynamic range indication may define whether the display is an HDR or LDR display, and may specifically indicate the maximum light output in nits. Thus, the display dynamic range indication may define whether the display is a 500 nit, 1000 nit, 2000 nit, 4000 nit, and so on.

[0285] The display receiver 1701 of the image processing device 103 is coupled to a dynamic range processor 203 which is fed with an indication of the display dynamic range. The dynamic range processor 203 can accordingly generate an output signal that corresponds directly to the particular display, rather than generating an output signal for an assumed or artificially set white point luminance.

[0286] The dynamic range processor 203 can adaptively adjust the dynamic range transform in response to the received display dynamic range indication. For example, the received encoded image may be an LDR image, and it can be assumed that the image has been optimized for a 500 nit display. If the display dynamic range indication indicates that the display is indeed a 500 nit display, the image processing device 103 can directly use the encoded image. However, if the display dynamic range indication indicates that the display is a 1000 nit display, a first dynamic transform can be applied. If the display dynamic range indication indicates that the display 107 is a 2000 nit display, a different transform can be applied, and so on. Similarly, if the received image is a 2000 nit optimized image, the image processing device 103 can directly use the image if the display dynamic range indication indicates that the display is a 2000 nit display. However, if the display dynamic range indication indicates that the display is a 1000 nit or 500 nit display, the image processing device 103 can perform an appropriate dynamic range transform to reduce the dynamic range.

[0287] For example, refer to Figure 18 , two different transformations can be defined for 1000 nit displays and 4000 nit displays, respectively, and a third one-to-one mapping can be defined for 500 nit displays. Figure 1 , the mapping for a 500 nit display is indicated by curve 1801, the mapping for a 1000 nit display is indicated by curve 1803, and the mapping for a 4000 nit display is indicated by curve 1805. Thus, in this example, it is assumed that the received encoded image is a 500 nit image, and this is automatically converted into an image suitable for a specific display. Thus, the image processing device 103 can automatically adapt and generate an optimized image for the specific display to which it is connected. In particular, the image processing device 103 can automatically adapt to whether the display is an HDR or LDR display, and can further adapt to the specific white illuminance of the display.

[0288] It will be appreciated that when mapping from a higher dynamic range to a lower dynamic range, an inverse mapping may be used.

[0289] If the monitor has Figure 18 If one of the three curves for white luminance corresponds to one of the three curves, then the corresponding mapping can be applied to the encoded image. If the display has different luminance values, then a combination of these transformations can be used.

[0290] Therefore, the dynamic range processor 203 can select an appropriate dynamic range transform based on the display dynamic range indication. As a low-complexity example, the dynamic range processor 203 can choose between these curves based on how closely the associated white point luminance matches the white point luminance indicated by the display dynamic range indication. In particular, the mapping associated with the white point luminance that is closest to the desired white point luminance indicated in the display dynamic range indication can be selected. Thus, if an LDR output image is generated, then the dynamic range transform can be performed using curve 1801. If an HDR image with a relatively low white point luminance is generated, then the mapping of curve 1803 is used. However, if an HDR image with a high white point luminance is generated, then curve 1805 is used.

[0291] If an image is to be generated for a white luminance between the dynamic range transforms for the two HDR settings (e.g., for a 2000 nit display), then these two mappings 1803, 1805 can be used. In particular, an interpolation between the transformed images for the two mappings can be performed. Such interpolation can be linear or nonlinear. The weighting of the transformed images can typically depend on how closely they correspond to the desired output maximum brightness.

[0292] For example, a first transformed image can be created by applying the first mapping 1803 to an encoded image (LDR image), and a second transformed image can be created by applying the second mapping to the encoded image. The first and second transformed images are then combined (e.g., summed) to generate an output image. The weights of the first and second transformed images are determined by how closely the white luminance associated with these different mappings matches the display white luminance indicated in the display dynamic range indication.

[0293] For example, for a 1500 nit display, the first transformed image may be weighted much higher than the second transformed image, and for a 3500 nit display, the second transformed image may be weighted much higher than the first transformed image.

[0294] In some embodiments, the dynamic range processor (203) may be arranged to select between generating the output image as the received encoded image and generating the output image as a transformed image of the received encoded image in response to the display dynamic range indication.

[0295] In particular, if the white point luminance indicated by the display dynamic range indication is sufficiently close to the white point luminance indicated or assumed for the received image, then the dynamic range transform can simply consist of not performing any processing on the received image; i.e., the input image can simply be used as the output image. However, if the white point luminance indicated by the display dynamic range indication differs from the white point luminance indicated or assumed for the received image, then the dynamic range transform can modify the received encoded image according to an appropriate mapping of input image pixels to output image pixels. In such cases, the mapping can be adaptively adjusted based on the received indication of the white point luminance of the end-user display. In other examples, one or more predetermined mappings can be used.

[0296] For example, the image processing device 103 may include a predetermined first mapping determined to provide an appropriate output image for doubling the white point level, and a predetermined second mapping determined to provide an appropriate output image for halving the white point level. In such an example, the image processing device 103 may select between the first mapping, the second mapping, and the unit mapping based on the white point luminance of the received image (e.g., as indicated by a target display reference) and the white point luminance of the end-user display as indicated by a display dynamic range indication. In particular, the image processing device 103 may select the mapping that most closely corresponds to the ratio between the white point luminance of the end-user display and the input image.

[0297] For example, if an input image is received with a target display reference indicating that it is optimized for a 1000 nit display, and the end-user display is a 2000 nit display, the image processing device 103 will select the first mapping. If, on the other hand, the display dynamic range indication indicates that the end-user display is a 1000 nit display, the image processing device 103 will select the unit mapping (i.e., use the input image directly). If the dynamic range indication indicates that the end-user display is a 500 nit display, the image processing device 103 will select the second mapping.

[0298] If intermediate values ​​of the white point luminances of the end-user display are received, the image processing device 103 may select the mapping that is closest to the ratio between these white point luminances, or may, for example, interpolate between these mappings.

[0299] exist Figure 2 In the example of , the image processing device 103 is arranged to perform the dynamic range transform based on a target display reference received from the content provider apparatus 101, but without any specific information or knowledge of the specific display 107 (i.e. it may simply generate an output image optimized for a given dynamic range / white point, but without explicitly knowing whether the connected display 107 has that value). Thus, an assumed or reference white point luminance may be used. Figure 17 In the example of , the image processing device 103 may perform the dynamic range transform based on the display dynamic range indication received from the display 107, but without any specific information or knowledge of the white point luminance and the specific dynamic range for which the received encoded image was generated (i.e., it may simply generate an output image based on a given dynamic range / white point luminance for the received encoded image, but without explicit knowledge of whether the image was actually generated for such a range and luminance). Thus, an assumed or reference white point luminance for the encoded image may be used. However, it should be understood that in many implementations, the image processing device 103 may be arranged to perform the dynamic range transform in response to information received from the content provider side and from the end-user display. Figure 19 An example of an image processing device 103 is shown, the device comprising a dynamic range processor 203 arranged to perform a dynamic range transform in response to both a target display reference and a display dynamic range indication. It will also be appreciated that for Figure 2 and Figure 17 The comments and descriptions provided for the independent methods apply (mutatis mutandis) to Figure 19 system.

[0300] These methods can be particularly advantageous in non-uniform content distribution systems (such as are increasingly being perceived for future television systems, for example). Indeed, the (peak) brightness of displays is currently increasing rapidly, and in the near future, displays with a wide variety of (peak) brightness are expected to coexist on the market. Since display brightness (and the electro-optical transfer function, which typically specifies how the display converts input pixel (color) drive values ​​into light values ​​that provide a specific psycho-visual impression to the viewer) is no longer known on the content generation side (and is moreover typically different from the reference monitor for which the content is intended / graded), providing the best / optimum picture quality on a display becomes challenging.

[0301] Therefore, in Figure 1 In a system with HDR, the display 107 (or sink device) can send information about its brightness capabilities (peak brightness, grayscale ( / color) reproduction transfer function or other grayscale reproduction properties over its HDR range, such as a specific electro-optical transfer function, etc.) to the image processing device 103.

[0302] In this particular example, the image processing device 103 is a BDROM player connected to a display via an HDMI interface, and thus the display dynamic range indication can be transmitted from the display to the image processing device 103 via the HDMI interface. Thus, the display dynamic range indication can specifically be transmitted as part of EDID information, which can be signaled from the display 107 to the image processing device 103 via HDMI. However, it will be appreciated that the method can be applied to many other video / graphics generating devices, such as DVB receivers, ATSC receivers, personal computers, tablet computers, smartphones, game consoles, and the like. It will also be appreciated that many other wired and wireless interfaces can be used, such as DisplayPort, USB, Ethernet, and WIFI, among others.

[0303] Image processing device 103 can then select, for example, one of the different versions of the content / signal based on, for example, the display brightness. For example, if the signal from content provider apparatus 101 includes both LDR and HDR images, image processing device 103 can select between these images based on the display dynamic range indication, indicating whether the display is an LDR display or an HDR display. As another example, image processing device 103 can interpolate / blend the different brightness versions of the content to derive a new signal that is approximately optimal for the display brightness. As another example, it can adaptively adjust the mapping from the encoded image to the output image.

[0304] It should be understood that in different implementations, different parameters and information may be provided in the display dynamic range indication. In particular, it should be noted that the comments and descriptions previously provided for the target display benchmark may equally apply to the display dynamic range indication. Thus, the parameters and information transmitted from the display 107 to the image processing device 103 may be similar to the parameters and information described for the transmission of information about the target display from the content provider apparatus 101 to the image processing device 103.

[0305] In particular, the display may deliver a maximum luminance / white point luminance for the display, and this may be used by the dynamic range processor 203 to adapt the output signal as previously described.

[0306] Alternatively or in addition, in some embodiments, the display dynamic range indication may include a blackpoint luminance for the display 107. The blackpoint luminance may typically indicate the luminance corresponding to the drive value corresponding to the darkest pixel value. For some displays, the inherent blackpoint luminance for the display may correspond to virtually no light output. However, for many displays, for example, the darkest setting of an LCD element still results in some light output from the display, resulting in black image areas that are perceived as lighter and lighter gray rather than deep black. For such displays, information about the blackpoint luminance may be used by the dynamic range processor 203 to perform tone mapping, where, for example, all black levels below the display's blackpoint luminance are converted to the darkest pixel value (or, for example, a more gradual transition is used). In some scenarios, the blackpoint luminance may include a contribution from ambient light. For example, the blackpoint luminance may reflect the amount of light reflected from the display.

[0307] Furthermore, for many displays, the display dynamic range indication may include more information characterizing the display's OETF. In particular, as previously mentioned, a display may include white point luminance and / or black point luminance. In many systems, the display dynamic range indication may also include more details about the display's OETF at intermediate light outputs. In particular, the display dynamic range indication may include the gamma for the display's OETF.

[0308] The dynamic range processor 203 can then use the information of the OETF to adapt the specific dynamic range transform to provide the desired performance, and in particular, the conversion to an HDR image can not only reflect that a brighter light output is possible, but also fully consider how the relationship between the drive values ​​should be generated to provide the desired light output in the increased brightness range. Similarly, the conversion to an LDR image can not only reflect that a less bright light output is available, but also fully consider how the relationship between the drive values ​​should be generated to provide the desired light output in the reduced brightness range.

[0309] The display dynamic range indication may thus in particular provide information that tells the dynamic range processor 203 how it should map input values ​​corresponding to one dynamic range to output values ​​corresponding to another, typically larger, dynamic range. The dynamic range processor 203 may take this into account and, for example, compensate for any variations or non-linearities in the reproduction of the display 107.

[0310] It will be appreciated that many different dynamic range transforms are possible, and that many different ways of adapting such a dynamic range transform based on a display dynamic range indication may be used. Indeed, it will be appreciated that most of the comments provided for dynamic range transforms based on a target display reference from the content provider apparatus 101 apply (mutatis mutandis) equally to dynamic range transforms based on information about the illuminance characteristics of the end-user display.

[0311] As a low complexity example, the dynamic range transform may simply apply a piecewise linear function to the input values ​​of an LDR image in order to generate improved HDR values ​​(or vice versa). In fact, in many scenarios, a dynamic range transform such as Figure 20 The mapping represents a direct mapping between input pixel values ​​and output pixel values ​​(or, in some scenarios, may reflect a (possibly continuous) mapping between input pixel luminances and output pixel luminances).

[0312] In particular, the method provides a dynamic range transform that maintains the dark areas of the image so that they remain dark, while at the same time allowing a greatly increased dynamic range to provide a much brighter reproduction of the bright areas, as well as a truly improved and more vivid-looking mid-range. However, the exact transform depends on the display on which it is to be reproduced. For example, when reproducing an image intended for a 500 nit display on a 1000 nit display, a relatively modest transform is required, and the stretching of the bright areas is relatively limited. However, if the same image is to be displayed on a 5000 nit display, a much more extreme transform is required to fully utilize the available brightness without brightening the dark areas too much. Figure 20 The figure shows how two different mappings can be used for a 1000 nit display (curve 2001, with a maximum value of 255 corresponding to 1000 nits) and a 5000 nit display (curve 2003, with a maximum value of 255 corresponding to 5000 nits) for a 5000 nit LDR input image (maximum value of 255 corresponding to 5000 nits). The image processing device 103 can further determine appropriate values ​​for other maximum illuminances by interpolating between the provided values. In some implementations, more points can be used to define a curve that is still piecewise linear but has more linear intervals.

[0313] It will be appreciated that the same mapping may be used when mapping from an HDR input image to an LDR output image.

[0314] In some embodiments, the dynamic range transform may include a color gamut transform or a color gamut transform, which may depend on the received display dynamic range indication. Therefore, in some embodiments, the dynamic range processor 203 may modify the chroma of the reproduced image according to the display dynamic range indication. For example, when the received HDR image is reproduced on an LDR display, compression may result in a flatter image with less variation and gradation in each image object. The dynamic range transform can compensate for this reduction by increasing the chroma variation. For example, when an image of an illuminated apple is optimized for reproduction on an HDR display, reproduction on an LDR display with a reduced dynamic range typically makes the apple appear less prominent and less clear and more matte. This can be compensated by the dynamic range transform by making the color of the apple more saturated. As another example, texture variations may become less noticeable perceptually due to reduced illumination variations, and this can be compensated by increasing the chroma variation of the texture.

[0315] In some instances or scenarios, the display dynamic range indication may provide general information for the display, such as standard manufacturing parameters, default EOTF, etc. In some instances or scenarios, the display dynamic range indication may further reflect the specific processing performed in the display and may particularly reflect user settings. Thus, in such instances, the display dynamic range indication does not merely provide fixed and unchanging information that depends solely on the display, but rather provides a time-varying function that may reflect the specific operation of the display.

[0316] For example, the display may be capable of operating in different image modes with different reproduction characteristics. For example, in a "vivid" display mode, the display may reproduce the image with bright areas brighter than normal, in a "silent" display mode, the display may reproduce the image with bright areas darker than normal, and so on. Information about the current mode, such as the specific gamma used for that mode, may be reported to the image processing device 103 as part of the display dynamic range indication, thereby allowing the image processing device 103 to adapt the dynamic range transform to reflect the reproduction characteristics. The image processing device 103 may, for example, override the display setting to compensate for this, or may optimize the transform to maintain a specific setting.

[0317] The display dynamic range indication may also reflect other processing settings for the display. For example, clipping levels, backlight power settings, color scheme mappings, etc. may be communicated to the image processing device 103 where they may be used by the dynamic range processor 203 to adapt the dynamic range transform.

[0318] Figure 21An example of elements of a display 107 is shown, wherein the display provides an indication of a display dynamic range to the image processing device 103 .

[0319] In this example, the display comprises a receiver 2101 that receives an image signal output from the image processing device 103. The received image signal is coupled to a driver 2103, which is further coupled to a display panel 2105 that reproduces the image. The display panel may be, for example, an LCD or plasma display panel known to those skilled in the art.

[0320] The driver 2103 is arranged to drive the display panel 2105 so that it reproduces the encoded image. In some embodiments, the driver 2103 may perform advanced and possibly adaptive signal processing algorithms, including tone mapping, color grading, etc. In other embodiments, the driver 2103 may be relatively low complexity and may, for example, only perform a standard mapping from input signal values ​​to drive values ​​for the pixels of the display panel 2105.

[0321] Furthermore, in the system, the display 107 comprises a transmitter 2107 arranged to transmit data signals to the image processing device 103. For an HDMI connection, as will be described later, the data signals may be transmitted in a DDC channel, for example using an E-EDID structure.

[0322] The transmitter 2107 generates a data signal comprising an indication of a display dynamic range for the display (107). Thus, in particular, the transmitter 2107 indicates, for example, the white point luminance and optionally the EOTF of the display. For example, a data value providing an index between several predetermined white point luminances or EOTFs may be generated and transmitted.

[0323] In some low-complexity embodiments, for example, the white point luminance can be a fixed value stored in transmitter 2107, with the transmitter simply transmitting this standard value. In more complex embodiments, the display dynamic range indication can be determined to reflect a dynamically changing and / or adaptive value. For example, driver 2103 can be arranged to operate in different display modes, and the display dynamic range indication can be adapted accordingly. As another example, a user setting for the display, such as the brightness level, can be reflected in the display dynamic range indication generated and transmitted by transmitter 2107.

[0324] As previously mentioned, the display dynamic range indication may include an ambient light metric, and the dynamic range processor may be arranged to adapt the dynamic range transform in response to the ambient light metric. The ambient light metric may be provided as explicit and separate data, or may be reflected in other parameters. For example, the ambient light metric may be reflected in the blackpoint luminance, which may include a contribution corresponding to light reflection from the display.

[0325] In many scenarios, the display may include a light detector placed in front of the display. This light detector may detect the overall ambient light level, or may specifically measure light reaching the display from a given direction that is likely to be reflected back toward the viewer. Based on this light detection, the display may generate an ambient light indication that generally reflects, for example, the ambient light level of the viewing environment, or specifically, for example, reflects an estimate of the light reflected from the screen. The display 107 may report this value to the image processing device 103 as a separate value, or, for example, by calculating an effective black illuminance level that reflects the amount of light reflection.

[0326] The dynamic range processor 203 can then adapt the dynamic range transformation accordingly. For example, when ambient light levels are high, more use of the HDR display's additional brightness level can be used more aggressively to produce a bright-looking image with high contrast. For example, the average light output can be set relatively high, and even mid-range illuminance can be pushed toward the HDR range. Bright areas can be reproduced using the full HDR range, and even dark areas will typically be reproduced at relatively high light levels. However, the increased dynamic range of the HDR image allows such relatively bright images to still exhibit large illuminance variations and thus still have high contrast.

[0327] Therefore, the HDR capability of the display is used to generate an image that is perceived as bright and has high contrast even when viewed in bright daylight, for example. Such an image is typically not suitable for a dark room because it would be overwhelming and appear too bright. Therefore, in a dark environment, the dynamic range conversion will perform a much more conservative LDR to HDR conversion, which, for example, maintains the same LDR light output for dark and mid-range values ​​and only increases the brightness for brighter areas.

[0328] This approach may allow the image processing device 103 to automatically adapt an LDR to HDR dynamic range conversion (or for example an HDR to HDR dynamic range conversion) to match the specific viewing environment of the display. Furthermore, this is possible without the image processing device 103 having to make any measurements of that environment or indeed even be placed in or near that environment.

[0329] The ambient light indication may typically be optional, and thus the image processing device 103 may use it if it is available, and otherwise just perform a default dynamic range transform for certain characteristics (eg the OETF of the display).

[0330] The optional extended information provided by the display about its viewing environment (especially the ambient light) is therefore used by the image processing device 103 to perform more complex image / video optimization transformations in order to present an optimal image / video to the display, where the optimization may include not only characteristics of the display but also characteristics of the viewing environment.

[0331] Therefore, further optimization can be performed when the display provides information about the viewing environment. The display will typically periodically measure the ambient light and send information about this (e.g., three parameters: brightness and color in XYZ format) to the image processing device 103. This information may typically not be provided as part of the EDID data or any other data type primarily used for a one-time transmission of information. Instead, it may be transmitted, for example, in a separate channel, such as using HDMI-CEC. This periodic measurement and updating can, for example, allow the image processing device 103 to automatically adapt the processing to provide an image more suitable for darker viewing conditions, such as by applying a different color / luminance mapping, if the user turns off the light near the display.

[0332] An example of a set of relevant parameters that may be reported by an end-user display in a display dynamic range indication includes:

[0333] The absolute maximum illuminance of the end-user display (white point illuminance).

[0334] · Gamma of the end-user display – factory setting.

[0335] The absolute maximum illuminance of an end-user display may be defined, for example, for typical display settings, factory default settings, or settings that produce the highest apparent brightness.

[0336] Another example of a set of related parameters that may be reported by an end-user display in a display dynamic range indication includes:

[0337] The maximum illuminance of the end-user display for the current settings of brightness, contrast, etc.

[0338] · Gamma of the end user's display - current setting.

[0339] The first parameter set is time-independent, while the second set varies over time because it depends on user settings. The use of one or the other set has an impact on the system's behavior and user experience, and it should be understood that the specific parameter set used in a particular system depends on the preferences and requirements of the system. In fact, parameters can be mixed between the two sets, and for example, factory default settings can be provided at power-up, and parameters that depend on user settings can be reported periodically thereafter.

[0340] It should also be understood that a particular set of parameters can characterize an EOTF for a display, which is either a factory default EOTF or an EOTF that depends on specific current user settings. Thus, these parameters can provide information about a mapping between the display's luminance output and drive values, which allows the image processing device 103 to generate drive values ​​that result in a desired output image. It should be understood that in other implementations, other parameters can be used to characterize part or all of the mapping between the display's luminance output and drive values.

[0341] It will be appreciated that many different methods may be used to communicate the display dynamic range indication from the display to the image processing device 103 .

[0342] For example, for parameters of a display that are independent of user settings and do not change over time, communication can be efficiently transmitted in the DDC channel using the E-EDID structure for an HDMI connection.

[0343] In a low complexity approach, a set of categories may be defined for end-user displays, each category defining a range of related parameters. In such an approach, only the category identification code for the end-user display needs to be transmitted.

[0344] A specific example of transmitting display dynamic range indication data in the E-EDID format will be described.

[0345] In this specific example, the first 128 bytes of the E-EDID should contain the EDID 1.3 structure (basic EDID block).

[0346] For the display dynamic range indication parameter, a new display descriptor block in the E-EDID data structure can be defined. Since current devices do not understand this new display descriptor block, they will simply ignore it, thereby providing backward compatibility. One possible format of this "Illuminance Behavior" descriptor is listed in the following table.

[0347] Byte Number Byte Value Description

[0348] 0,1 2 00h Indicates that the 18-byte descriptor is a display descriptor

[0349] 2 1 00h Reserved

[0350] 3 1F 6h Display descriptor tag number indicating this is an illuminance descriptor

[0351] 4 1 00h Reserved

[0352] 5 1 Peak_Luminance

[0353] 6-8 3 Transfer curve (optional; e.g., α, β, offset)

[0354] Peak_Luminance is a parameter having a value between 0 and 255 that indicates the peak luminance of the display according to the following formula:

[0355] Display peak illuminance (cd / m2) = 50 x Peak_Luminance,

[0356] Thus covering the range of 0 to 255*50=12750cd / m2 or 255*100

[0357] The transfer curve can be a gamma curve (as in ITU601, ITU709, etc.), but allows for much higher gamma (up to 10). Alternatively, in some scenarios, different transfer (or logarithmic) curve parameters may be more appropriate. For example, instead of a gamma function:

[0358]

[0359] You can use the power function:

[0360]

[0361] where the parameters α, β, and ∆ can be set to provide the desired characterization.

[0362] The additional information can thus be used by the image processing device 103 to make more advanced decisions about determining the different video and graphics (or multiple image components) gray levels, such as, for example, global processing such as gamma-based modifications. With more information about, for example, how the display gamma-remaps all gray levels, the dynamic range processor 203 can make more informed decisions about the final appearance of the video and secondary images (and how they may overlap in luminance, also based on geometric properties such as, for example, how large the sub-regions are, etc.).

[0363] In the previous examples, the display 107 provides a display dynamic range indication that tells the image processing device 103 how the display will display the incoming display signal. In particular, the display dynamic range indication can indicate a mapping between the light output applied by the display and the drive values. Thus, in these examples, the display dynamic range indication tells the image processing device 103 the available dynamic range and how it is presented, and the image processing device 103 is free to adapt the dynamic range transform as it sees fit.

[0364] However, in some systems the display may also be able to exert some control over the dynamic range transform performed by the image processing device 103. In particular, the display dynamic range indication may comprise dynamic range transform control data, and the dynamic range processor 203 may be arranged to perform the dynamic range transform in response to the dynamic range transform control data.

[0365] The control data may, for example, define the operations or parameters of the dynamic range transform that must be applied, can be applied, or is recommended to be applied. Furthermore, the control data may be differentiated for different characteristics of the images to be encoded. For example, separate control data may be provided for multiple possible initial images, such as one set for a 500 nit image, another set for a 1000 nit encoded image, and so on.

[0366] For example, the display may dictate, based on the dynamic range of the received image, which tone mapping should be performed by the dynamic range processor 203. For example, for a 2000 nit display, the control data may dictate that one mapping should be used when mapping from a 500 nit LDR image, another mapping should be used when mapping from a 1000 nit image, and so on.

[0367] In some scenarios, the control data may specify the boundaries between these mappings, where the mappings are predefined within various intervals (e.g., standardized or known both on the content provider side and on the renderer side). In some scenarios, the control data may further define the elements of the different mappings, or may specify the mappings more precisely, for example using gamma values, or specify a specific transfer function.

[0368] In some embodiments, the dynamic range transform control data may directly and explicitly specify the dynamic range transform that should be performed to transform the received image into an image having a dynamic range corresponding to the dynamic range of the display. For example, the control data may specify a direct mapping from input image values ​​to output image values ​​for a range of received image white points. The mapping may be provided as a simple parameter, allowing the dynamic range processor 203 to implement the appropriate transform, or detailed data may be provided, such as a specific lookup table or mathematical function.

[0369] As a low-complexity example, a dynamic range transform can simply apply a piecewise linear function to the input values ​​of an LDR image to generate improved HDR values ​​(or vice versa). In fact, in many scenarios, a dynamic range transform such as Figure 20 The simple mapping consisting of two linear relations shown in .

[0370] In particular, as previously described, such an approach can provide a dynamic range transform that maintains dark areas of an image so that they remain dark, while at the same time allowing a greatly increased dynamic range to provide a much brighter reproduction of bright areas, as well as a truly improved and more vivid-looking mid-range. However, the exact transform depends on the dynamic range of the received image and the dynamic range of the end target display. In some systems, the display can therefore dictate that the tone mapping to be performed by the image processing device 103 simply transfer the coordinates of the inflection point of the function (i.e., the intersection between the linear elements of the mapping).

[0371] One advantage of this simple relationship is that the desired tone mapping can be communicated with very low overhead. In fact, a simple two-component data value can specify the desired tone mapping performed by the image processing device 103 for different displays. Different coordinates of the "knee" point can be communicated for different input images, and the image processing device 103 can determine the appropriate value for other input images by interpolating between the provided values.

[0372] It will be appreciated that most of the comments provided in relation to the provision of dynamic range transform control data from the content provider apparatus 101 apply equally well (mutatis mutandis) to dynamic range transform control data received from the display 107 .

[0373] Thus, in some scenarios, the display 107 may control the dynamic range transform performed by the image processing device 103. One advantage of this approach is that it may allow a user to control the desired reproduced image, for example by controlling the display and without any requirement for user input or settings to the image processing device 103. This may be particularly advantageous in scenarios where multiple image processing devices are used with the same display, and in particular, it may help provide homogeneity between images from different image processing devices.

[0374] In many implementations, the control data from the display 107 may not specify a specific tone mapping that should be performed, but rather provide data defining boundaries within which the dynamic range transform / tone mapping may be freely adapted by the image processing device 103 .

[0375] For example, rather than specifying the maximum brightness level for a transition point, the control data may define limits for the transition point (possibly providing different limits for different maximum brightness levels). Figure 20 The image processing device 103 can thus individually determine the desired parameters for the dynamic range transform so that they can be set to provide a preferred transition for a particular display, taking into account, for example, specific user preferences. However, at the same time, the display can limit this freedom to an acceptable level.

[0376] Thus, the dynamic range transform control data may include data defining the transform parameters that must be applied for the dynamic range transform performed by the dynamic range processor 203 and / or defining limits for these transform parameters. The control data may provide such information for a range of input image dynamic ranges, thereby allowing the dynamic range transform to be adapted to different received images. Furthermore, for input images with a dynamic range not explicitly included in the control data, appropriate data values ​​may be generated from the available data values, for example, by interpolation. For example, if the inflection point between two straight line segments for 500 nit and 2000 nit input images is specified, then the appropriate value for a 1000 nit input image may be found by simple interpolation (e.g., in this particular example, by simple averaging).

[0377] It will be appreciated that many different and varying approaches for dynamic range conversion and for how this is limited, adapted and controlled from the display side via additional control data may be used in different systems, depending on the specific preferences and requirements of the individual application.

[0378] In some scenarios, the control data may only provide suggestions for appropriate mappings that can be applied, for example, in the mid-range region. In such cases, the display manufacturer can assist the image processing device 103 by providing recommended dynamic range transform parameters found (e.g., through manual optimization by the display manufacturer) to provide high image quality when viewed on a particular display. The image processing device 103 can use this to advantage, but is free to modify the mapping, for example, to suit individual user preferences.

[0379] In many scenarios, the mapping performed at least in part based on the control data represents a relatively low-complexity functional relationship, such as a gamma mapping, an S-curve, a combined mapping defined by partial specification for various ranges, etc. However, in some scenarios, more complex mappings may certainly be used.

[0380] As mentioned, the control data may provide mandatory or voluntary control data. Indeed, the received data may include one or more fields indicating whether the provided tone mapping parameters are mandatory, allowed or recommended.

[0381] In some systems, displays may be able to operate with different dynamic ranges. For example, a very bright HDR display with a white point luminance of, say, 5000 nits may also be able to operate in a display mode with a white point luminance of 4000 nits, another display with 3000 nits, one display with 2000 nits, another display with 1000 nits, and finally may be able to operate in an LDR mode with a white point luminance of only 500 nits.

[0382] In such a scenario, the data signal from the display may indicate multiple luminance dynamic ranges. Thus, each of these different luminance dynamic ranges may correspond to a dynamic range mode for the display. In such an arrangement, the dynamic range processor 203 may select one of the luminance dynamic ranges and proceed to perform a dynamic range transform in response to the selected display dynamic range. For example, the dynamic range processor 203 may select a dynamic range of 2000 nits and then proceed to perform a dynamic range transform to optimize the generated image for that white point luminance.

[0383] The selection of an appropriate luminance dynamic range for a display can depend on various factors. In some systems, the image processing device 103 can be arranged to select an appropriate display dynamic range based on the image type. For example, each range can be associated with a given image type, and the image processing device 103 can select the image type that most closely corresponds to the received image and then proceed to use the dynamic range associated with that image type.

[0384] For example, several image types corresponding to different content types can be defined. For example, one image type might be associated with cartoons, another with football matches, another with news programs, another with movies, and so on. The image processing device 103 can then determine the appropriate type for the received image (e.g., based on explicit metadata or based on content analysis) and proceed to apply the corresponding dynamic range. This can, for example, result in a very vivid rendering of cartoons with high contrast and brightness, while allowing, for example, dark movies to be reproduced without being unnatural.

[0385] The system can thus be adapted to the specific signal being reproduced. For example, a crudely produced consumer video, a brightly lit football game, a well-lit news program (e.g., with reduced contrast), etc. can be displayed differently, and in particular, the dynamic range of the reproduced image can be adapted to a range that is particularly suitable for the specific image.

[0386] It was mentioned previously that the display may provide control data to the image processing device 103. However, alternatively or in addition, in some systems it may be the image processing device 103 that provides control data to the display 107.

[0387] Therefore, if Figure 22 As illustrated in , the image processing device 103 may include a controller 2201 capable of outputting a display control data signal to the display 107 .

[0388] The display control signal can specifically instruct the display to operate in a specific dynamic range mode selected by the image processing device 103 for a particular image. Consequently, poorly lit amateur images will be reproduced using a low dynamic range, thereby avoiding the introduction of unacceptable errors caused by converting to a high dynamic range that was not actually present in the original image. Simultaneously, the system can automatically adapt so that high-quality images can be effectively converted to high-dynamic range images and presented as such. As a specific example, for amateur video sequences, the image processing device 103 and display can automatically adapt so that the video is presented with a 1000 nit dynamic range. However, for professionally captured high-quality images, the image processing device 103 and display 107 can automatically adapt so that the video is presented using the full 5000 nit dynamic range that the display 107 is capable of.

[0389] The display control signals may thus be generated to include commands such as "use 1000 nits dynamic range", "use LDR range", "use maximum dynamic range", and so on.

[0390] The display control data may be used to provide several commands in the forward direction (from the image processing device 103 to the display).For example, the control data may include image processing instructions for the display, and in particular may include tone mapping instructions for the display.

[0391] For example, the control data may specify a brightness setting, a clipping setting, or a contrast setting that should be applied by the display 107. The image processing instructions may thus define mandatory, voluntary, or recommended operations that should be performed on a received display signal by the display 107. The control data may thus allow the image processing device 103 to control some of the processing performed by the display 107.

[0392] The control data may, for example, specify specific filtering that should or should not be applied. As another example, the control data may specify how the backlight operation should be performed. For example, the display may be able to operate in a low-power mode that uses aggressive local dimming of the backlight, or it may be able to operate in a high-power mode in which local dimming is used only when it can improve the reproduction of dark areas. The control data may be used to switch the display between these operating modes.

[0393] In some examples, the control data may specify a particular tone mapping that should be performed by the display, or may in fact specify that the tone mapping function should be switched off (thereby allowing the image processing device 103 to have full control over all tone mapping).

[0394] It should be understood that in some embodiments, the system may use control data in both directions: forward from image processing device 103 to display 107 and backward from display 107 to image processing device 103. In such cases, it may be necessary to introduce rules and operating conditions to resolve potential conflicts. For example, the image processing device 103 may be arranged as a master device, controlling and dictating display 107 in the event of a conflict. As another example, control data in both directions may be limited to specific parameters to prevent conflicts.

[0395] As another example, the master-slave relationship can be user-configurable. For example, the image processing device 103 and the display 107 can each be configured to provide control data for the other entity, and can specifically both be capable of operating as a master device. In such a system, a user can designate one of these devices as the master device, with the other becoming a slave device. The user can specifically choose this based on their preference for whether to control the system from the image processing device 103 or the display 107.

[0396] The system described above can thus allow communication between a content provider and an image processing device and / or between an image processing device and a display. These methods can be applied to many systems featuring communication channels between a content provider and an image processing device and / or between an image processing device and a display. Examples include BDROM, ATSC and DVB, or the Internet, among others.

[0397] The system can utilize a communication channel between the image processing device and the display, such as an HDMI or DisplayPort communication interface. This communication can be in both directions. For example, if the intelligent display is doing all the optimal video and graphics mapping, then the image processing device can, for example, read the control parameters and reformat and transmit them in a similar HDMI structure.

[0398] The method is particularly applicable to BDROM systems. Thus, the method can augment the BDROM specification to allow the transmission of target display parameters and control commands. Using such data in conjunction with end-user display parameters can allow BDROM players to, for example:

[0399] Perform additional video and / or graphics tone mapping or other processing in the player based on the characteristics of the target display and the end-user display.

[0400] • Perform additional video and / or graphics tone mapping or other processing manipulated by commands in the data stream provided by the content provider.

[0401] In some embodiments, the image processing device 103 may also comprise a transmitter for transmitting the dynamic range control data to the content provider apparatus 101. Thus, the image processing device 103 may be able to control or at least influence the processing or operations performed at the content provider apparatus 101.

[0402] As a specific example, the control data may include an indication of a preferred dynamic range for the image, and may in particular include an indication of the dynamic range (eg white point luminance and optionally EOTF or gamma function) for the end-user display.

[0403] In some embodiments, the content provider apparatus 101 may be arranged to take into account the indication of a preferred dynamic range when performing tone mapping. However, in other embodiments, the content provider apparatus 101 may provide a number of predetermined tone mappings, for example involving manual tone mapping by a tone mapping expert. For example, a tone-mapped image may be generated for a 500 nit display, for a 1000 nit display, and for a 2000 nit display.

[0404] In such a scenario, the content provider apparatus 101 may be arranged to select which image to transmit to the image processing device 103 based on the received control data. In particular, the image that is closest to the dynamic range indicated by the control data may be selected and transmitted to the image processing device 103.

[0405] Such an approach may be particularly suitable for streaming applications, where the streamed signal may be dynamically updated in order to match the dynamic range of the end-user display as closely as possible.

[0406] This approach may reduce the extent of dynamic range transform that has to be applied in the image processing device 103 and, in particular for scenarios where the content provider apparatus 101 may provide an image that is tone mapped to the same dynamic range as the end user display, may allow the dynamic range transform to be a simple no-op (i.e. it may allow the received image to be used directly by the image processing device 103).

[0407] There are a variety of application scenarios in which the present embodiments can be useful. For example, encoding with a specific white point or expected white or similar value for pixel image content (e.g. DCT encoding of local object textures) allows for a more intelligent allocation of required code levels and expected output luminances for a variety of different possible output signals. One could, for example, encode the texture of a dark room as if it were well illuminated (i.e. pixel luminances up to 255, rather than having a maximum luminance of, e.g., 40 as in a dark scene image), but with the provision that "white", i.e., 255, must be treated in a specific way, i.e., it must be reproduced as dark. A simple way of doing this is to jointly encode the output luminance, e.g., to be reproduced on a display, with this 255 luminance code. The same thing can be done to encode significantly bright values, such as in, e.g., a foggy scene with strong light in it.

[0408] Regarding gamma, this can be used, for example, to indicate whether the material is encoded, such as based on gradient celluloid negative material or using a digital camera with a strong saturation setting. Or, typically, for any other reason that the gamma intention of one display deviates from another for the final display on which the reproduction will occur. EOTFs can typically encode, for example, rather crude grayscale value behavior, e.g., to compensate for displays with different gammas or different viewing environments that can serve as compensation for different gammas. Thus, information such as "a signal optimized for a reference display encoded for gamma = X / intended for this display, e.g., on this display" can be conveyed so that another display with other characteristics knows how to process it to achieve a more optimal reproduction towards the artistic intent. Tone mappings can be more general, as they can also convey, for example, what typical reproduction intent was applied to the image (e.g., an artist made clouds darker and more sinister, which should still appear at least approximately the same in the output displayed image using any final display reproduction mathematics).

[0409] We use Figure 23As a further example, we'll illustrate the principle of (approximately) encoding any HDR scene in an LDR image ("HDR_encoded_as_LDR"), which can be, for example, a 10-bit image standard, but we'll also explain an interesting variant encoding in a classic 8-bit image, i.e., an image compatible with standards such as MPEG2 or AVC and directly usable by classic reproduction techniques. While it might be desirable to use more bits for an HDR signal, such as 12, 16, or 22 bits, 8 bits for the luminance channel already conveys a lot of information for any reproduced peak white (especially for approximating complex textures and many possible colors). Furthermore, many HDR signals can allow for a significant degree of approximation, as, for example, the sun doesn't need to be encoded with its true brightness, as it will be approximated when reproduced on a display anyway. For the LDR range of luminance, even fewer bits are often quite sufficient, as, for example, 6 bits provide a reasonable approximation / quality of image quality (as known from printing).

[0410] In this example, we therefore encode an HDR image exactly within an 8-bit luminance structure by applying a typically simple appropriate mapping, i.e. a mathematical transformation, to at least the luminance of the pixels. The criterion is that, on the one hand (by jointly encoding the transform), an HDR image (i.e. an 8-bit or 12-bit interpolated approximation intended for reproduction on a 0.1-5000 nit display, for example) can be reconstructed from the LDR 8-bit coded image by reversing the jointly coded mapping (without any or significant post-correction), i.e. this HDR image will look (almost) psychovisually indistinguishable, or at least it will still be a good HDR image (i.e. typically exhibiting the appearance of the HDR scene, close to how it would be reproduced if it were generated directly from the original, e.g., 12-bit HDR image IM_HDR, where its HDR range HDR_Rng is the luminance to be reproduced). On the other hand, however, we want an LDR image, i.e. one that still allows good visual reproduction if the 8-bit signal is applied directly to an LDR display of, e.g., 0.1-400 nit. For example, an HDR image IM_HDR can be compressed linearly into the LDR range LDR_Rng, e.g., by discarding the least significant bits and assuming that white (maximum code value 255) is expected to be reproduced at 400 nits. However, since such HDR images typically contain very bright objects in the upper part of their luminance range, such an 8-bit image will appear too dark on an LDR display, since the relevant darker parts of the image / scene now end up at very low luminance codes, i.e., the display output luminance. However, a substantial improvement could have been achieved by applying an optimal gamma before encoding the HDR / 12-bit / 5000 nit image into an LDR / 8-bit / 400 nit classic (e.g., AVC) representation. That is, this gamma maps bright objects to brighter parts (e.g. making them less contrasty and pastellish, but still acceptably on an LDR display, still with enough information for a reasonable inverse mapping to HDR again), which is optimally balanced by at the same time not squashing the darker parts (e.g. dark trees) too much, so that these dark objects still look reasonably bright on an LDR display (and can also recreate good HDR dark parts for viewing with dark viewing surroundings; or enough texture data is available to encode these brighter on an HDR display).

[0411] Typically, such a mapping can be a general global transform on brightness (i.e., a mapping that does not take into account geometrically local specific details, such as where a pixel resides in the image, or what the brightness of its neighbors is, or what kind of scene object it belongs to, but instead takes only the brightness value of the pixel as input). Somewhat more complex mappings can be jointly encoded, such as transforms that are only used for delimited sub-regions or objects in the image (local mappings, in which case typically also jointly encode additional information, such as the boundaries defining the object). In general, however, while any transform that is effective for the embodiments we disclose can be envisioned, if it is only to reduce the workload of a typical human grader in defining these optimal mappings, then they will typically be few and simple (no local mappings are encoded if a general global function such as an S-curve or multi-point spline is sufficient).

[0412] This example is illustrated using a content creator-side image encoding device 510, in which a human optimizes the output image (typically encompassed by a transform / mapping function or algorithmic strategy, such as metadata MET in a certain image signal structure S as specified in, for example, AVC or HEVC) encoded into a memory (e.g., a Blu-ray disc 511 or temporary storage for final encoding of a signal to be stored or transmitted). The grader can check the image on one or more displays 530, for example to see if the LDR and restorable HDR images look good on various reference LDR and HDR displays, before sending their instructions to the image encoding unit 550 (which performs the mapping to 8-bit luminance) and the formatter 554, which formats the image and its color code according to the currently used image coding standard and jointly encodes the texture image and the transformed metadata into the output 512.

[0413] At the top we see how the HDR image IM_HDR (which is input via the input 511 of the image encoding device 510 ) with its HDR range is mapped to an LDR image (if on an LDR display) with the reproduced luminance with its LDR range.

[0414] While we've described "HDR_encoded_as_LDR" in terms of encoding on the content creation side for transmission to a content consumption side, such as a consumer's home, it's clear that the same "HDR_encoded_as_LDR" embodiment can also be used when transmitting (e.g., via transcoding) between different devices, such as two home devices on a home network. Thus, for example, an automatic image analysis and mapping unit can apply automatic image analysis and corresponding luminance mapping methods. This can be done, for example, by a content receiving or storage device, when having a first image representation, such as a 12-bit HDR image, and sending it to a television via HDMI or other network connection. Alternatively, an 8-bit LDR image can be encoded according to or for a wireless standard for streaming to a mobile display that has HDR capabilities, but at any rate has lower visual quality.

[0415] Typically, at least for new HDR standards, if such an 8-bit encoding (e.g., 8-bit luma and the normal 2×8-bit encoding for chroma) is implemented in a classic LDR (e.g., MPEG) scheme, the standard will annotate with metadata that the LDR image is not actually an LDR image intended primarily for an LDR display (although, as mentioned, it may have been constructed so that it still looks reasonable on an LDR display with, for example, 100 nits peak brightness or peak white), but rather an HDR image. This can be done with a normal HDR codec that is assumed to give a reasonable reproduction for an HDR display with, for example, a peak brightness of around 3500 nits. A somewhat more specific aspect of the jointly encoded first target display reference in the metadata may also be that the HDR signal was originally graded on, for example, a 5000 nit display. This will mean that the actual luminance of image objects (also when mapped to LDR luminance) will have values ​​that depend on what is typically reproduced on a 5000 nit display (e.g., retaining a certain sub-range of high luminance, pushing normal luminance scene objects in the scene, typically main objects, to deeper luminance values ​​already in, for example, a 16-bit HDR coarse gradation). In this case, a real 3500 nit or 2500 nit display can further optimize the color conversion function for best gamut mapping to the actual display gamut according to quality criteria (e.g., output luminance similarity, or psychovisual appearance quality metrics on HDR effects, etc.), rather than just using the HDR signal for driving the display under the assumption that it still gives a reasonable picture (peak luminance is in the range of the expected value around 5000 nits). One could even jointly encode a second peak luminance value for the final HDR_encoded_as_LDR image (e.g., looking most reasonable on a 250 nit display, starting to exhibit slight artifacts at higher and or lower peak luminances, and potentially even further preferably specifying such artifacts in a functional way (e.g., geometric position, etc.) so that a renderer can attempt to correct for them).

[0416] By HDR displays, we mean displays with a peak brightness greater than 750 nits. Displays with lower peak brightness, especially those below 500 nits, are LDR displays.

[0417] The predetermined quality criterion for determining whether the LDR reproduction and the HDR reproduction of the recovered HDR signal from the LDR image (typically derived simply by reversing the mapping of the joint encoding, but some additional processing may be performed, such as quantization boundary mitigation image processing applied by the receiving device, etc.) will be a mathematical algorithm or a human operator who determines that the final image encoding is good enough when encoding it for distribution. However, the quality estimator applied by humans and software encoding will apply such image analysis criteria as: whether there is sufficient (local) contrast in various blocks, especially in the block in the center of the image (i.e., sufficient visibility of the original, e.g., master celluloid negative scan 12 or 14 bit HDR image is still preserved), whether there are many artifacts such as quantization boundaries and how large or wide the steps are, whether there are sufficient spatial sub-patterns in the luminance histogram (whether the original film look / intent is preserved), whether spatially separated objects have sufficient intra-area contrast, etc. And in particular, if the original is present in a networked system such as a connected device, then the sending device (e.g. a set-top box) determines whether the recoverable HDR signal is sufficiently close to the original, e.g., 12-bit HDR signal present at that location (which may be done based on a mathematical criterion like MSE or PSNR, or a psycho-visually weighted difference, etc.). For example, after an automatic brightness conversion and corresponding automatic color adjustment (which may be, for example, a gamma function or similar power function, or an S-curve tuned for factors such as typical median brightness in a scene, or further image analysis such as detecting small bright areas and giving them their own sub-range and corresponding mapping function, etc.), the color grader (e.g., after first performing a main grading on the 16-bit original HDR) then further color grades the HDR_encoded_as_LDR image. On the one hand, this is therefore done so as to give a good usable LDR grading, but on the other hand, also give a recoverable HDR, so he can assign the important information of the containing area to a sub-area of ​​the LDR range, the sub-area having enough code values, but still converting them into an "average" brightness range (e.g., not too dark, so that the darker areas are still well visible, but dark enough to express emotions) that shows a good reproduction on the LDR. Typically, he can do this by fine-tuning the brightness / color mapping function from an automatic brightness / color mapping function. At least the brightness should be correctly placed, and then the color can be optimized from this point on. For example, if a certain background area appears dim in the LDR reproduction, he can still tune the global mapping function in the part corresponding to those pixels, as long as it does not get worse in other parts of the LDR reproduction, and of course, via inverse mapping, the recoverable HDR image will not suffer from subcritical quality.In principle, one could even choose to grade a spatially localized image region (encoding this region for the first or second time, in addition to the first image) so that it corresponds to a different display peak brightness, gamma, etc., than the rest of the image, forcing the renderer to account for this before rendering. This could be useful, for example, to emphasize dark areas, but generally, one would keep things simple and fixed to a single HDR-intended display encoding. Therefore, a global mapping function from HDR to LDR (or its inverse mapping from LDR to HDR) and, if applicable, further transform data are also encoded. Legacy systems are unaware of all this and, in principle, could use a classically encoded LDR image even if the first target display reference and other information were discarded from the image encoding signal, but generally, this would be recorded in a portion of the data, reserved for, for example, upscaling and ignored by older systems but used by newer ones. An HDR image decoding box can view this data regardless, even if it is connected to a legacy LDR display. It can then improve the LDR signal to some extent by color transforming it, rather than just applying the LDR signal to the display in order to drive the reproduction, given all local factors, and all this additional metadata information (whether using only the intended display type and whether "blindly" based on the similarity between the current reproduction environment used by the grader and the intended reproduction environment, or by also transforming the LDR input signal using some or all of the information in the mapping function between the LDR encoding and the original HDR encoding, which indicates something about the difference between the two, i.e. the HDR nature and composition of the original scene and / or the graded HDR signal).

[0418] Such a signal has the advantage that any HDR capable system knows that the HDR image is actually encoded as an LDR image and can optimally restore this HDR image before reproduction, however, backwards compatible legacy LDR systems can also use the LDR image directly for reproduction.

[0419] It will be clear to those skilled in the art which combinations can be derived from our teachings, such as encoding several HDR gradations for several HDR displays in several LDR encodings, re-grading, alternatives for different scenarios, and also variations in viewing environment that can be viewed as display reproduction types. Where we mention specific parameters, such as 8-bit legacy encoding, it should be understood that the same approach can be achieved using, for example, 10-bit LDR encoding technology, and we do not intend to exclude any variants, combinations, or simply alternative implementations. Therefore, our teachings can certainly be combined and are contemplated as such without tediously specifying each and every easily readable variant, unless it is clearly apparent from our description that this is not possible or intended. Of course, the encoding can be used in a variety of different scenarios, whether professional or, for example, mobile consumer, in a number of applications, such as security systems, news gathering, and so on. It can be used automatically and extensively within any technical system, such as within an IC or multi-chip, or a networked technical system. Some of the parts of the invention may form separate commercial applications, such as for example any grading may be performed as a re-scaling on an existing image encoding, whether it has already been graded in that way but is now improved, or lacks some kind of grading, such as for a new popular display or display method.

[0420] It should be understood that, for the sake of clarity, the above description describes embodiments of the present invention with reference to different functional circuits, units, and processors. However, it should be apparent that any suitable distribution of functionality between different functional circuits, units, or processors may be used without detracting from the present invention. For example, functions shown to be performed by separate processors or controllers may be performed by the same processor or controller. Therefore, references to specific functional units or circuits should be viewed merely as references to appropriate components for providing the described functionality, rather than as indicating a strict logical or physical structure or organization.

[0421] All method embodiments and teachings are corresponding to corresponding devices and potentially other products such as output signals, embodiments, and vice versa. The present invention can be implemented in any suitable form, including hardware, software, firmware or any combination of these. Alternatively, the present invention can be implemented at least in part as computer software running on one or more data processors and / or digital signal processors. The elements and components of the embodiments of the present invention can be implemented in any suitable manner physically, functionally and logically. In fact, the function can be implemented in a single unit, in multiple units or as the part of other functional units. Similarly, the present invention can be implemented in a single unit, or can be distributed between different units, circuits and processors physically and functionally.

[0422] Although the present invention has been described in conjunction with certain embodiments, it is not intended that the present invention be limited to the specific forms set forth herein. Rather, the scope of the present invention is limited solely by the appended claims. Furthermore, although features may appear to be described in conjunction with specific embodiments, it will be appreciated by those skilled in the art that various features of the described embodiments may be combined in accordance with the present invention. In the claims, the word "comprising" does not exclude the presence of other elements or steps.

[0423] Furthermore, although listed individually, multiple components, elements, circuits, or method steps may be implemented by, for example, a single circuit, unit, or processor. Furthermore, although individual features may be included in different claims, these features may potentially be advantageously combined, and inclusion in different claims does not imply that a combination of features is not feasible and / or advantageous. Furthermore, the inclusion of a feature in one claim category does not imply limitation to that category, but rather indicates that the feature is equally applicable to other claim categories, if appropriate. Furthermore, the order of features in a claim does not imply any specific order in which the features must function, and in particular, the order of steps in a method claim does not imply that the steps must be performed in that order. Rather, the steps may be performed in any appropriate order. Furthermore, singular references do not exclude plural references. Thus, references to "a," "an," "first," "second," etc., do not exclude plural references. Reference numerals in the claims are provided merely as examples for clarification and should not be construed as limiting the scope of the claims in any way.

Claims

1. An image processing device, comprising: a receiver (201) for receiving a high dynamic range image signal from a content provider apparatus (101), the image signal comprising at least a first encoded image and a first target display reference, the first target display reference indicating a dynamic range of a first target display for which the first encoded image is encoded; wherein the receiver is arranged to receive a first target display reference comprising an electro-optical transfer function indication of a first target display for which the first encoded image is encoded; And the image processing device includes: a dynamic range processor (203) arranged to generate an output image by applying a dynamic range transform to the first encoded image in response to an electro-optical transfer function indication of the first target display; as well as An output (205) is configured to output an output image signal including an output image.

2. The image processing apparatus of claim 1, wherein the first target display reference comprises a tone mapping indication representing a tone mapping used to generate the first encoded image for the first target display.

3. The image processing apparatus of claim 1 , wherein the image signal further comprises a data field containing dynamic range transform control data; and wherein the dynamic range processor ( 203 ) is further arranged to perform a dynamic range transform in response to the dynamic range transform control data.

4. The image processing apparatus of claim 3 , wherein the dynamic range transform control data comprises different tone mapping parameters for different display maximum illuminance levels, and wherein the dynamic range processor (203) is arranged to determine the tone mapping parameters for the dynamic range transform in response to the maximum illuminance for the output image signal and the different tone mapping parameters.

5. The image processing apparatus of claim 3, wherein the dynamic range transform control data comprises data defining a set of transform parameters that must be applied by the dynamic range transform. The image processing apparatus of claim 1 , wherein a maximum illuminance of a dynamic range of the first target display is not less than 1000 nits.

7. The image processing apparatus of claim 1 , wherein the image signal comprises a second encoded image and a second target display reference, the second target display reference indicating a dynamic range of a second target display for which the second encoded image is encoded, the dynamic range of the second target display being different from the dynamic range of the first target display; Wherein the dynamic range processor (203) is arranged to apply a dynamic range transform to the second encoded image in response to a second target display reference.

8. The image processing apparatus of claim 7, wherein the image dynamic range processor is arranged to generate the output image by combining the first encoded image and the second encoded image.

9. The image processing apparatus of claim 1 , further comprising: A receiver (1701) for receiving a data signal from a display, the data signal comprising a data field comprising a display dynamic range indication of the display, the display dynamic range indication comprising at least one luminance specification; and wherein a dynamic range processor (203) is arranged to apply a dynamic range transform to the first encoded image in response to the display dynamic range indication.

10. The image processing apparatus of claim 1, wherein the dynamic range transform comprises a color gamut transform.

11. The image processing apparatus of claim 1, further comprising a control data transmitter for transmitting the dynamic range control data to a source of the image signal.

12. An image signal encoding device, comprising: a receiver for receiving an encoded high dynamic range image from a content provider device (101); a generator for generating an image signal comprising an encoded image and a target display reference indicating a dynamic range of a target display for which the encoded image is encoded; as well as a transmitter for transmitting an image signal, The target display reference includes an electro-optical transfer function indication for the target display.

13. The image signal encoding apparatus of claim 12, wherein the target display reference comprises a tone mapping indication representing a tone mapping used to generate the first encoded image for the first target display.

14. The image signal encoding apparatus of claim 12, wherein the generator is further arranged to generate the image signal to include a data field containing dynamic range transform control data; the dynamic range transform control data indicating parameters of a dynamic range transform for encoding the image.

15. An image processing method, comprising: receiving a high dynamic range image signal from a content provider apparatus (101), the image signal comprising at least a first encoded image and a first target display reference, the first target display reference indicating a dynamic range of a first target display for which the first encoded image is encoded, and the first target display reference comprising an indication of an electro-optical transfer function of the first target display; generating an output image by applying a dynamic range transform to the first encoded image in response to an electro-optical transfer function indication of the first target display; as well as An output image signal including an output image is output.

16. A method for transmitting an image signal, the method comprising: receiving an encoded high dynamic range image from a content provider device (101); generating an image signal comprising an encoded image and a target display reference indicating a dynamic range of a target display for which the encoded image was encoded, the target display reference comprising an indication of an electro-optical transfer function; as well as Send image signal.

Images