Optimizing the display of HDR images

Abstract

1. In order to obtain a better and more viewable HDR image in a practical manner, several variants of an image processing device have been proposed for processing an input image Im_Comm of an input video to obtain an output image of an output video, the input image having pixels with an input luminance Ln_in_pix falling within a first luminance dynamic range DR_1, the first luminance dynamic range having a first maximum luminance PL_V_HDR, the output image having pixels with an output luminance that can be calculated from the input luminance and falling within a second luminance dynamic range, the second luminance dynamic range having a second maximum luminance PL_V_MDR, the image processing device comprises a video data input 227 configured to receive the input image and a reference luminance mapping function that is encoded as metadata associated with the input image, the reference luminance mapping function specifying a relationship between the luminance of the first reference image and the luminance of a second reference image, the first reference image being the input image and the second reference image having a second reference maximum luminance PL_V_SDR, the image processing device further comprising: a display adaptation unit 209 configured to determine an adapted luminance mapping function FL_DA based on F_L, the display adaptation unit using a pre-fixed display adaptation algorithm, the algorithm identifying, for each point on the diagonal line, a respective metric along a line segment starting on the diagonal line oriented in a pre-fixed direction in a coordinate system of an input luminance normalized to a maximum of 1 and an output luminance normalized to a maximum of 1, the respective metric at each position being normalized by giving a value of 1 to an intersection point between the line segment and a locus of the luminance function F_L in the coordinate system; determining identifying a position on each respective metric corresponding to a second maximum luminance PL_V_MDR, the set of positions on the respective metric being output as the adapted luminance mapping function FL_DA; the image processing device includes a boost decision circuit 700 configured to calculate a booster strength value BO, the boost decision circuit 700 comprising:a histogram calculation unit 721 configured to determine a histogram of intermediate luminances obtained by applying the adapted luminance mapping function FL_DA to the input luminance; an average calculation circuit 705 configured to calculate an average luminance magnitude AB based on the histogram; a first converter unit 706 configured to calculate a first intensity value PosB from the average luminance magnitude AB; a second measurement unit 711 configured to calculate a weighted sum BDE of pixel counts in at least two configurable upper bins of the histogram; and a second conversion unit 712 configured to calculate a second intensity value NegB from the weighted sum BDE; determines a booster intensity value BO based on a value DV equal to the first intensity value PosB minus the second intensity value NegB, and the image processing device is further configured to use a display adaptation unit to calculate an adjusted adapted luminance mapping function F_ALT_B, the adjusted adapted luminance mapping function being calculated as the adjusted adapted luminance mapping function F_ALT_B by using a display adaptation algorithm configured to output a locus of positions equal to the booster intensity value on a metric, and the image processing device includes a color converter 208 configured to apply the adjusted adapted luminance mapping function F_ALT_B to the input luminance to obtain an output luminance.

Description

[Technical field]

[0001] The present invention relates to a method and apparatus for adapting image pixel brightness of high dynamic range video to obtain a clearly visible image for a particular display. [Background technology]

[0002] For the past few decades, standard forms of video capturing, coding, and display (today called low dynamic range (LDR) or standard dynamic range (SDR)) have been quite sufficient to bring all kinds of images to end consumers (in television broadcasting, video conferencing, gaming, etc.).

[0003] The system was based on defining relative brightness (0% being the darkest black and 100% being the brightest white) with the luminance dynamic range of the display output typically being between 100:1 and 1000:1. This system can work for well-illuminated (almost uniformly lit) scenes, so that it essentially captures the properties of the objects, in other words their own brightness, i.e. the amount of light they reflect (+4% to +90%). In such scenes, such capture resembles what the brain should expect and what the eye should see.

[0004] From the creative side, a sufficient image results if a value for the camera iris is simply set based on the average amount of light in the scene, and the illumination of the subject is captured based on the percentage of photoelectron filling that each camera sensor pixel collects. For historical reasons, the signal itself (which, when normalized, is between 0 and 1) is not transmitted, but its square root. Whereas analog standards such as PAL represent the signal as a voltage between 0 and 700 mV, today's digital standards quantize it to 255 levels, i.e., convey the relative brightness of the pixel as a value between 0 and 255 inclusive.

[0005] However, there were several problems with this approach. Real-world scenes are not illuminated in such a tightly controlled manner. Although there is still generally a limit to the ratio between the darkest and brightest pixels (because bright entities will inevitably illuminate darker areas to some extent), there can be a fairly large dynamic range of luminance in a scene. For example, lighting levels indoors are typically 100 times less than outdoors, even with relatively large windows, so the dynamic range is already 100 illumination ratios times 100 subject reflectance, or 10,000:1; a small crack in a cave wall would result in a much larger dynamic range.

[0006] Until recently, displays simply did not have the ability to display a dynamic range much greater than 100:1, or a peak brightness, also called maximum luminance, much greater than 100 nits, so when an image (i.e., what will be the luminance displayed) was created, the luminance of all scene objects that could not be clearly displayed was typically clipped to maximum white (or darkest black).

[0007] However, a few years ago, much better displays emerged. For example, by placing separately controllable LEDs behind the LCD pixel locations, it is possible, at least in theory, to make the display area darker or brighter as desired, even though the LCD pixel limits how much LED light it can transmit. Thus, today it is possible to make displays with maximum brightness of over 2000 nits and dynamic ranges of over 10,000:1.

[0008] This allows for entirely new modalities of image display (eg, with a realistic visual impression of a differently lit environment) when it comes to creating these images.

[0009] However, even the traditional coder Rec. 709 is simply not capable of coding the large range of luminance that can be coded in the resulting images, so for starters, new ways of coding images (so-called high dynamic range (HDR) images), representing HDR images, and handling HDR images are also needed.

[0010] A few years ago, novel techniques for coding high dynamic range video were introduced, inter alia, by the applicant (see, for example, WO2017157977).

[0011] Coding a video is concerned with creating pixel color codes (e.g., one luma and two chromas per pixel) to describe an image. This is sometimes different from knowing how to optimally display an HDR image. For example, the simplest method can simply utilize a highly nonlinear optical-electrical transfer function OETF to convert the desired luminance to, for example, a 10-bit luma code, and vice versa, convert those video pixel luma codes to the luminance to be displayed, by mapping, for example, the 10-bit electrical luma code to the optical pixel luminance to be displayed, using an inversely shaped electro-optical transfer function EOTF, but more complex systems can deviate in several directions, especially by decoupling the coding of the image from the specific use of the coded image.

[0012] One new approach to specifying HDR images is to create them on the deployed dynamic range of the so-called target display (not to be confused with any actual display of the end user). The maximum luminance PL_V of that target display is co-transmitted in metadata along with the image, which indicates that the image was created in that range. Instead of loading the relative image directly from the camera, it is now possible to precisely specify the luminance required in some master HDR image.

[0013] So now we can answer precisely how bright, for example, a fireball explosion should be compared to the surrounding pixel brightness in the room (in SDR we would typically vary the iris exposed to the fireball, i.e. so that many of the pixels end up in the range 0-100%).

[0014] Thus, a first video creator might create, for example, a 4000 nit PL_V video of this scene. Creating a 4000 nit video, as opposed to, for example, a 2000 nit PL_V video, would typically result in creating a pixel somewhere in the video, i.e., for a given moment, with a luma code that codes for a pixel that is approximately 4000 nit (i.e., a pixel that should ideally be displayed as long as it is acceptable to display it as 4000 nit). Knowing that one has a range available up to 4000 nit, and taking into account, for example, the impact of a fireball in terms of its size in terms of the amount of pixels, or the percentage of the image area that is included as an object, one might decide that the brightest pixel of the fireball shall be 3000 nit, and reserve 4000 nit for a laser beam later in the movie.

[0015] A manufacturer of an equivalent 2000 nit version of the same movie, when balancing all aspects (the visual impact of the fireball as well as its relationship to the luminance of the surrounding pixels and the capabilities of its 2000 nit object display), might, for example, place the brightest point of the fireball at 1950 nit. This defines various characteristics of the input image, and then further requires that any particular display, for example with an (actual) display maximum luminance PL_D of 600 nit, must work out how to map all the luminance values ​​of the image object pixels into its available dynamic range.

[0016] But this new way of defining images paves the way to enable more predictable ways of displaying those images. Displays that do not have sufficient dynamic range will in fact need to do some brightness downgrading, but can do so in a way that respects the needs of the image, or more precisely, how the creator defined their regrading needs. In the SDR era, images were simply displayed using a simple heuristic: the brightest pixel in the image (white, i.e., luma code 255) was mapped to the brightest displayable pixel. Then, on a 200 nit PL_D display, the image would look twice as bright as on a 100 nit display. When the ratio of maximum brightness of the displays is so small, this is acceptable to both viewers and creators (who bought a brighter, less expensive display only to have a somewhat more beautiful image of the same scene, because the eyes and brain recalibrate out some of the difference).

[0017] For example, when we have brighter HDR displays where PL_D is 1000 nits or more, not only does showing a 1000 nit white pixel become too extreme, but more importantly, by using new techniques for coding and handling HDR images, thus showing real impressive HDR images, we can more fully use that range of the display.

[0018] Using FIG. 1, it becomes clear not only what the differences between the HDR images are exactly (despite the higher PL_V values), but also how such differences can be handled (reduced) when creating an equivalent lower dynamic range image, e.g. a 100 nit PL_V SDR image that corresponds to the HDR master image.

[0019] Grading in this application is intended to mean either the activity or the resulting image where a pixel is given a desired brightness, for example by a human color grader or by automation (e.g., in the graded master image, the fireball is 5 times brighter than the surrounding pixels, instead of 4 or 6). When one looks at an image, for example designing an image, there will be several image objects, and one ideally wants to give the pixels of those objects a brightness spread of approximately the average brightness that is optimal for that object, given the entire image and scene. For example, if a person has available image capabilities such that the brightest codable pixel of an image is 1000 nits (the maximum brightness of the image or video PL_V), a human grader may choose to give the pixels an explosion brightness value between 800 nits and 1000 nits in order to make the explosion look quite impressive, while another movie maker may choose to make the explosion less than 500 nits bright, for example, so that it does not interfere too much with the rest of the image at that moment in time (of course, the technology should be able to handle both situations, so that the dynamic range of the subject display should be defined in a stable manner for both graders).

[0020] The maximum brightness of an HDR image or video (i.e. of its associated target display) can vary considerably, for example typical values ​​are e.g. 1000nit or 4000nit or even 10,000nit, with no limit (one generally says that one has an HDR image when PL_V is at least 600nit).

[0021] HDR displays have a maximum capability, i.e., a highest displayable pixel brightness of, for example, 600 nits or 1000 nits, or a number N×1000 nits (starting with lower-end HDR displays).

[0022] Video creators generally cannot create optimal video for each possible end-user display (i.e., the capabilities of the end-user display are best utilized by a video where the maximum luminance of the video never exceeds the maximum luminance of the end consumer display, nor is it lowered to fully utilize the available dynamic range). That is, there are implementations of an attribute similar to the "display maximum image brightness as maximum displayable brightness" of the SDR era, but this is no longer implemented electronically by chance, but is a precisely mathematically definable attribute (and yet one can choose to give only one pixel a luminance of 4000 nits, and grade all other scene objects at any desired lower luminance value).

[0023] A second question then arises of how to best display an image with a peak luminance PL_V on a display with a (often much) lower display peak luminance PL_D, which is referred to in this text as display adaptation (or sometimes as display optimization). In particular, the term will be used for non-arbitrary adaptation, but as shown in the following guide with information on the required regrading of pixel lumas in display optimization, the display optimization described below is done in the simplest technical manner by the video creator creating and communicating at least one luma mapping function. Even in the future, there will still be displays that require the produced dynamic range images to be lower than, for example, a PL_V image of 2000 nits. In theory, a display can always regrade, i.e. map, the luminance of image pixels so that the luminance of the image pixels is displayable by its own internal heuristics, but if the video creator has done enough to determine the pixel luminance, it would be beneficial for him to be able to further indicate how his image should be displayed adapted to lower the PL_D value, and ideally the display will largely comply with these technical requirements. If the display manufacturer re-grades the fireball according to any of its display internal algorithms, it will not look as impressive as originally intended by the creator.

[0024] However, there is some uncertainty about who has control over the display, i.e., who has the final say on how bright all image objects should be displayed. One could say that it should be the creator of the video, since the video creator defines the look. However, the end user could also decide, for example, that they do not like the video to be as bright as they do. Finally, display manufacturers could also say on this issue, for example, not to be slaves to the creator, but to make his display pop out by creating a video display that is slightly better than that of the competitor. By defining such possibilities in the framework of the new HDR coding and handling techniques deployed, it is at least guaranteed that the video creator has more technical possibilities to have the say on the final look of the image, rather than the display manufacturers doing arbitrary processing with complete disregard for what the image was supposed to be, so that no one has any better idea anymore about how the image will ultimately look on the various displays.

[0025] The steps involved in creating and encoding an HDR image are summarized below. a) A person must define the desired pixel absolute luminance (of the master HDR image, i.e. the first image to be graded against the HDR scene) and for this person defines the dynamic range of the object display ending at PL_V (this can be called grading the image) b) Since typical video coding uses YCbCr pixel color coding, one needs to encode these luminances as, for example, 10-bit luma, which requires selecting one of a variety of possible HDR Electro-Optical Transfer Functions (EOTFs), which selection is typically further co-communicated as metadata for the target display.

[0026] This is the basic creation of the HDR image itself, which can then be reconstructed into a decrypted HDR image at the receiving end, for example at a TV display.

[0027] However, in a professional future-proof system, video creators will add additional metadata.

[0028] This typically requires specifying at least a luma mapping function (component c in addition to a and b above) and specifying how a secondary reference grading can be obtained from a master HDR image, for example at a PL_V of 500 nits, by applying a luminance mapping (typically specified as a luma mapping function, the two being mathematically related whether one knows the EOTF that defines the master HDR and the EOTF that defines the luma of the secondary image).

[0029] Advantageously, one wants the second image to be an SDR image. Then one not only ensures that the luminance of all image objects knows which relative position (among the relative positions that other intermediate dynamic range gradations can have) should be with respect to the most extreme grading (one typically encounters, i.e., actually needs), but also, if one uses this SDR image as the image to be transmitted as a representative of the grading pair, it can be used as a backward compatibility system, where a conventional display can directly display this received SDR image without needing knowledge of HDR video coding or HDR video handling. This SDR image can be specified to have PL_V_SDR=100nit, but traditionally it will be interpreted as a dimensionless image where 100% is SDR white.

[0030] To illustrate the components of the methodology, the reader may focus on a few image subjects in some representative images in FIG.

[0031] The luminance axis on the left is the luminance of the master HDR image, i.e. the "best image" that the creator created and would ideally like to see displayed.

[0032] For example, in the outdoor Wester scene ImSCN1, the creator wants to take full advantage of the increased dynamic range to render objects that burn in the sun. He considers 500 nits to be enough for those objects (e.g., the cowboy's white hat) (the viewer will then encounter the sun-burned scene as it appears 5 times brighter than the normal LDR rendering that may be the previous scene of the movie, which took place indoors, for example). Of course, on the right SDR luminance range, which has a secondary luminance dynamic range DR_2 that is much smaller than the primary dynamic range DR_1, the luminance of the cowboy hat must be mapped at a convenient position, for example, 18 nits, between 0.1 nit and 100 nit. The set of all these projection arrows defines a luminance-to-luminance mapping function for defining an SDR image based on an HDR image (or, i.e., SDR to HDR, if the function can be reversed, without, for example, clipping HDR luminance to the same SDR luminance). The same can be done for a night scene ImSCN2, or for the cave with an opening through which one can see the daylit outside world ImSCN3.

[0033] Now, if one wants to create a mid-dynamic range image (MDR) with, say, a PL_V_MDR of 800 nits ready to drive, say, a PL_D display of 800 nits, the question is about where the various object luminances should end up. This can be a complex problem, since depending on what objects are in what kind of scene, one wants to place any particular object, like a cowboy, in various possible positions according to the video creator (see arrow FL_DA), because on such a dynamic range one not only considers, among other things, the relocation of the various object luminances to be different depending on the degree of the available third-order dynamic range DR_3, and how far it deviates from the first-order DR_1, but also the type of image content (a plain type of object like a cowboy has different needs than, say, an explosion or clouds in the sky).

[0034] However, it may be assumed that one can actually use a theoretically not perfect system and a pre-designed method, symbolically shown by simply connecting lines, linking the three positions by some fixed algorithm, and finding interesting positions on the cubic luminance axis, where the display adaptation from the 1st HDR image (also called master image) to the 2nd grading only requires the application of a luma mapping function, which is specified for this re-grading by a color ranking unit or an automatic grading algorithm (which will define an averagely good mapping based on the analysis of various characteristics of the master HDR image, such as its luma histogram), and using the display adaptation to obtain a cubic (MDR) image based on this luma mapping function and some pre-deployed display adaptation algorithm, which will convert the initial, e.g., HDR-to-SDR luma mapping function to a HDR-to-MDR luma mapping function (see the explanation below in conjunction with Figures 4 and 5).

[0035] Regarding component b, a simple HDR coder, the HDR10 coder, has been introduced to the market. This HDR10 video coder uses as OETF (inverse EOTF) the so-called perceptual quantizer (PQ) function standardized in SMPTE 2084. Instead of being limited to 1000:1 as in the Rec. 709 OETF, this PQ OETF allows to specify luma (typically 10 bits) for many more luminances (to be displayed as possible), i.e. between 1 / 10,000nit and 10,000nit, which is enough for any practical HDR video production.

[0036] Please note that the reader should not be too simplistic and confuse HDR with a large number of bits in a luma code word. That is true for linear systems, such as the amount of bits in an analog-to-digital converter, and in fact the amount of bits follows the dynamic range as a logarithm to the base 2. However, since the code assignment function can theoretically have a significant non-linear shape even if desired, HDR images can be defined using only 10-bit luma (plus 8-bit HDR images per color component), which leads to the advantage of reusability of systems already deployed (e.g. ICs with a certain bit depth or video cables, etc.).

[0037] After luma calculation, one has a 10-bit pixel luma plane Y_code, to which two reference color components Cb and Cr per pixel are added as reference color pixel planes. This image can be treated "as if" it were a traditional and even future technically SDR image, e.g. compressed MPEG-HEVC. The video compressor does not actually need to care about the color or brightness of the pixels. However, further brightness or tone mapping may be in place, since the colorimetry of HDR PQ-defined images will of course not result in the correct colors as specified under Rec.709 colorimetry.

[0038] Thus, accepting devices, e.g. displays (or indeed decoders thereof), typically need to perform a proper color interpretation of the {Y,Cb,Cr} pixel colors in order to display a proper looking image rather than, e.g., an image having bleached colors.

[0039] This is typically handled by co-transmitting further image-specific metadata in addition to the pixel color component matrix of the video image, which specifies the image coding, such as an indication of which EOTF is used (for which we shall currently assume, without limitation, that the PQ EOTF (or OETF) is used), and the value of the maximum luminance of the target display, associated with e.g. the PL_V of the HDR image.

[0040] More sophisticated future-proof HDR coders include further image-specific metadata, e.g. handling metadata, e.g. functions that specify how to map (or in other words re-grade) the normalized luminance of the first image up to PL_V=1000 nit to the normalized luminance of a secondary reference image, e.g. an SDR reference image with PL_V=100 nit (as explained in more detail in FIG. 2). These mapping functions vary from image to image, since a cave scene would require a different luminance remapping to SDR luminance than, e.g., a sun-bleached desert scene. That is, one associates with each successive image of the video a separate global luminance mapping function F_L (or equivalently embodied as a luma mapping function via the EOTF of the input and output lumas) in the associated metadata.

[0041] 2 shows a full HDR video communication system, which uses, for example and without any limitation, HDR video image coding as described in the standard ETSI TS103433 (High Performance Single Layer High Dynamic Range System for Use in Consumer Electronics (SL-HDR)). On the sender side there is a source of image 201. This can range from a hard disk to a cable output from, for example, a TV studio, depending on whether one has a video produced offline, for example from an internet delivery company or a real life broadcast.

[0042] Some video generation rules result in a master HDR video (MAST_HDR), which is, for example, a color graded version by a human color rater or shaded by a camera-captured or automatic brightness redistribution algorithm.

[0043] This would be, for example, the 5000 nit PL_V video shown on the left side of FIG. 1, e.g. a Western movie with 5000 nit sunlight brightness.

[0044] From this master HDR image, a communication image is derived to actually convey the HDR image. This can be done in various ways. For example, the 5000nit HDR image itself can be transmitted by calculating a 10-bit luma code using the PQ EOTF. We will explain how another grading can be further transmitted as a proxy for the grading of the master HDR. For example, transmit an SDR communication image Im_SDR, which is an SDR regraded version of the master HDR image. If a strictly increasing luminance or luma mapping function F_L is used, since the luma mapping function F_L is invertible, at any receiver, the HDR master image can be reconstructed by applying an inverse function to the Im_SDR pixel luma. In general, taking into account that a person may also want to make, for example, saturation changes, there will be a color mapping function F_ct that can be optimized for each image. The color processing of the mapping is performed by the color converter 202.

[0045] Since the communication image essentially just looks like any other 10-bit image, and in particular the SDR communication image Im_SDR is a true SDR image (in the sense that it will give a good-looking picture when it is directly displayed on a conventional LDR display without further color optimization), this image can be compressed in a video compressor 203, which applies, depending on which compression format is deployed on any communication medium (whether it is optical disc, Internet subscription, terrestrial digital television, or any professional video communication system for example for surveillance or security, etc.), an MPEG algorithm such as HEVC or VVC, or some other video coder such as AVI, etc., to obtain a compressed HDR image Im_COD. The formatter 204 applies formatting typical for communication, such as packetization, frequency modulation, (handshaking), etc. The mapping function, which may be specified in various forms, such as parameters that set the shape of the function or an exact specification of the function, e.g., as a LUT, can be easily conveyed because there are existing mechanisms for conveying various forms of specific data, where one simply subscribes to some video communication standard where a particular, e.g., supplemental enhancement information (SEI) message (SEI(F_ct)) contains a luma mapping function specified in a specific manner, and each compliant receiver knows how to understand that SEI message.

[0046] Thus, the reader again sees something similar to the explanation using FIG. 1 and sees two components of future-proof HDR coding: a first technical circuit or process which includes the colorimetric specifications of the desired image (master image, more precisely the communication image), and a second circuit or process which includes the coding aspects itself, in particular to generate an image which is at least formally (not necessarily colorimetrically) compliant with existing communication technologies, so that one can send HDR images over existing communication channels having certain defined coding techniques, conditions such as available bandwidth, etc.

[0047] At any receiving location, a corresponding video signal unformatter 206 applies the necessary unformatting method to recapture the compressed video, e.g., as a set of compressed HEVC images (i.e., HEVC image data), e.g., modulation, etc.

[0048] The video decompressor 207 performs, for example, HEVC decompression to obtain a stream of pixelated uncompressed images Im_USDR, which in this example are SDR images, but in another mode are HDR images. The video data input 227 is any medium suitable for carrying video pixel data and metadata. For example, it may be an external cable such as HDMI, and an internal data bus within the device, a connection to a network, etc. The video decompressor will further extract the required luminance mapping function F_L, for example from the SEI message, or a general color conversion function F_ct.

[0049] The image and function are input to a (decoder) color converter 208 configured to convert the SDR image into an image of any non-SDR dynamic range (i.e., a PL_V higher than 100 nits, and typically at least several times higher, e.g., 5 times higher).

[0050] For example, a receiver decoder can perform a simple reconstruction of the master 5000 nit image by calculating a reconstructed HDR image Im_RHDR from Im_USDR by applying the inverse color transform IF_ct of the color transform F_ct used on the encoding side to create Im_LDR from MAST_HDR.

[0051] This image may then be sent for further display application, e.g. to the display 210 to obtain a PL_V image of e.g. 700 nits for optimally driving a connected PL_D end-user display of 700 nits, while creating the display-adapted image Im_DA_MDR may also be done once during decoding in the color converter by using the calculated FL_DA function (e.g. specified in firmware, in an offline loop) instead of the F_L function. In such an embodiment, the color converter further comprises a display adaptation unit 209 to derive the FL_DA function based on the received F_L function using a pre-baked (or configured) display adaptation algorithm.

[0052] The optimized, e.g., 700 nit display-adaptive image Im_DA_MDR is sent, e.g., to the display 210 if the video decoder 220 is included, e.g., in a set-top box or computer, it is sent to a display panel if the decoder is, e.g., in a mobile phone, or it is communicated to a cinema projector if the decoder is, e.g., in some internet-connected server.

[0053] FIG. 3 is shown to further illustrate a more detailed useful variant of the internal processing of an HDR decoder (or encoder, which for the most part typically has the same topology, but uses an inverse function and typically does not include display adaptation as circuit 350 shown in dotted lines in FIG. 3), i.e., color converter 300 corresponding to FIG. 2, 208.

[0054] The luminance of a pixel, in this example an SDR image pixel, is input as the corresponding luma Y'SDR (alternative inverse regrading to reduce the dynamic range will be explained with further figures). The chroma components Cb and Cr, also called reference colors, are input to the lower processing path of the color converter 300.

[0055] The luma Y'SDR is mapped to the required output luminance L'HDR, for example to the master HDR reconstruction luminance or some other HDR image luminance, by a luminance mapping circuit 310. It applies a suitable function, for example a display adapted luminance mapping function FL_DA(t), for the particular image and for example the maximum display luminance PL_D obtained from a display adaptation function calculator 350 that uses as input a metadata broadcast reference luminance mapping function F_L(t). In an embodiment, an upper processing track (circuit 310) which consists by itself of several successive sub-processings, however for the purposes of the description of the present technical contribution to the field, it can be assumed that circuit 310 applies a single luma mapping function to all pixels of the currently converted video image.

[0056] The display compatibility function calculator 350 also determines a suitable function for processing the reference color. For now, it will simply be assumed that a set of multiplication coefficients mC[Y] for each possible input image pixel luma Y (i.e., for example, 0 to 1023) is stored, for example, in the color LUT 301. The exact nature of the color processing varies. For example, one may want to keep pixel saturation constant by first normalizing the reference color by the input luma (corresponding hyperbola in the color LUT) and then correcting for the output luma, but any differential saturation processing may also be used. Hue will typically be maintained since both reference colors are multiplied by the same multiplier.

[0057] When indexing the color LUT 301 with the luma value Y of the currently color transformed (luminance mapped) pixel, the required multiplication coefficient mC is provided as the LUT output. The multiplication coefficient mC is used by multiplier 302, which multiplies the two reference color values ​​of the current pixel, i.e., provides the color transformed output reference color. Cbo=mC*Cb Cro=mC*Cr [Equation 1] Via a fixed color matrixing processor 303, applying standard colorimetry calculations, the reference colors can be transformed into lightness-deficient normalized non-linear R'G'B' coordinates R' / L', G' / L', and B' / L'.

[0058] The R'G'B' coordinates that give the appropriate brightness to the output image are obtained by multiplier 311, which: R'_HDR=(R' / L')*L'_HDR, G'_HDR=(G' / L')*L'_HDR, B'_HDR=(B' / L')*L'_HDR, [Equation 2] Calculate They can be summarized in the color triplet R'G'B'_HDR.

[0059] Finally, there is further mapping to the format required by the display by the display mapping circuit 320. This not only results in the display driving color D_C, which in turn provides the colorimetry formulation desired by the display (e.g., also the HLG OEFT format), but this display mapping circuit 320 is in some variants configured to do some specific color processing for that display, i.e. it further remaps some of the pixel luminances, for example.

[0060] Some examples illustrating some suitable display adaptation algorithms for the creator's rating unit to derive a corresponding FL_DA function for any possible F_L function that it has determined are taught in WO2016 / 091406 or ETSI TS103433-2V1.1.1(2018-01).

[0061] The display adaptation approach is summarized in FIG. 4 by showing its movement on a plot of possible normalized input luminance Ln_in versus normalized output luminance Ln_out (which would be converted to actual luminance by multiplication by the normalized luminance, i.e., the maximum luminance of the display associated with the PL_V value). Note that such a plot may be made as a luma-luma plot, where a perceptually uniformed luma is advantageous. For example, the following equation in the present application example is used to derive such a perceptually uniformed luma: Y_P=v(L_in;PL_V_in)=log[1+(RHO-1)*power(L_in;p)] / log[RHO], where RHO is a constant that depends on the input maximum luminance PL_V_in according to the following equation: RHO(PL_V_in)=1+32*power((PL_V_in / 10,000);p), where p is a constant that is typically set equal to 2.4. [Equation 3]

[0062] Converting the horizontal axis to perceptual luma, the input luminance L_in would be the normalized luminance in Figure 4, and PL_V_in would be the PL_V for those luminances, i.e., 5000 nits, if we were converting a 5000 nit PL_V master HDR image to a secondary reference grading. If the secondary reference grading of perceptually equalized luma between 0 and 1 required on the vertical axis is a 100 nit SDR image, then when using Equation 3 to calculate the equalized output luma, the value PL_V_in=100 would be used. The advantage of such an equation is that both axes are already properly equalized as the eye properly wants to see different gray values ​​for each respective range, making the mapping function more visually well defined.

[0063] Such a display adaptation algorithm, i.e. an algorithm for calculating a final luma mapping based on an input luma mapping, typically from a video creator, to be applied to the luma of an input image to obtain a final display-compatible luma of, e.g., a 700 nit MDR image, typically has the following technical characteristics:

[0064] The input luma mapping function F_L can be contrasted with the identity transform, which corresponds to mapping the input image onto itself, i.e., on the horizontal input axis, we will have a perceptually uniform luma for, say, a 5000 nit span of luminance (i.e., PL_V_in=5000), and on the vertical axis, we will have an exactly the same scaled and divided set of perceptually uniform lumas. Moreover, the identity transform intends that there should be no brightening or darkening of pixels, so an input value of, say, 0.6 on the horizontal axis for a particular pixel will map exactly to the same value 0.6 on the vertical axis. Thus, the identity transform will geometrically be at the 45 degree diagonal in the perceptually uniform luma plot. It makes sense that the function is scaled with intermediate PL_Vs mapped to some intermediate grading. Thus, a 5000 to 700 nit display adapted luma mapping function (FL_DA) typically has the same shape as the 5000 to 100 nit input function (F_L), i.e., it has the same ridges, but placed closer to the diagonal lines.

[0065] This "opening" of the "fan of functions" is formally formulated as follows: a) One specifies a direction to position the location of an intermediate point of any display adapted luma mapping function to be calculated, i.e. forming the end points of a line segment, any point (pos) on the diagonal line corresponding to the input image luma on the function F_L corresponding to the secondary grading, in this example SDR grading, will fall on the line segment along that direction. b) One defines a metric to place any PL_V value of the intermediate image to be calculated (i.e., a display-adapted image for a 700 nit display, for example, with PL_V equal to 700 nit) on the line segment halfway between the end points corresponding to the PL_V values ​​of the two starting reference gradings (one of them is the transmitted and received image, in this example the master HDR image with a PL_V of, for example, 5000 nit, corresponding to the diagonal line identity mapping, and the other is a secondary grading that can be calculated by applying the luma mapping function received as metadata to the primary / master HDR image luma with a value of, for example, PL_V=100 nit). According to some embodiments, the direction is 135 degrees clockwise from the horizontal axis of the input luma, and in other embodiments, 90 degrees counterclockwise, with a display adaptation algorithm using the first direction to calculate the first partial luma mapping function and the second direction for the second partial function.

[0066] For example, a video creator designs a luminance mapping strategy between two reference gradings as described with Fig. 1. Thus, for any possible normalized luminance Ln_in of a pixel in an input image, e.g., a master HDR image, this normalized input luminance must be mapped to a normalized output luminance Ln_out of a second reference grading, which is the output image. This re-grading of all luminances corresponds to some function F_L, which can have many different shapes depending on the human grading unit or the grading automation, and the shape of this function is co-communicated with the dynamic metadata.

[0067] The question now is what shape should the quadratic version of the derived F_L function, the display adaptation mapping function FL_DA, have in this simple display adaptation protocol to map the input image luma to the MDR image luma on a metric that can calculate that, for example, an 800 nit display should have 50% of the grading effect (full 100% is a regrading of the master HDR image to a 100 nit PL_V SDR image). In general, via the metric, any point between no regrading at all and full regrading to a second reference image can be determined, but this is for any possible normalized input luminance of a pixel (Ln_in_pix). The resulting MDR luminance (or luma) is denoted as the display adaptation luminance L_P_n, whose location of course depends not only on the input normalized luminance but also on the value of the maximum luminance associated with the display adaptation output image (PL_V_out) to be calculated. Those skilled in the art will appreciate that, although one may represent a function of a normalized luminance representation, one may equally represent a function of any normalized luma representation defined by any OETF.

[0068] The metric is advantageously typically logarithmic in nature, which means that instead of values ​​between 100 nit and 1000 nit being spread equally far on a line segment for every 100 nit, the values ​​are first transformed by a non-linear function before assigning them to equally distant positions. For example, it may be determined that a 950 nit display is close enough to the PL_D=1000 nit value, so that a PL_V=1000 nit input image does not need almost any luminance mapping, while a 150 nit image needs relatively more changes in the mapping compared to a 1000 to 100 nit mapping. Typically, the creator of the coding ecosystem, or at least the creator of the receiving device such as a television display, will choose a fixed value for all options, for example, he will use a 90 degree orientation with the metric pos=1-{log(1+[(PL_V-100)]) / log(1+[(1000-100)])} [Equation 4].

[0069] The corresponding display adapted luminance mapping FL_DA can be determined as follows (see FIG. 4): Take all input luminances, e.g. any one of Ln_in_pix. This corresponds to a starting position on a diagonal line with equal angles to the input and output axes of normalized luminance (shown as a square). For each position along the diagonal line, place a scaled version of the metric at each point on the diagonal line (the scaled metric SM) so that it starts at the diagonal line orthogonal to the diagonal line (or 135 degrees counterclockwise from the input axis) and ends at its 100% level, or normalized pos=1, a point on the F_L curve, i.e., the intersection of the F_L curve with the orthogonal scaled metric SM (shown as a pentagon). Place the point at the 50% level of the metric (in this example, for this PL_D value of the display for which the image has to be calculated), i.e. in the middle, or in general at any normalized position obtained by evaluating Equation 4, or any similar non-linear equation to specify the required severity of regrading for any deviation between PL_D and PL_V_in. By doing this for all points on the diagonal line corresponding to all Ln_in values, an FL_DA curve is obtained, which is shaped similarly to the original one, i.e., regraded in the same way, but preferably rescaled / adjusted to a lower maximum luminance. This function is now ready to be applied to calculate the required corresponding optimally regraded / display-adapted 700nit PL_V pixel luminance given any input HDR luminance value Ln_in. This function FL_DA will be applied by the luminance mapping circuit 310 after receiving this display-adapted luma mapping function and immediately prior to initiating colorimetric conversion on all active pixels of the currently processed video image.

[0070] Figure 5 generally formalizes the technical components of display adaptation.

[0071] For example, a display adaptation circuit 510 in a television or set-top box or the like includes a configuration processor 511. The configuration processor 511 sets values ​​for the processing of an image before the live pixel colors of that image are processed. For example, the maximum luminance value PL_V_out of the display optimized output image is set once in the set-top box by polling it from the connected display (i.e. the display communicates its maximum displayable luminance PL_D to the set-top box), or when the circuit is in a television, it is set by the manufacturer or the like.

[0072] The luminance mapping function F_L, in some embodiments, varies for each received image (in other variations it is fixed for multiple images) and is input from some source 512 of metadata information (e.g., it is broadcast as an SEI message, read from a sector of memory such as a Blu-ray disc, etc.). With this data, a normalized height of the normalized metric (Sm1, Sm2, etc.) can be established on which the desired location of the PL_D value can be found from the mathematical equation of the metric.

[0073] When an input image 513 is input, successive pixel intensities (e.g., Ln_in_pix_33 and Ln_in_pix_34, or luma) run through a color processing pipeline that applies display adaptation, resulting in corresponding output intensities such as Ln_out_pix_33. Summary of the Invention [Problem to be solved by the invention]

[0074] While pre-establishing such a fixed well-behaved function deviation algorithm as the display adaptation algorithm above has the advantage of technical simplification (i.e., little human grader input is required, for example, when there are no control parameters to change the metric equation for different positions along the diagonal), establishing a stable regrading allows the grader to predict what will be displayed on different end-user displays owned by different consumers as well. In fact, since the display adaptation algorithm works based on the input F_L function in a manner that maintains its specific shape, including the brightness regrading needs of various objects in the image, it is guaranteed that his graded appearance of the master image will reveal very much on the display what is not able to fully display all the luminance of the master HDR image. The disadvantage of a fixed algorithm is, of course, that it does not always produce a 100% perfect image due to any particular preference, so one hopes to improve upon it.

[0075] Thus, with this deployed technology, a video creator can specify the video of HDR images and further how these video images should be automatically regraded for various displays with different dynamic ranges, and he can specifically adapt the shape of the luma mapping function to the needs of any particular image, i.e., the distribution of inputs outputs lumas along the dynamic range of luminance for all image objects, respectively. However, according to the inventors, this approach still suffers from being somewhat crude, and therefore it would be advantageous for the display manufacturer and / or end viewer to be able to perform some more amount of regrading, especially in or on this framework, even if it potentially deviates slightly from the creator's graded appearance.

[0076] US20140368531 teaches an algorithm for enhancing the transmissivity of an LCD panel when a relatively dark image is to be shown, so that the backlighting can be reduced to save energy that would otherwise be consumed by closed pixels. Instead of doing this via voltage control, US20140368531 teaches that this can be done when an image processing operation changes the pixel luma code to a higher value, and uses a standard LDR display power function of gamma 2.2. The enhanced transmissivity function can be defined as desired which primarily boosts the darkest colors to be brighter, while leaving the brighter colors largely unchanged. For example, a boost of up to 4 times can be applied to the darkest colors (with a slope of the curve of zero). Such a strong boost is used for dark images, but less for bright images. The boost can be quickly achieved by interpolating between the maximum boost and the minimum boost (for bright images where there will be no boost, i.e. the image is left as is). That is, we therefore quickly interpolate the required (intermediate) output for each pixel luma code, e.g. 166, as an interpolation of the two corresponding output values. The amount of interpolation, e.g. 25%, depends on an analysis of the input image histogram (e.g. how many pixels are in the darkest bin out of 8 bins). [Means for solving the problem]

[0077] A visually better looking image may be obtained by a method of processing an input image (Im_Comm) of an input video to obtain an output image of an output video, the input image has pixels having an input luminance (Ln_in_pix) that falls within a first luminance dynamic range (DR_1), the first luminance dynamic range having a first maximum luminance (PL_V_HDR); An output image may be calculated from the input luminance and have pixels having an output luminance that falls within a second luminance dynamic range, the second luminance dynamic range having a second maximum luminance (PL_V_MDR). A reference luminance mapping function (F_L) is received as metadata associated with the input image; the reference intensity mapping function specifies a relationship between the intensity of the first reference image and the intensity of the second reference image; The first reference image is an input image; The second reference image has a second reference maximum luminance (PL_V_SDR); The processing includes determining an adapted luminance mapping function (FL_DA) based on a reference luminance mapping function (F_L); The determining step uses a pre-fixed display adaptation algorithm that, in a coordinate system of input luminance normalized to have a maximum of 1 and output luminance normalized to have a maximum of 1, identifies, for each point on the diagonal line, a respective metric along a line segment starting on the diagonal line and oriented in a pre-fixed direction; Each metric at each location is normalized by giving a value of 1 to the intersection point between the line segment and the locus of the luminance function (F_L) in the coordinate system, The determining includes identifying a location on each respective metric that corresponds to a second maximum luminance (PL_V_MDR); Each set of metric locations is output as a fitted luminance mapping function (FL_DA), The visually superior looking image may be processed by calculating a booster strength value (BO), the calculating a booster strength value (BO) being: - determining a histogram (hist) of intermediate luminance values ​​obtained by applying the adapted luminance mapping function (FL_DA) to the input luminance values; - calculating an average brightness value (AB) based on the histogram; - calculating a first intensity value (PosB) from the average brightness magnitude (AB); - calculating a weighted sum (BDE) of pixel counts in at least two configurable top bins of the histogram; - calculating a second intensity value (NegB) from the weighted sum (BDE); - determining a booster strength value (BO) based on a value (DV) equal to the first strength value (PosB) minus the second strength value (NegB), The processing further includes calculating an adjusted adapted luminance mapping function (F_ALT_B) by using a display adaptation algorithm with a setting that outputs a locus of locations where the adjusted adapted luminance mapping function is equal to the booster intensity value on the metric, the adjusted adapted luminance mapping function being calculated as the adjusted adapted luminance mapping function (F_ALT_B); and applying the adjusted adapted luminance mapping function (F_ALT_B) to the input luminance to obtain an output luminance.

[0078] Note that when we consider luminance, we intend a representation of the technical concept of luminance. This does not mean that the reader should limit it to any particular representation, especially normalized physical luminance, but that typically luminance will be represented in a luma representation. For example, an embodiment that works with a psychovisually equalized luma system according to Equation 3 will work well, but other representations will also work if various factors (e.g., metric scale) are set appropriately.

[0079] Once the metric equation is defined, any PL_V_MDR value will be located at some location of the metric between 0 and 1 by calculating the equation, depending on which input image comes in and whether upgrading or downgrading is involved, typically the values ​​PL_V_HDR (e.g. 5000nit) and PL_V_SDR (100nit). While one function F_L is used to explain, several functions are used in conjunction to convert one image (interested readers can find useful embodiments in ETSI TS103433, which is incorporated by reference). Display adaptation algorithms will typically be fixed once per video communication ecosystem. For example, in terrestrial ATSC broadcasting, the algorithm is selected by the standard. An extraordinary provider of movies from an Internet server may decide to use the same display adaptation algorithm or a similar but different one. In any PL_V mastering of HDR images, the second reference image is usually a 100 nit SDR image, since it has a better redistribution of image subject luma for most typically resulting display systems of lower quality. The PosB and PosN values ​​are an easy way to indicate the need and risk of increasing the intended regrading in a competitive manner. The reuse of pre-fixed display adaptation algorithms ensures predictability and a large preservation of the artistic intent of the original content rating part of the image. Several parameters are available, for example, to shape the transformation function so that the end-point entity, e.g., the display manufacturer, can tailor the system to what he aspires to (e.g., change the intended grading more aggressively, but enhance some desirable, pictorial aspects, e.g., from the intensity of compressive downgrading).

[0080] In Fig. 6 we can see diagrammatically (in an exaggerated way) what can be done for at least some of the images of some videos. Let us assume that an input HDR image with PL_V=1000nit is mapped to a standard 100nit (or maybe 200nit) output. The diagonal lines then represent what the inherent linear scaling in normalized luminance coordinates (e.g., according to the traditional SDR display paradigm "map the brightest pixel coded in the image to the brightest displayable color (display white)") introduces as darkening. All colors are attenuated by a factor of 10 compared to what they are supposed to be. That is not a problem (or even makes auto-tone mapping rather handy) for already quite bright subjects like sunlit clouds or our strongly lit cowboy. For example, a pixel originally supposed to be 500nit will be displayed at 50nit. This is well worth seeing, since on the one hand it is a sensible value in the more limited dynamic range of an SDR display, but on the other hand it is still respectably bright even in SDR. It is the relatively darker objects that are more problematic. In the input range R_i, the grading part of the master 1000nit HDR image will have carefully graded the darker objects while also dealing with the presence of contrasting brighter HDR objects. These are, for example, indoor objects, and especially importantly, indoor objects in areas of the scene that are not very well lit. In the real world (i.e. when you start measuring with a luminance meter), indoor objects are typically 100 times darker than outdoor objects, for example when viewed through a window in the image view. In a master HDR grading ending at PL_V=1000nit, a ratio of 100:1 will typically not be used, because on the one hand it is not the best use of the dynamic range, and on the other hand it is not what a typical viewer is expected to see (not seeing a real scene, but seeing a small rectangle during an evening viewing in his living room).So for example, a master grading that gives exterior pixels viewed through a window a value between 10 nit and 1000 nit, and indoor pixels a value between 0 nit and 100 nit, uses an illumination ratio of 10:1.

[0081] It actually looks pretty accurate when shown on a 1000nit HDR display. However, when scaled by a factor of 10, pixels that are supposed to display at 10nits end up displaying at 1nit, a deep black. When you factor in the additional effects of ambient lighting at the viewing location, such as reflections on the display's front glass, these darker areas rarely become worth looking at.

[0082] This is why the ranker, in combination with a display adaptation algorithm, produces a luminance (or luma) mapping function FL_DA, which will give a fairly good remapping of all pixel luminances, for example, especially for those indoor pixels of the mentioned indoor scene. It turns out that the mapping curve stretches the input range R_i relative to the output range R_o, which must further be converted to absolute luminance by multiplying by 100. However, in any case, in the example, approximately 50% of the available range is used, which ensures better visibility.

[0083] However, in some situations one might want to apply an even stronger brightening regrading, stretching the dimmest luminances over a larger second output range R_o2 in this example.

[0084] It is important to build this possibility on the existing regrading framework. Otherwise, the TV manufacturer, who is one of the first entities wanting to apply this additional brightening, may do whatever luminance mapping is necessary to make their display brand look better. And this not only destroys the video creator's focus (which goes through all the trouble to create a particularly impressive looking HDR scene image by placing all subject luminance at a desired value in the master grading image, and then specifies how to slightly change this distribution when going to a smaller dynamic range), but also brings one of the simple and unpredictable ways of creating, coding, handling, and displaying HDR back to square one. In fact, it can be expected that at least some end users will apply fairly aggressive luminance remapping functions, making their images impressively pop out in a potentially overly elaborate and sterile way according to the creator. However, there is a worse effect to this. For simplicity, we have described what happens to the darkest luminances of the image, but all luminances along the range will need to be properly placed to have a credible and impressive HDR effect in the image (see, for example, the steeper slope between the two ridges). If one deviates strongly from the desired display adaptation curve, one will strongly deteriorate the image, and instead of an impressive HDR image, it will be more like an SDR image without all the good HDR effects (not even scaled down somewhat to fit the smaller dynamic range capability, but simply missing or at least severely distorted). The output image will look flashy and, depending on the taste, can be more appropriately varied by the customer, but will not look perfect, i.e., will not look like it was produced by the grading part of the master HDR video.

[0085] Therefore, an alternative display adaptation function F_ALT_B that suits the different preferences of non-creators, eg display manufacturers, needs to be produced in a carefully designed technical manner.

[0086] A method for processing an input image is advantageous if it exploits when one of two configurable upper bins contains 1% of the brightest pixels in the intermediate luminance histogram. This superbin can then be advantageously ignored as non-critical by setting its ci multiplier equal to 0 (see below).

[0087] It is advantageous if the method for processing the input image has a booster strength value (BO) determined as a customizable multiplier value (mf) multiplied by a relative boost value (RB) normalized between zero and 1. This is an easy way for the implementer to set a customary strength of the amount of original grading deviation, while balancing with a general improvement in the aesthetics of the initial appearance of the final displayed image.

[0088] It is advantageous if the processing of the input image includes calculation of the booster strength value after temporal filtering in order to preserve limited or reduced variations with previously determined booster strength values ​​compared to direct unfiltered variations.

[0089] If asymmetric temporal filtering is used, which filters positive differences differently from negative differences, it is advantageous to change quickly to smaller booster strength values ​​and slowly to higher booster strength values ​​(as images cannot tolerate large boosting and will create errors and require more careful treatment immediately).

[0090] The technique may further be embodied in an image processing device for processing an input image (Im_Comm) of an input video to obtain an output image of an output video; the input image has pixels having an input luminance (Ln_in_pix) that falls within a first luminance dynamic range (DR_1), the first luminance dynamic range having a first maximum luminance (PL_V_HDR); The output image may be calculated from the input luminances: having pixels having an output luminance falling within a second luminance dynamic range, the second luminance dynamic range having a second maximum luminance (PL_V_MDR); The image processing device comprises a video data input (227) configured to receive an input image and a reference luminance mapping function (F_L) encoded as metadata associated with the input image; the reference intensity mapping function specifies a relationship between the intensity of the first reference image and the intensity of the second reference image; The first reference image is an input image; The second reference image has a second reference maximum luminance (PL_V_SDR); The image processing device includes a display adaptation unit (209) configured to determine an adapted luminance mapping function (FL_DA) based on a reference luminance mapping function (F_L); The display adaptation unit uses a pre-fixed display adaptation algorithm, which determines, for each point on the diagonal line in a coordinate system of input luminance normalized to a maximum of 1 and output luminance normalized to a maximum of 1, a respective metric along a line segment starting on the diagonal line and oriented in a pre-fixed direction; Each metric at each location is normalized by giving a value of 1 to the intersection point between the line segment and the locus of the luminance function (F_L) in the coordinate system, The determining includes identifying a location on each respective metric that corresponds to a second maximum luminance (PL_V_MDR); Each set of metric locations is output as a fitted luminance mapping function (FL_DA), The method includes an image processing apparatus including a process, the process including a boost decision circuit (700) configured to calculate a booster strength value (BO), the boost decision circuit (700) comprising: a histogram calculation unit (721) configured to determine a histogram (hist) of intermediate luminances obtained by applying the adapted luminance mapping function (FL_DA) to the input luminances; an average calculation circuit (705) configured to calculate an average brightness magnitude (AB) based on the histogram; a first converter unit (706) configured to calculate a first intensity value (PosB) from the average brightness magnitude (AB); a second measurement unit (711) configured to calculate a weighted sum (BDE) of pixel counts in at least two configurable upper bins of the histogram; a second conversion unit (712) configured to calculate a second intensity value (NegB) from the weighted sum (BDE), a boost decision circuit for determining a booster strength value (BO) based on a value (DV) equal to a first strength value (PosB) minus a second strength value (NegB); Image processing device The image processing device is configured to use a display adaptation unit to calculate an adjusted adapted luminance mapping function (F_ALT_B), and the adjusted adapted luminance mapping function is calculated as the adjusted adapted luminance mapping function (F_ALT_B) by using a display adaptation algorithm configured to output a locus of positions equal to the booster intensity value on a metric; a color converter (208) configured to apply the adjusted adapted luminance mapping function (F_ALT_B) to the input luminance to obtain an output luminance.

[0091] The image processor includes within the boost decision circuit (700) a temporal filter (740) configured to calculate a booster strength value that varies less.

[0092] These and other aspects of the method and apparatus according to the invention will become apparent from and will be explained with reference to the implementations and embodiments described hereinafter with reference to the accompanying drawings, which serve merely as non-limiting, specific illustrations illustrating more general concepts, in which dashed lines are used to indicate that components are optional, and non-dashed components are not necessarily required. Dashed lines may also be used to indicate elements that are required but are described as hidden inside an object, or for intangible things, such as, for example, selection of an object / area. [Brief description of the drawings]

[0093] [Figure 1] FIG. 1 is a schematic diagram of some typical color transformations that occur when a person optimally maps a high dynamic range image to a corresponding optimally color graded and similar looking lower dynamic range image (as desired and feasible, considering the difference in the first and second dynamic ranges Dr_1 and DR_2, respectively), e.g., a standard dynamic range image with a maximum brightness of 100 nits, which, if it is lossless, actually corresponds to a mapping of the SDR image to a reconstructed HDR image of that scene, as a received SDR image encoding an HDR scene. Brightness is shown as a location on the vertical axis from darkest black to maximum brightness PL_V. The brightness mapping function is symbolically shown by an arrow that maps the average subject brightness from its brightness on the first dynamic range to the second dynamic range (those skilled in the art know how to equivalently depict this as a traditional function, e.g., on an axis normalized to 1, which is normalized by dividing by the respective maximum brightness). [Diagram 2]FIG. 1 is a schematic diagram illustrating an example of a high-level diagram of a technique recently developed by the applicant for encoding high dynamic range images, i.e., images that may typically have a brightness of at least 600 nits or more (typically 1000 nits or more), which may in fact be conveyed as an HDR image, either on its own or as a corresponding brightness regraded SDR image plus a metadata-encoded color conversion function that includes at least an appropriate determined brightness mapping function (F_L) for pixel colors, for use by a decoder to convert the received SDR image(s) into HDR image(s). [Diagram 3] FIG. 2 shows the internal details of an image decoder, in particular the pixel color processing (transformation) engine, in a (non-limiting) preferred embodiment. [Figure 4] FIG. 1 shows how display adaptation works in a graphical fashion, derived from a reference luminance mapping function F_L that organizes the luminance re-grading needs between two reference images, and a final display adapted luminance mapping function FL_DA that is used to calculate an optimal display adapted version of an input image for a particular display capability (PL_D). [Diagram 5] FIG. 1 is a diagram summarizing the principle of display adaptation more generally, in order to more easily understand it as a component in the formulation of the present embodiments and claims. [Figure 6] FIG. 10 shows a schematic diagram of how further regrading possibilities can be succinctly formulated in the framework of display adaptation according to the present insight. [Figure 7] FIG. 7 is a schematic diagram of an exemplary embodiment of the present technique that is practically easy to build, in particular a boost decision circuit (700). [Figure 8] FIG. 8 shows useful variations of possible conversion functions for use in the unit of FIG. 7. [Figure 9]FIG. 8 shows useful variations of possible conversion functions for use in the unit of FIG. 7. [Figure 10] FIG. 1 illustrates an example of an asymmetric temporal filter for producing a more optimal value of the booster strength value (BO). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0094] FIG. 7 illustrates aspects of a fully functional embodiment of the current approach and boost decision circuit 700. Complex image analysis is expensive in simpler ICs, for example when done in cheaper cell phones. Furthermore, images are often so complex that while initial insight is relatively easy to obtain, more detailed insight into the image requires exponentially more analysis calculations before deeper knowledge of the image is gained. Note that ideally the calculations are performed on the current image being processed just before making the final colorimetric conversion, since then only a delay of one image time (i.e., 1 / 50th of a second, for example) is required.

[0095] What the IC can typically do is determine the histogram of the image. For example, if a 128-bin histogram is calculated at 1024 luma levels, one has a count of the amount of pixels in the image with an accuracy of approximately 10 luma units. From such a histogram, already a lot of interesting image information can be obtained to create a well-functioning algorithm. It is assumed in the present description without limitation that the transmitted received image (Im_Comm) is the master HDR itself with the downgrading function F_L as it was created on the video origination side.

[0096] It is assumed that the IC performs a first histogram calculation, for example in the first histogram calculation unit 720, for the intrinsic coding of the input image, i.e. for the coding with which the input image enters the decoder (this description is taught without any intentional limitation, since those skilled in the art can understand how to obtain the histogram in other ways). Let us assume that this is a PQ luma domain histogram. In fact, we need to have a histogram of the display-adapted luma (reference display adaptation by FL_DA, not including the current improvement). Because it is an image optimized for a specific display, even if it is not yet completely as desired, it will be measured to understand what (semi-)automatic improvements are needed. This histogram (hist) will be determined by the second histogram calculation unit 721. This unit does not actually need to apply the display adaptation to the whole image to calculate the histogram. It can simply apply the function FL_DA to the histogram (hi) obtained from the first histogram calculation unit 720. Those skilled in the art will understand how a histogram in a second domain can be derived from a histogram in a first domain, e.g., the luminance domain, or any luma domain defined according to some EOTF. For example, the PQ luma can first be converted to normalized luminance by applying the SMPTE2084 perceptual quantizer EOTF EOTF_PQ, and then the function FL_DA is applied. The counts will then be moved to different histogram bins. In a preferred embodiment, our luminance representation, and therefore the histogram his, will be expressed in the psychovisually equalized luma domain according to equation 3 above, using the maximum of the image to be determined as the value of PL_V_in, i.e., as the value of PL_D_MDR for the MDR display for which the image is display-adapted, e.g., 700 nit or 1500 nit, etc.

[0097] The Brightness Promoter unit 701 calculates (in an average calculation circuit 705) the brightness measure of the image (AB) based on some average value of the histogram. There are various equivalent calculations, but for now it is assumed to be the average of the histogram in the perceptually equalized luma domain, which works well in practice.

[0098] This value is converted to an intensity value, which is ideally normalized between 0 and 1.

[0099] The reader can see how this works by looking at FIG. 6. As taught, one can grade in to the desired appearance starting from an HDR image input, and downgrade to any degree required between 0 and 1 on the metric (the reader will understand that upgrading starting from a received SDR image, for example, works similarly albeit in the other direction, such as a concave curve below the diagonal line), with 1 being the highest regrading to obtain the most extreme other reference grading (e.g., SDR grading). Here, the metric can be continued to grade in to something greater than what is required. This is not theoretically the video creator's optimal grading, but it is applying a little more of the useful grading desired by, for example, a display manufacturer, and at least follows this prescribed grading. Advanced embodiments can further modify the creator's reference regrading over only a portion of the input luminance range, for example the lower half only brightens to match the upper half which remains roughly the same or less variable, but for illustration purposes we will stick to the simpler embodiment.

[0100] Therefore, this additional technical option can be mathematically formulated by extending the metric somewhat (not too much, otherwise the grading would be overly distorted, but a maximum value can be realized by the developer of the coding framework). Let us assume that we can go to the maximum metric position, which is a Booster Strength value (BO) of 2.0.

[0101] It is advantageous to stamp this value, for example, as a customizable setting by the display manufacturer. The appropriate equation should use a multiplier fixed for a value which is preferably a relative boost value (RB) between zero and one that will result from the boost decision circuit 700. Therefore, BO = mf * RB, where 1 < mf <= VAL_Ph, [Equation 5] VAL_PH is, for example, a maximum value determined by the applicant of this application, for example 2.0.

[0102] Therefore, the reader should understand that in order to apply a preselected display adaptation algorithm, the metric position, which is 1.0 on the outside here, needs to be determined here, but this value should be appropriately determined automatically (or semi-automatically in the sense that an adoptee, for example a mobile phone manufacturer, can set some control parameters such as mf) based on the characteristics of the current image.

[0103] Thereby, no unwanted image or artifact should be introduced for each image or for all of the video.

[0104] To convert the magnitude AB of the average brightness into a positive intensity value, the first converter unit 706 applies a suitable conversion function.

[0105] This function is designed by the developer of the entire HDR handling framework, that is, for example, the applicant of this patent application, who allows a specific display manufacturer to fine-tune it according to their own preferences with a few control parameters.

[0106] This function aggregates the requirements as to when one wants to enhance the regrading or leave it as it was (i.e. F_ALT_B remains at FL_DA) and to what intensity a particular image should be enhanced according to the automatic algorithm.

[0107] FIG. 8 shows a useful embodiment of such a first conversion function. MxPU is a typical value predefined over a large set of images, beyond which no boosting of display-adaptive regrading (i.e. boosting FL_DA becomes F_ALT_B) is performed. For example, if all pixels are in the top 10 histogram bins, boosting makes little sense, since the person already has such a bright image. In fact, in the practical example of FIG. 8, the person already stops encouraging boosting above control point 2 (Cp2). These control points (as well as the first control point Cp1) can be dragged to allow the display manufacturer to adjust the behavior of the fixed booster algorithm, but the designer presets them to, for example, 25% of MxPU for Cp1 and 75% for Cp2. Thus, if the mean value of the histogram hist is really low (lower than the first value ABV1), one would want to do maximum boosting, i.e. if the display manufacturer has set mf equal to 2, then a value RB of 1 will be output from the boost decision circuit 700 and a value BO=2 will be used in the display adaptation algorithm to calculate F_ALT_B. If the image is already a little brighter, one would want to use a smaller downgrading, i.e. higher than ABV1, the strength of the boosting would decrease, or even be the exact value of the first (positive / promoting) strength value (PosB).

[0108] That is why the algorithm, which over-promotes boosting, is thus suboptimal.

[0109] Therefore, the boost decision circuit 700 includes a boost reduction decision circuit 710. This unit protects the rendering of brighter areas.

[0110] It first further calculates the histogram magnitude in a second measurement unit 711 .

[0111] This unit creates a number of (constructed) superbins, i.e., collections of bins of the histogram hist.

[0112] It creates, for example, two bins, the top 1% and the top 5% of pixels.

[0113] Or, it may group the top two bins of the computed histogram hist into a top super-bin and, for example, group the bottom four bins into a second super-bin, etc.

[0114] For example, using four superbins works well.

[0115] The counting equations are then calculated with appropriately set coefficients.

[0116] The result DBE is equal to the sum {ci*SB_i} [Equation 6]. SB_i is the count of pixels that fall into superbin i of at least two superbins, and ci is the corresponding coefficient. The coefficient can be set to a value that is more or less exactly the brightest pixel, for example, if the brightest pixel only gives the image some sparkle, e.g. metallic reflection, or if one expects a very dark image, for example for sports content, one wants a stronger but less colorimetrically accurate rendering (thus, although the parameters will generally be set once to give an averagely clear-looking result, they can be varied for further aspects such as the type of image, the TV channel watched, professional vs. consumer content, etc.). Typically, the ci value for the highest superbin will be set to zero, and the image content in the superbin or superbins below it is measured, and then the size of the highest superbin is set to include the brightest 1% of pixels. Since there is clipping in the equation, the extreme values ​​are clipped to 0, resulting in no reduction effect, or to 1, or more precisely, to -1, resulting in a strong reduction. Further non-linearities may be present, for example when DBE=1, resulting in no boost even for high promotion PosB, but a simpler subtractive counting is accounted for.

[0117] This value is converted in a second conversion unit 712 to a (negative / reducing) second intensity value (NegB) via a conversion function, for example, as shown in FIG. 9. There can be further control points, a third control point Cp3, a fourth control point Cp4, to adjust different shapes, but the example shows how increasing DBE values ​​translate to increasing normalized NegB values. Finally, NegB is subtracted from PosB by a subtractor 730 to obtain the final relative boost value RB to be used. Typically, the algorithm decides to clip negative values ​​to zero, resulting in no boost or a metric position of 1.0.

[0118] Since we are dealing with video processing, it is advantageous to add a temporal filter 740 to filter the relative boost value based on the value of the previous boost value or the difference with one or more previous boost values. Then, there will not be too much deviation between the boost of temporally consecutive video images, which may lead to visual artifacts. This filter will typically have an asymmetric nature, taking into account the increase versus decrease in the required relative boost, as shown in FIG. 10.

[0119] The following types of exemplary filtering are used for illustration (those skilled in the art will understand that equivalent filters may be used):

[0120] The difference with the previous relative boost value RB_tp (or a combination of several weighted previous relative boost values ​​RB_tp) is calculated. DEL_RB=DV-RB_tp; The retained weight (Blnd) is calculated. Blnd=FF(DEL_PB), where FF is the filtering function shape. Output relative boost value RB=(1-Blnd)*DV+Blnd*RB_tp [Equation 7] It is calculated as:

[0121] The algorithmic components disclosed in this text may actually be implemented (fully or partially) in hardware (e.g., as part of an application-specific IC) or as software running on a dedicated digital signal processor or a general-purpose processor, etc.

[0122] It should be possible for a person skilled in the art to understand from our presentation which components are optional improvements and can be realized in combination with other components, as well as how the (optional) steps of the method correspond to the respective device means and vice versa. The word "device" in this application is used in its broadest sense, i.e. a group of means enabling the realization of a particular purpose, and thus may be, for example, a (small circuit part of) an IC or a dedicated instrument (such as an instrument including a display) or a part of a networked system. The word "composition" is also intended to be used in the broadest sense, and thus it includes, inter alia, a single device, a part of an instrument, a collection of (parts of) cooperating instruments.

[0123] The explicit meaning of a computer program product should be understood to encompass any physical realization of a set of commands that, after a series of loading steps (including intermediate conversion steps such as translation into an intermediate language and a final processor language), enables a general-purpose or special-purpose processor to enter the commands into the processor and to execute any of the characteristic functions of the invention. In particular, a computer program product is realized as, for example, data on a carrier such as a disk or tape, data present in a memory, data moving via a wired or wireless network connection, or program code on paper. Apart from the program code, the characteristic data required for the program are also embodied as a computer program product.

[0124] Some of the steps required for the operation of the method, such as data input steps and data output steps, are already present in the functionality of the processor instead of being written into the computer program product.

[0125] It should be noted that the above-mentioned embodiments are illustrative rather than limiting the present invention. Those skilled in the art can easily realize the mapping of the presented examples to other areas of the claims, and for the sake of brevity, not all those options are mentioned in depth. Apart from the combination of the elements of the present invention, such as the combination within the claims, other combinations of elements are possible. Any combination of elements can be realized in a single dedicated element.

[0126] Any reference signs between parentheses in the claims are not intended to limit the scope of the claims. The word "comprising" does not exclude the presence of elements or aspects not listed in the claims. The word "a" or "an" does not exclude the presence of a plurality of such elements.

Claims

1. 1. A method for processing input images of an input video to obtain output images of an output video, comprising the steps of: the input image has pixels having an input luminance that falls within a first luminance dynamic range, the first luminance dynamic range having a first maximum luminance; The output image may be calculated from the input luminance and has pixels having an output luminance that falls within a second luminance dynamic range, the second luminance dynamic range having a second maximum luminance; A reference intensity mapping function is received as metadata associated with the input image; the reference intensity mapping function specifies a relationship between an intensity of a first reference image and an intensity of a second reference image; the first reference image is the input image; the second reference image has a second reference maximum luminance; Processing the input image includes determining an adapted luminance mapping function based on the reference luminance mapping function; Determining the luminance mapping function uses a pre-fixed display adaptation algorithm, which determines, for each point on a diagonal line in a coordinate system of the input luminance normalized to a maximum of 1 and the output luminance normalized to a maximum of 1, a respective metric along a line segment starting on the diagonal line and oriented in a pre-fixed direction; the respective metric at each position is normalized in the coordinate system by assigning a value of 1 to the intersection point between the line segment and the locus of the intensity mapping function; Determining the luminance mapping function includes identifying a location on each respective metric that corresponds to the second maximum luminance; The set of positions on each metric is output as the adapted intensity mapping function, The method further comprises processing the input image, A step of calculating a booster strength value (BO), the step of calculating the booster strength value comprising: determining an intermediate luminance histogram obtained by applying the adapted luminance mapping function to the input luminance; Calculating an average brightness value based on the histogram; calculating a first intensity value from the average brightness magnitude; calculating a weighted sum of pixel counts in at least two configurable top bins of the histogram; calculating a second intensity value from the weighted sum; and determining the booster strength value based on a value equal to the first strength value minus the second strength value. calculating the booster strength value (BO); Calculating the adjusted adapted luminance mapping function, calculating the adjusted adapted luminance mapping function by using a display adaptation algorithm with a setting that outputs a locus of positions equal to the booster strength value on a metric, the adjusted adapted luminance mapping function being calculated as the adjusted adapted luminance mapping function; and applying the adjusted adapted luminance mapping function to the input luminance to obtain the output luminance.

2. The method of claim 1 , wherein one of two configurable upper bins contains 1% of the brightest pixels in the histogram of intermediate luminance.

3. The method of claim 1 or 2, wherein the booster strength value is determined as a customizable multiplier value multiplied by a relative boost value normalized between zero and one.

4. 4. The method according to claim 1, wherein the calculation of the booster strength value requires temporal filtering in order to preserve changes with previously determined limited booster strength values.

5. 5. The method of claim 4, wherein the temporal filtering is asymmetric, changing rapidly to smaller booster strength values ​​and changing more slowly to higher booster strength values.

6. 1. An image processing device for processing input images of an input video to obtain output images of an output video, comprising: the input image has pixels having an input luminance that falls within a first luminance dynamic range, the first luminance dynamic range having a first maximum luminance; The output image may be calculated from the input luminance and has pixels having an output luminance that falls within a second luminance dynamic range, the second luminance dynamic range having a second maximum luminance; the image processing device comprises a video data input for receiving the input image and a reference intensity mapping function encoded as metadata associated with the input image; the reference intensity mapping function specifies a relationship between an intensity of a first reference image and an intensity of a second reference image; the first reference image is the input image; the second reference image has a second reference maximum luminance; the image processing device includes a display adaptation unit for determining an adapted luminance mapping function based on the reference luminance mapping function; the display adaptation unit uses a pre-fixed display adaptation algorithm, which determines for each point on a diagonal line, in a coordinate system of the input luminance normalized to a maximum of 1 and the output luminance normalized to a maximum of 1, a respective metric along a line segment starting on the diagonal line and oriented in a pre-fixed direction; the respective metric at each position is normalized in the coordinate system by assigning a value of 1 to the intersection point between the line segment and the locus of the reference intensity mapping function; Determining the luminance mapping function includes identifying a location on each respective metric that corresponds to the second reference maximum luminance; The set of positions on each metric is output as the adapted intensity mapping function. the image processing device having a processor including a boost decision circuit for calculating a booster strength value; The boost determination circuit, a histogram calculation unit for determining an intermediate luminance histogram obtained by applying the adapted luminance mapping function to the input luminance; an average calculation circuit for calculating an average brightness value based on the histogram; a first converter unit for calculating a first intensity value from the average brightness magnitude; a second measurement unit for calculating a weighted sum of pixel counts in at least two configurable top bins of the histogram; a second transformation unit configured to calculate a second intensity value from the weighted sum; the boost decision circuit determines the booster strength value based on a value equal to the first strength value minus the second strength value; The image processing device further comprises: using the display adaptation unit to calculate an adjusted adapted luminance mapping function, the adjusted adapted luminance mapping function being calculated by using the display adaptation algorithm at a setting that outputs a locus of positions on a metric that are equal to the booster intensity value as the adjusted adapted luminance mapping function; 4. An image processing apparatus comprising: a color converter that applies the adjusted adapted luminance mapping function to the input luminance to obtain the output luminance.

7. 7. An image processing apparatus as claimed in claim 6, wherein the boost decision circuitry includes a temporal filter configured to calculate a booster strength value that varies less.

Images