Graphics-safe HDR image luminance regrading

Abstract

To provide an image processing apparatus.SOLUTION: An image processing apparatus includes: an output image connection section for connection to a display to enable practical insertion of a secondary image by a device connected to the final display; an input section for receiving an input image IM and metadata specifying at least one luminance mapping function F _ Lt; a graphics generation unit where the luminance mapping function identifies a relationship between the luminance in the input image and a second image with high or low maximum luminance and determines the secondary image IMG; and an image synthesis unit that synthesizes an output image IMC based on pixel colors of the input image and the secondary image, the image processing apparatus being configured to further include a luminance function selection unit, which outputs a copy of at least one luminance mapping function F_Lt to the metadata output unit if the colors of the secondary image are not mixed with the input image, and which outputs a specified mapping function F3 if the output image is not identical to the input image because some pixel colors of the secondary image are used and changed the colors of the input image.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a method and apparatus for manipulating pixel colors whose luminance needs to be adjusted from a first situation, e.g. typically the original encoding of an HDR image, and a second situation, e.g. typically the optimization of said image for presentation on a display having a dynamic range, in particular a display peak luminance PB_D, different from the peak luminance, also known as maximum luminance (PB_C), present in the encoded HDR image. [Background technology]

[0002] Until a few years ago, all video and most still images were encoded according to the so-called low dynamic range (LDR) principle, also called standard dynamic range (SDR). That meant that whatever the original scene captured, the maximum value of the code (typically 8-bit luma Y'=255, and similarly for the non-linear approximately square root R'G'B' color components) corresponded, i.e. should be rendered, by a standardized definition, to a display with a display peak brightness PB_D (i.e. the brightest white that the display can render), which by standard agreement is 100 nits. The reason this was done was that all displays were practically identical in terms of their color capabilities, i.e. they could render (only) luminances between 0.1 and 100 nits, and one had to make do with that to make the color of any image desired. Nevertheless, it seemed to the users of the system that for decades any image that could have been generated or desired could be displayed reasonably convincingly and with sufficient color or visual quality. But it also simplified things, because for any scene, even a naturally HDR scene like a cave, it had a well-defined single gamut in which a reasonably looking image and their colors could be defined, for example, by brightening the cave interior and typically restricting the exterior pixels to white or light pastel colors.

[0003] Such displays can render fairly good images, and consumers do not complain about image quality, especially since the reflectance of natural objects ranges between approximately 95% and 0.5%, so long as care is taken with the lighting of the scene during content creation by making it reasonably uniform, the end viewer will in fact at least see all objects reasonably well, with realistic colors and relative brightnesses. However, this may not be so evident, especially when scenes containing either bright light areas or ranges of dark shades must be rendered realistically so that the viewer gets a realistic impression of different lighting regimes throughout the scene.

[0004] The expected and now realized advent of high dynamic range (HDR) displays capable of rendering much brighter pixels (typically 10 times brighter for good quality HDR displays, but as much as 100 times brighter for high-end HDR displays) has also necessitated new techniques for encoding and manipulating such HDR images, i.e. images that contain areas significantly brighter than 100 nits SDR white (e.g., 900 nits bright city lighting lamp rendering) and / or darker areas (OLEDs under the correct viewing environment should be as dark as ±1 / 1000 nits). The dynamic range of cameras has also become increasingly better, but for the sake of understanding the present technology, the reader may simply assume that pixel brightness is generated by a computer, i.e. according to optional but of course required technical principles, e.g., the details of a specific HDR video codec. The reader will understand that in practice HDR video can be used in various areas of technology, from consumer television to security, etc.

[0005] In this text, when an HDR image or video is mentioned, it is assumed that it has a corresponding encoded peak brightness PB_C or maximum luminance for the highest luma code (or equivalently the highest R', G', B' value in case of RGB encoding instead of YCbCr encoding) higher than the SDR PB value of 100 nits, typically at least 6 times higher. Therefore, there can be image pixel color information encoded in HDR image encoding of such bright pixel luminance (e.g. 1000 nits instead of being limited to 100 nits as in SDR image / video encoding), even though the manner in which the encoding is done can be diverse. Correspondingly, the maximum display luminance rendered to make the HDR image look optimal is e.g. 1000 nits, 5000 nits or 10000 nits. This should not be confused with the obvious composite concept to be detailed below, that such an HDR image or video may in fact be encoded as an SDR image or video that is then communicated to a receiver, in which case the SDR image can be directly rendered on a 100 nits display, but importantly note that such a communicated SDR image also contains all the information to create an HDR image with, for example, a PB_C of 1000 nits and thus such a bright HDR pixel brightness (not directly encoded as only SDR luma), when simultaneously communicating the corresponding associated metadata that encodes the brightness transformation to recover the HDR image brightness from the SDR image pixel brightness.

[0006] Once one takes steps to accommodate HDR video encoding, for example in a typical television distribution chain or in a typical television signal handling device, such as a BD player or a set-top box (STB), one finds that much of the common prior art has to be redeveloped and common SDR-era technical beliefs have to be rethought, and often even reinvented.

[0007] Thus, high dynamic range encoding of a high dynamic range master image can encode images with a to-be-displayed brightness of, for example, up to 1000 nits (or higher for other HDR image encodings), allowing for the display of good quality HDR images, such as a bright explosion or a holiday photo that looks really sunny compared to the surrounding displayed scene.

[0008] In reality, there are scenes in the world that can have ultra-high dynamic ranges as measurable with a luminance meter (e.g., viewing an exterior sunlit object through a window with a luminance of over 10,000 nits, while simultaneously viewing an indoor capture with an object as dark as 1 nit or less, gives a dynamic range of ≧10000:1, which is 10 times greater than a dynamic range (DR) of 1000:1 and 100 times greater than a dynamic range of 100:1, and for example TV viewing has a DR of less than 30:1 in some typical situations, e.g. daytime viewing).

[0009] As displays become better than ever (PB_Ds that are 2-3 times brighter than 100 nits, 1000 nits have been available for televisions for almost a year now and even monitors since this year, and PB_Ds of several thousand nits are already expected and appearing), the aim is to be able to render images of various HDR scenes correspondingly more and more beautifully, and at least very naturally, or at least satisfactorily, if not exactly the same as the real thing due to factors such as different viewing conditions. And this requires something that was lacking in the SDR video coding era, namely, a good practical HDR video coding technique that encodes how to render those images. In addition to this good operation method for the coded images, so too is, for example, a new optimal display method. The coding should also satisfy as much as possible the many practical requirements of the various players in the market, such as cable TV providers, IC manufacturers, content creators, etc.

[0010] The reader should also understand that there is no identity between luminances in a scene and those that are ultimately rendered on a TV (or other display) because viewers are typically viewing content in different conditions (e.g., sitting in a dimly lit living room at night, or in a darkened home theater or movie theater, rather than actually standing in the bright African landscape that was captured, so bright objects can suddenly appear overly bright in darker viewing conditions). There is no identity even between relative or normalized luminances, i.e., all scene luminances divided by some maximum scene luminance versus the display rendered luminances divided by PB_D (also on a 0 to 1.0 scale). Thus, for the purposes of this discussion, the reader may begin his or her research by ignoring the original scene as it existed during camera capture, and focus only on the image as it will be displayed (e.g., on a high quality reference display of, say, 5000 nits PB_D, as a representation of what an actual HDR image will look like). This formation of the final HDR image, which will be displayed corresponding to the original scene luminance and lightness impression, can be manipulated, among other things, by having a human color grader manually determine the optimal color on the coded dynamic range C_DR of an available, i.e. relevant, reference display (e.g. PB_D=5000 nits), for example by specifying that the sun in the scene should be rendered in the image at 5000 nits (and not its actual measured value of 1 billion nits, which cannot be rendered on any display).

[0011] This is called HDR master grading of the content (and the resulting image is a master HDR image or grading), and obviously how it is done in practice again depends on various practical factors, such as, for example, whether the content is created as a live stream from the event being broadcast, etc. For example, instead of human intervention and as far as the present application is concerned, an automatic algorithm performs such a conversion, for example from the raw camera capture to what would be (generally) called in the text a (master) HDR grading. This means that this master grading can then be displayed on a 5000 nits PB_D HDR display where it is available (directly, without further colorimetric optimization, since it is consistent with said 5000 nits display and therefore optimal when rendered on such a 5000 nits display). Automatic brightness determination will often occur somewhere in the technical operation chain of HDR video communication, from the original creator to the final consumer.

[0012] However, at the same time, over the next few years there will likely be a large installed base of people with 100 nits PB_D legacy SDR displays, or at least some displays that cannot produce 5000 nits white (e.g. with a dynamic range of display peak brightness PB_D=500 nits) e.g. because they are portable, but they still somehow need to be able to view HDR movies as well, ideally as optimally as possible (i.e. typically with all object brightnesses looking reasonably close to what they were in the HDR master, so that at least the e.g. atmosphere or mood of the image is preserved). So there needs to be some mechanism to convert from a 5000 nits PB_C HDR image of the same scene to a 100 nits SDR look image.

[0013] For the convenience of the reader and to quickly familiarize the reader with some of the relevant aspects, Figure 1 illustrates some prototypical illustrative examples of many possible HDR scenes that a future HDR system (e.g., connected to a 1000 nit PB_D display) needs to be able to handle correctly, i.e., by rendering the appropriate brightness for all objects / pixels in the image. For example, ImSCN1 is a sunny outdoor image from a Western movie (which often has a bright range and should ideally be rendered somewhat brighter on a 100 nit display, providing a sunny rather than rainy look), while ImSCN2 is a nighttime image.

[0014] What makes such an image bright and dark on the other? Not necessarily the relative luminance, at least not in the SDR paradigm. What makes HDR image rendering different from the way it used to be in the SDR era that ended only a few years ago is that mostly only the intrinsic reflectance of objects could be shown in SDR (fitting between 90% of good white and 1% of good black) since SDR had a very limited dynamic range (about PB=100 nits, and black level around 0.1-1 nits). That is good for recognizing objects (with a certain amount of brightness from their reflections, and of course their chromaticity) under uniform technologically controlled lighting, but it cannot even have beautiful variations in the lighting itself in a natural scene, and have any impact on the viewer. The luma histogram could make the night image somewhat darker, but otherwise it would just render it as too dark and unpleasant, and there is no room available for anything too bright on a 100 nit TV or with a 100 nit encoding. So objects had to be shown independent of their illumination, but all possible, sometimes highly contrasty, illuminations of the scene could not be faithfully shown at the same time. In practice, that meant that a very bright sunny scene had to be rendered at about the same display luminance (0-100 nits) as an overcast rainy scene. And even night scenes could not be rendered too dark, but still their night luminances were rendered over a range between 0-100 nits, because otherwise the viewer could not well distinguish the darkest parts of the image. The traditional solution to that was to make the night scenes blue, so that the viewer understood that he was not looking at a daytime scene. Now, of course in reality human vision also adapts to the amount of light available, but not by much (most people in reality realize that it is getting darker, or that they are in a darker or significantly brighter environment). So we want to render an image with all the spectacular local and even temporal lighting effects that can be artistically designed into it, to get a much more realistic rendered image, at least if HDR displays are available.It is left to the color grader creating the master grade (or shader or color processing automatic algorithms in broadcast, etc.) to decide what exactly is the appropriate brightness for, say, a light saber in a dark room, and this application focuses on the technical possibilities required to create and manipulate such images.

[0015] On the left axis of Figure 1 is the object luminance that one would like to see with a 5000 nits PB master HDR grading, for a 5000 nits PB_D display (i.e., a grader might create an image assuming that a typical high quality HDR TV in a home has 5000 nits PB_D, and then grade on such a grading display while actually sitting in a representation of such a home viewing room). If one wants to convey, not just an illusion, but the actual sense that the cowboys are in a bright, daylit environment, then their pixel luminance must be specified and rendered bright enough (but not annoyingly overly bright, a typical pitfall of HDR image creation and manipulation), say around 500 nits. For night scenes, we mostly want a dark luminance, but the main character on the motorcycle should be well recognizable, i.e. not too dark (say around 5 nits), and at the same time there can be fairly bright pixels, for example street lights, for example around 3000 nits on a 5000 nits display, or around the peak brightness on any HDR display (say 1000 nits). A third example, ImSCN3, illustrates what is currently possible with HDR displays, which can render very bright and very dark pixels at the same time. It shows a dark cave, through which a small opening can be seen to a clear sky outside. For this scene, we want sunlit objects such as trees to be somewhat less bright, for example around 400 nits, than in a scene where we want to render the impression of a bright clear sky, which should better match the essentially dark nature of the cave interior. The color grader wants to optimally blend the luminance of all objects (already in the PB_HDR=5000 nits master HDR image) so that nothing appears inappropriately dark or bright and the contrast is good, e.g., this person standing in the dark of the cave is encoded at around 0.05 nits in the master HDR grading image (assuming the HDR rendering can render dark areas as well as bright highlights).

[0016] Now, secondly, it is often necessary to have an SDR regrading image for an HDR image, which can be briefly summarized as follows: whereas a capable HDR display (or an image coded accordingly, for example as an HDR image with PB_C=5000 nits) spreads far-away object luminances along a large luminance range to their optimally looking luminance positions, in the SDR luminance range they need to be compressed together and fit along a small luminance range. Nevertheless, in such a manner that the SDR image still preferably conveys the HDR look as far as possible, for that reason we will assume for now that the secondary grading derived from the HDR master grading is still also the master grading made by the color grader, i.e. the SDR master grading. That is, it is necessary to apply a content-dependent optimal luminance mapping function, which is also currently described as dynamic metadata. The reader can appreciate that there are some complexities involved in optimizing all pixel / object luminances to any secondary image of any dynamic range (particularly the peak luminance PB_C) corresponding to a master HDR image, or in general any second image of a significantly different dynamic range derived from any first image, but for simplicity the reader can assume that a simple luminance function is used that, when converting to a low dynamic range, makes darker pixel luminances relatively brighter (i.e., a slope higher than the diagonal 1.0) and compresses brighter colors at the top of the luminance range of the low dynamic range image, i.e., with a reduced slope (this graph can be formulated with two [0-1.0] normalized luminance axes for the input or output image, respectively).

[0017] All the corresponding SDR image pixels / object luminances are illustrated in the right SDR luminance range in Fig. 1. Note that between the two grading images, we can also calculate other HDR images with different encoding peak luminance PB_C, which we call medium dynamic range (MDR) images (e.g., for 800 nits).

[0018] It can be understood that mapping all object luminances for all these very different kinds of HDR scenes to the optimal luminance available in the much smaller SDR dynamic range (DR_1) illustrated on the right of Fig. 1 is not always a trivial task, which is why preferably a human color grader may be involved to determine the color transformation (which includes at least the luminance transformation, or equivalently the luma transformation when performed on the luma code, which may actually consist of several functions formulating technically interesting sub-strategies, but for the sake of simplicity of understanding, the reader may consider it to be a single function L_out_SDR=F_L(L_in_HDR) for the purposes of this application). However, one can always choose to use an automatically determined transformation, for example based on analyzing color characteristics of the image content, such as its luminance histogram, which is a preferred option for applications where human grading is less preferred, for example for simpler kinds of HDR footage, or for applications where human grading is less preferred, for example in real-time content production (in this patent application, without limitation, it is assumed that grading may also involve simple setting of a few color transformation function parameters for the entire production, for example quickly before the start of capture).

[0019] Moreover, merely to illustrate some technical video coding possibilities, and for the sake of clarity, an exemplary HDR video coding system designed by the applicant for HDR image coding and in particular for HDR video coding is described (so that the reader will understand that the principles of the present invention are applicable to systems other than the illustrative exemplary system), which is not only capable of handling the communication (encoding) of only a single standardized HDR video (e.g., a 10-bit perceptual quantizer used as the luma code defining the EOTF for encoding) for a single type of display typical in the field (e.g., an image defined at PB_C=1000 nits, under the assumption that every end viewer has a 1000 nits PB_D display), but is also capable of communicating and manipulating video with optimal look / grading for various other possible display types with various other peak brightness in the field, in particular SDR images for a 100 nits PB_D SDR display.

[0020] That is, in such an HDR video communication system, the pixelated images transmitted are in fact only one type of graded image, typically but not exclusively an SDR image in this application (or alternatively an HDR image, typically together with a brightness modification function for deriving pixel brightness for the image with the next lower PB_C), but in this system one or more functions defining the HDR image pixel color and especially brightness from those SDR images are also added to the metadata, so that the HDR image look for the scene is also communicated at the same time (indeed, without the need to communicate an HDR image, or at least a second layer of pixelated HDR image data, as in dual image communication). And we want to emphasize that this is not just a duplex of two images, but that it brings about a new perspective from which images can be thought about and manipulated: by communicating the look, i.e., the various pixel intensities, of two far apart dynamic range images (even though in reality only one of the two is communicated to the receiver as a pixelated, e.g. MPEG-HEVC image), the content creator also communicates his artistic perspective on how the various scene object pixel intensities in a representative image (master HDR image) should be luminance-wise re-graded for a particular scene (i.e., in effect a bunch of images are communicated, or instructions for at least the receiver on how to re-grade any desired image for any display of the PB_D that matches the look or pixel intensities in the master HDR image). That additional benefit, however, comes at the cost of increased complexity in at least some practical situations.

[0021] It is not too difficult to encode just a series of time-consecutive HDR images, i.e. with the correct look, i.e. image object luminances, for rendering on, e.g., a 1000 nit HDR monitor, e.g., with 10-bit legacy MPEG HEVC or similar video coding technologies. One only needs to establish an optimal luma code assignment function, aka OETF (Opto-Electronic Transfer Function), for a new class of images with a fairly large dynamic range, i.e., those that do not show banding in relatively dark areas compared to most whites, and then calculate luma codes for all pixels / object luminances.

[0022] Applicant has, however, designed a system whereby an HDR image can actually be communicated as an SDR image (standard dynamic range, which means a legacy Rec. 709 OETF-based encoding targeted to a 100 nits PB reference display and often optimally color graded on such a reference display), which can then be quickly and immediately used to render a correctly looking SDR look on a legacy 100 nits PB_D SDR display.

[0023] This means that the receiver can assume that a) the relative luminances of all objects have been determined such that they will appear correct or at least plausible on a 100 nit PB_D display, and b) the lumas to produce such luminances have been defined by the Rec. 709 OETF, which is approximately a square root function.

[0024] In this way, for each type of HDR scene in a specific one of the consecutive images at each time, an optimal functional relationship (function shape, typically a one-dimensional function defining L_out for each possible L_in, e.g., 0.1 / 100 < L_in < 100 / 100) between the HDR image pixel luminance and the LDR image pixel luminance can be defined so that both HDR and SDR images that can be restored on the receiving side look their best (e.g., optimize the brightness inside the cave so that people inside the cave can be seen, or optimize the luminance of the outdoor area visible through the window so that it looks sufficiently imposing regardless of the capabilities of the display DR).

[0025] On the other hand, as illustrated in FIG. 2, a set of appropriate reversible color conversion functions F_ct are defined. FIG. 2 illustratively and non-limitingly shows a typical system of the SDR communication type for the purpose of explaining the basic concept. These functions are defined by a human color grader in order to obtain a moderately looking SDR image (Im_LDR) corresponding to the HDR master image MAST_HDR, while at the same time, by using the inverse function IF_ct, it is guaranteed that the original master HDR (MAST_HDR) image can be reconstructed with sufficient accuracy as the reconstructed HDR image (Im_RHDR), or an automatic analysis algorithm is used on the content creation side to determine an appropriate such color conversion function F_ct. The IF_ct function can be determined from the transmitted forward HDR-SDR mapping F_ct function, or the system may even directly transmit the IF_ct function.

[0026] The color converter 202 typically applies an F_ct luminance mapping of the relative luminance of the master HDR image (MAST_HDR) pixels, i.e., normalized so that the maximum luminance is 1.0. To understand the concept of the present invention in a simple form, for simplicity, it is assumed that it uses a fourth-power luminance mapping function (L_out_SDR=power(L_in_HDR;1 / 4)) to derive the normalized SDR output luminance (i.e., the right side of FIG. 1) of the pixels of the 100 nit PB_C SDR output image Im_LDR, i.e., that such a function gives the master HDR image of the scene a reasonable look for the SDR graded image (reasonable means that for a particular scene, most of the shadow ranges do not look too dark, and lamps and other light-emitting objects appear as desired, at least as far as the SDR luminance dynamic range allows, by virtue of the fact that they still have a reasonable inter-area contrast with the darker image regions even in the SDR image. For other images, other factors contribute, but such details are neither necessary nor limiting to elucidating the technical components of the present invention).

[0027] Since the receiver must be able to reconstruct the master HDR image from the received corresponding SDR image, or at least an approximate reconstruction free of some compression related artifacts, a color mapping function must also enter the video encoder 203 apart from the actual pixelated image. Without limitation, we assume that the video is compressed with an MPEG HEVC video compressor and that the function is stored in the metadata, for example by means of the SEI mechanism or a similar technique.

[0028] So, after the operation of the content creation device 221, from an image communication technology point of view, the video encoder 203 pretends that it gets a normal SDR image as input, and more importantly outputs what is technically an SDR image, according to the Rec.709 standard SDR luma specification. Then, further technology, such as the transmission formatter 204, which applies all the necessary transformations to format the data to go through some transmission medium 205 (e.g. encoding for storage on a BD disc or frequency encoding for cable transmission, etc.), can apply just all the typical steps it used to work in the SDR encoding paradigm.

[0029] The image data then travels to one or more recipients via some transmission medium 205, for example satellite, cable or internet transmission according to ATSC3.0, DVB or any video signal communication principle.

[0030] At any consumer or professional end, a receiver 206 is embedded in various physical devices, such as a set-top box, television or computer, and undoes the channel encoding by applying unformatting and channel decoding. A video decoder 207 then applies, for example, HEVC decoding, resulting in a decoded SDR image Im_RLDR and color transformation function metadata F_ct. A color converter 208 is then configured to convert the SDR image into an image of any non-SDR dynamic range. For example, the original master image Im_RHDR of 5000 nits is reconstructed by applying the inverse color transformation IF_ct of the color transformation F_ct used at the encoding end to make Im_LDR from MAST_HDR. Alternatively, a display adjustment unit 209 is provided to convert the SDR image Im_RLDR into a different dynamic range, such as Im3000 nits, or 1500 nits or 1000 nits PB image, which is optimally graded when the display 210 is a 3000 nits PB display. This display adjustment or display adaptation is within this framework and is based on a co-communicated luminance mapping function, which is conveniently derived by calculating a display adjustment luminance mapping function to be applied to the decoded image, which is calculated based on the luminance mapping function received in F_ct and the display capabilities of the display to which it is connected or to which the image needs to be supplied.

[0031] It has been assumed, without limitation, that the video decoder and color converter are within a single image redeterminator 220. The skilled reader will appreciate that one can similarly design a topology to communicate an HDR image having, for example, PB_C=10,000 nits, and the color converter produces an output HDR image having, for example, PB_C=2500 nits to a corresponding TV or monitor.

[0032] Additional issues beyond basic encoding / decoding: The system works perfectly if, for example, a content creator (consider for example a television broadcaster) creates an SDR image and an SDR-HDR_10000 brightness increasing function, and any receiver simply applies these functions to decode the received SDR image to an HDR_10000 image (typically assuming that all three receiver decode processing units 206, 207 and 208 are equipped in an HDR television). This is entirely consistent with what those skilled in the art usually think of from an encoding / decoding framework, i.e., the decoder just suitably undoes what the encoder did as a mathematical color transformation to generate an optimal image for communication according to whatever the main desire for a certain technical image communication system is. However, in a real video communication chain, there may be devices in between (and in fact, what is worse, the devices know little or nothing about the color manipulation operations of the devices before and after them in the processing order or the hardware-connected video manipulation system chain), and, unfortunately, those intermediate devices may change image characteristics, particularly the brightness of some pixels of at least some images. The pixel luminance is then no longer correct or formulated differently for the luminance mapping function F_ct, and the luminance mapping function F_ct suddenly becomes non-optimal (possibly far from optimal) for at least some parts of some images at some moments. And the question is then: what to do; how to deal with this big problem deftly? This is a question that can suddenly arise and can be dealt with in various ways, as illustrated in the prior art patent applications outlined below. We emphasize that the reason this question never arises and thus was ignored without incident in the SDR imaging era is because nobody worked within a stable 0.1-100 nits luminance range framework and nobody did (rather unusually shaped) luminance transformations to represent images in at least two very different luminance dynamic ranges (potentially the higher dynamic range is typically 100 times larger than the SDR dynamic range, meaning that rather unpleasant visual errors can occur, potentially even periodic flickering, etc.).

[0033] WO2016074999A also addresses the problem of inserting graphics into a video (with corresponding brightness re-grading operations simultaneously supplied as a brightness mapping function in metadata associated with and simultaneously receivable from the video) when it involves an intermediate device and a final display that does brightness processing and does not fully know what the other device is doing and / or is not fully able to coordinate the required pixel brightness processing operations, but the problem is addressed in a different manner to that proposed in this specification.

[0034] The fact that there are some graphic insertion pixels (where the graphic is some secondary image for which the luminance mapping function does not correspond correctly, and typically some simple graphics with a limited set of colors, such as logos, perhaps some insertion of an app showing weather maps, subtitles, etc.) inserted by some intermediate device between the source of the content (i.e. the device of the content creator or at least the device of the content provider, e.g. a TV broadcaster) and the device that will ultimately use the video content, which is typically a display or equipped with one, is signaled by putting some code in the lower bits, e.g. the RGB color code, of the image pixels. The final device, i.e. the display that does the final processing to obtain an image that looks correct for its display peak brightness, can then try to use a safer mapping method for the graphic pixels than simply applying a luminance mapping function (55 in FIG. 6, or 1009 in FIG. 10) that was optimal for the image surrounding the graphic object area but incorrect for the graphic pixel color. There are various kinds of graphic insertions in the field of graphic compositing, and therefore some variants are taught. They roughly fit into two categories. For simple replacement of video pixels by graphic pixels (or recombined within such a framework as opaque blending, where the mixture has 100% graphic and 0% video), it is sufficient to put a binary code in the least significant bit to indicate A=0 if the pixel is the original, e.g., an SDR video pixel, and A=1 if it is, e.g., a subtitle pixel. The receiving display then applies the received SDR-HDR luminance mapping function to the video pixel, and may decide, for example, to render the subtitle pixel at a fixed relative luminance factor of its peak brightness, e.g., 20% of PB_D=1000 nits.More advanced embodiments provide for more challenging situations where video and graphic input pixel colors are blended in some proportion, and the least significant bit encoding conveys some aspect of that blend, e.g., what % of video brightness or color was preserved in the final blended output, which the receiving display can then take into account to design a more advanced brightness mapping for the graphics area (e.g., trying to unmix by estimating the split graphics and video, or at least trying to create a reasonably working brightness mapping based on that estimated blending or splitting information). In either case, all variants involve communicating the original brightness mapping function (55) to the display along with the image that is not yet display optimized (e.g., the encoded image as received, or at least re-encoded after graphic insertion, i.e., the final brightness of the pixels still needs to be established, e.g., by the receiver of the HDMI link), in addition to a new signal classifying aspects of the graphics mixing situation, to let the display work out what the best or at least better display optimization process should be. In the following embodiments, the sender, i.e., the intermediate device, handles the decision of the better post-processing approach. WO2017089146A is a much more advanced system (the following embodiments can work even with simple systems) and deals with the general matching of two different input signals with significantly different dynamic ranges, e.g. a 2000 nits PB_C HDR first image to be matched with a 100 nits PB_C SDR image, and the underlying techniques that need to be calculated for this, i.e. determining the optimal mixing dynamic range of the mixed image content presentation (which may be significantly different from the dynamic range of both input images, but in some situations does not have to be the final dynamic range of the display that will show the mixed image, for example), and also the matching of various representative (HDR scene situation summary) luminance anchors, and the determination of the final gamut function that maps the luminance of the various images to be mixed to an output luminance on the mixing dynamic range.This is a technology, at least some of its aspects may be combined with some of the aspects below, but it is a very different system and technical reasoning.

[0035] US20150156469A is a set of HDR teachings that are similar at a high level (i.e., the combination with graphics is also mentioned), but there are also many differences. In order to allow the reader to get the various key points and differences, such HDR approaches are outlined in FIG. 6. The EDR (their term for HDR) decoder in '469 is scalable, also known as two-layer (base layer data and enhancement layer), or in fact two images per image. The last part means that '469 needs to send two images per image, unlike the typical embodiment of this application, which communicates only one image per video instant. Not even an HDR master grade and an SDR grade per se (one could ask "then can you just send me those two images?") from which the image is derived, and both grades could usually be encoded anyway, and possibly with fewer bits (although this becomes particularly interesting when display adjustments are required, as the two image transformations require more data over the communication channel, and also more complexity).

[0036] So that the different technical components (and tasks) can be fully understood, one should first look at FIG. 6A of this patent application.

[0037] This clearly illustrates that an HDR video manipulation processing chain (or device structure) involves more than what one skilled in the art typically thinks of from coding and decoding (at least coming from the framework and perspective of the LDR era). A decoder 601 (present in some HDR video signal receiving device) generates, in this (just) example, from a received LDR image (BL_LDR) a 1000 nit HDR image (Im_REF_H, which is the starting point for further display optimization). It will therefore typically apply a fairly steep brightness mapping function, mainly brightening the brighter pixels of the input image BL_LDR as shown. After this decoding to a reference image situation (which is typically a reconstruction of a master HDR image created or created at the sender), an optimization of that image, i.e. all its pixel brightnesses, to the dynamic range capabilities of the connected display must occur by the display adaptation unit 602 before sending the final display-adapted image Im_DA to the connected display 603. If the display has a PB_D of 800 nits, then we should create an image with a maximum encoded peak brightness of PB_C=800 nits. Since this is quite close to the 1000 nits of the 1000 nits intermediate reference image Im_REF_H, in this example we could in principle skip the display adaptation and just render the 1000 nit image (so that the maximum image brightness PB_C is mapped to the maximum displayable brightness PB_D of the display, i.e. what should be displayed as a bright 1000 nit pixel becomes a dark 800 nit pixel), which could also be considered a brightness error, but if we want to do it right, some slight brightness-correction brightness mapping curve should be applied, as shown in the rectangle 602.

[0038] This overall aspect is similar at a high level to the present approach below, except that the display adaptation and of course the video decoding are quite different, as will be illustrated by the exploded view of a prior art scalable HDR decoder in Figure 6B. The present equivalent of Display Adaptation (DA) in '469 is Display Mapping (DM) metadata, which is typically more general information about the situation that should guide display optimization, such as general gamut mapping based on aspects such as color space or gamut information.

[0039] The signal S_HDR has four types of data (probably three in a simplified version, but described in general terms): a first, base layer picture (BL_LDR), which is an LDR picture directly displayable on a legacy 100 nit SDR display (called VES1 in '469); a second, enhancement layer picture (Im2_ENH; VES2), which is decoded by another, e.g., AVC video decoder 120; metadata about the first picture (MET(F_L2ImP)), which in some embodiments is typically a luminance mapping function for predicting the intermediate picture Imp; and metadata about the second, enhancement picture (MET(DecEnh)).

[0040] What exactly is in the image and metadata depends on the scalable HDR encoding transformation used, as there are some transformations (usually minor ones regarding multiplicative vs. additive transformations).

[0041] The older one is a multiplicative variant, which was based on the fact that the first HDR displays consisted on the one hand of a locally modulatable 2D backlight (i.e. various LEDs behind it could be dimmed or brightened to illuminate small areas of the LCD pixels) illuminating an LCD panel where the pixels acted as valves letting through a certain percentage of the local light. With a fixed backlight, the older LCD panels could only let through, for example, 0.5% to 100% of the light, giving a displayable dynamic range of 200:1. If we could reduce the 0.5% light output to, say, 0.05% light by dimming the LED light behind those pixels by a factor of 10 for darker areas of the image, and on the other hand brighten the LEDs behind the brighter image object areas by a factor of 10 (meaning now locally 100% of the light, although the LCD pixel would still only be able to let through at most 100% of the light), we could already achieve a more reasonable DR of 20,000:1 on the one hand, and a PB_D of 1000 nits instead of the usual 100 nits on the other hand (because that is the luminance that can now be generated for the brightest image area pixels). So, combining this on the other hand with the more general observation that, also in the real world, the luminance emanating from an object point can be modelled as a variable incident illuminance times an object reflectance on the order of 0.5% to 99%, if this is what is needed to generate "arbitrarily, more realistically lit" object pixels on a display, one can imagine that they could be driven by an encoding consisting of L_displayed[x,y]=L_LDR[x,y]xI[x,y] for each pixel, where L represents luminance and x and y are the spatial coordinates of the pixel (arranged in the output image to be displayed, i.e. the decoded image, and the two input images constitute a dual image HDR encoding: L_LDR is the base image used to drive the LCD pixel transmittance, and I[x,y] is the backlight illumination image, and some agreed definition specifies which I values ​​should correspond to which amounts of brightening or dimming of the various backlight LEDs behind position [x,y]).Data compression savings can be made in this encoding technique because the eye is not only less sensitive to the spatial resolution of varying illumination, but also because real HDR displays of that kind have a lower LED backlight matrix resolution than, for example, 2K HD LCD resolution (OLED HDR displays were not really of interest at the time). On the other hand, an example of the second metadata MET(DecEnh) for that HDR codec variant defines the required backlight illumination value I via a logarithmic function, and that definition is needed to be able to determine at the receiver what perceived amount of LED dimming or boosting is required to show a decent HDR image (whether it be an approximate reconstruction of the master HDR image, or just some good looking displayed HDR image that matches the LDR image BL_LDR, which many people with legacy LDR displays at home will see).

[0042] The LDR image BL_LDR is generated by applying a down-mapping curve, typically the inverse of the one shown in 601, to the normalized HDR luminance if the video chain starts, for example, with an HDR camera capture.

[0043] The other class is the additive two-image HDR coding class, which appears to be more popular recently and is the one used in various implementations of '469.

[0044] In this case, the BL_LDR image is again calculated by the sender by applying at will some (typically convex) HDR-LDR luminance mapping function dynamically adjusted to the specific image or at least one of the consecutive images of a single HDR scene, adjusted to what is required for such image to optimally convey HDR luminance as LDR luminance. The first, base layer metadata typically includes some additive prediction luminance mapping function, which calculates an intermediate predicted image (Imp) luminance from the LDR input image luminance, performed by the image predictor 610. The correction unit 611 adds the correction image (VES2) after appropriate decoding and potentially pre-processing. Hence, the two-image additive HDR decoder works as L_HDR[x,y]=F_L2ImP(BL_LDR[x,y])+Im2_ENH[x,y][Eq.1].

[0045] Then we apply a display mapping strategy to that image: L_disp[x,y]=F_DM[L_HDR[x,y]] (still assuming for now that there is no graphics information), whereby the display mapping can in principle be many things, such as a global mapping that takes into account only the local value of the input luminance L_HDR at location [x,y], or more complex local mapping techniques that may adjust the final image for display based on the luminance of objects surrounding location [x,y], the details of which are not given in '469. But interestingly, such image feature-based local processing can look at the display for graphics aspects that it has and take them into account in its display-wide final display mapping strategy, whereas other variants of the technique cannot.

[0046] Now given that basic knowledge of two-layer HDR encoding, '469 describes what can be done with those principles when you also have graphics in such an encoding situation. This is particularly with regard to the question of which of the two connected devices, the digital media player and the display / TV, should do which of the various required processing, which is decoding (and the corresponding luminance mapping of 601), display mapping, and graphics mixing, and potentially in what order.

[0047] To that end, two solution category teachings are presented (whereas Figures 2B, 3B and 4 would have simply included a right-eye image, here we teach a specific embodiment of the latter variant in which a graphic image is communicated in a second image placeholder that is repurposed for 2D applications, the details of which are not relevant to this application).

[0048] It is proposed to do part of the decoding (typically involving MPEG decompression, which should not be confused on the one hand, and color representation conversion to obtain the correct luminance for the final or intermediate image on the other hand) in the media player (i.e. an intermediate device acting as receiver of the creator's metadata, e.g. through a satellite or Internet connection, and supplier of the image to the display, e.g. through HDMI®, i.e. this receiver is e.g. a set-top box), and part in the display. In fact, in this paragraph, sending all information, i.e. HDR image information, graphic information and metadata, in a suitable format, e.g. through an HDMI® interface, there is no problem in that case, since the display can decide at its end how to optimally create the final output image. In fact, since it will first apply a display mapping strategy to the decoded HDR image, and only afterwards mix the graphics, not only can it choose how to mix the graphics (e.g. adjust the luminance), but also, since in any case the dynamic mapping has already been applied to determine the optimized image before mixing with the graphics, flickering of the mixed graphics due to the application of a dynamically changing luminance mapping can no longer occur. On the other hand, of course, in this system there is no guarantee that the display will actually do the graphic mixing as the set-top box or BD player wanted (e.g. the player's internal menus must now be communicated down to the display in order for it to do another device's interface functionality), and there is no problem if everything already happens exclusively in the video player, including optimizing the video to the PB_D of the connected display, since the TV then simply gets the pre-optimized picture and simply displays it without any further processing, acting simply as a dumb monitor.

[0049] In version 1 of Fig. 1, HDR video decoding is done by the composer 125 in the receiver (e.g. set-top box) according to Eq. 1 above. It is produced in a high quality common color space, e.g. 12-bit YCbCr. The output signal 137 over HDMI comprises the blended video and graphics produced by the blender, as well as some (possibly static) display management data. Graphics and display management appear to be handled jointly in component 145. Little or no details are given about unit 145, but there is still the issue of flicker if the dynamic processing still occurs in the TV - not if it already occurs in the player (and in any case is not proposed to be addressed in the direction towards the present invention).

[0050] A more practical solution is proposed in Fig. 2A (it is assumed that, at least for the next few years, expensive TVs will have more powerful HDR processing capabilities, and cheaper players will simply hand over all the required data to the TV, perhaps doing some initial management like channel adjustment, but leaving all the complex HDR calculations to the TV, but if the TV does all the processing, the clash problem will not exist again). All the data is simply interleaved, which actually teaches to allow decoding EDR, aka HDR images, and graphic images, where both two images per moment are communicated (unless communicated via 3D video tricks), for example as temporally interleaved frames of higher frame rate video. Also, all the metadata is communicated, including instructions on how to mix the graphics as desired by the player, i.e. alpha blending metadata. It can be seen again in Fig. 3A that the TV can choose the correct processing order so that the dynamic flicker problem underlying the embodiment of the present invention cannot occur, i.e., the TV first decodes the HDR image in the composer 325, then applies the color gamut mapping in the display management processor 330, and finally blends with the graphics in unit 340. Note also the difference between those display managements that are typically just color gamut mapping determined at the receiver side based only on the identified characteristics of the image and some global characteristics such as the maximum luminance in the received image (e.g., not the same as the maximum theoretically codable luminance PB_C, which the reader can probably imagine for a dark cave scene that does not contain 5000 nit pixels), and not the luminance mapping function scheme derived by this application based on the author's artistic luminance mapping guidance. But in any case, the processing topology between the devices is always different, and even the flicker problem does not exist. Summary of the Invention

[0051] 1, an image processing device (100) comprising: an output image connection (506) for connecting to a display (550), an input unit (510) for receiving metadata specifying an input image (IM) and at least one luminance mapping function (F_Lt) to enable good operation of the entire image manipulation chain, i.e., a decent image to be displayed on the final display; a graphics generation unit (502) configured to generate a secondary image (IMG), e.g. a CGI image, e.g. a logo or text; and an image synthesis unit (504) configured to synthesize an output image (IMC) based on pixel colors of the input image and of the secondary image IMG. In accordance with the present invention, there has been invented an image processing device according to the invention, characterized in that the image processing device (301 in Fig. 3 and 501 in Fig. 5) comprises a luminance function selection unit (505) configured to output (in the output metadata) a copy of at least one luminance mapping function (F_Lt) to the metadata output section (507) if the colors of the secondary image are not mixed with the input image, and configured to output a predefined luminance mapping function (F3) in the output metadata if some pixel colors of the secondary image have been used to modify the colors of the input image so that the output image is not identical to the input image (i.e. for those pixels the colors in the output image IMG are different, and typically non-trivial related, to the corresponding co-located pixel colors in the input image or its decoded IMD).

[0052] Such devices are intermediate image processing devices (e.g. BD player or set-top box) that exist in an image processing chain or system between the input stream of images from the content creator (e.g. direct TV broadcast) and the final display, which performs dynamic range conversion luminance mapping, also called re-grading, on the received images to obtain a final image that is optimal for its dynamic range capabilities (i.e. e.g. SDR to 2000 nits PB_C decoded HDR and then to 1500 nits PB_D, or SDR to 1500 HDR in one step). The intermediate devices do not necessarily know the contents of the images (even though in some embodiments they have as much knowledge of the image characteristics as high-end displays, lesser HDR TVs do not), nor do they know what will happen luminance-wise on the final display, let alone the content creator's intention (which is encoded in the luminance remapping function shape). A very practical solution is needed.

[0053] What the device / STB knows is that it can change some pixel colours during operation, especially their brightness (e.g. by replacing a dark street HDR scene image pixel by a white graphic arrow pixel, or by alpha blending its video pixels with white, etc.), i.e. it knows the facts (about what it has done, i.e. what has happened under its control, e.g. whether the user has pressed a remote control switch on the user interface which initiates image composition of video and secondary images, in this typical case a graphic image (IMG)). The Luminance Function Selection Unit (505) then acts as a kind of selection switch: if no graphics were mixed, the pixel colors would not have been changed (i.e. IMC = IMD, where D is optionally the decompressed image (similarly the process can operate on images that do not need to be decoded or decompressed) if MPEG video compression was involved, otherwise IMC = IM), then the content creator's original luminance mapping function F_Lt is perfectly correct for each pixel of all images and can simply be passed to the display (more advanced intermediate devices may calculate their own versions of better functions, and new images with possibly different pixel luminance distributions, but the principle remains the same).But if graphics are mixed, the device 501 can instead put into the stream a predefined function (F3), i.e. a function that it can predetermine by itself to be a safe function for the current or typical graphics insertion situation, or at least one selection of a set of functions predefined, e.g. at the factory, by the developer of this device (1 or 1 out of N functions loaded, e.g. from memory), or even the content creator may have added to his communicated normal dynamic luminance mapping function a well-working predefined fixed luminance mapping function to be used for typical or non-typical graphics mixing situations (but taking into account the luminance distribution details of the current HDR scene image), and the STB can then load and use the predefined function at its creator side and output it directly, or it can then determine its own adjusted predefined function (for final, communication to the display). So the original content creator may or may not know this (i.e. he may not specify a well-working fallback function for his movie), but at least the device 501 picks a well-working smart function. The degree of smartness can vary in different embodiments. A simple embodiment would simply use a single simple function that works well for graphics in general or at least one prototypical graphic insertion situation (e.g. for white subtitles, the device would decide to focus on the final luminance of those subtitles rather than the luminance mapping operation of the various image pixels of any particular HDR or SDR image, whereas a more advanced version would adjust the function of a fixed pre-defined luminance mapping function that will be used as long as the graphic is mixed, or at least for a significant portion of the mixing period, to better suit the relationship between graphic and video pixel luminance or color). For example, the device would try to follow to some extent the desired luminance mapping operation communicated in the dynamic function, e.g. it might keep the luminance mapping operation, i.e. the function shape for the darkest pixels, the same or similar to the shape of the originally input dynamic luminance mapping function F_Lt, but it might want to be conservative with respect to increasing the luminance of brighter pixels, etc.This will typically produce a sub-optimal HDR image for display (i.e. not necessarily the master HDR image look) with, for example, somewhat darker highlights, e.g. street lights (e.g. boosted by a factor of 4, i.e. 400 nits instead of 1000 nits), but at least as long as the UI is displayed, the UI will look and be displayed reasonably well, and typically the HDR image will still be fairly reasonable (and perhaps the viewer will be concentrating on the UI, e.g. while reading comments, menus or some information about the image, and not the final full HDR image pixel brightness. The shape of function F3 can be selected by device 501 so that, on average, the UI Colors will look reasonable in most typically occurring kinds of HDR images without even taking the final display into account (the image will probably be somewhat brighter on a bright display, but with such an approach at least the relationship between original image pixel luminance and graphic modification luminance is reasonably controlled, i.e. how much brighter for example a white arrow will be than other typical luminances in the image; even discounting the final display and the viewer's eye adaptation, the relationship between graphic luminance and image luminance in the luma code representation of the image is already important and can therefore be handled more carefully by higher quality embodiments of the apparatus.

[0054] In some embodiments, the device outputs a fixed, predefined luminance mapping function with a shape that is predefined to give acceptable visual results regardless of the maximum luminance of the connected display (550). So, for example, if a subtitle is placed at 90% of some luminance reference level (e.g., 100% white), a function shape can be chosen that does not raise the maximum possible luminance occurring in the video, for example, above 150% or 200%. Although some graphics may be quite bright on a high-end high PB_D HDR display, the result is less unpleasant because at least the eye can visually see the relationship that a significant amount of the video pixels are brighter. The predefined curves can also be communicated in absolute luminance representations rather than 1.0 normalized representations, for example, a mapping curve ends up at 2000 nits for its brightest pixel (whether video or graphic), so that even a 10,000 nits display does not render large graphics too brightly that may be considered annoying by some viewers.

[0055] Another embodiment outputs a predefined luminance mapping function whose shape depends on the maximum luminance of the connected display (550). This is useful for relative representation, whereby the STB first simply polls the peak brightness of the display, then ends its mapping function at the brightest, e.g. 100% (i.e. 1.0) for a 1000 nits PB_D display relative to the input SDR luminance or luma, 70% for a 2000 nits PB_D, 50% for a 3000 nits PB_D, etc., or whatever allocation is desired according to the intermediate device / STB manufacturer.

[0056] An alternative embodiment of the image processing device (301, 501) communicates an indicator (FLG), such as a flag or function type number, in the output metadata to the display when the display is connected to receive the output image (IMC), indicating that a predefined mapping function should be used by the display to convert the output image (IMC) received by the connected display to an image of a different luminance dynamic range (typically a display-compatible image). This is of interest for smart displays, since sending a good predefined function can accommodate any dumb display that does not have any capabilities beyond simply applying the fixed or dynamic metadata it receives, whereas smart displays will want to do part of the procedure themselves, i.e., whatever the device outputs. By only getting an indication that a safe fixed luminance mapping is needed, the display can read a fixed function from its own memory for some time indicated by successive images and their associated flags or other similar metadata (it should be noted that in principle it makes no big difference whether the same function is loaded on the device or on the display, however differences can arise when a smarter function decision is needed, for example based on the creator's F_ct or image characteristics, and furthermore the device and display can be made by different manufacturers, so the variations are larger than others, or at least sufficient coordination is needed). If the device and display are made by the same manufacturer for example, such synchronization can be useful, since an integer (such as using a given function number 1 or number 2) can be a communication to the display to select from its own memory, rather than the corresponding function being retrieved from the device function memory and then communicated (this still gives the same rough indication in visual quality of what has happened inside the device in terms of graphics mixing, and which function is safe or likely to be safer to apply as luminance re-grading inside the display). This is, by reference, the equivalent of function communication. Flags, for example, may function in a system that is found to be a function of a single high alert that is qualitatively sufficient.

[0057] Some of the device embodiments work with existing HDR displays that lack the processing power, but there are also smart HDR displays that support it, such as:

[0058] A display (550) comprising a dynamic metadata analyzer (551), which is configured to read from the output metadata received from an image processing device according to any of the taught variants, when such an image processing device is connected to the image input (552) of the display, that such an image processing device communicated a predefined fallback luminance mapping function (F3) instead of a dynamically varying luminance mapping function. If such is indicated by any mechanism, such as a flag or writing dynamic metadata functions in placeholders of the received video signal other than the predefined fixed function (i.e. keeping a fixed shape for several consecutive images), by reserving data for that function, such as the code PRED_FL in the signal standard, then the 0<[i,j]<1 coordinates of the multilinear representation of the function shape, etc., the display can not only apply the fixed function, but also be aware of the situation and adjust its further behavior (e.g. not do excessive internal contrast stretching, etc.).

[0059] A display (550) that identifies in its output metadata that a first type of metadata (MET_1) predefined to communicate a predetermined fallback luminance mapping function is present.

[0060] The display (550) identifies that a second type of metadata (MET_2) predefined to communicate a time-varying luminance mapping function is not present in the output metadata. This can be a check for the absence of an original dynamic luminance mapping function.

[0061] A display (550) that identifies that an indicator (FLG) is present in the output metadata that a predefined mapping function should be used by the display to transform the output image (IMC).

[0062] These devices also have corresponding methods of operation.

[0063] 1. A method for providing a high dynamic range image signal for driving a display, comprising a step of receiving an input image (IM) and metadata specifying at least one luminance mapping function (F_Lt), the luminance mapping function specifying a relationship between luminance in the input image and in a second image having a maximum luminance that is at least six times higher or lower, the method comprising a step of generating a secondary image (IMG) and a subsequent step of synthesizing an output image (IMC) based on pixel colors of the input image and of the secondary image IMG, the method comprising a step of selecting a luminance function, the selecting step being configured to output a copy of the at least one luminance mapping function (F_Lt) in the output metadata to a metadata output unit (507) if the colors of the secondary image are not mixed with the input image, and to output a predefined luminance mapping function (F3) in the output metadata if the output image is not identical to the input image because some pixel colors of the secondary image have been used to modify the colors of the input image.

[0064] The predetermined luminance mapping function is to be used by a receiving device, e.g. a display such as a television, to transform the image in the output image signal of the method to derive an optimized image. This can be, for example, an upgrading of a received SDR image (which contains luminance information of a higher dynamic range, in that such an image allows a significant amount of luminance stretching to obtain an image close to the original HDR representation of the corresponding HDR scene) by using a function that typically boosts the luminance of brighter pixels in the input image compared to the darker luminances. However, the function can also be used to downgrade from a received HDR image, e.g. by receiving a 1000 nits PB_C image and downgrading it to obtain an image for a 500 nits PB_D display.

[0065] A method for receiving an image signal having a sequence of pixelated images, the sequence having pixels with colors having luminance, and at least one luminance mapping function (F_Lt) per image, comprising a step of analyzing whether the luminance mapping function (F_Lt) varies dynamically for successive images or is pre-determined and fixed for successive images by reading from the received metadata, which is output metadata from a combined image processing device according to one of the embodiments of the image processing device, that such image processing device has communicated a pre-defined fallback luminance mapping function (F3) instead of a dynamically varying luminance mapping function. Those skilled in the art may find inspiration from the present teachings that in practice this can be communicated / tuned in various (typically standardized) ways, for example some descriptors may identify what type of luminance mapping function is (dynamic or fixed predefined) (typically for several consecutive images, but alternatively a function and corresponding description can be sent for each of the consecutive images, which can then communicate the status without actually needing duration information for the applicability of a particular fixed predefined function), or reserve different placeholders in the metadata of the image signal, for example a first section comprising a LUT-defined variable luminance mapping function, and a second placeholder that enables the possibility of communicating a predefined function according to any of the embodiments of the present application (thus in effect communicating a general "change" status of a graphic or secondary image).

[0066] These and other aspects of the methods and apparatus according to the invention will become apparent from and with reference to the implementations and embodiments described herein, and with reference to the accompanying drawings, which are merely non-limiting illustrative examples illustrating more general concepts, and in which dashed lines are used to indicate that components are optional, whereas non-dashed components are not necessarily required. Dashed lines can also be used to indicate elements that are described as required but are not visible within an object, or for intangible things, such as the selection of objects / areas (and which are illustrated on a display). [Brief description of the drawings]

[0067] [Figure 1] FIG. 10 is a diagram illustrating, in schematic form, some typical color transformations that occur when optimally mapping a high dynamic range image to a corresponding optimally color graded and similar looking (as similar as desired and as possible taking into account the differences in the first and second dynamic ranges DR_1 or DR_2) low or more accurately standard dynamic range image, which in the lossless case also corresponds to the mapping of an SDR image encoding an HDR scene to a reconstructed HDR image of the same scene. [Diagram 2] FIG. 1 is a schematic illustration of an example of a technique recently developed by the applicant for encoding high dynamic range images, i.e. images capable of having a brightness of typically at least 700 nits (i.e. at least 7 times the PB_C of the SDR image) or more (typically 1000 nits or more), in effect communicating the HDR image as metadata encoding a color transformation function having at least an appropriate determined brightness transformation for pixel colors in addition to the SDR image, which will be used by a decoder to convert the received SDR image into an HDR image that is a faithful reconstruction of the original master HDR image created at the image creation side. [Diagram 3] FIG. 1 illustrates a system where an intermediate device capable of image processing that changes some pixel brightness resides between the source of the image from the content creator and the display that performs the final brightness mapping to get a good output image on the display. [Figure 4A] FIG. 13 is a diagram illustrating generally some of the challenges when arbitrarily mixing, for example, graphic imagettes (such as UI windows) with video, where a luminance mapping that changes significantly over time is then applied. [Figure 4B] FIG. 13 is a diagram illustrating generally some of the challenges when arbitrarily mixing, for example, graphic imagettes (such as UI windows) with video, where a luminance mapping that changes significantly over time is then applied. [Figure 4C]FIG. 13 is a diagram illustrating generally some of the challenges when arbitrarily mixing, for example, graphic imagettes (such as UI windows) with video, where a luminance mapping that changes significantly over time is then applied. [Diagram 5] FIG. 1 illustrates some possible technical arrangements according to the present concept in one schematic image. [Figure 6] FIG. 1 illustrates some concepts of a possible prior art scalable dual-layer HDR video encoding. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0068] Some aspects of the present invention are explained by Fig. 3. There is a content creation device 301, which the reader can assume, without limitation, as a television broadcaster's video device in an OB van that creates and communicates the video, or a computer of a computer graphic production company, etc. There is a created encoded video IM (i.e., for example, a Rec. 709 encoded SDR image, or alternatively, an SMPTE 2084 EOTF encoded HDR image, etc.), and a function F_ct_or (where "or" indicates the original, because according to function-based HDR image coding, this is the function that belongs to the pixelated image that is output, so that at the receiving site at least one image of a different dynamic range can be calculated) that is needed for the luminance remapping to obtain an image of a different dynamic range (for example, its reconstruction by the original master HDR image, which is represented by the SDR image that is channeled and received, for example, at the receiving side of a cable television video communication system). This F_ct_or is a color transformation or especially a luminance mapping function specified by the original content creator (e.g., indicating how his content should be luminance adapted so that the receiver obtains an image that still looks optimal under very different dynamic range capabilities, for example by calculating what the luma or luminance of the corresponding image should be to render at a 750 nits display peak brightness PB_D based on specific luminance processing instructions encoded in the luminance mapping function shape F_ct_or originally created by the content creator as specifying how a corresponding specific HDR scene image should be luminance regraded to, for example, a 100 nits SDR image), and it is communicated through the video communication medium 302 in metadata, e.g., SEI message, output by the content creation device. For now, we will assume that this medium is a portable memory and especially a Blu-ray Disc (BD), but the skilled reader can imagine, for example, a satellite TV connection or an Internet connection as well.

[0069] The video stored on the BD will not be input directly to a display 305 (e.g. an HDR TV), but will typically be input to an intermediate image processor 303 which in operation is connected via a second video communication interface 304, e.g. an HDMI cable or wireless connection or the like (this raises interesting additional issues as the intermediate device may do some image processing operations, whereas otherwise a direct feed to a display would mean any operations would occur at the display). In the BD example, the intermediate image processor 303 would be a BD player, but those skilled in the art should understand how the same technology could similarly be embodied in a STB or computer etc. That the intermediate image processor 303 is typically co-located with the display, usually in the end viewer's home, is symbolically illustrated by the dotted rectangle.

[0070] The problem is now further explained by Fig. 4. It illustrates a typical example of a time-varying luminance mapping that changes between successive shots of a video in response to the changing luminance characteristics of various HDR scenes in a movie (for more information than the simple explanation given in this application, the reader is referred to WO2012035476). While some may consider SDR<>HDR conversion to be relatively straightforward, in this simple example the luminance mapping function shape already shows how general purpose dynamic range remapping is.

[0071] In Fig. 4A, an example of an HDR scene from which successive images are shown is shown. A person walks from a nighttime street overview street scene into a well-lit house, where only the interior composition frame is shown (the exterior seen through the window can be limited to black). Related problems occur when a person walks from a sunlit daytime outdoor to a fairly dark, but not yet dark, indoor.

[0072] In FIG. 4B, we list several example (e.g., artistically determined by the content creator's color grader) desired luminance mappings (i.e., corresponding mapping function shapes) between the left HDR luminance L_HDR (assuming a typical HDR10 range up to PB_C=10,000 nits) and the SDR luminance (normalized to 1.0) on the right axis.

[0073] For example, a human wearing white clothing can be a good reference luminance object for white objects (but luminance changes when walking between light and dark environments).

[0074] In well-lit indoor scenes, we can have a situation close to SDR (in fact, if the room is well lit, it is known that the SDR representation usually works quite well, except perhaps for some too dark corners that map to invisible black, or clipping some lamps, etc., and therefore not much of a problem with SDR color representation and / or display errors). We can therefore have a luminance up to the SDR white level, i.e. the white of the person's clothing can be mapped close to, or for illustration purposes exactly to, 100 nits, which is 100 nits in the communicated SDR image (and if, in terms of lighting, this seems to correspond to a 100 nits luminance level in the HDR image as well, then the SDR image seems to be a pretty good representation of the HDR image, at least for that partial situation at that moment).

[0075] Formulated in reverse, in such a situation it still makes sense to give the person's white shirt pixels a luminance of ±100 nits in the (reconstructible or master) HDR image, which is an isoluminance mapping. In the normalization function definition of Fig. 4C (i.e. the luminance mapping function is defined equally between normalized SDR and HDR luminances that can go up to 1.0), such an operation corresponds to the first mapping function 420, which has (relative) dimming properties. For example, if PB_C of the HDR image is 1000 nits (i.e., the actual luminance corresponding to 1.0), then the first mapping function 420 corresponds to dimming / 10 for all possible SDR input luminances (because a 100 nit white shirt is 1 / 10 of the 1000 nit HDR image PB_C) (which in this example means that there is nothing above 100 nits or 0.1 in the HDR image, which is indeed the case for a less high dynamic range uniformly lit scene represented as an HDR image, and in fact there are some bright outliers, but their mapping depends on how the SDR image represents them, e.g., clipping). In Figure 4B, the corresponding mapping of the white person's clothing is represented diagrammatically by the dotted luminance reassignment arrows 410.

[0076] However, when a person walks outside, a night street may have very dark corners as well as very bright pixels, e.g. street lights (and objects near them, such as walls), at the same time. Therefore, this rather high dynamic range of luminance must be compressed to the still quite small 0.1-100 nits SDR range. The relative brightness of the person in the SDR range must be dimmed (arrow 411) to obtain space for brighter objects, such as street lights (arrow 412). Typically, a convex function is obtained, the inverse of which appears on the decoding side (i.e., mapping from the received SDR image to the HDR_1000 image) as a strictly increasing concave second luminance mapping function 421 (e.g., defined by linear pieces for the darkest and brightest pixels). This is obviously a very heterogeneous function with a very different visual behavior, but corresponding to a series of images over time.

[0077] From the point of view of the scene objects, the content creator would have optimized those functions so that the entire movie looks optimal not only image by image, but also as a time sequence. For example, as can be seen from the seemingly oddly low use of the HDR luminance range for the first function 420, this produces HDR pixel luminance that is similar to when the person is walking outdoors, yet the person is reasonably bright and visible in both the indoor and outdoor images with respect to the SDR image. Such functions can be adjusted and tuned to give the person a similar look, a certain amount of dimming or brightening, when moving to dark or bright areas of the scene, in both the SDR and HDR images, and in principle the relationship can be whatever is artistically required for the two gradings, depending on the content creator's ability to specify the function shape he or she needs for each moment. Hence, the scene pixels will look "perfect" for this video content.

[0078] However, problems arise when the intermediate image processor 303 starts to create new pixel values, and although it can deal with them intelligently (e.g. anticipating subsequent typical or actual brightness modification operations), it often does so in a rigid manner, without taking into account the specific colorimetric aspects of the function brightness re-grading (often a BD player wants to modify an SDR image, but simply passes the function on to the display and lets that display perform the required SDR-HDR conversion, or other generation of an image with a different dynamic range).

[0079] Suppose the BD player overlays a graphic element 402 that indicates the state of its user interface. For simplicity, let's say it's just pixel substitution, but when transparency is involved, things can get more complicated, e.g. mixing alpha x graphic luminance + (1-alpha)x lower HDR scene image luminance, or even temporary effects such as fades.

[0080] The arrow (i.e., the small image) in the graphic element 402 is typically given the color white by the designer of the UI, for example. When that designer of the UI specifies white, they are implicitly thinking of SDR white (i.e., luma 255, which displays as 100 nits, hence the 1.0 value in the SDR image being communicated, i.e., the person's shirt is in the part of the image that corresponds to at least some of the illuminated areas), and are not even aware that there may be several significantly different whites in the HDR image encoding or display. If function 420 is used, things will be fine, since the arrow will appear as 100 nits on the reconstructed HDR image. However, if F_Lt is changed to function 421 for another time, the display will raise all colors defined as white in the SDR image to the peak brightness of the HDR image, i.e., 1000 nits. Although 1000 nits is a fair value for the corresponding object in the HDR scene image, such as a lamp, the arrow will start to shine excessively. And, even more disconcertingly, every time the dynamically variable luminance mapping function shape changes, it starts to flicker and becomes significantly brighter or darker, already by a factor of 10 in this example (and thus is detrimental to readability, and therefore not suitable for e.g. user interface menus, subtitles, etc.).

[0081] FIG. 5 illustrates briefly some possible implementations of the inventive concept in one overall system configuration.

[0082] The invention can be implemented with just an intermediate image pre-processing device (501), in which case a standard, e.g., HDR TV can be connected and is unaware of the fact that something special has occurred. It will now apply a new alternative luminance mapping function F3 (i.e., instead of passing the optimally determined luminance mapping function F_Lt for the current image, which changes dynamically with time, the device 501 communicates a (fallback) function F3, which is not optimal for the current HDR scene image, but works well in practice, and is less important for the surrounding image luminance, since the viewer is probably looking at menu information). There is no good theoretical solution to this problem, because the graphic position can create any kind of luminance, which requires a different output luminance for the same input luminance value (inside the graphic) than the output luminance for a similar value input luminance of the image below where no graphic is present (i.e., it cannot be fully realized by a 1D luminance mapping function), and one would like, for example, to brighten a lamp rather than a white arrow. Indeed, it would be necessary to design complex new luminance tuning techniques like those described in WO2017089146A in the introduction, which is not always possible, for example, in devices where manufacturers want to keep things simple. However, the device 501 may at least communicate (in at least one metadata) a function that at least ensures that, for example, a graphic pixel is nearly 100% mixed with the underlying image, i.e., does not become too bright even when it is opaque, and that this output luminance, especially for a graphic pixel, does not change too much over time. For example, referring to FIG. 4, a display (or a device connected to the intermediate device 501 in general) may use a linear function similar to the function 420, but three times higher, i.e., maps a 1.0 SDR input luminance (L_in_SDR) to a HDR output luminance L_out_HDR=0.3. This already gives a nicely brightened HDR version of the SDR image.However, there are no annoyingly bright graphic elements, and there is no flickering rendering, especially since this function continues to be used for continuous images (the image brightness changes somewhat, but it is not too annoying in this graphics display scenario, and the image changes are natural and expected, but here it is displayed with a lessened HDR look than theoretically optimal). More advanced functions have a concave shape like function 421, i.e., they mostly brighten the brighter pixels of the image, but the maximum value maps not to 1.0, but e.g. to 0.3 (at least for 1000 nits PB_D displays).

[0083] This function may have a shape with two linear parts, for example, with a determined position (L_in_SDR_t) where the stronger brightening of highlights starts. The intermediate device may apply some smarts in some embodiments, for example when using disk media, it may look ahead to check what kind of image function is needed for a continuous, for example, 2 minutes, and use some average function based on that as a predetermined function, for example, to try to keep dark luminances appropriately rendered (not too bright or dim) and bright luminances not overly brightened, but also not too dull with low object contrast. Or it may just use a simple function pre-designed in the manufacturer's lab, known to give a suitable display behavior for all or most graphics in the image situation (the function may depend on the type of graphic, for example a small logo vs. a large window that overlaps a significant part of the image).

[0084] For a function that depends on the PB_D of the display, the angle of the segment of the darkest pixel can depend on the PB_D (and if it is determined by a pre-agreed algorithm, then only a flag should be communicated to the display indicating that such a function should be applied to it to make an HDR version of the output image IMC, since the display already knows its own PB_D and the pre-agreed algorithm).

[0085] In some cases, the display communicates its PB_D value to device 501 to perform a reasonable fallback function depending on PB_D, and then communicates, for example, the exact shape of this function (which can be done, for example, by encoding a LUT in the metadata, or by providing some parameters that uniquely specify the function shape).

[0086] That is, the image processing device (301 in Fig. 3 and 501 in Fig. 5) has an output image connection (506) for connection to a display (550) and an input (510) for receiving metadata specifying an input image (IM) and at least one luminance mapping function (F_Lt), although the output connection may in some embodiments be bidirectional (or have a second input for) feedback information from the display. Typically (but optionally) the input images are MPEG compressed (e.g. HEVC), in which case the HEVC decoder 503 decodes them into a normal pixel image (IMD) with pixel colors with pixel luminances. If the input comes from a BD, the luminance mapping function is burned into the BD and then read out etc. from there. The output image is decompressed in some scenarios, but in principle it would work just as a compressed image communicated to the display.

[0087] The image processor (301, 501) can output a fixed, pre-defined luminance mapping function with a shape that is predetermined to give acceptable results regardless of the maximum luminance of the connected display (550). The function is specified in an absolute manner and is shown to end at, for example, 1000 nits, even for a 5000 nits display.

[0088] This may mean, for example, that the contrast of the image is not optimal, but the function may be designed, for example, to map the darkest pixels so that they are at least visible (in the example of downgrading a received HDR image to a lower dynamic range), or, for example, to use an isoluminance approach for the darkest pixels (for example, to 0.3 at normalized SDR luminance).The device may make no assumptions about how the graphic alone will look when the display applies its given luminance mapping function F3, without the need to view the image.

[0089] The image processor (301, 501) can alternatively output a predefined brightness mapping function whose shape depends on the maximum brightness of the connected display (550). This is useful, for example, to maintain better darkness in the darkest image ranges and to adjust how bright UI appears on the display.

[0090] The way in which device 501 actually communicates the status to the display can vary, depending for example on how one wants to standardize the HDMI interface. For example, it could be agreed that the luminance mapping function to be applied is simply communicated (single metadata), and then the display applies it blindly and relies on device 501 to choose it safely.

[0091] Alternatively, the communication can alternate between two different types of metadata, MET_1 for a predefined fallback function and MET_2 when a dynamic variation function is communicated, in which case the display can check the situation with its dynamic metadata analyzer (551) configured to recognise the situation where an image source device connected to its image input section (552) has communicated a predefined fallback luminance mapping function (F3).

[0092] Such a metadata analyzer (551) can also see if an indicator has been filled to indicate that a predefined fallback luminance mapping function should be used because it has been universally pre-agreed upon (as indicated by that indicator, e.g., Fallback_type_1), even if the shape of that function is not communicated but is known by the display.

[0093] That is, the claimed image processing device (301, 501) additionally or alternatively communicates to the display an indicator (FLG), such as a flag or function type number (e.g., a predetermined function shape 1 out of 3 appropriate for a certain type of graphic mixing and / or original image luminance distribution), indicating that a predetermined mapping function should be used by the display to convert the output image (IMC) received by the display into an image with a different luminance dynamic range.

[0094] In addition to intermediate devices serving dumb displays, some aspects of the invention can also lead to more advanced displays, and embodiments of the intermediate device can also be designed to work with such more advanced displays. Such a display (550) comprises a dynamic metadata analyzer (551) configured to recognize the situation where an image source device connected to its image input (552) has communicated a predefined fallback luminance mapping function (F3), for example, it identifies that a first type of metadata (MET_1) predefined to communicate a predefined fallback luminance mapping function has been met, or that a second type of metadata (MET_2) predefined to communicate a time-varying luminance mapping function has not been met (meaning there is no optimization metadata and there will be fallback metadata).

[0095] Note that functions are taught for ease of understanding, and that in some embodiments a set of functions on an image (moment) can be communicated and concatenated and applied, but this does not significantly change the essence of the new technology. Since luminance is one dimension of a three-dimensional color characterization, other color processing aspects exist and are different for different color representations, but the present invention can be fully explained by focusing on luminance operations. Whether the device 501 mixes its pixels in luma or luminance representations is an optional detail that is not relevant to the general present teaching, but often the device sends uncompressed video through a connection to a display (e.g., through HDMI). If the device communicates its fallback situation by not communicating metadata, the display also understands it by applying a fixed, predefined mapping function.

[0096] The algorithmic components disclosed in this text may in fact be implemented (in whole or in part) as hardware (e.g., parts of an application specific integrated circuit), as software running on a special purpose digital signal processor or a general purpose processor, or the like.

[0097] It should be possible for a person skilled in the art to understand from this presentation which components are optional improvements and can be realized in combination with other components, as well as how the (optional) steps of the method correspond to the respective means of the device, and vice versa. The term "device" in this application is used in the broadest sense, i.e. for a group of means enabling the realization of a specific purpose, and thus for example an IC (a small circuit part of an IC), a dedicated device (such as a device with a display), or a part of a networked system, etc. An "arrangement" is also intended to be used in the broadest sense, so that it comprises, inter alia, a single device, a part of a device, a collection of (parts of) cooperating devices, etc.

[0098] The computer program product indication should be understood to encompass any physical realization of a set of commands that enables a general-purpose or special-purpose processor, after a series of loading steps (including intermediate translation steps such as translation into an intermediate language and a final processor language), to input the commands to the processor and to execute any of the characteristic functions of the invention. In particular, a computer program product is realized as program code, for example data on a carrier such as a disk or tape, data present in a memory, data traveling via a network connection - wired or wireless - or on paper. Apart from the program code, characteristic data required for a program are also embodied as a computer program product.

[0099] Some of the steps required for the operation of the method may already be present in the functionality of the processor instead of being described in the computer program product, such as data input / output steps.

[0100] It should be noted that the above-mentioned embodiments illustrate rather than limit the present invention. Those skilled in the art can easily realize the mapping of the examples presented in the claims to other areas, but not all these options are described in detail for the sake of clarity. Apart from the combinations of the elements of the present invention combined in the claims, other combinations of elements are possible. Any combination of elements can be realized in a single dedicated element.

[0101] Any reference signs between parentheses in a claim are not intended to limit the claim. The word "comprising" does not exclude the presence of elements or aspects not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. [Explanation of symbols]

[0102] 501 Image Processing Device 502 Graphics Generation Unit 504 Image Synthesis Unit 505 Luminance Function Selection Unit 506 Output image connection part 507 Metadata Output Unit 510 Input section 550 Display

Claims

1. An image processing device, an output image connection for connecting to a display; an input unit configured to receive an input image and input metadata specifying at least one luminance mapping function, the luminance mapping function specifying a relationship between luminance in the input image and a second image having a maximum luminance higher or lower than a maximum luminance of the input image; a graphics generation unit for generating a secondary image; an image synthesis unit for synthesizing an output image based on pixel colors of the input image and pixel colors of the secondary image; a luminance function selection unit, which selectively outputs to the metadata output one of a first type of output metadata representative of the at least one luminance mapping function if the colors of a secondary image are not mixed with the input image, and a second type of output metadata representative of a predefined luminance mapping function of the output image, which is fixed over several successive images as long as the secondary images are synthesized, if the output image is not identical to the input image because some pixel colors of the secondary image are used to modify the colors of the input image; 2. An image processing device comprising:

2. 2. An image processing apparatus as claimed in claim 1, which outputs a fixed, pre-defined luminance mapping function with a shape that is pre-determined to give acceptable visual results regardless of the maximum luminance of the connected display.

3. The image processing device according to claim 1 , which outputs a predetermined luminance mapping function whose shape depends on the maximum luminance of the display connected thereto.

4. 4. An image processing device according to claim 1, further comprising: a display device configured to receive the output image from the display device; a display device configured to receive the output image from the display device; a display device configured to receive the output image from the display device;

5. 5. A display comprising a dynamic metadata analyzer, the dynamic metadata analyzer reading from output metadata received from an image processing device according to any one of claims 1 to 4 when the image processing device is connected to an image input of the display that the image processing device has communicated a predefined fallback luminance mapping function instead of a dynamically varying luminance mapping function.

6. 6. The display of claim 5, further comprising: identifying the presence of a first type of metadata in the output metadata predefined to communicate a predetermined fallback luminance mapping function.

7. 6. The display of claim 5, further comprising: identifying an absence of a second type of metadata predefined in the output metadata for communicating a time-varying luminance mapping function.

8. 6. The display of claim 5, further comprising: a display unit configured to display an image of a source and a sink;

9. 1. A method for an image processor providing an image signal for driving a display, comprising the steps of: receiving an input image and input metadata specifying at least one luminance mapping function, the luminance mapping function specifying a relationship between luminance in the input image and a second image having a maximum luminance higher or lower than a maximum luminance of the input image; generating a secondary image; synthesizing an output image based on pixel colors of the input image and pixel colors of the secondary image; selecting, for output to a metadata output unit, one of a first type of output metadata representative of the at least one luminance mapping function if the colors of a secondary image are not mixed with the input image, and a second type of output metadata representative of a predefined luminance mapping function of the output image if the output image is not identical to the input image because some pixel colors of the secondary image have been used to modify the colors of the input image; The method includes:

10. 5. A method for receiving an image signal having a sequence of pixelated images, the sequence having pixels with colors having luminance and at least one luminance mapping function per image, the method comprising the step of analysing whether the luminance mapping function varies dynamically for successive images or is pre-determined and fixed for successive images by reading from the received metadata, the metadata being output metadata from a combined image processing device according to any one of claims 1 to 4, that the image processing device has communicated a pre-defined fall-back luminance mapping function instead of a dynamically varying luminance mapping function.

Images