Blending secondary graphic elements in HDR images

Abstract

In order to obtain much better predictable graphic insertion and better coordinate the pixel brightness and encoded luma of primary graphic elements and secondary graphic elements inserted at different times in HDR video, the inventors propose a method (or device) for determining a second luma of pixels of secondary graphic image elements (216) to be mixed with at least one high dynamic range input image (206) in a circuit for processing digital images, the method comprising: receiving a high dynamic range image signal (S_im) comprising the at least one high dynamic range input image, characterized in that the method comprises: analyzing the high dynamic range image signal to determine a primary graphic luma range (R gra) of primary graphic elements of the at least one high dynamic range input image, the primary graphic luma range being a subrange of the brightness range of the high dynamic range input image, wherein determining the primary graphic luma range comprises determining a lower limit luma (Y low) and an upper limit luma (Y low) specifying the endpoints of the primary graphic luma range (R_gra). high); brightness mapping the secondary graphic element, at least the brightest subset of the pixels luma of the secondary graphic element being included in the graphic luma range; and blending the secondary graphic image element with the high dynamic range input image.

Description

Technical Field

[0001] The present invention relates to a method and apparatus for constructing a high dynamic range (HDR) image, which is a novel way of encoding an image compared to a standard low dynamic range image, which specifically includes graphic elements. In particular, the present invention can cater to more advanced HDR applications involving brightness or tone mapping, which brightness or tone mapping can potentially be specified differently according to different scene images of a video, so that a composite color grading of different brightness dynamic ranges ending at different maximum brightnesses can be calculated from an input image with primary grading (grading generally includes a redistribution of the normalized brightness of pixels of various objects in the image from starting relative values ​​in the input image to different relative values ​​in the output image; if associated with the normalized brightness, the relative redistribution also affects the absolute brightness value of the pixels). In particular, the present invention can be useful when secondary graphic elements need to be mixed in a previously created HDR image by some video processing device (the secondary graphic elements should be distinguished from the primary graphic elements that were already present in (one or more) images before mixing at least one secondary graphic element). Background Art

[0002] Until the first studies around 2010 (and before the first purchasable HDR-decoding TVs were deployed in 2015), at least for video, all video was produced according to a common low dynamic range (LDR), also known as standard dynamic range (SDR) encoding framework. This had a number of properties. First, there was only one video produced, which looked good on all displays. The system was relative, where white was the maximum (100%) signal, encoded with the maximum luma code (255 in 8-bit YCbCr encoding), corresponding to the maximum non-linear RGB value: R'=G'=B'=255. There was nothing brighter than white, but all typical reflected colors could be represented as darker than this brightest white (e.g., a piece of paper will at best reflect all incident light, or absorb some red wavelengths, leaving blue and green to reflect back to the eye, resulting in a cyan local color, which is by definition slightly darker than paper white). Each display displays this whitest white (as a "drive request") as any brightest color it is technically configured to render, for example, 80 nit on a computer monitor (which is an easier written name for the SI quantity Cd / m^2) and 200 nit on an LCD display with a TL backlight. The viewer's eyes quickly compensate for differences in brightness, so when not side by side in a store, at home, all viewers see approximately the same image (regardless of display differences).

[0003] The desire is to improve the perceived appearance of an image not only by making colors (like they can be printed or painted on paper) but by making real luminous pixels that are much brighter than "paper white" (also called "diffuse white"). In practice this will mean, for example, that whereas in the past images were made as if in a studio, with everything important well lit from a set of lights on the ceiling, now one can simply shoot with a strong backlight and have it look good. Cameras keep improving so they can capture multiple scenes simultaneously at brightnesses sufficient for many scenes, and the brightness can be further tuned or supplemented in computers, and displays keep getting better too, even consumer TV displays.

[0004] Some systems, such as the BBC's Hybrid Loggamma (HLG), achieve this by defining values ​​above white (white is given a reference level of "1" or 100%) in the encoded image (e.g., up to 10x white), which can define 10x brighter glowing pixels to be displayed.

[0005] So far, most systems have also moved to a paradigm where a video creator can define an absolute nit value for his image at a chosen target display dynamic range capability (i.e. not 2x or 10x compared to an undefined white level that is converted to a variable actual nit output on each endpoint display). The target display is a virtual (intended) display on the video creation side, e.g. a 4000nit (ML_C) target display for defining a 4000nit specific video, and any actual consumer endpoint display may have a lower display maximum luminance (ML_D), e.g. 750nit. In this case, the end display will still need to contain luminance remapping hardware or software, which is usually implemented as luma mapping, which somehow makes the pixel luminances in the HDR input image, which have too high a luminance dynamic range (especially their maximum luminance) to be faithfully displayed, conform to values ​​within the dynamic range of the end display. The simplest mapping simply clips all luminance above 750nit to 750nit, but this is the worst way to handle dynamic range mapping, since from then on, the beautiful structure in a 4000nit image of a sunset with sunlit clouds in the range of 1000-2000nit is clipped away and displayed as a white uniform 750 tile. A better luma mapping will move the 1000-2000nit sub-range in the HDR input image to e.g. 650-740 of the final display dynamic range by a suitably determined function (which can be determined autonomously inside the receiving device (e.g. TV, STB, etc.); or determined by the video creator as the most suitable for his art film or program, and communicated together with the video signal, e.g. as standardized format metadata). Luma means any encoding of luminance, e.g. in 10 bits, using a function that assigns 0-1023 luma codes to e.g. 0.001-4000nit video luminances by means of a so-called electro-optical transfer function (EOTF).

[0006] The simplest system simply sends an HDR, e.g. 4000nit maximum brightness image by itself (with a suitably defined EOTF) (i.e. providing the image to the receiver without, e.g., indicating how it should be degraded by brightness mapping for a lower maximum brightness capable display). This is what the HDR10 standard does. More advanced systems (such as HDR10+) may also transmit a function for down-mapping a 4000nit image to a lower dynamic range (e.g. 750nit). They make this easy by defining a mapping function between two different maximum brightness versions of the same scene image or a reference grading and then using an algorithm to calculate a deformed version of that reference brightness or luma mapping function to calculate other brightness mapping functions (e.g., endpoint functions that calculate other display maximum brightness). For example, a reference brightness mapping function may specify how to change the distribution of normalized brightness in a 2000nit input image (e.g., a master HDR graded video created by the creator) to obtain a 100nit reference video, and the display adaptation algorithm calculates a final brightness mapping function based on the reference brightness mapping function so that a 700nit TV can map 0-2000nit brightness to output a brightness to be displayed within its 0-700nit range. For example, if it is agreed to define an SDR image that, when newly interpreted not as a relative image but as an absolute nit image, always achieves a maximum pixel brightness of 100 nits, then the video creator can define and jointly transmit a function that specifies how to map the 0.001 (or 0) -4000 nit brightness as a first reference image grading to the corresponding expected SDR 0-100 nit brightness (as a secondary reference grading), which is called display tuning or adaptation. If a function is defined that boosts the darkest 20% of colors by e.g. a factor of 3 in a graph normalized to 1.0 for both a 4000nit ML_C input image (horizontal axis) and a 100nit ML_C secondary graded / reference image, i.e. when going all the way down from 4000nit to 100nit, if in some specific end-user TVs a drop down to 750nit is needed, the required boost might be e.g. only 2x (depending on which definition EOFT is used for luma, since as mentioned above luminance mapping is typically actually implemented as luma mapping in color processing ICs / pipelines, e.g. using a psychovisually uniform EOTF, since this allows the impact of luminance variations along the range to be defined to be more visually uniform, i.e. more relevant and visually impactful for humans).

[0007] A third and even more advanced class of HDR encoders employs the principle of defining the imperative need for re-grading based on two reference graded images to the next level by restating it in another way. If one restricts oneself to using largely reversible functions, one can in fact transmit an LDR image which can be calculated on the transmission side by mapping the brightness or luma of, for example, a 4000nit HDR image to an SDR image, as a proxy for the actual master HDR image created by the video creator (e.g. a Hollywood studio for BD or OTT supply, or a broadcaster of sports programming, etc.). The receiving side device can then apply the inversion function to reconstruct a close reconstruction of the master HDR image. In contrast to a system which transmits the HDR image itself (when it is created), which we will call “mode-HDR”, a system which transmits an LDR image will be referred to as a “mode-LDR coding framework”.

[0008] exist Figure 1 A typical example is shown in (eg summarizing principles previously patented by the present applicant in eg WO 2015 / 180854), including the decoding function itself and the display adaptation as the following blocks, the two techniques should not be confused.

[0009] Figure 1 An exemplary video encoding and communication and processing (display) system is schematically shown. On the creation side, we show a typical encoder (100) embodiment. Those skilled in the art will appreciate that we first show the per-pixel processing pipeline for luminance processing (i.e., all pixels of an input HDR image IM_HDR (usually one of several master HDR images of a video created by a video creator, creation details such as camera capture and shadows or offline color grading are omitted as they are understandable to those skilled in the art and do not improve this description) are processed sequentially), and thereafter the video processing circuitry that operates on the entire image is shown.

[0010] We assume that we start with HDR pixel brightness L_HDR (note that some systems may already work from luma), which is sent through the selected HDR inverse EOTF in the luma conversion circuit 101 to obtain the corresponding HDR luma Y_HDR. For example, a perceptual quantizer EOTF may be used. The input HDR luma is luma mapped by the luma mapper 102 to obtain the corresponding SDR luma Y_SDR. In this unit, the creation side device such as the color details corresponding to the input image apply appropriate color changes in the output image, including brightness re-grading suitable for any particular scene image, for example, maintaining an appropriate average brightness for the cave image. There will be an input connection UI to obtain an appropriately determined shape for the luma mapping function (LMF). For example, when creating a lower maximum brightness output image for a cave image, the normalized version of the luma mapping function may need to boost the darkest luma of the input by a factor of 2.0, for example as determined by a human color grader or automaton, which means that the luma mapping function is concave or so-called r-shaped (as shown inside the rectangular representation unit 102).

[0011] Broadly speaking, there can be two categories. An offline system may employ a human color grader to determine the best LMF via a color grading software according to his artistic preferences. We assume, but are not limited to, that the LMF is defined as a LUT defined with the coordinates of several nodes (e.g. (x1, y1) for the first node). The human grader may set the slope of the first line segment, i.e. the position of the first node, because, for example, there is dark content in the image and he wants to keep it visible enough when displayed on a lower dynamic range display (e.g., specifically on a 100nit ML_C image for a 100nit ML_D LDR display).

[0012] The second category uses automata to obtain secondary gradings from primary gradings with different maximum brightness. These automata analyze the image and propose the best LMF (for example, they can use a neural network trained on a set of training aspects of the image to produce normalized coefficients of some parameter functions in its output layer, or any deterministic algorithm can be used). The automata of particular interest (the so-called inverse tone mapping (ITM)) do not analyze the main HDR image, but analyze the input LDR image on the creation side and make a pseudo HDR image for the LDR image. This is very useful because most videos are displayed as LDR and can even be generated as SDR today or in the near future (at least, for example, some cameras in multi-camera productions can output SDR, for example, drones capture side pictures of sports games, and the side pictures may need to be converted to the HDR format of the main program). When the capabilities of the mode-LDR encoding system are appropriately combined with the ITM system, the applicant and its partners have been able to define a system that can perform double inversion, that is, the upgrade function of the pseudo HDR image generated from the original LDR input is basically the inverse function of the LMF used when encoding the LDR communication agent. In this way, a system can be made that actually transmits the original LDR image, but also transmits information for making a good HDR image for it (and if the content creation customer so automatically desires, or in other versions also utilizes human input, for example, to adjust automatic settings). The automaton can use all kinds of rules, for example, to determine where there are light sources in the image, which need a certain amount of relative brightness compared to the average image brightness, but the exact details are irrelevant to the discussion of the present invention. Importantly, any system that adopts or collaborates with this innovative embodiment can generate some brightness (or luma) mapping functions LMF. The luma mapping function determined by the automaton can be similarly input via a connection UI to be applied to the luma mapper 102. Note that in our simplest embodiment description, there is only one (degraded) luma mapper 102. This does not need to be a limitation: since both EOTF and luma mapping usually map the normalized input domain [0, 1] to the normalized output domain [0, 1], there can be one or more intermediate normalized mappings that (basically) map 0 input to 0 output and 1 to 1. In this case, the former intermediate luma map is then used as a base map, and the (second inline) luma mapper 102 then uses as a correction map based on the first map.

[0013] The encoder has now obtained a set of LDR images luma Y_SDR, which corresponds to the HDR image luma Y_HDR (via a mapping function). For example, the darkest pixel in the scene can be defined as being displayed as substantially the same brightness on HDR and SDR displays, but the brighter HDR brightness can be compressed into the upper range of the SDR image, as shown by the convex shape of the LMF function shown inside the luma mapper 102, with a reduced slope (or reconverging toward the diagonal of the normalized axis system). Note that normalization is easily understood by those skilled in the art: it is only necessary to divide the luma code by a power (2; number of bits (number_of_bits)). If so desired, normalized brightness can also be defined by dividing any pixel brightness by the maximum value ML_C (e.g., 4000nit) of its associated target display. That is, by driving until the maximum brightness of the target display (which is the upper limit point of the brightness dynamic range of the target display) and fully using the capacity of the associated target display, the brightest possible pixel of the target display is represented as a normalized brightness of 1.0 (used as an input brightness value, which is shown on the horizontal axis of the curve diagram in unit 102). One skilled in the art will appreciate how various different normalizations can be represented as different maximum values, and how a function can be specified on the vertical axis that maps a normalized input value to a normalized luma (of any bit length and any selected EOTF) or to any normalized luminance (of any associated maximum luminance).

[0014] Thus, an example of an indoor-outdoor scene can be mentally depicted. Since in the real world, outdoor brightness is typically 100 times brighter than indoor pixels, a traditional LDR image will show that indoor objects are very bright and colorful, but everything outside the window is hard-clipped to a uniform white (i.e., invisible). Now, when transmitting HDR video using reversible proxy images, we will ensure that the bright outside areas seen through the window are brightened (possibly even desaturated), but in a controlled manner so that enough information is still available for reconstruction back to HDR. This has advantages for both outputs, because any system that wants to use only the LDR image as is will still see a good rendering of the outdoor scene as much as possible over the limited LDR dynamic range.

[0015] Therefore, in the sense that the later circuit does not always need to be particularly concerned whether this LDR image is generated intelligently or simply directly from the camera as in a traditional LDR system, a set of Y_SDR pixel luminances (and their chrominances, the details of which are unnecessary for this explanation) will form a "classic LDR image". Therefore, the video compressor 103 applies an algorithm such as, for example, MPEGHEVC or VVC compression. This is a set of data reduction techniques that, in particular, transforms blocks of, for example, 8x8 pixels into a finite set of spatial frequencies using a discrete cosine transform, which requires less information to represent them. The amount of information required can be adjusted by determining a quantization factor, which determines how many DCT frequencies are retained and the accuracy with which they are represented. The disadvantage is that the compressed LDR image (Im_C) is not as accurate as the input SDR image (IM_SDR), and in particular there will be block artifacts. Depending on the broadcaster's choice, it may be so severe that some blocks in the sky are represented only by their average luminance, so they may become uniform squares. This is not usually a problem, since the compressor will determine all its settings (including the quantization factor) in such a way that quantization errors are largely invisible, or at least unobjectionable, to the human visual system.

[0016] The formatter 104 performs any signal formatting required for the communication channel (which may be different when communicating, for example, via storage on a Blu-ray disc, than when broadcasting, for example, with DVB-T). In general, all variants will have the following property: the compressed video image Im_C is bundled in the output image signal S_im with a luma mapping function LMF which may change (or not) per image.

[0017] The deformatter 151 performs the reverse operation of formatting, so that a compressed LDR image and a function LMF are obtained to reconstruct an HDR image, or other useful dynamic range mapping processing (e.g., optimization for a specific connected display) can be completed in a later circuit. The decompressor 152 undoes, for example, VVC or VP9 compression to obtain a sequence of approximate LDR luma Ya SDR to be sent to the HDR image reconstruction pipeline. In order to basically perform the inverse process of encoding the HDR video to be transmitted as an LDR proxy video, the upgrade luma mapper 153 transforms the SDR luma into a reconstructed HDR luma YR_HDR (which uses the inverse luma mapping function ILMF, which is (essentially) the inverse of LMF). We have shown two possible receiver (150) devices in an illustrative figure, which can exist as dual functions in one physical device (where, for example, the end user can choose which of the parallel processing is applied), or some devices may have only one of the parallel processing (e.g., some set-top boxes may only perform reconstruction of the main HDR image to store it in a memory 155 (such as, for example, a hard disk)).

[0018] If we have a display panel connected to a receiver embodiment, for example a 750nit ML_D end-user display 190, the receiver may have a display adaptation circuit 180 to calculate a 750nit output image instead of, for example, a 4000nit reconstructed image (we have shown this dashed line to show its optional component that is tangential to the teachings of the present invention, but the two techniques will typically be used in combination). Without going into detail about the many variations in which display adaptation can be achieved, there is typically a function determination circuit 157 that will propose an adapted luma mapping function F_ADAP based on the shape of the inverse of the LMF (which will typically be closer to the diagonal if the difference between the maximum brightness of the starting image and the maximum brightness of the desired image is less than the difference in maximum brightness of two reference gradations (e.g. a 4000nit main HDR image and a corresponding 100nit LDR image)). This function will be loaded into the display adaptation luma mapper 156 to calculate a less boosted HDR luma L_MDR over a typically smaller dynamic range, which ends up at ML_D=750nit instead of ML_C=4000nit. By regrading we mean any brightness (or luma) mapping from a first image of a first dynamic range to a second image of a second dynamic range, where at least some pixels will be given different absolute or relative brightness (e.g., compressing the brightest luma into a smaller sub-range for a smaller output dynamic range). Such a graded image may also be referred to as a graded. If only reconstruction is required, the EOTF conversion circuit 154 will be sufficient, resulting in a reconstructed (or reconstructed) HDR image IM_R_HDR comprising pixel colors including the reconstructed HDR brightness LR_HDR.

[0019] If the video content consisted only of natural images (i.e. captured by a camera, for example), high dynamic range techniques would already be complex given the various ways of computing the rescaled pixel luminances of the input luminances. Additional complexity comes from the details of the various encoding standards, which may even be present in the mix, i.e., for example, some HDR10 video data needs to be mixed with HLG captures, such as in picture-in-picture.

[0020] In addition, often content creators also want to add graphics to the image. By the term graphics, we mean anything that is not a natural image element / area, i.e., is typically simpler in nature, and typically has a more limited set of discrete colors (e.g., no photon noise). Another way graphics can be characterized is that they do not form part of an image that is similarly illuminated, but other considerations (such as easy readability or attention grabbing) may be important. Graphics are typically computer-generated, and we usually mean graphics that are not natural, such as CG furniture that is visually indistinguishable from furniture that is actually in the scene captured by the camera. For example, there may be a bright logo of a company, or other graphic elements such as a plot, a weather map, an information banner, etc.

[0021] Graphics started in the old LDR days, simulating the way to make graphic elements on paper. That is, there was usually a limited set of pens or crayons, etc. In fact, in early systems like teletext, 3 primary colors (red, green, blue), two secondary colors (yellow Ye, cyan Cy and magenta Mg), and black and white were originally defined. Any graphics (such as an information page about scheduled flights at an airport) must be generated from a graphic primitive (such as a pixel block) with pixels in one of these 8 colors. However, "red" (but it needs to look reddish to human viewers) is not the only defined color, in particular it does not have a unique brightness predefined for it. More advanced palettes may contain, for example, 256 predefined colors. For example, graphical user interfaces (such as the X-Windows system for Unix) define many optional colors in X11, such as "honeydew", or "gainsboro gray", or "lemon chiffon". These colors must be defined as some representative code that produces approximately a certain color appearance when displayed, and in the LDR color space as percentages of standard red, green, and blue to be mixed. For example, honeydew consists of 94% red, 100% green, and 94% blue (or hex code #F0FFF), and therefore appears as a light greenish white.

[0022] The problem is that, while these colors are defined in a stable way in the limited (and universal) LDR color gamut, they are not defined or at least not clear in HDR images. There are several reasons for this. First, there is only one white in LDR, so all graphic colors compared to this white can be defined. In fact, just like when white is painted on a monitor with an appropriately colored marker to serve as a white canvas, the code that imitates the absorption of certain hues defines the creation of certain chromaticities (or color nuances, such as desaturated Cabernet Sauvignon). But in HDR, there is no unique white. This is something we know from the real world. If you look at an interior wall painted with white, although it may have slight gray hints in shadow locations, all in all, this has the visual impression of looking white to a person. But if you look at a white garage door in the sun outside the window, this also looks "white" of the color type. But another, much brighter white. In a high quality HDR handling system (handling generally involves aspects such as optimally creating, encoding and processing for proper display), instead of simply wanting to extend the LDR look with a few pure HDR aspects, one might want to also render various whites on the display (but typically at lower brightness than in the real world). In technical terms, such considerations lead in particular to the useful definition of a framework, where various HDR images with different encoded maximum brightness ML_C (e.g. 1000nit or 5000nit, etc.) can be defined.

[0023] Further complicating the issue, the fact that there are different kinds of HDR images (with different MI_Cs, and even related systems often have the same problem) means that a luma mapping will be needed to map luminance along the dynamic range of the input image to luminance along the output dynamic range of, for example, an end-consumer display. Luma mapping is typically defined and calculated as a corresponding luma mapping function, which maps luma codes that uniquely represent luminance, for example via a perceptual quantizer EOTF. So, for example, imagine that an input image luminance between 0 and 4000 needs to be mapped to an output luminance between 0 and 100 via a function F_H2S_L, i.e., L_out = F_H2S_L(L_in). Both input and output luminance can be expressed as PQ-defined luma, since the perceptual quantizer EOTF (EOTF_PQ) can encode luminances up to 10,000 nit. It is found that Y_out = OETF_PQ(L_out), and L_in = OETF_PQ(L_in). Therefore, the luma mapping F_H2S_Y can be found because: Y_out = F_H2S_Y (Y_in) = F_H2S_Y (OETF_PQ (L_in)). Or, the brightness mapping function and the luma mapping function are related by: L_out = EOTF_PQ (Y_out) = EOTF_PQ [F_H2S_Y (OETF_PQ (L_in))], so:

[0024] F_H2S_L=EOTF_PQ(o)F_H2S_Y(o)OETF_PQ, where (o) means the function composition (usually represented by a small circle symbol). It is even possible to define a mapping of luma specified by another EOTF, such as from input PQ luma to output Rec.709SDR luma, etc.

[0025] With the help of Figure 2 A possible graphics insertion pipeline is illustrated.

[0026] Figure 2An HDR image communication and processing pipeline illustrating a similar video communication system is shown. This is a sports program (horse racing) for a classic broadcast service. One or more cameras 201 capture sporting events, and their images are mixed in a production room 202. In the production room 202, the feeds from various cameras can be mixed and a total HDR video graded (or shaded) according to preference can be produced. The way in which the pixel luma of a continuous video image is standardized is beyond the scope of the present invention. The broadcaster will produce an original mixed HDR image 205, which includes local video content 206 (feeds from (one or more) cameras, after appropriate coloring) and the broadcaster's own graphics (an example of a primary graphic). The graphic can be many things in different applications, but in this example, it is a list of the names of the two horses at the head of the race. We call this original graphic that is pre-mixed and transmitted within the video image a primary graphic element 207. It will be distributed to a local distributor's device 210, for example, via a satellite dish 203. This can be, for example, a Dutch broadcaster or a redistributor, which will be distributed to consumers, for example, via terrestrial broadcast or cable 219. The local distributor may mix its own secondary graphics, such as the logo of the broadcaster ("NL6") under the sun. We will assume that the pixel color of the primary graphics is, for example, all white, and the pixel color of the secondary graphics depends on the color of white, such as a yellow sun with 90% white brightness, with a red text color of 60% brightness inside. The broadcaster may also mix other kinds of secondary graphics, such as a preamble 217 of an upcoming sports program. The further mixed video (215) (also referred to as the secondary mixed image / video) is further distributed to the end customer via a television communication cable (CATV) 219. The end customer may have a set-top box (or computer, etc.) 220 that can mix tertiary graphics (etc.). In this example, subtitle information is transmitted together in the video signal (usually as plain text information, which still needs to be rendered into video pixels to constitute the generated video element), which is rendered by the set-top box as a tertiary graphic 226, and before they are transmitted to a consumer display 230 via, for example, an HDMI cable 229, they are mixed into the video image again, thereby generating a tertiary video 225. In fact, instead of (or in addition to) a set-top box (which may not be present in some processing pipelines), the terminal display may also insert its own coordination graphics.

[0027] In this document (when referring to some graphics at a higher level concept), we will call all additional graphics (in this example, secondary and tertiary graphics) "secondary graphics" to distinguish them from primary graphics, which are the previous graphics case, and are usually the first graphics put into the video (usually put into the video by the original creator of the video). Typically, the (one or more) images themselves (i.e., the pixel color matrix, such as YCbCr) Im_HDR are communicated together with metadata (MET) in the HDR image or video signal S_im, which metadata can encode several things in various HDR video encoding variants, such as the encoded maximum brightness of the HDR image (e.g. ML_C_HDR=1000nit), and the luma mapping function that is usually used to calculate the secondary image colors of the re-graded version of the image, etc.

[0028] It is desirable that the various graphics are not discordant across the entire luminance dynamic range, or even worse, vary in luminance over time (e.g., compared to each other), but rather have, for example, the same average luminance, as shown over, for example, the 950nit ML_D luminance dynamic range of the end consumer display 230. In this way, the technical functionality of, for example, a television display (or other device receiving a primary video-graphic mix) with respect to the rendering of its secondary graphics can be improved. If you have a simple system with a display that has already calculated a video image suitable for display with its optimal pixel luma, and simply want to add the primary graphics to this already established range, then doing so may not be that difficult or critical, but things may be more complicated, especially in the case of different kinds of graphics being added by different devices in different locations in the HDR video processing pipeline.

[0029] US20180018932 teaches multiple blending techniques for primary graphics elements (secondary graphics are not blended if primary graphics are already blended). There is also no proper location of the graphics sub-range of the primary graphics established in the main HDR video (graphics maximum is simply mapped to the video maximum, i.e. the entire range).

[0030] The most complex variant to understand is in US'932- Figure 2 The video priority mode described in A has been implemented in this application with the help of Figure 8Summarized. If we want stable graphics in the output of an end-user display-adapted image to be displayed (e.g., an image composed for an end-user display maximum brightness ML_endUSR_disp of 750 to be displayed), this can be achieved by giving the same value to all pixel brightnesses of, say, monochrome white text (TXT) for multiple consecutive images (e.g., several scenes, or if it is a user interface graphic, as long as the graphic is to be presented). It is assumed that a good final value for the text brightness is the position where the cloud object of the video is projected within the display dynamic range. Graphics typically do not have the same dynamic range (in particular, their graphics maximum brightness ML_gra) as the input video to be mixed with it (the video maximum brightness ML_vid), nor do they typically have the same dynamic range as the maximum brightness of the end-user display, because various users may have purchased different displays. In fact, associating graphics with a certain dynamic range is already a slightly advanced HDR technique, because in LDR time, graphics typically have a certain relative 3x8-bit code (e.g., 255 / 255 / 255). The problem is that with dynamic metadata (which conveys differently shaped brightness mapping functions for successive shots of an image to be applied by an end-user display), there is no guarantee of what the premixed text will do. If the original text (TXT_OR) is to be mixed into the video at its original brightness (e.g. 390nit), the display may shift this brightness up with some mapping functions and down with others, i.e. it may start to flicker. However, if one knows exactly the dynamic mapping function that the display will apply to the video (e.g. a first dynamic mapping function F_dyn1), then one can pre-compensate for this, whether or not it has graphics mixed. The graphic can then be pre-mapped (with a first pre-compensation function F_precomp1) to exactly the brightness it will need to have in the HDR image to be sent to the display from, e.g. a set-top box, so that it is mapped by F_dyn1 to the desired final brightness in the image to be displayed (as bright as clouds). Perhaps in the next scene, a piece of paper is mapped up to the same final brightness of TXT by a second dynamic mapping function F_dyn2, so then we use the second pre-compensation function F_precomp2. Typically, which function the display applies depends on its maximum brightness ML_endUSR_disp and a dynamic, time-varying reference brightness mapping function that specifies how 4000nit video brightness should be mapped to, for example, a 100nit reference image. A predetermined algorithm then morphs this reference brightness mapping function into a final display-adapted brightness mapping function that will be used by tv for any consecutive images until a new, dynamically changing function is entered as metadata.But if the set-top box is codec-compatible, i.e. it knows the reference luminance mapping function and the algorithm how to transform it into a final display-adapted luminance mapping function for any value of ML_endUSR_disp, and polls ML_endUSR_disp from the connected display, it can do any such pre-compensation.

[0031] The prior art shows several other examples of blending primary graphics, which are more straightforward.

[0032] US'932- Figure 2 The graphics priority mode shown in B simply maps the maximum value of the graphics to the maximum value of the display. Then, it can down-map the video to the range already in the set-top box. If everything is prepared in the STB, and the best display-adapted image is simply transmitted to the display via HDMI (which does not require further processing), then the graphics mixing is relatively simple. The STB has everything in hand, and no variable mapping occurs later. But these two options are not always available. For example, the STB may not license the technology of a certain codec because it is quite expensive and the STB sells it with minimal profit. Then, the display (such as a consumer TV) will need to perform dynamic HDR brightness mapping anyway. Pre-compensation is not always possible, for example, if the TV refuses to transmit its ML_endUSR_disp value to the STB. Even if these methods may be useful for home appliances or computers and monitors such as STBs and TVs, there may be more locations where graphics need to be inserted in the video communication pipeline from the "camera" to the final consumer display (for example, in a shopping mall).

[0033] If TV can perform graphics fusion (such as US'932- Figure 2 D), then things can also be relatively simple. The TV can apply its usual display-adapted brightness mapping function to the video and thereafter place the graphics wherever desired, e.g., output brightness ending with clouds. This may be OK for pass-through DVB subtitles with standardized encoding, but the STB needs a mechanism to transmit graphics for its own UI graphical elements over the HDMI interface.

[0034] US2020193935 switches the mapping of the display to static when the graphics are blended, so the graphics will always end up at the same (shifted) brightness position, which also has its advantages and disadvantages.

[0035] It is clear that in the state-of-the-art field of high dynamic range video / television, there is still a need for good graphics processing technology. Summary of the invention

[0036] The problems existing in simple methods of graphics blending / interpolation are addressed by a method of determining a second luma of pixels of a secondary graphics image element (216) to be blended with at least one high dynamic range input image (206) in a circuit for processing a digital image, the method comprising:

[0037] Receiving a high dynamic range image signal (S_im) comprising the at least one high dynamic range input image, wherein the method comprises:

[0038] analyzing the high dynamic range image signal to determine a primary graphic luma range (R_gra) of a primary graphic element of the at least one high dynamic range input image, the primary graphic luma range being a subrange of a luminance range of the high dynamic range input image, wherein determining the primary graphic luma range comprises determining a lower limit luma (Y_low) and an upper limit luma (Y_high) specifying endpoints of the primary graphic luma range (R_gra);

[0039] brightness mapping the secondary graphic element, at least a brightest subset of the pixels luma of the secondary graphic element being included within the graphic luma range; and

[0040] The secondary graphics image element is blended with the high dynamic range input image.

[0041] The luminance mapping to the graphics sub-range in the input (video + primary graphics mix) HDR image and (casuquo) any output variant image from which its luma may be derived by mapping it using some luma mapping function can in principle occur at the same time as the pixels of the graphics are created (i.e. defined). However, in general, even if the graphics are not read from a storage device in which they are predefined to be contained and are generated by any method or device embodiment, the graphics will be generated in some different format (such as but not limited to LDR format or 1000nit maximum brightness format), but the luma (or any code, such as a language code such as Chartreuse) of the pixels making up the secondary graphics will then be mapped to the appropriate position within the established graphics range (or possibly slightly outside, typically at the lower / darker end), where they become the respective corresponding secondary luma of the various graphics pixels.

[0042] The blend can often be as simple as replacing the original video pixel by a graphics pixel (but with a coordinated color, and in particular a coordinated luma). The method also lends itself to more advanced blending, in particular if more of the graphics is retained, and less of the graphics is blended. For example, a linear weighting called blending can be used, where a certain percentage of the determined graphics luma (Y_gra) is blended with the complementary percentage of the video:

[0043] Y_out=alpha*Y_gra+(1-alpha)*Y_HDR[Equation 1]

[0044] The alpha is usually above 0.5 (or 50%), so that essentially the pixel contains mostly graphics, and the graphics are well visible. In fact, ideally / preferably, such blending does not happen on luma (especially for highly non-linear luma definitions like PQ), but on the derived luminance itself:

[0045] L_out = alpha*L_gra + (1-alpha)*L_HDR [Equation 2]

[0046] In embodiments where a proxy SDR image is delivered for an HDR image, such an HDR video luminance L_HDR may be obtained by reconverting the SDR luma to an HDR reconstruction luminance using, for example, the maximum luminance ML_C delivered with the metadata of the HDR representation being delivered, and downgrading this luminance to a corresponding luminance within the display dynamic range using display adaptation if necessary. Figure 3 It is seen that the elegance of at least this method of constructing a luma mapping function (or corresponding brightness mapping function) is that graphics can be mapped / mixed in both the input range as well as the output range, because the curve maps the graphics range in a stable manner. More generally, the present method can take advantage of the fact that the content creator - for his primary graphics - will decide what the best position of the graphics is, or at least check out reasonable positions for the graphics. Therefore, if this is properly coordinated with the primary graphics, it will also include appropriate brightness for the secondary graphics.

[0047] If no specific information about such primary graphics is available, for example because the video creator does not want to spend the effort to encode and transmit it, the method or device can analyze the situation of the incoming HDR video signal S_im and determine, through various embodiments, a suitable range R_gra of luma (or brightness) for locating at least most pixel colors in the secondary graphic elements.

[0048] How exactly this is done depends on the nature of the graphic elements.

[0049] If we have a secondary graphic (like subtitles) that has only one or a few (like original, before brightness mapping for coordination) colors, then the secondary graphic can be rendered with just any luma within the graphics range. More complex images can limit the darkness of black, such as not darker than 10% of the brightness of white, or place some darker colors outside / below the range of the primary graphics (but the effect of stable graphics presentation is largely maintained given the luma of brighter colors). For example, a slightly changing black surround rectangle on top of the underlying HDR video may not be too annoying if the subtitle colors are stably coordinated with the video object luma of one or more consecutive scenes. Sometimes you may want to avoid relatively large colors that are significantly darker than the average from oscillating too much in brightness, but if that is a problem, this can be handled separately, such as by brightening those darker colors to approach the lower point of the appropriate graphics range (i.e., slightly below or above their minimum brightness). It is desirable to keep the brightest colors of the secondary graphics within the graphics range R_gra, for example, this brighter sub-range is defined (e.g. by a TV receiver construction engineer) as all colors brighter than 50% of the brightness of the secondary graphics element (this percentage may depend on the kind of graphics one wants to add, e.g. menu items may require less precision than e.g. some logos). Typically, the brighter sub-range will contain enough colors to (geometrically) represent most of the secondary graphics elements so that, for example, their shape can be recognized.

[0050] The full luma distribution of the primary graphics need not be completely known, as long as there is a reasonable idea of ​​the typical luma of at least the brightest color in the primary graphics. In order to analyze the brightness or luma of the HDR video signal and its images and its object pixels, several methods can be used alone or in combination, in the latter case, some algorithm or circuit will determine the final best determination of the graphics range based on the input of one or more circuits applying the analysis techniques (in the case of only one circuit / method in any device or method, no such final merging step or circuit is required).

[0051] It is advantageous if the method for analyzing the high dynamic range image signal includes: detecting one or more primary graphic elements in the high dynamic range input image, establishing the luma of the pixels of the one or more primary graphic elements, and summarizing the luma into the lower limit luma (Y_low) and the upper limit luma (Y_high), wherein the lower limit luma is lower than all or most of the luma of the pixels of the one or more primary graphic elements, and the upper limit luma is higher than all or most of the luma of the pixels of the one or more primary graphic elements.

[0052] If no further information is available or can be relied upon, analyzing the image itself can be a robust option. For example, the algorithm (or circuit) can detect that there are two primary graphic elements in the "original" version of the video content. Assume that, for three speakers, some subtitles are present in three colors (white, light yellow and light blue), and some logos have multiple colors, for example, several primary colors and 5% brightness black compared to the white of the subtitles or logos themselves. In general, it can be expected that the white of the logo and the subtitles (or at least it can be expected that the white brightness is extrapolated from, for example, the yellow pixels of the logo) will be the same or approximately the same. In particular, if they are not too different (for example, the subtitles are 5 times brighter than the white of the logo, or vice versa), the present method can still define the graphic range R_gr relatively easily by then treating the darker whites as if they were "some gray" called brighter white. That is, the upper part of the primary graphic range R_gra can then be the white of the brightest pre-mixed graphic, and all graphics can still be recognized as one range, rather than having twice the application of the recognized differential range (in this case, for example, the secondary graphics can be mixed in either range or preferred range). If the brightness (white brightness) of two graphic elements differ greatly, the brighter primary graphic element can be retained in the graphic analysis and the darker primary graphic element can be discarded so that R_gra is based on the brightest primary element. Figure 3 and 4 This embodiment may be useful if the illustrated security pattern range is instead, for example, a simple power function (which has no special region and only acts as a continuous compression function).

[0053] It is advantageous if the method includes reading from the metadata in the high dynamic range image signal the values ​​of the lower limit luma (Y_low) and ( / or) upper limit luma (Y_high) written into the metadata of the high dynamic range image signal. The participant of at least one high dynamic range input image can use this mechanism to communicate what he (or in the case of an automaton) thinks is the appropriate graphic range for this image or this shot of the video image (for example, coordinated with the bright pixels of the nearby explosion, which should not appear to be reduced for the human visual system in view of its recognition of, for example, the white of the nearby subtitles). He will put his primary graphics at such brightness. Note that the human visual system can make a variety of estimates about the content in the image (for example, local lighting), and this can be quite complicated, but the present technical mechanism provides a simple solution for those who should know (for example, the human creator of the video or the color grader and / or post-producer). For example, the range transmitted may be slightly darker than the brightness actually used in the primary graphics, thereby ensuring a certain coordination and higher brightness of the primary graphics. Then, any merging step or circuit can simply discard or turn off other functions, such as the analysis of the HDR image itself. Or, it can use any available mechanism.

[0054] It is advantageous if the method includes analyzing the high dynamic range image signal, comprising: reading two or more luma mapping functions associated with temporally consecutive images from metadata in the high dynamic range image signal, and establishing the graphic luma range as a range that satisfies a first condition regarding the two or more mapping functions, wherein the two or more luma mapping functions map input luma within the graphic range to corresponding output luma, and the first condition is that, for each input luma, the corresponding two or more output lumas of the input luma are substantially the same, and the corresponding two or more output lumas can be obtained by applying corresponding mapping functions from the two or more luma mapping functions to the input luma. Figure 7 An example is shown where two functions have a sub-range of their input range where the functions are (substantially) identical, i.e. they create (approximately) the same output value for any input value in that range. This can be detected because the content creator has had the intention of making a graphically robust dynamic brightness mapping function.

[0055] Thus, not only is the range similarly mapped to a stable output range, but the various lumas therein are mapped by both functions to essentially the same output luma (except for perhaps some minor deviations, usually not too visible, e.g. within 2% of a typical human visually noticeable difference, or 10% if some more flicker is still acceptable). You might want to supplement such an analysis with further analysis.

[0056] Specifically, this can also involve the verification of the second condition that the primary graphic luma determined based on the function falls within the secondary range that will be suitable for the luma of the graphic. For at least a series of shooting (such as indoor shooting, outdoor shooting thereafter), just finding the area of ​​various functions does not necessarily mean that you will want to put the graphic there. For example, if only the darkest 10% of the total range (input or output) is identified, this will result in a dark graphic, which is usually not preferred. For example, 1 / 2 to 1 / 10 of the high brightness dynamic range (for example, maximum brightness 4000nit or higher) may be a suitable place to place subtitles or other graphics. However, 1 / 2 or more of 4000nit may be suboptimal. Since this will produce a fairly bright maximum brightness of the graphic, and for graphics like text, where the brightest color is usually normal (diffuse) white, compared with more general graphics in which there may be many different whites including super white, the 2000nit level can be considered high. However, for the 5000nit main grading, the creator of the mapping function may have thought that it is best to place the stable graphic range at, for example, 0.8*0.5-1.2*0.5. If the curve is used to degrade to an output maximum of up to, say, 1000 nit, then 500 nit graphics (or even subtitles) may be considered reasonable, i.e., satisfy the suitability criterion. If the second criterion is not met, the graphics mixing device may, for example, try to stay close to the identified range, but decide to mix the graphics with luma below the lowest value of the identified graphics range that depends on the function shape (i.e., where two or more functions have a sub-range that projects the same input sub-range to the same output sub-range). Various secondary criteria may be used, such as a criterion of how high the percentage of the output range maximum of the upper limit point of the stable graphics range identified by the function shape is (i.e., also depending on the absolute maximum, i.e., the higher the maximum value of the output range, the lower the allowed percentage may be, e.g., a 500 nit upper limit for a 1000 nit range, and a 4000 nit output range maximum or higher upper limit luma of 1200 nit) and / or how high the absolute brightness value of the output range upper limit will be displayed. Thus, typically with respect to the relevance of the identification of a range determined by the function, one would look at values ​​along the range of the input luma (i.e. the input luma of the received image), but with respect to suitability the envisioned output range and the maximum brightness of the output range (i.e. what a transcode that will also be mixed in the graphics for example will produce) may also play a role in determining what is suitable from the input luma values ​​alone as an extension.For example a raw graphic brighter than 2000nit in the input image presented as pre-mixed graphics, to become an output below 1000nit maximum (which would for example have a levelling soft clipping mapping function starting from 4000nit maximum input), for example ends up at 900nit in the output (which is quite bright subtitles compared to the HDR effect of the brightest video objects, which ideally should be more impressive than the graphics white level, but at least it can be a level that is not discounted in itself, as 900nit graphics may still be acceptable to some users in some scenes). A good secondary criterion would be that the maximum value of the secondary range must be below a predetermined percentage of the maximum value of the range of the mixed image and graphics.

[0057] Any device can be programmed with limits on what graphics can be expected. For example, they can be expected to be at or slightly above the expected sub-range of normal ("LDR") image objects in an HDR image. For a 1000nit ML_C defined HDR image, the whites of the graphics can be expected to fall somewhere in the range between typically 80nit and 350nit, and the blacks below that range, also depending on whether there are several primary graphics elements and whether they have the same brightness (or luma) characteristics.

[0058] Therefore, an embodiment of the verification can be performed as follows. If the analysis of the luma mapping curve finds an upper limit luma of a graphic such as 150nit, this will indeed fall within the range of expected luma values ​​for bright graphic pixels because it falls within the larger range of [80.350]. Similar considerations can be made for lower levels, but as we have described, the lower levels are less critical and can, for example, be simply set at 10% of the determined upper limit brightness (or the corresponding x% of the upper limit luma when looking for it on the applicable EOTF of the pixel color encoding). If the method, for example, estimates the brightness of the upper limit luma of, for example, 950nit, this may be an accident of analysis because such an overly bright graphic is not ideally desired. But it may be true that a certain content provider has created such bright subtitles in some way. In this case, instead of simply rejecting the value and concluding that the graphic range cannot be determined with sufficient certainty, further analysis can be performed, for example, checking which graphic luma the creator has actually transmitted together in the video signal, or looking for such luma values ​​in the video signal, and attempting further image analysis, for example, how and how large the connection of those pixel sets is, etc.

[0059] Note that the HDR images do not always need to actually have the primary graphics rendered at their respective appropriate luma positions (e.g., a first shot of an image with a first luma mapping curve coordinated for mapping a substantially bright HDR scene may actually contain a graphic element, but a second subsequent shot with a different luma mapping curve for mapping normal and bright image objects may not have the actual graphics inserted, but if it does, place them at approximately the same luma value).

[0060] It is advantageous if the method comprises a step of mapping said pixel luma signals of said at least one high dynamic range input image to corresponding output luma signals of at least one corresponding output image, wherein said mixing occurs in said at least one corresponding output image according to at least one luma mapping function obtained from metadata of said high dynamic range image signal.

[0061] Especially when there is Figure 3 When the function in , this relationship can lead to the ability to mix in both the input and output luma domains. Whether to mix in the input domain or the output domain may not always be the same (and in some embodiments that meet some HDR codecs, it may be useful to know where to mix in the input luma domain). In any case, the video pixels are usually mapped to the output domain by the luma mapping function received by the metadata. The method or device can use the mapping function to determine the appropriate luma position in the output domain (e.g., 350nit maximum value ML_C) for mixing the graphics determined by the output. This creates flexibility about which device can mix what and when. In particular, with respect to stable sub-region functions, it is easier to understand what will happen to the graphics mixing that has been performed, even if these functions are distorted in the display adaptation scene (i.e., by making the shape of the original function more similar to the shape of the secondary function, the stable sub-range will be stretched or compressed, but if the adaptation is performed correctly, the same coordinated mapping properties are maintained for the luma within the range, where both the main function and the reference function are about the sub-range).

[0062] It is advantageous if the method comprises designing said secondary graphic image element having a set of colors spanning a scale of colors of different brightness, some of said colors appearing brighter than average to the human eye and some of said colors appearing darker than average, wherein at least the brighter than average colors are mixed into said at least one high dynamic range input image with a luma within said graphic luma range.

[0063] A color scale is a collection of colors that increase (or decrease) in brightness, for example, from some colors that look black to some light colors, and usually some intermediate brightness colors. The scale does not necessarily need to contain all the steps of a primary color (such as blue), but a limited color palette can, for example, have several light blue steps and several dark greens, but no dark blue (then green defines the darker steps of the scale). If the eye examines the approximate average brightness of the graphic elements, some colors usually appear dark and some colors appear light. A brightness metric of about 50% or a brightness of about 25% can be an atypical value that is considered a midpoint, or can be used as a midpoint. Therefore, pixels with brightness below, for example, 25% can be considered darker colors and no longer need to meet the graphic range criteria. If a tighter system requires more colors to be within the graphic range, the secondary graphics can be designed to have less dark colors.

[0064] Some embodiments of the method's secondary graphics insertion may blend through a more predictable blending method using pixel replacement (i.e., drawing graphics pixels with appropriately established luma where there were video pixels of the original video), or blending with a percentage less than 50% of the brightness of at least one high dynamic range input image (the percentage is preferably 30% or less), so that the video is slightly shiny and still visible, but the brightness of the graphics is dominant and is largely preserved in its luma determination.

[0065] Advantageously, if the method designs a color set of (one or more) secondary graphic elements, it includes selecting a finite set of relatively bright, lower-than-average dark colors, the relatively bright, lower-than-average dark colors having a luma higher than a secondary lower limit luma, the secondary lower limit luma being a percentage of the lower limit luma, the secondary lower limit luma preferably being higher than 70% of the lower limit luma. If the graphic range R_gra is determined, a suitable secondary graphic color can be determined. For example, the method may decide not to go below the determined safety lower limit, such as only 30 darker, to reduce the impact of potentially significantly changing the luma mapping. The percentage may depend on aspects such as how many dark pixels there are in the secondary graphic or at which locations. For example, if there are only 5 dark pixels in a 100x100 pixel graphic, the average perceived brightness of the secondary graphic will not be affected too much by these few dark pixels, no matter how they are ultimately displayed in the output image.

[0066] The method may also be implemented as an apparatus, such as an apparatus (500) for determining a second luma of a pixel of a secondary graphic image element (216) and blending the secondary graphic image element with at least one high dynamic range input image (206), the at least one high dynamic range input image comprising the primary graphic element (207), the apparatus comprising:

[0067] An input terminal (501) for receiving a high dynamic range image signal (S_im) comprising the at least one high dynamic range input image;

[0068] an image signal analysis circuit (510) arranged to analyze the high dynamic range image signal to determine a graphic luma range (R_gra) of the at least one high dynamic range input image, the graphic luma range being a range specifying luma of pixels representing a primary graphic, the range being a sub-range of a luminance range of the high dynamic range input image, wherein determining the graphic luma range comprises determining a lower limit luma (Y_low) and an upper limit luma (Y_high) specifying endpoints of the primary graphic luma range (R_gra);

[0069] Graphics generation circuitry (520) arranged to generate or read a secondary graphic element and brightness map the secondary graphic element, at least the brightest subset of the pixels luma of the secondary graphic element being included in the graphic luma range;

[0070] an image mixer (530) arranged to mix the secondary graphics image elements with the high dynamic range input image to produce pixels having a mixed brightness (Lmax_fi); and

[0071] An output terminal (599) for outputting at least one mixed image (Im_out) comprising said pixels having a mixed brightness (Lmax_fi).

[0072] or device, wherein the image signal analysis circuit (510) comprises an image graphic analysis circuit (511), and the image graphic analysis circuit is arranged as follows:

[0073] detecting one or more primary graphic elements in the high dynamic range input image,

[0074] establishing luma of pixels of the one or more primary graphics elements, and

[0075] The luma is summarized into the lower limit luma (Y_low) and the upper limit luma (Y_high), wherein the lower limit luma is lower than all or most of the luma of the pixels of the one or more primary graphic elements, and the upper limit luma is higher than all or most of the luma of the pixels of the one or more primary graphic elements.

[0076] or device, wherein the image signal analysis circuit (510) includes a metadata extraction circuit (513), which is arranged to read from metadata in the high dynamic range image signal, for example the values ​​of the lower limit luma (Y_low) and the upper limit luma (Y_high) written into the high dynamic range image signal by a creator of the at least one high dynamic range input image.

[0077] or device, wherein the image signal analysis circuit (510) includes a luma mapping function analysis unit (512), which is arranged to: read two or more luma mapping functions (LMF(t), LMF_p) of temporally consecutive images from metadata in the high dynamic range image signal (S_im), and establish the graphic luma range as a range (R_id) that satisfies a first condition regarding the two or more luma mapping functions, wherein the two or more luma mapping functions map input luma within the graphic range to corresponding output luma, and the first condition is that, for each input luma, the corresponding two or more output lumas of the input luma are substantially the same, and the corresponding two or more output lumas can be obtained by applying corresponding mapping functions from the two or more luma mapping functions to the input luma.

[0078] or device, wherein the image mixer (530) includes a luma mapper (533), the luma mapper being arranged to map the luma of pixels of the at least one high dynamic range input image to corresponding output luma of at least one corresponding output image, the at least one corresponding output image having a dynamic range, such as a maximum brightness, different from the dynamic range of the at least one high dynamic range input image, and the luma mapper being arranged to perform the mixing in the at least one corresponding output image according to at least one luma mapping function obtained from metadata of the high dynamic range image signal.

[0079] or apparatus, wherein the graphics generation circuit (520) is arranged to design the secondary graphic image element having a set of colors spanning a scale of colors of different light brightnesses, some of the colors appearing brighter than average to the human eye and some of the colors appearing darker than average, wherein at least the brighter than average colors are mixed into the at least one high dynamic range input image having a luma within the graphics luma range.

[0080] or device, wherein the image mixer (530) is arranged to mix by pixel replacement of video pixels of the at least one high dynamic range input image by secondary graphic element pixels, or blending with a percentage less than 50% of the brightness of the at least one high dynamic range input image, the percentage preferably being 30% or less.

[0081] In particular, those skilled in the art understand that these technical elements can be embodied in various processing elements (such as ASICs (application specific integrated circuits, i.e., typically IC designers will make (a part of) the IC perform the method), FPGAs, programmed processors, etc.), and can reside in various consumer or non-consumer devices, whether including a display (e.g., a mobile phone that encodes consumer video) or a non-display device that can be externally connected to a display, and that images and metadata can be transmitted through various image communication technologies (such as wireless broadcast, cable-based communication), and that the devices can be used in various image communication and / or usage ecosystems, such as, for example, television broadcast, on-demand on the Internet, video monitoring systems, video-based communication systems, etc. The innovative coded HDR signal can correspond to various methods, for example, transmitting at least one lower and upper luma value of at least one primary (premixed) graphics range. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] These and other aspects of the method and apparatus according to the invention will become apparent from the embodiments and examples described below, and are explained with reference to the accompanying drawings, which are intended only as non-limiting specific illustrations of the more general concepts, and in which dashed lines are used to indicate that components are optional, and non-dashed line components are not necessarily required. Dashed lines can also be used to indicate elements that are interpreted as necessary but hidden inside an object, or for intangible things, such as the selection of an object / area.

[0083] In the attached picture:

[0084] Figure 1 schematically illustrates one of the possible HDR processing chains (including encoding, decoding and potentially display) for which the present innovation may provide benefits;

[0085] Figure 2A typical situation in which a primary graphic may appear but a secondary graphic needs to be inserted at one or more locations is schematically illustrated;

[0086] Figure 3 A possibility of defining an image content adaptive luma mapping function providing a more stable and deterministic graphics insertion is schematically shown;

[0087] Figure 4 Schematically shows another example of the possibility of providing a more stable, deterministic image content adaptive luma mapping function for the definition of different brightness content scenes;

[0088] Figure 5 There are shown schematically typical components of an apparatus illustrating some of the principles of the present invention;

[0089] Figure 6 schematically illustrates one possible embodiment of defining at least an upper limit luma and typically also a lower limit luma of the identified primary graphics range based on an analysis of the primary graphics in at least one received high dynamic range image;

[0090] Figure 7 Schematically shows the Figure 3 or a possible embodiment of at least two continuous luma mapping functions formulated according to the principles set forth in 4 defining at least an upper luma limit and typically also defining a lower luma limit; and

[0091] Figure 8 Pre-compensated brightness mapping based mixing of primary graphics with HDR video according to the prior art is explained. DETAILED DESCRIPTION

[0092] Figure 3 It is shown in more detail how an HDR video image with graphics can best be constructed. It shows two typical HDR scene images (which may for example be temporally adjacent shots of an image) for which one might want to specifically grade the pixel brightness to obtain an impressive HDR effect. In particular, HDR images allow pixel colors to be made both bright and saturated colors, whereas in LDR bright colors would need to be desaturated and less beautiful. Furthermore, a color grader will typically look at the psycho-visual impact of various image objects, so that for example a dragon's fire looks bright, but not too bright.

[0093] Without wishing to be limited, we have two scenes that the creator can typically process as follows. Although the two scenes (with the only similarity that they contain important areas of high brightness) have completely different content and will be brightness-specific, i.e., graded differently, they both contain several normal reflective objects (i.e., objects that reflect a certain percentage of local illumination light). These will reflect the average amount of light present in the scene and therefore get a relatively low brightness, comparable to the brightness they would get in LDR. The idea is that - after human visual adaptation or when assuming such adaptation to make an image to be displayed - those will produce darker colors below the high brightness range of luminous colors, such as light bulbs, clouds that strongly reflect sunlight, etc. For the fire-breathing dragon image 301, this will be the body of the dragon. For the sunset image 302 at sea, this will be the ship. For these scenes, those normal objects will fall into the darker sub-range R_Lrefl of the lowest brightness available on the input dynamic range (in this example, the total range is from 0 to MaxL_HDR=3000nit), which reaches, for example, 200nit. If the dragon has a red body, it can for example have 130nit pixels on average, but if it has a black body, it can have for example 20nit pixels. Graphic pixels, for example for subtitles, can often be conveniently placed slightly above this darker sub-range. This has the advantage that they are very bright and noticeable (like LDR white), but on the other hand, they do not overwhelm the HDR effect of the video itself. It can be argued that - especially if the two images come from separate HDR sources that are mixed - then the upper brightness limit (or its luma, which encodes this brightness, for example in the perceptual quantizer luma definition) does not need to be the same for the two ranges of normal scene colors. But it can be constrained to be around it by coordination before the mixing of the video images (for example, the dragon range is maximally 1.5 times brighter than the ship and sea ranges at the upper end of the range), and this is taken into account by setting the graphics range, for example, starting at the brighter upper end of the two (for example, starting at about 300nit instead of 200nit, or starting at 220nit instead of 150nit). HDR effects are e.g. the orange-yellow fire of a dragon, which may be rendered at a pixel brightness of about 1500 nit (this is a relatively large part of the image, brighter brightness seems convenient). For the sunlit clouds of a sunset, those may reach e.g. 2000 nit, and the sun disk itself about 3000 nit (e.g. 95% of 3000 nit to make it yellow). All those HDR effects are clearly well above the graphics range, which may be specified by the video creator over the master HDR range of m1_c=3000 nit to be between e.g. 400 nit and 500 nit (or 50-500 nit when considering the smaller visual impact of darker graphics colors).For graphics that use only a limited subset of brighter colors, it is possible to focus on such a limited range without covering too much of the normal and effects video colors that may fluctuate in their grading and, in particular, in the regrading through the brightness mapping. When darker colors are needed, it is possible to balance how much below the 400 level one might want to reach, for example, based on expected or communicated variations in the lower portion of the mapping function (for example, in an offline scenario, one can look at all the curves that will appear during the movie before processing and determine some characteristic values ​​of the fluctuations in the upper portion of the lower curve - shown as the second linear segment in the example - and in the running display of the transmitted video, such characteristics can be transmitted and taken into account when deciding where to map the deepest black of the graphic). For example, additional luma or brightness values ​​(below L_low or Y_low) can be transmitted where the curve begins to fluctuate by more than, for example, 20% or 40% compared to the average curve starting from the lower graphics range point (which is a fixed point) downward. More critical applications (or types of graphics, such as, for example, a black text box around a monochrome subtitle) could then choose the luminance or luma of this 20% or less fluctuation point for their darkest colors, and less critical applications could use the 40% point. Often, the upper point may be more critical than the lower point (e.g., if bright pixels will be strongly compressed when regrading occurs). The lower luminance L_low of 400nit is represented as some lower luma code Y_low (e.g., 0.65 in PQ multiplied by a factor that depends on the bit depth, such as 1023). In this paper, we will make the unrestricted assumption that all pixel luminances are always encoded in the popular perceptual quantizer encoding, but other EOTFs or OETFs such as Hybrid Log-Gamma (HLG) can be used, and our techniques can even work with a mixture of EOTF / luma code definitions (e.g., finding a first primary graphics element in a PQ luma-defined pixel color encoding and a second primary graphics element as a secondary reference in some portion of an HLG-encoded image). An upper primary graphics range luminance L_high and an upper primary graphics range luma Y_high will usually be determined, for example, in cases where not too many key HDR bright objects are expected to appear, so there is no specific, e.g. strong enhancement or compression of the local curve shape. It can be defined as a percentage multiplier of the lower limit point (e.g. 150%), because if the graphics stay within this range, the continuity of the curve from a fixed point will not create too many visual mapping inconsistencies. The amount of graphics will usually depend on the maximum brightness of the input video, because we expect the need to be able to compress the higher range of various HDR objects in a generic way.For example, if the input image will only be up to 1000nit, you may often not want to choose an upper limit point of 1.5×400nit, as it will either leave little room for bright HDR objects (which may require grading them away from normal objects, which may not always be desired), or if it is juxtaposed with the established primary graphics range, there will be considerable overlap between the HDR video object brightness and at least the higher brightness possible in the secondary graphics range.

[0094] When only having the HDR image itself (i.e. only as a final image, to be displayed isoluminantly, i.e. where each encoded image pixel brightness is displayed exactly as encoded), the graphics problem is not so complex, but typically (e.g. using Figure 1 The HDR input image (explained) may need to be luma mapped (or equivalently brightness mapped). We show a first brightness mapping function 310 for the dragon image (the corresponding luma mapping function may be obtained by converting both the input and output ranges to ranges of PQ luma or converting the input to PQ luma axes and the output to Rec.709 classic LDR luma, etc.) to obtain a corresponding LDR graded image as output. We see that the function conceptually consists of three parts: a suitably shaped mapping for darker normal objects that fall within the input range R_Lrefl, a mapping for the range R_Heffs for HDR effects, and a mapping for the intermediate part (i.e., the graphics sub-range R_gra). The next scene in the video may be a sunset scene with multiple (i.e., captured) consecutive sunset images. This may have another preferred re-grading luma mapping function (315), such as Figure 1 As explained, it is usually transmitted together as metadata of the video signal to guide the display optimization of the terminal device (such as a consumer TV display). But the video creator can define his re-grading function so that the graphics part remains stable, that is, all those brightnesses are mapped to the same corresponding output brightness for the two HDR scenes. Then, the graphics will look the same on both scenes in HDR and LDR (or any other intermediate display-adapted image, such as a 950nit ML_C image optimized for a 950nit ML_D display). Moreover, the video creator (i.e., the creator of the brightness mapping function) can choose which graphics sub-ranges are reasonable in both HDR (3000nit main HDR in this example) and LDR. Note that if HDR videos of different maximum brightness need to be mixed and the graphics coordinated, this can be done via the mapped output stable graphics range (75-80nitLDR range in this example). Note that the graphics can be freely configured, so they can depend on what is within the effect range of the HDR movie.

[0095] Now, if Figure 4 As explained, the rest of the movie contains two essentially dark scenes. The first is a night scene 401 showing a motorcycle riding through the city. The motorcycle can be rendered at, for example, 30 nit to give it a relatively dark impression but still be well visible (in LDR, a trick of making the pixels blue must be used to simulate the night, in HDR, it is possible to play more on the darkness of the pixels themselves, but still stay on the brighter side of the sub-range of the darkest pixels, for example, HDR movies are often expected to be watched under relatively bright lighting). The only bright object in this image is the street lamp, which can be given a pixel brightness of 900 nit to make it not annoyingly bright and distracting compared to the rest of the image. This is followed by another dark scene 402. It is not exactly a night scene, but it is still typically dark (rainy day, so outdoor pixels will appear dim). The really dark part is the clown hiding in the sewer, which will get a brightness equal to or lower than 10 nit, for example, to make him almost invisible. For these two sets of images, the video creator may have considered that lower typical levels for the primary graphics are more appropriate, such as another exemplary lower brightness limit L_low2=80nit in HDR (represented by Y_low2 as luma code in the HDR image), and another upper brightness limit L_high2 equal to 90nit (in HDR grading). We also show the corresponding appropriate LDR lower brightness limit (Ll_low=60) and LDR upper brightness limit Ll_high=65nit, which will depend on or vice versa determine the graphic part of the two brightness mapping functions (e.g., the third brightness mapping function 405 for the motorcycle night image 401) for these two dark scenes. The darkest segment of the brightness mapping function 405, which is shown as a line segment for simplicity, can be customized at will so that the clown also looks appropriately dark in the LDR re-graded image, which can be used, for example, to drive a traditional LDR tv. The fact that the graphics brightness changes abruptly as the scene changes (e.g., a bright fire-breathing dragon in the daytime to a nighttime cityscape) is not a problem, as the video creator so desires, as long as the change is not "arbitrary". The current mechanism allows the video creator to select all brightness of various image objects as desired, rather than arbitrarily changing them, for example, when using simpler curves (such as a power function brightness mapping function with a variable power coefficient). Note that not all colors used in the graphics need to remain within sub-ranges that are equally mapped between the various different luma mapping curves. Typically, for at least some applications or systems, it is sufficient if the brighter colors of the graphics remain stable and the darker colors can vary slightly. A device mixing secondary graphics may decide to use graphics luma in a smaller range than the primary graphics, for example, only such luma that is indeed not differentially mapped in multiple consecutive differently re-graded scenes.

[0096] Figure 5 A potential device embodying the concepts of the invention is illustrated (at a high level).

[0097] In this exemplary embodiment, it is assumed (without limitation) that the HDR image signal input terminal 501 receives not only the HDR image itself (i.e., the encoded matrix of pixel colors, where the information to be decoded is commonly transmitted or known to the receiver, such as the selected EOTF), but also receives at least one brightness mapping function LMF at a time t corresponding to one of the images of the video, which can have a variable shape for the capture of consecutive images or similar images of the same scene (e.g., a cave scene). There may also be (or not) explicit metadata about the lower and upper luma limits of the primary graphics (MET (YLo, Yhi); note that we use L for brightness and Y for luma). Let's assume that those are defined in the input domain, i.e., for the received HDR image. An image graphics analysis circuit (511) may be present in the device. How this unit works may depend on the device, for example, some simpler devices may only detect text, and the graphics range from this detected text, for example, including the range of all lumas used in all found texts, from the darkest text pixels to the brightest text pixels, or some parts of its range that are quite large, for example, text pixels brighter than the average text luma, etc.

[0098] Figure 6 A possible advantageous circuit for analyzing primary graphics in images is explained (other analysis algorithms may be used). A person skilled in the art understands that other equivalent graphic analysis techniques may exist as an alternative or in addition to the various configurations of the units used to illustrate the innovation as taught.

[0099] The detector (601) for areas of limited color variation is arranged to check for typical graphic colors in typical graphic low-level elements. Some graphics may be complex, but many of them will have limited subsets that may be illuminated differently, but for example have only two different chromaticities. For example, since there may be text in the HDR image (for example, the name of a horse may have the same white or single pixel color as its colored head shape), the text detector 602 may typically exist to have made a good or first determination of the primary graphic color. Segmentation maps (such as the first segmentation map (SEG_MAP1)) are a simple way to summarize areas that have been identified as possible representative primary graphic pixels so that the colors of those areas / maps can be easily determined later. Therefore, when there is, for example, a recognized text character "G", the following pixels will, for example, obtain the value "1" in a matrix of initial zeros. A property-based graphic analyzer 610 may also exist and can be used for more complex graphics, for example in an iterative manner (graphic color properties / multiple properties are identified, sets are determined, properties are re-identified). This will use, for example, a color type characterizer 611. For example, highly saturated colors (like a strange purple) may usually (unless it is a flower, etc.) indicate that the pixel may be a graphic (graphics usually contain at least some primary colors (which have high or even maximum some of the RGB components, and zero or close to zero one or two components), and natural image content ideally usually does not contain such colors, but rather muted colors). These candidates can be further verified by other units (such as the basic graphic geometry analyzer 612). Graphics, such as shapes that can be characterized, can be matched with the shape characterizer.

[0100] If a group of connected or adjacent (e.g., repeated) pixels with a particular color property is small, it can be a graphic element, because it is usually desirable to make them non-intrusive, just enough to read or see, especially if there are several graphics. And the main movie objects are usually enlarged and large (e.g., a purple coat or a magic fireball may include more pixels). The location can also be heuristic: graphics are usually close to the borders of the image, such as a logo at the top or a ticker tape at the bottom. Such pixels can form further initial candidates in the first segmentation map SEGMAP_1 to be verified again by further analysis or vice versa removed. A skilled reader can understand how to also use, for example, a neural network, which may have been trained on, for example, the simplicity or frequency of changes of the graphics relative to natural video (such as, for example, a texture metric). A more advanced shape analysis circuit 620 can analyze additional properties of the initially assumed graphic region (i.e., for example, starting with a segmentation map based on the initial identification of potential graphic pixels) to obtain a more stable set of pixels for summarizing its luma. As described, it is not necessary to identify all primary graphic pixels with maximum accuracy. For example, there may be an edge detector 621 to detect edge pixels between the graphic shape and the surrounding video.

[0101] The G criterion can be used to detect such edges (M. Mertens et al.: A robust nonlinear segment-edge finder, 1997 IEEE Workshop on Nonlinear Signal and Image Processing).

[0102] The G criterion can be used when you define properties as desired (e.g., bright saturated colors contrasting with more common natural object colors) and count their co-occurrence in two regions. Since counts are involved, the shapes of the regions can also be adjusted as desired.

[0103] For example, using two chromaticities of the pixel color, and defining the first property:

[0104] P1=1000*Cb+Cr

[0105] The property may be transformed under another function, for example, the deviation from a property based on a locally determined average chromaticity, e.g.

[0106] Delta_P1 = P1_pixel minus P1_determined

[0107] P2=Function(Delta_P1)

[0108] The function may be classified, for example, as zero if the delta value is below a first threshold, and from 1 to 9 for intermediate delta values, and 10 if the delta values ​​are sufficiently different (i.e., abs(1000*Cb–1000*Cb_reference)>1000*Threshold1or abs(Cr–Cr_reference)>Threshold2).

[0109] Then two regions are defined, e.g. two adjacent rectangles, which are shifted on the image until they also fall left and right on the horizontal border of e.g. the ticker tape. The amount of shifting may depend on the amount of matching.

[0110] The G criterion is equal to: For all possible different values ​​of P2, G = the sum of the absolute values ​​of the following: the number of pixels in rectangle 1 with any value P2_i (e.g., P2_0 means the red component of the pixel being counted = 0, P2_1 means red = 10, etc.) minus the number of pixels in rectangle 2 with the same value P2_i, and finally dividing the sum of this absolute difference by a normalization factor, which is usually the number of pixels in the two rectangles (or area-adjusted if the regions have unequal sizes). When comparing two equal-sized test rectangles that fall at adjacent locations in the image, the equation becomes:

[0111] G=sum_i{abs[count(P2_i)_right_rectangle-count(P2_i)_left_rectangle]} / 2*L*W[Equation 3]

[0112] L and W are the length and width of the two rectangles

[0113] If the detector is inside the graphics, there will be approximately the same color in either rectangle, and we will see all zeros, i.e. no edge. If one rectangle falls on the graphics, e.g. saturated yellow, and the other side falls on the video, e.g. desaturated green (small excess of green components), then the moving average from the green side (e.g. above the green graphics) will be green, resulting in a P2 value of approximately zero in the upper rectangle, and in contrast to this continuous green, in the other sampled rectangle, the P2 values ​​are all, e.g., below about 10. Then there will be L*W different pixels in the left rectangle, approximately all zero, and there will be L*W pixels with the P2 property in the right rectangle (e.g., with a determined value of 10 as input to the G criterion), and these different feature colors have no matches found on the left. That is, when falling on an edge, the G criterion will be close to the value 1. The benefit of the G criterion is that any property can be added for comparison, such as texture-based metrics, etc. Alternatively, other more classical edge detectors can be used to find a set of candidate points at the edges of graphics areas and the beginnings of natural video areas. Edge detectors typically have the property that they are noisy, i.e. there may be both gaps and pseudo-edge pixels. In addition, the shape analyzer 622 may include a pre-processing circuit to identify connected shapes in the HDR image from the edge pixels found. Various techniques are known to image analysis technicians, for example, Hough transforms can be used to detect lines, circles can be matched, splines or snakes can be used, etc. If the analysis reveals that the four lines (or a line and the image boundary) form a rectangle, in particular with specific properties, such as (exactly or approximately) as wide as the image, and at the bottom, the internal color is a good candidate for a graphic pixel (such as a scroll bar in a news program). Therefore, the entire rectangle can be added to the second segmentation map SEGMAP_2. Based on the symmetry of the expected graphic boundary found, it can be confirmed that the primary graphic element actually exists. A simpler embodiment may, for example, focus only on a few simple geometric elements, like a rectangle detected at the bottom of the image (this is usually enough to get a good first idea of ​​the graphic range R_gr), and only if no such elements are found, for example, the algorithm may further search for more complex graphic objects, such as stars, small areas that fly in, which remain for a few seconds in several consecutive video shots and then disappear again (i.e., also taking into account the time-varying behavior of the graphics). Or on the contrary, the candidate identified graphic areas / objects may remain particularly unchanged over several different shots of a movie, etc. The gradient analyzer circuit 623 may further analyze the case of graphics that do not consist of a set of fixed colors but have internal gradients (usually long-range gradients, slowly varying over tens or hundredths of a pixel). That is, such gradients contain, for example, all yellow, but the saturation varies from left to right. It may verify positively that some gradients may be graphic-generated gradients, or negatively.For example, it may discard a blue gradient in the upper portion of the image as possible sky, depending in some embodiments on further geometric properties of the gradient, such as, for example, the size of the gradient region, the steepness of the gradient, the amount of colors spanned by the gradient, and particularly, again, if the gradient ends at a complex lower boundary (e.g., a tree). In the case of a blue line that may depict water from the sky, pixels may be discarded if the position of that line is too far below the upper boundary of the image.

[0114] The pseudo image analysis circuit 630 can provide further certainty about the robustly determined graphic pixels in the third segmentation map SEGMAP_3. Modern, more pleasing graphics may contain, for example, a (graphically generated) cloud shape in the lower banner, which almost looks like a natural image and may therefore be confused. Even if it is correctly identified as a graphic, it may not contribute significant new insights to the pixel luma of the graphic as can be determined from other parts of the graphic. If it is indeed part of the graphic, it will have a coordinated color anyway, which will roughly overlap with the additionally determined graphic range R_gra, possibly resulting in a slightly higher upper limit luma Y_high or slightly lower Y_LOW as determined, but the method is not so critical. Such an area attached to the identified graphic area, such as in the lower left corner of the banner rectangle, can be discarded again (or in an advanced embodiment, if verified as a graphic element associated with the rest of the banner, it is retained, for example if it has a related color, but these pseudo graphics may be complex, and then they can be better discarded). By discarding, a more credible segmentation map can be obtained for the purpose of determining the appropriate graphic range R_gra. More advanced embodiments may apply texture or pattern recognition algorithms to special regions of interest, i.e., regions that are not a few colors and their simple gradients (i.e., deviating from, e.g., saturation by less than 20%) (usually geometrically simple, such as rectangular or substantially circular, and usually symmetrical). For example, a busyness metric may be computed that indicates how often and how quickly (e.g., which color changes) the pixel color changes in each region of, e.g., 10×10 pixels. Computing the angular spread of lines (e.g., the center of mass of objects with small color changes) is another metric that may be useful in distinguishing some complex natural patterns (e.g., leaves) from simple geometric patterns when they occur in graphics (text typically has only two or a few stroke directions). Simpler embodiments may compute gradients. If there are short-range gradients (i.e., significant changes over a few pixels, as opposed to long-range gradients (slowly changing over tens or hundredths of pixels) that often occur in graphics, such areas can be excluded from the graph for robustness reasons. In addition, if the colors in an area of ​​large-scale surrounding geometry of the graphics area (e.g., a text box) deviate significantly from the average color of the rest of the graphics area, even if they are complementary colors (meaning that they may have been purposefully chosen for the graphic, e.g., blue to complement orange), the processing logic of circuit 630 can remove those pixels from the set that ultimately determines the graphics range, i.e., remove those pixels from SEGMAP_3.

[0115] However, there are many pixel region analysis subcircuits or processes (more or less than the three illustrated), and the final processing is usually performed by the luma histogram analysis circuit 650. This circuit will look at all the lumas of the pixels identified as primary graphics in the third segmentation map SEGMAP_3 (or any equivalent segmentation map or mechanism). It will output a lower limit luma Y_low_ima based on image analysis, which is at or near the lowest luma in the luma histogram of the pixels identified as graphics in SEGMAP_3. If the luma histogram contains many dark colors, the output can be higher than the minimum value, for example, set to no less than 10% of the highest luma to have a practical graphics range. If, on the contrary, only white graphic pixels are detected in SEGMAP_3, the luma histogram analysis circuit 650 can again output a lower limit luma Y_low_ima based on the image analysis, which is, for example, 25% of the upper limit luma Y_high_ima based on the image analysis (which as a more important parameter is usually determined first, which can be, for example, the luma of white text), instead of setting the lower and upper limits to the same (which is useless). In addition, Y_high_ima does not need to always be exactly the same as the maximum luma found in SEGMAP_3. If, for example, only yellow is found as the brightest color, knowing that they usually have 90% of the brightness of white, a luma of 110% of the brightness of the maximum luma found in the histogram can be set as the upper limit luma. Or if brighter color identities coexist with darker white subtitles, a value corresponding to the white value suitable for the brightest primary graphic element can be set.

[0116] Back to Figure 5 There may also be a luma mapping function analysis unit 512 to analyze the similarity regions of the running mapping functions of (consecutive or adjacent) images of the video (i.e., where a subset of lumas are mapped to approximately the same corresponding output luma by all different optimal image-dependent brightness mapping functions), wherein the exemplary algorithm utilizes Figure 7 Explanation. That is, any device can have unit 511 or unit 512 or both, and then usually have some circuitry to summarize the range of patterns found, such as overlap or joint. The same is true for unit 513.

[0117] The various luma mapping functions (LMF(t)) valid for certain moments in time are input to the identity analysis circuit 701. One or more previous luma mapping functions LMF_p are retrieved from the memory 702. If a function of a different shape is determined (i.e. the incoming LMF(t) has a different shape from the stored LMF_p at least at some points), it can replace the old LMF_p, or supplement it when more than two functions are compared. The identity analysis circuit 701 is arranged to first check whether there is an overall identity of the function, rather than just in a sub-region of the input luma. Several HDR codecs send one function per image, and those functions may have the same shape for all images in the same scene, so those functions should be discarded, which is shown by the builder 703 finding that the identity Boolean FID is "yes" or "1". The next function is then simply read until a new function is actually read (e.g. the function for dragon is the old function, and the function for Figure 3 The sunlight exposure function 302 is a new function).

[0118] The sub-range identity circuit 710 calculates the difference of the output value DEL of substantially all input luma values ​​Y_in. Assume that at least one outer region where there is a difference (e.g., below the graphic range) is found, and there is an intermediate range R_id of the output identity. It can determine the bottom luma Yb and top lumaYt of this range. These can be output as the final lower limit luma and upper limit luma of the identified graphic range R_gra suitable for mixing secondary graphics or not suitable for mixing secondary graphics. Typically, the graphic range determination circuit 720 can be further analyzed before it outputs the lower limit luma Y_lowfu determined by the function and the upper limit luma Y_highfu determined by the function. As explained, this can be based on checking whether, for example, the upper limit luma falls within an atypical range, i.e., below the typical upper limit HighTyp from the (dashed line is therefore optional) range supply circuit 740. The same situation may occur with respect to the typical lower limit luma LowTyp. If we have a 5000nit main HDR image, then finding a graphic range around 5000nit may not be a good graphic range because those graphics are found to be too bright by several viewers. If the analysis fails, for example the determined Yt is much higher than HighTyp, an embodiment may propose some average good working value for Y_highfu, but typically it will throw an error condition ERR2. Additionally, a curve may be passed that does not have an intermediate identity range (R_id), in which case an error condition (first error condition ERR) may be thrown again. For simplicity, we have shown that for both curves for all points in the range, the graphical range is exactly the same mapping, but typically this can be relaxed to be found to be similar if within a certain tolerance (e.g. 10% brightness deviation maximum).

[0119] Back to Figure 5, in which case the merging logic circuit 515 cannot use the inputs of the lower luma and upper luma from the luma mapping function analysis unit 512. In other cases, it can check the similarity of Y_highfu and Y_high_ima, and if they give close values, it can use either value or, for example, the average value as the final upper limit luma value Y_high, etc. If present, it can also directly use the metadata-based upper limit value Yhi as provided by the metadata extractor 513 (and if present, also use the metadata lower limit lumaY_Lo, if not, it can be replaced by a percentage of Y_hi (e.g., 1 / 3 of the corresponding brightness L_hi_met)). In any case, at least the upper limit luma Y_high and usually also the lower limit luma Y_low will be transmitted to the graphics generation circuit 520 so that it can take these lumas into account when selecting (or transforming, if an already pre-created graphic element needs to be mixed) the color of the secondary graphic element. As explained, it will determine the appropriate graph lumaY_gra for each pixel so that it generally falls within the graph range (R_gra), or not too far below or above. The actual value of Y_gra will depend on what the graph pattern contains as pixel color and luma, for example, if the darkest color (Y_orig_graph_min) is mapped to Y_low, and the brightest color in a pre-existing or to-be-made graph (Y_orig_graph_max) is mapped to Y_high, then pixel colors between Y_orig_graph_min and Y_orig_graph_max can be proportionally (or non-linearly) mapped between Y_low and Y_high.

[0120] Finally, for (substantially) all pixels in the image, image mixer 530 will mix the graphics color (with graphics pixel luminance Lgra, since mixing is generally more elegant in the linear domain, or graphics pixel luma Ygra, for example, if the mixing involves a simple pixel replacement) and the video pixel color.

[0121] A luma mapper (533) may be present in some embodiments, such as those that can blend in the output domain (some embodiments of the device may have the ability to blend only in the output domain, or only in the input domain (potentially before the final luma mapping), or be able to do both and switch as desired). If the appropriate graphics are blended in the input domain, a brightness mapping function like that used for the dragon will still be applied to the secondary graphics pre-mixed image, and the function can be scaled for a variety of different end-user displays with different end-user display maximum brightness capabilities. But with the present innovation, the graphics will also be relatively stable when subjected to further brightness mapping. But as mentioned above, we can also do this in the output domain of any function (i.e. Figure 3 The vertical axis of the image is not mixed in the output domain, but when the brightness mapping function to be applied is known, then the brightness mapper will know how to blend within the output graphics range. The reader should still understand that this is the upcoming dynamic mapping that has not yet been applied (which can significantly change the brightness or relative brightness), which makes blending easier. Typical applications will send pre-adjusted graphics to the output domain (which can be encoded in PQ, for example) for final blending for example end-user TV.

[0122] The final mixed output image Im_out having for its pixels the mixed brightness Lmix_fi (or generally the luma encoding them) will be provided via the image or video signal output terminal 599 .

[0123] The algorithmic components disclosed herein may (in whole or in part) actually be implemented as hardware (eg, part of a program-specific IC) or as software running on a special digital signal processor or a general purpose processor or the like.

[0124] From our introduction the skilled person should be able to understand which components can be optional modifications and can be implemented in combination with other components, and how the (optional) steps of the method correspond to the corresponding units of the device, and vice versa. The term "device" in this application is used in its broadest sense, i.e. a group of units allowing a specific goal to be achieved, and can therefore be, for example, (a small circuit part of an IC), or a dedicated appliance (e.g. an appliance with a display), or a part of a network system, etc. "Arrangement" is also intended to be used in the broadest sense, so it can in particular include a single device, a part of a device, a collection of (parts of) cooperating devices, etc.

[0125] Computer program product refers to any physical implementation that should be understood as including a set of commands that enables a general or special processor to input commands into the processor and perform any of the characteristic functions of the present invention after a series of loading steps (which may include intermediate conversion steps, such as translation into an intermediate language, and a final processor language). In particular, a computer program product can be implemented as data on a carrier (e.g., a disk or tape), data present in a memory, data transmitted via a network connection - wired or wireless - or program code on paper. In addition to the program code, the characteristic data required for the program can also be embodied as a computer program product.

[0126] Some of the steps required to operate the method may already exist in the functionality of the processor rather than being described in the computer program product, such as data input and output steps.

[0127] It should be noted that the above embodiments illustrate rather than limit the present invention. Where a skilled person can easily implement the mapping of the presented examples to other areas of the claims, we have not mentioned all of these options in depth for the sake of brevity. In addition to the combination of elements of the present invention combined in the claims, other combinations of elements are also possible. Any combination of elements can be implemented in a single dedicated element.

[0128] Any reference marks in brackets in the claims are not intended to be used to limit the claims. The word "comprising" does not exclude the presence of elements or aspects not listed in the claims. The word "a" or "an" in front of an element does not exclude the presence of multiple such elements. In a particular embodiment, the unit of the device in the present teaching may be formed by a circuit on a dedicated integrated circuit (e.g., a color processing pipeline that applies technical color modification to the incoming pixel color (such as YCbCr)), which may be a software-defined algorithm running on a CPU or GPU in a mobile phone or running on an FPGA, etc. Typically, computing hardware (whether a general-purpose bit processing calculator or a specific digital processing unit) will be connected to a memory unit under the control of an operating command, and the memory unit may be on some circuits, or off-board, connected via a digital bus, etc. Some of these calculators may be directly connected to a larger device, such as a panel controller for a display, or a hard disk controller for longer storage, etc., or connected to a physical medium, such as a Blu-ray disc or a USB stick. Some functions may be distributed over various devices via a network, for example, some calculations may be performed on a server in the cloud, etc.

Claims

1. A method of determining, in a circuit for processing a digital image, a second luma of pixels of a secondary graphic image element (216) to be mixed with at least one high dynamic range input image (206), the method comprising: Receiving a high dynamic range image signal (S_im) comprising the at least one high dynamic range input image, wherein the method comprises: analyzing the high dynamic range image signal to determine a primary graphic luma range (R_gra) of a primary graphic element of the at least one high dynamic range input image, the primary graphic luma range being a subrange of a luma range encoding the brightness of the high dynamic range input image, wherein determining the primary graphic luma range comprises determining a lower limit luma (Y_low) and an upper limit luma (Y_high) specifying endpoints of the primary graphic luma range (R_gra); brightness mapping the secondary graphic image element, at least a brightest subset of the pixels luma of the secondary graphic image element being included within the graphic luma range; and The secondary graphics image element is blended with the high dynamic range input image.

2. The method according to claim 1, wherein: Analyzing the high dynamic range image signal includes: detecting one or more primary graphic elements in the high dynamic range input image, establishing luma of pixels of the one or more primary graphic elements, and summarizing the luma into the lower limit luma (Y_low) and the upper limit luma (Y_high), wherein the lower limit luma is lower than all or most of the luma of pixels of the one or more primary graphic elements, and the upper limit luma is higher than all or most of the luma of pixels of the one or more primary graphic elements.

3. The method according to claim 1, wherein: Analyzing the high dynamic range image signal includes reading, from metadata in the high dynamic range image signal, values ​​of the lower limit luma (Y_low) and the upper limit luma (Y_high) written as metadata in the high dynamic range image signal.

4. The method according to claim 1, wherein: Analyzing the high dynamic range image signal includes: reading two or more luma mapping functions associated with temporally consecutive images from metadata in the high dynamic range image signal, and establishing the graphic luma range as a range that satisfies a first condition regarding the two or more mapping functions, wherein the two or more luma mapping functions map input luma within the graphic range to corresponding output luma, and the first condition is that, for each input luma, the corresponding two or more output lumas of the input luma are substantially the same, and the corresponding two or more output lumas can be obtained by applying corresponding mapping functions from the two or more luma mapping functions to the input luma.

5. The method according to claim 4, wherein: The graphics luma range satisfies a verification of a second condition of falling within a suitable graphics luma secondary range, wherein the maximum value of the secondary range must be lower than a predetermined percentage of the maximum value of the mixed image and graphics range.

6. The method according to any one of claims 1 to 5, wherein: The method comprises the step of mapping the pixel luma of the at least one high dynamic range input image to the corresponding output luma of at least one corresponding output image, wherein the blending occurs in the at least one corresponding output image by mapping the pixel luma using at least one luma mapping function obtained from metadata of the high dynamic range image signal.

7. The method of any one of claims 1 to 5, comprising determining the secondary graphic image element having a set of colors spanning a scale of colors of different lightness, some of the colors appearing brighter than average and some of the colors appearing darker than average to the human eye, wherein At least above average bright colors are blended into the at least one high dynamic range input image at a luma within the graphic luma range.

8. The method according to any one of claims 1 to 5, wherein: The blending comprises pixel replacement or blending with a percentage less than 50% of the pixel brightness of the at least one high dynamic range input image.

9. The method according to claim 7, wherein: The determination of the set of colors includes selecting a finite set of relatively bright, below-average dark colors having a luma above a secondary lower limit luma that is a percentage of the lower limit luma.

10. The method according to claim 9, wherein: The determination of the set of colors includes selecting a finite set of relatively bright below-average dark colors having a luma above a secondary lower limit luma that is higher than 70% of the lower limit luma.

11. An apparatus (500) for determining a second luma of pixels of a secondary graphical image element (216) and blending the secondary graphical image element with at least one high dynamic range input image (206), the at least one high dynamic range input image comprising a primary graphical element (207), the apparatus comprising: An input terminal (501) for receiving a high dynamic range image signal (S_im) comprising the at least one high dynamic range input image; an image signal analysis circuit (510) arranged to analyze the high dynamic range image signal to determine a graphic luma range (R_gra) of the at least one high dynamic range input image, the graphic luma range being a range specifying luma of pixels representing a primary graphic, the range being a sub-range of a luma range encoding brightness of the high dynamic range input image, wherein determining the graphic luma range comprises determining a lower limit luma (Y_low) and an upper limit luma (Y_high) specifying endpoints of the primary graphic luma range (R_gra); Graphics generation circuitry (520) arranged to generate or read secondary graphic image elements and brightness map said secondary graphic image elements, at least the brightest subset of the pixels luma of said secondary graphic image elements being included within said graphic luma range; an image mixer (530) arranged to mix the secondary graphics image elements with the high dynamic range input image to produce pixels having a mixed brightness (Lmax_fi); as well as An output terminal (599) for outputting at least one mixed image (Im_out) comprising said pixels having a mixed brightness (Lmax_fi).

12. The device according to claim 11, wherein: The image signal analysis circuit (510) comprises an image pattern analysis circuit (511), wherein the image pattern analysis circuit is arranged as follows: detecting one or more primary graphic elements in the high dynamic range input image, establishing luma of pixels of the one or more primary graphics elements, and The luma is summarized into the lower limit luma (Y_low) and the upper limit luma (Y_high), wherein the lower limit luma is lower than all or most of the luma of the pixels of the one or more primary graphic elements, and the upper limit luma is higher than all or most of the luma of the pixels of the one or more primary graphic elements.

13. The device according to claim 11, wherein: The image signal analysis circuit (510) includes a metadata extraction circuit (513), which is arranged to read the values ​​of the lower limit luma (Y_low) and the upper limit luma (Y_high) written in the metadata of the high dynamic range image signal from the metadata in the high dynamic range image signal.

14. The device according to claim 11, wherein: The image signal analysis circuit (510) includes a luma mapping function analysis unit (512), which is arranged to: read two or more luma mapping functions (LMF(t), LMF_p) of temporally consecutive images from metadata in the high dynamic range image signal (S_im), and establish the graphic luma range as a range (R_id) that satisfies a first condition regarding the two or more luma mapping functions, wherein the two or more luma mapping functions map input luma within the graphic range to corresponding output luma, and the first condition is that, for each input luma, the corresponding two or more output lumas of the input luma are substantially the same, and the corresponding two or more output lumas can be obtained by applying corresponding mapping functions from the two or more luma mapping functions to the input luma.

15. The device according to claim 11, wherein The image mixer (530) comprises a luma mapper (533) arranged to map the luma of pixels of the at least one high dynamic range input image to corresponding output luma of at least one corresponding output image, the at least one corresponding output image having a dynamic range different from the dynamic range of the at least one high dynamic range input image and the luma mapper being arranged to perform the mixing in the at least one corresponding output image according to at least one luma mapping function obtained from metadata of the high dynamic range image signal.

16. The device according to claim 15, wherein: The at least one corresponding output image has maximum brightness.

17. The device according to claim 11, wherein: The graphics generation circuit (520) is arranged to design the secondary graphic image element having a set of colors spanning a scale of colors of different light brightnesses, some of the colors appearing brighter than average to the human eye and some of the colors appearing darker than average, wherein at least the brighter than average colors are mixed into the at least one high dynamic range input image with a luma within the graphic luma range.

18. The device according to claim 11, wherein The image mixer (530) is arranged to perform mixing by pixel replacement of video pixels of the at least one high dynamic range input image by secondary graphic image element pixels, or blending with a percentage less than 50% of the pixel brightness of the at least one high dynamic range input image.

Images