Adjustment of display optimization behavior for HDR images

Abstract

In order to enable a better and more adjustable method of HDR video display adjustment, a connection part 501 to a provided or connectable video decoder 207, where the video decoder receives an encoded high dynamic range image Im_COD, which is encoded according to the first maximum encodable luminance PB_H, and the video decoder receives metadata specifying at least one luma mapping function F_ct, FL_50t1_1. The at least one luma mapping function specifies the offset of the luminance of the secondary image corresponding to the encoded high dynamic range image compared to the luminance at the same pixel position when encoded in the encoded high dynamic range image. The secondary image preferably has a second maximum encodable luminance PB_S that is at least 4 times smaller or larger than the first maximum encodable luminance PB_H. The video decoder is configured to output a decoded high dynamic range image Im_RHDR and the luma mapping function. A display adjustment unit 401 configured to receive the value of the display maximum luminance PB_D that a specific display can display as the brightest pixel color and the input luma mapping function. The display adjustment unit applies an algorithm to calculate at least one display-adjusted luma mapping function based on the input luma mapping function and the display maximum luminance PD_D. This at least one display-adjusted luma mapping function corresponds to the input luma mapping function in shape but is close to a 45-degree increasing diagonal of the graph of the input luma mapping function in a perceptually uniformized axis related to the difference between the value of the display maximum luminance PB_D and the difference between the second maximum encodable luminance PB_S and the first maximum encodable luminance PB_H. An image pixel luminance adjustment device 500 including a display adjustment unit 401. The image pixel luminance adjustment device includes an alternative luma mapping function determination unit 502 for determining an alternative luma mapping function ALT_FL_50t1_1. The display adjustment unit 401 is configured to obtain a combined luma mapping function from the at least one luma mapping function F_ct,A combination unit 503 is configured to be combined with FL_50t_1 and the alternative rum mapping function ALT_FL_50t1_1. The display adjustment unit applies the algorithm to the combined rum mapping function as the input rum mapping function. The image pixel luminance adjustment device receives the pixel rum of the decoded high dynamic range image Im_RHDR and applies the combined rum mapping function to obtain the output rum of the output image Im_DA through a rum mapping unit 510. The image pixel luminance adjustment device includes an output image or a video transmission cable or a wireless channel, and a display can be connected to the output image or the video transmission cable or the wireless channel. It is further characterized by further including an output signal formatter 230 for transmitting the output image Im_DA. An image pixel luminance adjustment device 500 is provided.

Description

Technical Field

[0001] The present invention relates to a method and apparatus for adjusting the high dynamic range image pixel luminance of an HDR video for a specific display having a specific luminance dynamic range, particularly the maximum displayable luminance (PB_D).

Background Art

[0002] A few years ago, new technologies for high dynamic range video coding were introduced, particularly by the applicant (see, for example, WO2017157977).

[0003] The coding and processing of HDR video is very different from the conventional video coding, which until a few years ago, all videos were encoded and are now called standard dynamic range (SDR) video coding (known as low dynamic range (LDR) video coding), that is, PAL in the analog era and, digitally, for example, Rec.709 MPEG2. In fact, starting from very bright and sometimes dark image objects that need to be codable (i.e., a larger range of pixel luminances in the original master HDR image created by the content creator), all the rules of video technology have been rethought one by one and often reinvented.

[0004] Regarding coding, the difference between HDR and SDR is not only a physical difference (there is a large difference in pixel luminance and it is displayed on a display with larger dynamic range characteristics), but also a technical difference in developing different luma assignment functions (OETFs), which are additional (for each shot of the image) dynamic change metadata that specifies techniques for regrading the pixel luminances of various image object pixels to obtain an image with a secondary dynamic range different from the original image dynamic range.

[0005] The Rec.709 SDR luma coding definition encodes an approximate square root OETF function shape (luma: Y = square root of luminance L) to match the normal rendering capabilities of all displays at that time, enabling encoding of only an approximate 1000:1 luminance dynamic range (for 8- or 10-bit luma). In the SDR era, no one cared to specify or use the maximum luminance, known as the coding peak luminance PB_C, but the luminance of the various LDR displays in the 20th century market was approximately between a minimum displayable luminance of about 0.1 (in simple terms, "black") and 100 nits ("white"), i.e., the display peak luminance (PB_D).

[0006] The first HDR codec introduced to the market was the "HDR10" codec, which was used, for example, to create HDR Blu-rays in black-ribbon jewel cases, and only changed the OETF to the more logarithmic perceptual quantization (PQ) function standardized in SMPTE2084, enabling definition of luma for a much wider range of luminances, i.e., between 1 / 10,000 nit and 10,000 nit, sufficient for any practical HDR image specification for video creation (e.g., movies, TV broadcasts, etc.).

[0007] HDR should not be confused with simply a larger (more bits) amount of bits for luma codewords (e.g., 16 bits instead of 8). Such a thing may apply to linear systems such as the bit amount of an analog-to-digital converter, but since the code assignment function can be highly non-linear, in this case, HDR images can actually be defined with 10-bit luma, leading to the advantage of reusability of already developed systems (e.g., an IC may have a specific bit depth or video cable, etc.).

[0008] After the luma calculation, only 10-bit plane pixels are obtained (or 10-bit planes of two chrominance planes, Cb and Cr3, are obtained), and it can be further processed classically down to the line, as if it were mathematically an SDR image, for example, as if it were MPEG-HEVC compressed.

[0009] Of course, the receiving side needs to recognize that it is obtaining an HDR image rather than an SDR image; otherwise, incorrect rendering will occur. For example, if only linear mapping is performed (the image maximum luminance PB_C is coded to the SDR display peak luminance PB_D = 100 nit), an image with PB_C = 1000 nit may appear 10 times too dark, which means that night scenes may become invisible. By using other luminance mappings, brighter pixels can be created, but generally, great care is needed regarding what is best to do.

[0010] Due to the logarithmic characteristic of the PQ OETF, HDR10 images are in principle visible (when the luma codes are interpreted as if they were normal SDR codes, i.e., when they are displayed after applying an approximately squared EOTF), but will have a visually unappealing and degraded contrast, and in particular, the contrast will be reduced and they will appear to have incorrect luminance (for example, a suspect who is thought to be hidden in the dark will suddenly appear as if illuminated by ceiling lighting).

[0011] The problem of simply coding and processing such "mere HDR" video images (also called HDR master gradings, where the term grading indicates what luminance each scene object / pixel should have in that representation in order for the HDR scene to look optimal in an image representation with, for example, a PB_C of 1000 nits (or more)) is to be displayed correctly only on a display with the same display peak luminance PB_D = 1000 nits. Therefore, since both the display and the PB_C of various contents can change ("uncontrollably"), for example, such HDR Blu-ray discs are not always displayed perfectly, and in this situation, it was immediately found that night scenes become invisible.

[0012] In addition, more advanced HDR video encoders encode two differently graded images, for example, a high-dynamic-range image with PB_C = 5000 nits and a low-dynamic-range image, which is usually an SDR image with PB_C = 100 nits and is known as the grading of the HDR scene. This is because the image can be immediately displayed on a conventional LDR display. As will be explained below, when transmitting all the mathematical information that allows, for example, a second image to be calculated from one of the pairs (for each display time of the video image) actually transmitted as an MPEG-HEVC image, the reader should understand that transmitting two differently graded images to the receiving side does not necessarily mean actually transmitting two images, that is, two sets of DCT transformation matrices of YCbCr pixel colors.

[0013] An actual scene (however, a uniformly illuminated scene has only a lower dynamic range due to the reflectance of objects at a ratio of 100:1) can have a very high dynamic range. For example, a cave with a small opening to the outside sunlight has a luminance of less than 1 nit for the pixels of the cave and includes a luminance up to 10,000 nit for at least some of the outdoor pixels on a 10,000 nit PB_C reference representation including appropriate HDR grading for the scene of watching TV at home. Such a difficult HDR image is not very clearly converted to a very low dynamic range (for example, at least 100 times when it becomes SDR), especially when the content creator still wants to convey a very similar HDR look in the SDR grading, but in principle it can be conveyed as explained in FIG. 1.

[0014] For convenience in reading this specification and to quickly understand some of the further related aspects, FIG. 1 is a diagram showing several exemplary illustrations of many conceivable HDR scenes that a future HDR system (connected, for example, to a 1000 nit PB_D display) needs to be able to process correctly, i.e., by rendering appropriate luminance for all objects / pixels in the image.

[0015] Illustrate typical problems related to dynamic range mapping, e.g., dynamic range mapping from a large pixel luminance dynamic range to a small pixel luminance dynamic range (assuming at least the peak luminance characteristics change). This problem can be likened to the task of packing many things into a very small suitcase. For the largest suitcase, one can simply throw all the things in without sorting, and somehow it will fit. For a medium-sized suitcase, some optimization needs to be determined. For example, when packing clothes neatly into small partitions such as individual bags, less space may be required, and in this case, everything will fit easily. When using only the smallest suitcase, a new set of more drastic actions is needed. For example, when packing books related to a destination, if the same information is explained in more detail in another book, tear a few pages from the second book to somehow reduce the amount of things to be packed (thus becoming "equivalent," but not exactly the same as what is in the largest suitcase). When only packing clothes, it is not preferable to tear a part of an expensive piece of clothing, so it is preferably not to apply the best compression method for the books described above. Similarly, in video coding technology, "image packing" is performed during execution, and it is necessary to find a stable, universally applicable, and fast computable method that may also vary according to the needs of different sub-ecosystems (e.g., very artistic Hollywood movies and crudely created news materials, such as videos from amateur contributors).

[0016] For example, ImSCN1 is an outdoor image on a sunny day from a western movie (characterized by, for example, showing more of the sunny look rather than the rainy-day look using an average luminance of about 500 nits, and being rendered slightly brighter on an HDR display than on a 100-nit display, showing mostly bright areas or only bright areas), while ImSCN2 is a nighttime image.

[0017] What makes an image sunny and what makes an image dark? It is not necessarily relative luminance, at least not relative luminance in the SDR paradigm. The difference between HDR image rendering and the rendering that has always been done in the SDR era is that SDR had a limited dynamic range (about PB = 100 nits, with a minimum black (MB) level of approximately 0.1 to 1 nit), and almost only the intrinsic reflectivity of objects is shown in SDR (in the range between 90% for good white and 1% for good black). That is good for recognizing an object (with a certain amount of brightness and, of course, chromaticity from its reflection) under uniform and technically controlled lighting, but it is not very suitable for conveying the beautiful variations in the lighting itself that a natural scene can have and the effects that can be given to the viewer.

[0018] When the display is compatible, and thus when the image coding and processing technology are compatible, a person walking in the forest will actually see the sunlight coming through the trees, that is, not only the slightly yellowish impression of some patches on the SDR display, but also expect to see clothes brightly colored by sunlight when walking from the shadow towards the sun. Similarly, fires and explosions also have optimal visual effects if at least PB_D is compatible.

[0019] In SDR, the night-time images are only slightly darkened, and in the luma histogram, it is not so much that they would otherwise be too dark and render the image unpleasantly (at least partially invisible). Furthermore, on a 100 nit TV or in 100 nit encoding, there is no room for overly bright elements. Therefore, objects must be displayed regardless of their lighting, and at the same time, even with high-contrast lighting in a scene, not everything could be faithfully displayed. In fact, this meant that very bright sunny-day scenes had to be rendered with approximately the same display luminance (0 to 100 nit) as overcast rainy-day scenes. The viewer had to calculate from other clues what the situation was, for example, that the cactus was probably brightly lit by the sun and thus expected to be bright. Night-time scenes should not be rendered overly dark either, otherwise the viewer could not clearly distinguish the darkest parts of the image, and thus, also in this case, the night-time luminance was rendered in the range approximately between 1 nit and 100 nit. The conventional solution to this was to color night-time scenes blue, by which it was thought that the viewer would understand that they were not looking at a daytime scene. These are indeed serious limitations of SDR imaging (another example is, for instance, clipping everything outside a window to a single white, by which there is no longer anything visible there), but viewers and industry users had somehow "gotten used to it", which does not mean that there is no room for improvement.

[0020] Here, of course, the actual human vision also adapts to the available amount of light, but not to that extent (most people in reality recognize that they are in a dark or darker or very bright environment). Furthermore, the adjustments by technical TV systems are not the same as the adaptation by the human eye and brain, and the adjustments to the display of content at home are not the same as the adjustments to the original scene when outside in the desert.

[0021] Therefore, when at least an HDR display is available, it may be desirable to render an image using all of the excellent local and temporal lighting effects that can be artistically designed in order to obtain a very realistically rendered image. What exactly shows this is, for example, the appropriate luminance for a light saber in a dark room, leaving the judgment to the color grader who is creating the master grading (when talking about the color grader, it means an equivalent concept for each ecosystem, and thus, there is no need for a human to spend a lot of time defining the pixel luminance for both the master HDR image and the SDR image, nor is an automatic grading for real-time broadcasts, etc. necessary). This application focuses on the necessary technical elements for creating and processing such images.

[0022] On the left axis of Figure 1, the luminance of the object is the luminance as seen with a 5000 nit PB master HDR grading for a 5000 nit PB_D display (i.e., the grader creates an image assuming a typical high-quality HDR TV in a home has 5000 nit PB_D, and actually sits in a representation of such a viewing room in such a home and grades on such a 5000 nit PB_D grading standard display). If one also wants to convey not only the illusion but also the realism of an actual cowboy in an environment lit by a bright sun, those pixel luminances need to be specified and rendered to be bright enough, for example, approximately 500 nit (compared to the previous scene which was, for example, an indoor scene), while (being careful of a typical pitfall in HDR image creation and processing) not being unpleasantly too bright. For a night scene, a mostly dark luminance is desirable, but the main character riding a motorcycle must be clearly recognizable, i.e., not too dark (for example, at approximately 5 nit it would not be recognizable), and at the same time, there can be pixels with very high luminance, such as around the peak luminance on a 5000 nit display, for example, approximately 3000 nit, or on any HDR display (for example, 1000 nit).

[0023] The third example, ImSCN3, also currently shows what has been achieved on an HDR display and can render both very bright and very dark pixels simultaneously. ImSCN3 shows a dark cave with a small opening through which one can see outside on a sunny day. Regarding this scene, for example, when one wants to render the impression of a sunny scene that is approximately 400 nits bright, there may be a case where one wants to make the objects illuminated by the sun, such as trees, slightly lower in luminance than the scene, and it is necessary to adjust more precisely with the almost dark characters inside the cave (because scattering in the human eye that would visually degrade the objects in the dark cave is also undesirable). The color grader optically adjusts the luminance of all objects (already present in the master HDR image with PB_HDR = 5000 nits), so that, for example, a person standing in the dark part of the cave is encoded in the master HDR gray-scale image at approximately 0.05 nits (assuming that HDR rendering can render not only bright highlights but also dark areas). In any case, it is desirable that they do not appear inappropriately dark or bright and that the contrast is good.

[0024] However, here, the question is what needs to happen on a dynamic range that terminates at, for example, 1500 nits when having these master HDR object pixel luminances (although it is possible to faithfully represent all luminances below 1500 nits even with this small dynamic range, what about pixel luminances above 1500 nits? In typical behavior, all are clipped to the same PB_C value of 1500 nits, which is far from ideal).

[0025] Therefore, in principle, content creators are tasked with defining a large set of re-graded images starting from a master 5000 nit HDR image, such as, for example, PB_C images at 3000 nit, 2000, 1500, 1000, 750, 500, 300, and 100 nit images, as in the above packing example of the suitcase. Such a large task is, of course, commercially infeasible. Therefore, a technology has been invented (see WO2017108906) that allows content creators to only grade their master HDR images and corresponding SDR images, and all intermediate images based thereon can be determined by an automatic, technical, so-called display optimization system.

[0026] For the explanation of some components of the concept of the present invention as detailed below, which is important to understand deeply, and to show some technical video coding possibilities, an exemplary HDR video coding system designed for HDR images will be described, particularly with respect to HDR video coding (it should be understood that the principles of the present invention are applicable to systems other than the exemplary system selected for simplicity of explanation in this description).

[0027] This video coding system can handle the transmission (encoding) of only a single standardized HDR video for a typical single type of display in this field (for example, 10-bit perceptual quantization is used as the luma code defining the EOTF for encoding), (for example, on the premise that each end viewer has a 1000 nit PB_D display, the image is defined at PB_C = 1000 nit), and at the same time, it can transmit and process videos with optimal looks / gradings for various other possible display types with various other peak luminances in the industry, particularly SDR images for 100 nit PB_D SDR displays.

[0028] That is, in such an HDR video transmission system, only one kind of graded image is actually transmitted as the transmitted pixelated image, but it has various variations. In this example, among them, the SDR image is transmitted to the receiver by any video transmission system (however, a variation for transmitting the HDR image may also be used). In the metadata, one or more luminance remappings, also known as re-grading functions, that define the HDR image pixels' colors, particularly a specific luminance among their SDR image pixel luminances, are added. At the same time, the HDR image look for the corresponding scene is also transmitted (it is not actually necessary to transmit the HDR image or at least a second layer of pixelated HDR image data like double image transmission).

[0029] In addition, as illustrated in FIG. 2, a set of appropriate reversible color conversion functions F_ct is defined on the encoding side, for example, by a human color grader.

[0030] The above function defines how to obtain a proper-looking SDR image (Im_LDR) corresponding to the HDR master image MAST_HDR starting from the master HDR pixel luminance (or the corresponding luma code transparently). At the same time, by using the inverse function IF_ct, it is ensured that the original master HDR (MAST_HDR) image can be reconstructed with sufficient accuracy as the reconstructed HDR image (Im_RHDR) at any receiving side. The IF_ct function can be determined from the F_ct function of the transmitted forward HDR-to-SDR mapping, or the system can directly transmit the IF_ct function, for example, by the MPEG mechanism of the SEI message or any other appropriate metadata transmission mechanism.

[0031] The color converter 202 typically applies an F_ct luminance mapping to the luminance of the master HDR image (MAST_HDR) pixels, assuming that the luminance is normalized, such that the maximum luminance becomes 1.0 (note that thereafter, it is possible to overlay the entire HDR and SDR regions, meaning that the DR luminance conversion corresponds to a shift of the color in the normalized entire region of the color that can be encoded). For the sake of simple understanding of the concept of the present invention, for simplicity, the luminance mapping from HDR to SDR of F_ct is assumed to be a quarter-power luminance mapping function (L_out_SDR = (L_in_HDR, i.e., 1 / 4) power) to derive the normalized SDR output luminance of the pixels of the SDR output image Im_LDR of PB_C at 100 nits (i.e., the luminance on the right side of FIG. 1).

[0032] Since the receiver needs to be able to reconstruct the master HDR image from the received corresponding SDR image, or due to some compression-related artifacts, at least an approximate reconstruction needs to be possible away from the actual pixelated image, so the color mapping function also needs to be input to the video encoder 203 thereafter. Without limitation, the video is compressed by this video encoder 203 using an MPEG HEVC video compressor to generate an encoded (SDR) output image Im_COD, assuming that those functions are stored in the metadata using, for example, the SEI mechanism or a similar technique.

[0033] Therefore, after the operation of the content creation device 221, from the perspective of the image transmission technology, the rest of the transmission chain pretends to obtain a "normal SDR" image as an input. Therefore, for example, the transmission formatter 204 applies all the necessary conversions to format the data to pass through some transmission medium 205 (for example, channel coding for storage on a BD disc, or frequency coding for cable transmission, etc.).

[0034] Thereafter, the image data moves to one or more receivers, such as a television set, or a consumer video device such as a set-top box, or a professional system such as a cinema receiving unit, via some transmission medium 205, such as ATSC 3.0 or DVB, or satellite or cable or Internet transmission according to any video signal communication principle.

[0035] On either the consumer side or the professional side, a receiving dematformer 206 incorporated in various physical devices such as a set-top box, a television, or a computer, for example, cancels the channel coding (if present) by applying dematforming and channel decoding. Thereafter, a video decoder 207 applies, for example, HEVC decoding to generate a decoded SDR image Im_RLDR and unpacks the color conversion function metadata F_ct. Thereafter, a color converter 208 is configured to convert the SDR image into an image with a non-SDR dynamic range (i.e., higher than 100 nits and usually at least four times higher with PB_C).

[0036] For example, the original master image Im_RHDR of 5000 nits is reconstructed by applying the inverse color conversion IF_ct of the color conversion F_ct used on the encoding side to create Im_LDR from MAST_HDR. However, a display adjustment unit 209 for converting the SDR image Im_RLDR to a different dynamic range may also be provided. For example, when the display 210 is a 3000 nit PB display, or a 1500 nit PB, or a 1000 nit PB image, etc., Im3000nit is optimally graded. Assuming non-limitingly that the video decoder and the color converter are present within a single video remapping device 220. A skilled reader can similarly design a topology for transmitting, for example, an HDR image with PB_C = 10,000 nits, and the color converter can generate an output HDR image with, for example, PB_C = 2500 nits for the corresponding television or monitor, and further understand that various units are connected via a network for operation, for example, on various servers.

[0037] The technical components (innovative components and / or prior art according to this disclosure where connections, collaborations, integrations, etc. take place) are embodied or realized as various technical systems that are typical in image or video technology, i.e., for example, in various hardware devices. For example, the video resolution device 220 may have any technical video supply output unit 231, for example, an HDMI (registered trademark) cable connectable to a TV display (furthermore, for example, a network cable that transmits the output image Im_RHDR of 3000 nits to another possible remote device or system, such as a storage device), etc. Depending on the selected physical transformation, there may be an image or video output signal formatter 230 that converts an image into a signal as required in any technical situation (for example, the following core colorimetric calculations generate a linear R, G, B representation of pixel colors as output, and for example, the final image signal (I_out) transmitted to the display 210 is, according to its technical configuration, formatted, developed, or compressed, such as in HLG format, MPEG, or AV1, and the signal formatter 230 plays a role in processing all signal derivation steps as described above (either fixed as a single option or configurable), and may include units such as a typical integrated circuit).

[0038] FIG. 3 illustrates an example that is helpful for understanding generally what the color converters 202 (and respectively the color converters 208) are in the present invention. This particular colorimetric core appropriately realizes all deformations as one configurable core, but as far as this innovative proposal regarding the technology is concerned, it is considered that the elements are also designed in various different topologies. Color is input in a very classical color format for video color representation, i.e., YCbCr (as an addendum, in various embodiments, this space may be defined from various non-linear R, G, B color triplets, such as gamma 2.0 definition, PQ definition, etc.).

[0039] The chroma multiplier 301 determines an appropriate multiplication scaling factor according to the luma (Y) value of any of the processed consecutive image pixels, and the multiplier 302 uses this multiplication scaling factor to multiply both chroma coordinates Cb and Cr of this scaling factor s(Y), that is, the output red chroma Cro = s(Y) * Cr and the output blue chroma Cbo = s(Y) * Cb of the input color. For multiplication, the same s(Y) coefficient keeps the hue of the output color the same as that of the input color (while appropriately affecting the saturation of the pixel color). The chroma multiplier 301 may be configured to be normally transmitted together with the transmitted SDR (or other) image and read from the metadata MET(F_CLUT) including the LUT of the s coefficient for each possible Y value that the SDR image has, for example. The output chroma values Cro and Cbo are input together with the luma Y to the matrix calculator 303, and the matrix calculator 303 uses standard 3×3 matrix coefficients (for example, according to generally known colorimetry methods depending on which primary color is selected, such as Rec.709 or Rec.2020) to calculate the normalized RGB representation, that is, the normalized red input component RnI, the normalized green input component GnI, and the normalized blue input component BnI. The normalized RGB representation is converted to the required (normalized) output RGB values RnO, GnO, BnO, that is, the (normalized) HDR reconstruction components in this example.

[0040] When the input luma Y of the processed pixel is given, for example, the increase in the luminance of the pixel triplet is affected by multiplying the same luma multiplication value g(Y) to the above three input components (by the multiplier 305). In the above, prior art has been described that can convert any given mapping function shape between normalized luma (for example, a parabola starting from (0,0) and ending at (1,1)) into a corresponding set of g multipliers for all possible normalized luma Yn. This operation is performed by the luma multiplier 304 that reads the metadata MET(F_PLUT) as input as transmitted by the content creator and encodes the shape of the luminance mapping curve, or equivalently the correlated luma mapping curve, or any selected one.

[0041] After the multiplier 305, although still in the region of normalization to 1.0, the pixel luminance is correctly shifted to its HDR-reconstruction relative position. Finally, the output color calculator 306 performs all the calculations necessary to technically format as required by a technical output component, such as a display connected via, for example, an HDMI (registered trademark) cable, a wireless communication channel, etc. The output color format RGB_DF may be determined to be, for example, the PQ-RGB format, but if it is known that the connected display is a specific physical type, for example, all types of optimizations may be applied (if the output is, for example, stored for later viewing, such as on a hard disk recording, it may not be necessary to perform them, or they may be performed differently), and the details of these are not important in the description of the present invention.

[0042] FIG. 4 is a diagram for explaining how display optimization (also known as display adjustment) works for any re-grading function shape (and situation, coding embodiment) selected by the content creator for a specific HDR scene image (for example, the left part is a relatively dark image, so a relative luminance increase is required in the normalized SDR representation of the darker region, and thus a person hidden in the shadow is semi-visible in both the HDR image display and the SDR image display, while for a person in the fog in the brighter part of the image continuous with that scene, sufficient contrast is simultaneously achieved, and a double curve for optimizing the contrast in two regions, such as the FL_50t1_2 curve shown in FIG. 4, is obtained).

[0043] Referring to FIG. 4, it is assumed that there is a current image (or shot of a continuous image) such that (for example, the received HDR image) is optimally re-graded to the corresponding SDR image by a specific normalization luma mapping function FL_50t_1 (readers with skills in video should note that they can understand how the above teachings can be configured in both the normalized luminance and the normalized luma by simply changing the axes and the corresponding shape of the curve).

[0044] It is not desirable to limit unnecessarily, and the examples in this description will continue the explanation on the premise that both are converted to an axis where the luminance is visually uniform (that is, the equally spaced graduations on the horizontal and vertical axes approximately correspond to visually equal luminance differences).

[0045] According to the applicant, the following formula can be used to change the luminance (or, if desired, any quantity of the color coordinates, such as the linear contribution rate of red of the primary colors) to the perceptually uniform luma v as described above. v(Ln;Lmax)=log[1+(RHO(Lmax)-1)*power(Ln;1 / 2.4)] / log[RHO(Lmax)] RHO(Lmax)=1+32*power(L / 10000;1 / (2.4)) [Equation 1]

[0046] In the above formula, L is the luminance of the pixel (normalized, absolute, in nit units == Cd / m^2), Ln is the luminance normalized to a maximum of 1.0, that is, the coded image PB_C, for example, having an Lmax equal to the peak luminance of 5000 nit, and then the luminance obtained by dividing Ln = L / PB_C.

[0047] Therefore, for example, when it is necessary to map 5000 nit content to SDR luminance, it becomes the output luma v_SDR as shown on the vertical axis of Figure 4B, and the output luma v_SDR can be converted to luminance by using the reciprocal of Equation 1. Due to the almost logarithmic characteristic of the visually uniform luma representation of luminance, on the vertical axis, the decades up to PB_C_SDR = 100 nit are almost equally spaced, that is, v = 0.6 approximately corresponds to 10 nit, etc.

[0048] Similarly, for the case of HDR input with PB_C defined as 5000 nit, the normalized luma on the horizontal axis approximately corresponds to 0, 1, 10, 100, 1000 at equal intervals, and the end point 5000 is slightly closer to the position where the 10k position exists.

[0049] Corresponding to this axis system, the content creator can then define the luminance remapping function shape as needed for the current image (for example, a dark street with some streetlights, in the corresponding dark PB_C image of the master HDR image for rendering on a darker PB_D display, needs to brighten up the darkest parts of the houses or shrubs on that street somehow to maintain a sufficiently visible state, which, along with a slope greater than 45 degrees at the dark end of the first exemplary lumamapping (corresponding to luminance mapping) curve FL_50t1_1, indicates a display adjustment strategy for automatically deriving a curve shape that maps 5000 nit luma to, for example, 650 nit luma (when a 650 nit PB_D connected display needs to supply a properly optimized / re-graded version of the master HDR image as received or reconstructed when the SDR representation image is transmitted and received), as described below.

[0050] It is possible that there are several variations of the display adjustment, and if all of them function with the present innovative technical features of this patent application, to keep the complex explanation simple, it is assumed that the specific display adjustment mechanism used is the one standardized by the applicant Koninklijke Philips together with Technicolor of ETSI TS 103 433-2 V1.1.1 (2018-01), and for the purpose of that, the overview is again briefly described in a somewhat more general way.

[0051] The idea is that, for example, content creators such as human color graders who determine at least one of the mapping curve shape FL_50t_1 or the master HDR and SDR graded images do not have much time (if the grader creates only two images, the applicant can use a technique to derive how the luminance of the SDR image is related to one of the HDR images by the automatically derived FL_50t_1 function, but since the details are not sufficiently relevant to this description in this case either, the grader, for example, uses a color processing user interface tool to draw the shape of the FL_50t1_1 curve, checks whether this actually realizes the SDR image with the correct look, and otherwise changes the function again until a shape that generates the desired SDR image output is obtained, which a skilled reader will understand).

[0052] Therefore, content creators do not want to create many different re-grading curves (for example, after creating a master HDR image of the highest quality at 5000 nits, they do not want to create a function for a re-grading method that optimally re-grades to 2000 nits and another function for a re-grading method to 1000 nits, because those functions operate on the same master HDR image with the same image objects that require very similar luminance re-grading, so those two functions usually look relatively similar, and the mapping curve from 5000 to 2000 performs a "slightly weaker" normalized luminance shift than a mapping from 5000 to 1000, etc.).

[0053] Therefore, in theory, there are many things that can be said about all types of re-graded functions and images. However, from a technical and practical perspective, when using an automatic display adjustment algorithm that automatically calculates the intermediate function for the mapping from v_HDR to v_MDR (similar to the first exemplary display-adjusted function F_DA50t6_1 for creating a 600 nit MDR image of PB_C), it has been argued that for most, if not all, HDR images, a sufficiently good quality intermediate dynamic range (MDR) image can be obtained for the PB_D display between 100 nit and PB_C_master_HDR, which is 5000 nit in this example. This alleviates the grader's workload because it can be seen that the grader only needs to create a single HDR-to-SDR mapping function, such as the first SDR mapping function FL_50t1_1, for example (since a cave requires a very different lumamapping function shape than, for example, a barbershop with blue lighting panels, and thus different typical image situations will naturally require different functions).

[0054] The display adjustment unit 401 (usually an integrated circuit that performs technical numerical operations as described by the ESTI standard, or an equivalent system) then derives, based on the above input luma mapping function and the PB_D value (600 nits in this example), the required first exemplary display-adjusted function F_DA50t6_1 for mapping the HDR luma to the MDR luma of the appropriate 600 nit PB_C (i.e., accurately taking into account the need for a specific luminance re-grading of the current image as transmitted by the content creator as the first SDR mapping function FL_50t1_1 transmitted together with the coded HDR image in the metadata as received by the unformatter 206). Similarly, for other images, if the second SDR mapping function FL_50t1_2 has a different shape (and usually the same PB_D value) as shown, as a result, a properly optimized second exemplary display-adjusted function F_DA50t6_2 that has the same general shape but is somewhat "weaker / intermediate" in the appropriate amount resulting from the difference between PB_C_master_HDR and PB_C_MDR = PB_D_available display (indicating the need for specific re-grading of different regions or objects in the image, such as a dark shadowed corner and an area on a table under strong illumination) is obtained.

[0055] Figure 4B shows how such automatic display adjustment typically works properly. For example, for any point on the diagonal corresponding to some input luma such as the first input luma VnH1 or the second VnH2, the input function, in this example the second SDR mapping function FL_50t1_2, is followed along a scale direction orthogonal to the diagonal until it enters. This point corresponds to the function that needs to regrade the master HDR input luma to the corresponding 100 nit output luma. In this scale (which is inclined 45 degrees to the left compared to the upper vertical direction), it corresponds to the 100 nit position. In this example, the PB_C_master_HDR position at 5000 nit corresponds to the diagonal (starting) position (skilled readers will understand that the regrading from Ln_5000 to Ln_5000 corresponds to the identity transformation). Here, if a scale such as a logarithmic scale is defined for the intermediate positions where there are large intervals starting from the 100 nit position on the input luma mapping curve and small intervals towards the diagonal, this scale can identify all positions of the desired PB_C, such as 600 nit, and thus generate all points of the curve with the shape of the required second exemplary display-adjusted function F_DA50t6_2 shown other than by the dashed line. Thereafter, this ("new" / optimized) curve shape can be sent to the luma multiplier determiner 304, and then, using the color calculation engine shown in Figure 3, all pixel luma for the 600 nit MDR image starting from the master_HDR image pixel color as input can be calculated.

[0056] This system operates very appropriately and has been executed at a satisfactory level many times for many types of HDR content. However, the accompanying problem is that due to the characteristics of the automatic fixed algorithm, it is somewhat static. In addition to the transformation selected for illustration, for example, other metrics, that is, along the distance between the diagonal and the input luma mapping function position, or other transformations using different arrangements of 100 PB_C_master_HDR positions in directions other than 45 degrees of the scale axis distributed along the diagonal, or transformations with more advanced changes to the display adjustment algorithm may be used. However, because the in-operation reconfiguration seems somewhat cumbersome, any such selected algorithm is usually incorporated into the integrated circuit of the receiving device that performs display adjustment, and the result of the display-adjusted function is always "fixed" as it is. This may lead to the inconvenience that for at least some customers, in at least some of the possible display peak brightness ranges, for example, PB_D < 350 nit, some of the images are still recognized as being too dark or having too little contrast, etc. It is important to understand that the grader wants to maintain the luma mapping function that has not been changed as a technical framework constraint. When a brighter image is desired, the content creator may claim that they should start from the steeper function FL_50t1_1. For example, assume temporarily that there are no other problems on a bright display (e.g., PB_C > 2500) that may be slightly or slightly more than too bright, or no other problems near the upper part of the display area.

[0057] However, content creators may claim that the above function is their "gold standard." The overall specification for the entire set of all MDR images on which all secondary processing is based is too important to be modified or is simply considered "correct" (i.e., perhaps it may be recognized that a particular display under certain conditions is slightly too dark, but that does not particularly mean that the reference display of the 100 nit image is incorrect. On the contrary, the 100 nit image itself, i.e., the image defining the content, is not incorrect, nor is the method by which the content creator optimized this image). Therefore, in practical terms, it is desirable to leave all incoming image information, including the SDR mapping function received in the SEI metadata for each consecutive video image, unmodified while enabling adjustment or improvement of MDR image generation by simply changing the display adjustment on the receiving side.

SUMMARY OF THE INVENTION

[0058] A practical approach to solve the static characteristics of prior art display adjustments and achieve some additional customizability on the receiving side is a connection part (501) to a provided or connectable video decoder (207), wherein the video decoder receives a high dynamic range encoded image (Im_COD), the high dynamic range encoded image is encoded according to a first maximum codable luminance (PB_H), the video decoder receives metadata specifying at least one luma mapping function (F_ct, FL_50t1_1), the at least one luma mapping function specifies an offset of the luminance of a secondary image corresponding to the high dynamic range encoded image by comparing with the luminance at the same pixel position when encoded in the high dynamic range encoded image, the secondary image has a second maximum codable luminance (PB_S) smaller or larger than the first maximum codable luminance (PB_H), and outputs a decoded high dynamic range image (Im_RHDR) and the luma mapping function; A display adjustment unit (401) that receives a value of the maximum display luminance (PB_D) that a specific display can display as the brightest pixel color and an input luma mapping function, where the display adjustment unit applies an algorithm that calculates at least one adjusted luma mapping function based on the input luma mapping function and the display maximum luminance (PD_D), and this at least one adjusted luma mapping function corresponds in shape to the input luma mapping function, but depending on the difference between the value of the display maximum luminance (PB_D) and the first maximum codable luminance (PB_H) related to the difference between the second maximum codable luminance (PB_S) and the first maximum codable luminance (PB_H), exists close to a diagonal line that increases by 45 degrees in the graph of the input luma mapping function on a perceptually uniform axis, characterized by the display adjustment unit (401), and an image pixel luminance adjustment device comprising The image pixel luminance adjustment device comprises an alternative luma mapping function determination unit (502) that determines an alternative luma mapping function (ALT_FL_50t1_1). The display adjustment unit (401) comprises a combination unit (503) that combines the at least one luma mapping function (F_ct, FL_50t1_1) and the alternative luma mapping function (ALT_FL_50t1_1) to obtain a combined luma mapping function (CMB_FL_50t1_1), and the display adjustment unit applies its algorithm to the combined luma mapping function as the input luma mapping function. The image pixel luminance adjustment device comprises a luma mapping unit (510) that receives the pixel luma of the decoded high dynamic range image (Im_RHDR) and applies the adjusted combined luma mapping function (ADJ_F_DA50t6_1) to those pixel luma to obtain the output luma of the output image (Im_DA). The image pixel luminance adjustment device includes an output image or a video transmission cable or a wireless channel, a display can be connected to the output image or the video transmission cable or the wireless channel, and further includes an output signal formatter (230) for transmitting the output image (Im300nit), which is realized by an image pixel luminance adjustment device (500).

[0059] First, regarding the details of the display adjustment unit's algorithm and hardware, those with skills can understand that there are several alternative methods that can be calculated based on the determination of whether the resulting display-adjusted luma mapping function is close to or far from the re-grading luma mapping function from HDR to SDR, depending on where the value of the display maximum luminance (PB_D) exists, which the display adjustment unit (401) needs to calculate the optimally adjusted image and its pixel luminance or luma between the first maximum codable luminance (PB_H) and the second maximum codable luminance (PB_S). Furthermore, there are several variations in how accurately (substantially) perceptually uniform luma is defined, and although functions like the above are approximately logarithmic as characteristics, as shown in the example, the parameters of the logarithmic function can be changed and the algorithm still operates similarly and generates an image with a good look. Therefore, those with skills understand that the above details do not form the essence of this innovative proposal and can still be changed while realizing a device that can be regarded as of the same kind. Additionally, those with skills understand that the device can be a stand-alone, i.e., an image color optimization device that is operatively connected to an individual video decoder (e.g., a linear RGB representation calculated using a well-known matrix based on PQ-encoded R, G, B non-linear component values, or, for the sake of brevity, assuming it is YCbCr for all data, i.e., image pixel color data and metadata including at least one luma mapping function 'used' by the final luma mapping calculation algorithm of the display adjustment unit and decoded to obtain and supply as needed), or the device can be a complete system that includes everything (e.g., all when embodied as a TV display, etc.).The secondary (graded reference) image is the image at the other end of the desired dynamic range to be covered. Thus, for example, in the case of a PB_C master HDR image of 4000 nits, it is usually a PB_C SDR image of 100 nits, and it doesn't matter which of the two is actually transmitted (the display-adjusted final MDR image can also be calculated starting from the SDR image, but for simplicity, in this description, unless there is an extrapolation to a slightly higher dynamic range that is better than the initially created image, it is assumed that it is calculated to an HDR image, i.e., a low dynamic range (or at least the maximum luminance assuming the minimum black of the input and output images is the same)). Usually, it makes sense for the two reference gradings of the content creator, i.e., their PB_C values, to differ by at least a factor of 4. Otherwise, in principle, the same principle can be applied even if there is a slight difference between various image gradings, but there is little meaning in performing at least technically high-quality display optimization. The display maximum luminance is also input to the display adjustment unit (401) in various ways depending on the technical implementation variations in this case. For example, a set-top box to which one of various displays is connected may perform polling, and then that display may return the PB_D value to the STB before starting color optimization and image or video output for the connected TV display (or the user may input a value that they think it is or a value that works at least well for their TV via an STB user interface or the like). On the other hand, when the device is the TV itself, its unique PB_D value (for example, the PB_D value related to the backlight behind the LCD display panel, or a value that the TV manufacturer considers safe to use without overheating an OLED panel, etc.) may already be stored in non-modifiable memory inside the display adjustment unit (401), i.e., compared to a general and merely schematic description when the display adjustment unit 401 receives the PB_D value from itself. Also in this case, such variable aspects are not actually important for identifying the case where the device is of the kind invented and described here.

[0060] The alternative luma mapping function determination unit (502) may use a number of (simple) fixed strategies to determine an excellent alternative luma mapping function (ALT_FL_50t1_1). For example, for many applicants of the present technology, something as simple as brightening the dark part, such as the Para Shadowgain exemplified in FIG. 7 (where "Para" is the name of a specific luma mapping curve having a linear slope at both the dark end 0,0 and the bright end 1,1 and having a smooth parabolic connection of those linear parts in the middle), or a simple contrast correction, etc. is already sufficient. Currently, instead of doing it according to PB_D, that is, "to the content" (i.e., to the pair of HDR and SDR reference gradings at both ends and the luma mapping function connecting them), control can be executed according to which part of the entire range of the PB_D-compatible display requires some correction. It is advantageous to perform the adjustment before the fixed display adjustment rather than later (for example, both the Para received from the content creator in the metadata of the transmitted signal and the alternative Para proposed by the receiving side, i.e., for example, by the TV itself, may be further display-adjusted, and the display-adjusted Para may be corrected using the display-adjusted alternative). Doing it beforehand is because there is still a function to pass through the logarithmic positions on various PB_D scales that change the closer SDR when, for example, many adjustments are required and the high PB_D display has "almost perfect HDR" in terms of the standard display-adjusted behavior.

[0061] Readers with skills will not find it too difficult to understand the case where several functions may exist (as usual, the claims should be read regarding the execution of operations set for the least detailed embodiments, i.e., for a specific single display of PB_D that needs to be supplied together with the display-adjusted MDR image, in order to derive the only final optimal display-adjusted luma mapping function that is ready to be loaded or applied in the color processing unit to obtain the highest MDR image corresponding to the HDR master image in terms of look (i.e., the relative luma position and overall color impression of various image objects), a single image is processed using a single original luma mapping function derived by the content creator and a single alternative luma mapping function.

[0062] Multiple different luma mapping functions can be created for the sequential images to be presented sequentially in time, and then the display adjustment unit creates several sequential display-adjusted luma mapping functions corresponding to each of the input luma mapping functions created by the content creator. The display adjustment unit further creates several output display-adjusted luma mapping functions for a single input luma mapping function when it is necessary to supply two output video streams, for example, supplying one of the two output video streams to a high-quality 1500 nit PB_D HDR display and simultaneously supplying the other to a portable display where other family members are watching the program in the kitchen.

[0063] Somewhat complex to understand (although it is not necessary to understand the present technical proposal by itself, at least from the perspective of patent acquisition, but it is valid for the sake of completeness) is that the lum mapping function consists of a number of partial lum mapping functions defined to be applied continuously. For example, the content grader applies Para to perform a rough grading to roughly balance the dark and bright areas. That is, when mapping a certain scene, for example, a scene that is usually 100 times darker outdoors by nature, to two very differently illuminated areas such as indoors that is probably 10 times darker in the HDR reference grading depending on how the content creator mapped the actual situation to their master HDR image, Para can be used to brighten the relatively dark parts compared to the bright parts, thereby generating some contrast in the bright areas, which is an excellent and simple method for creating a low dynamic range version of a high dynamic range image (moreover, it is often effective for most of the already very good content). However, outdoors, there may be cases where there is a commercial billboard embodied as white text (sandblasted) attached to a glass panel. If the contrast is reduced by an overly simplistic approach, such as a small upward slope of Para that is necessary to create a margin in the small luminance range of the SDR second reference grading for all dark object / pixel luma, the readability of the above text may decrease. According to the applicant's principle, the content creator / coder can solve this by applying, after Para, a customizable curve (CC) that realizes a higher contrast again precisely around the luma positions of the whiteish colors surrounding, behind, or reflecting the white text and the glass, thereby making the text highly readable again (note that the differences in calculation accuracy and coding word length, etc. also improve, but such details do not need to be explained here).

[0064] Regarding the new teachings herein, while the continuity of a number of luma mapping curves can be understood to be, in itself, again a luma mapping curve, there may be cases where it may be appropriate to consider that there is only one "complete" curve (in fact, if the reader wishes to keep things simple for the understanding of this patent application, the grader may assume that only Para is used without CC). However, not only for the complete mapping function, but also with specific calculations, it is possible to perform display adjustments on the partial luma mapping curves themselves. If the reader is interested in this, reference may be made to ETSI TS 103 433-2 V1.1.1 (2018-01), paragraph 7.3, "metadata recomputation".

[0065] Regardless of how the above details are implemented, this innovative improvement relates to having a very excellent adjustment mechanism for what is preferably realized relatively statically (as described above) due to technical framework limitations.

[0066] In a more advanced embodiment, the function determination unit 502 can analyze various details of the input image (such as the overall luminance dispersion characteristics of things like integrated derivatives, texture characteristics, etc., segmentation into various regions, analysis of geometric structures and contrast scales), and from this, derive adjustments configured as alternative luma mapping function shapes.

[0067] The combination unit 503 can further apply combinations in some variations that can be understood by a reader with skills based on the examples in this specification. Although such simple linear weighted combinations are often sufficiently good in practice (engineers prefer simpler variations that require fewer transmitters and power, but alternative embodiments can of course also be derived), the more interesting part is to control how this combination depends on the situation of PB_D. Regarding the combination, it does not mean the broadest possible concept such as only a part of a curve being exchanged with a part of another curve. Instead, all or most of the luma obtains a new curve output that depends on both an output composed of the first curve and an output composed of the second curve or a regulation.

[0068] The image pixel luminance adjustment device (500) causes the combination unit to determine that the combination luma mapping functions are more similar in shape to the respective alternative mapping functions, and excellent advantages are obtained when at least one of the luma mapping functions depends on the value of the display maximum luminance (PB_D). Thereafter, for example, more mixing of alternative behaviors for modifying a particular relative luma rearrangement to obtain a smaller PB_D value can be controlled, and the reader can understand that it is desirable to use a complex configuration of what is specifically done in which situation and execute it in several ways (for example, the alternative luma function corrects for two luma rearrangement modes, and the first luma rearrangement mode is controlled to be at a first angle that is not higher than the PB_D_limPB_D value only within a specific partial range of the entire PB_D range to be processed, and the second mode, which usually corresponds to other luma sub-regions, is controlled in a different way, that is, controlled to have a different PB_D-dependent combination behavior, etc.). For example, the combination unit can check the PB_D value and then any combination mapping function behavior determination it applies, and if the PB_D value is within a specific percentage deviation from the maximum PB_D (for example, PB_HDR of the input image), then perhaps slightly disrupt the original function according to the shape of the alternative function, for example, check whether it stays within a specific band around the first function. For PB_D values less than PB_D1, a serious deviation may start, and values less than PB_D2 mostly follow the shape of the alternative function. Other algorithms resulting in such PB_D-dependent function combination behaviors are also possible. This makes it more possible to control the difficult task of displaying a high dynamic range image on a display with a low dynamic range, actually having only the display peak luminance.

[0069] A variation of the image pixel luminance adjustment device 500 that is practically simple but operates well has a combined luma mapping function determined by the combining unit by linear weighting for each luma value defined as CMB_FL_50t1_1(Vn)=(1 - A)*FL_50t1_1(Vn)+A*ALT_FL_50t1_1(Vn), where Vn is a perceptually uniform luma representation of the pixel luminance that can be applied by applying a logarithmic function to the luminance, A is a weighting value between zero and one, A equal to zero is set below the low display maximum luminance (PLOW), A equal to one is set above the high display maximum luminance (PHIG), and a function is applied to set A equal to a value between zero and one when the display peak luminance is between the low display maximum luminance (PLOW) and the high display maximum luminance (PHIG) according to a predetermined weighting profile shape, and the value of A is derived based on the value of the display maximum luminance (PB_D). The values of PLOW and PHIG are then fixed or optimized in an intelligent manner depending on what gives good results, either generically with respect to the average of the entire image or for a specific type of image (e.g., classified based on the details of the luma histogram such as mostly bright with dark small objects, i.e., an image with a small number of dark pixel luma and mostly dark).

[0070] The reader will understand that other configurations of adjustments as alternative luma mapping functions and weight definitions are possible.

[0071] In such cases, the function determination unit 502 (in relation to what the combining unit executes) overall considers that the alternative function has mostly the shape required to perform a proper HDR to MDR re - grading. In many cases, the method is applied in any case to perform some fine - tuning. For (almost) HDR to MDR images, the effect of the display adjustment itself ensures correct behavior (near the diagonal) due to the logarithmic nature of the scale, regardless of the alternative used for each final function.

[0072] A practical and simple embodiment of the image pixel luminance adjustment device may use a shape that linearly increases between zero and one when defined on the input axis measured in the perceptually uniform luma representation as a predetermined weighting function shape. Also in this case, one of several (mostly similarly acting) perceptually luma representations may be fixed, and the device functions in a similar manner, and the selection depends on variables that are not important in this patent application (therefore, without intending to limit, the reader may assume that the representation of luminance as perceptually uniform luma is calculable by the exemplary formula 1).

[0073] The at least one luma mapping function (F_ct, FL_50t1_1) is at least partially defined using a luma mapping function comprising a first linear segment for the darkest part range of the total input luma range, the linear form being satisfied in the perceptually uniform luma representation, a second linear segment for the lightest part range of the total input luma range, and a non-linear non-decreasing segment for the intermediate part range between the darkest part range and the lightest part range, connecting at both ends to the first linear segment and the second linear segment. A specific embodiment of the image pixel luminance adjustment device (500) according to any one of claims 1 to 4, wherein the alternative luma mapping function (ALT_FL_50t1_1) includes at least a first alternative linear segment of the darkest part range having a slope different from the slope of the first linear segment of the darkest part range of the at least one luma mapping function, already achieves sufficient adjustment of the display adjustment quality for many customers and / or market situations. Therefore, that is, due to the fact that the adjusted Para performs a rough balancing, that is, a relative rearrangement of the partial ranges, and thus shifts to a new luma partial range compared to the originally specified partial range, when considering the detailed grading CC second partial luma mapping curve for grading in detail the luma of the same object, only the rough Para is corrected.

[0074] The luma at the ends of the darkest (and brightest) linear segments may be the same as or different from the original author's Para and the alternative Para.

[0075] Note that (even when transmitted as a corresponding SDR image), the first maximum luminance of PB_H of the master HDR image is usually further transmitted, and the SDR maximum luminance may be fixed in advance as described above, usually equal to 100 nit, but may be changed and transmitted. It should be noted that the present embodiment operates in the same manner.

[0076] Various technical realizations are Receiving a coded high dynamic range image (Im_COD), the coded high dynamic range image being coded according to a first maximum codable luminance (PB_H), and receiving metadata specifying at least one luma mapping function (F_ct, FL_50t1_1), the at least one luma mapping function specifying an offset of the luminance of a secondary image corresponding to the coded high dynamic range image compared to the luminance at the same pixel position when coded in the coded high dynamic range image, the secondary image preferably having a second maximum codable luminance (PB_S) that is at least 4 times smaller or larger than the codable luminance (PB_H); Decoding the coded high dynamic range image (Im_COD) into a decoded high dynamic range image (Im_RHDR); In the display adjustment step, receiving the value of the maximum displayable luminance (PB_D) as the brightest pixel color of a specific display and the luma mapping function, where the display adjustment step applies an algorithm for calculating at least one adjusted luma mapping function based on the luma mapping function and the display maximum luminance (PD_D), and this at least one adjusted luma mapping function corresponds in shape to the input luma mapping function, but exists close to a diagonal line that increases by 45 degrees of the graph of the input luma mapping function on a perceptually uniform axis, and the proximity to the diagonal line corresponds to the difference between the display maximum luminance (PB_D) and the first maximum codable luminance (PB_H) related to the difference between the first maximum codable luminance (PB_H) and the second maximum codable luminance (PB_S), including the step of, a method for pixel luminance adjustment, The method for pixel luminance adjustment includes the step of determining an alternative luma mapping function (ALT_FL_50t1_1). The display adjustment step includes combining the at least one luma mapping function (F_ct, FL_50t1_1) and the alternative luma mapping function (ALT_FL_50t1_1) to obtain a combined luma mapping function (CMB_FL_50t1_1), and the display adjustment step applies its algorithm to the combined luma mapping function as the input luma mapping function to generate an adjusted combined luma mapping function (ADJ_F_DA50t6_1). Receiving the pixel luma of the decoded high dynamic range image (Im_RHDR) and applying the adjusted combined luma mapping function (ADJ_F_DA50t6_1) to those pixel lumas to obtain the output luma of the output image (Im_DA). Outputting, on an image or video transmission cable or wireless channel to which a display is connectable, an output image (Im3000nit) obtained by applying the adjusted combined luma mapping function to the pixel luma of the decoded high-dynamic range image (Im_RHDR). A method for adjusting pixel luminance may be embodied as such.

[0077] In a general-purpose image pixel luminance adjustment method, further, the combined luma mapping function is determined to be more similar in shape to the alternative mapping function, respectively, and the at least one luma mapping function depends on the value of the display maximum luminance (PB_D).

[0078] In the image pixel luminance adjustment method as described above, the combined luma mapping function is determined by linear weighting for each luma value defined as CMB_FL_50t1_1(Vn) = (1 - A)*FL_50t1_1(Vn) + A*ALT_FL_50t1_1(Vn), where Vn is a perceptually uniform luma representation of pixel luminance that can be applied by applying a logarithmic function to the luminance, A is a weight value between zero and one, A equal to zero is set below the low display maximum luminance (PLOW), A equal to one is set above the high display maximum luminance (PHIG), and A equal to a value between zero and one is set when the display peak luminance is between the low display maximum luminance (PLOW) and the high display maximum luminance (PHIG) by applying a function that derives based on the value of the display maximum luminance (PB_D).

[0079] In the image pixel luminance adjustment method, the predetermined weighting function shape is a linearly increasing shape between zero and one when defined on the input axis measured in the perceptually uniform luma representation.

[0080] In the method for adjusting the luminance of image pixels, the at least one luma mapping function (F_ct, FL_50t1_1) includes a first linear segment for the darkest part range of the total input luma range, the linear form being satisfied with a luma representation in which the perception is made uniform, a second linear segment for the brightest part range of the total input luma range, and a non-linear non-decreasing segment for the intermediate part range between the darkest part range and the brightest part range, which is connected at both ends to the inner ends of the first linear segment and the second linear segment. The method is at least partially defined using a luma mapping function, and the alternative luma mapping function (ALT_FL_50t1_1) includes at least a first alternative linear segment of the darkest part range having a slope different from the slope of the first linear segment of the darkest part range of the at least one luma mapping function.

[0081] The maximum codable luminance means the physical luminance corresponding to the maximum codable pixel color, i.e., the highest luma code (e.g., 1023 for 10 bits), i.e., the actual luminance of the whitest white when ideally displayed on any display, and may be displayed on a corresponding virtual display associated with the image. That is, the virtual display indicates that the brightest codable luminance of the image is, for example, 1200 nit, and ideally, if the actual receiving-side (e.g., consumer) display has a maximum displayable white of 1200 nit or more, such a display needs to render the highest luma code colorless pixels with a display luminance of 1200 nit. Therefore, the function transmitted together can map the highest image luma below the highest possible displayable luminance, for example, by using a larger partial range for darker colors than for brighter colors, to show how such image luminance (typically, actually luma) needs to be displayed on a display with a smaller maximum luminance characteristic, such as a 600 nit display. The content creator optimization function shape can have, for example, the content creator can configure the relative luminance positions of various image objects differently according to the maximum codable luminance PB_C, such as 4000 nit PB_C, which can make the illumination significantly brighter compared to the non-emissive image object pixel luminance of PB_C of 900 nit. Further, this can be regarded as the pixel luminance of one object, such as illumination, offset to a new relative position compared to, for example, the indoor chair object pixel luminance, on an axis normalized to 1.0 luminance (or luma, according to the EOTF defining the luminance corresponding to various luma codes) for various PB_C image codings.

[0082] The above and other aspects of the method and apparatus according to the present invention will be apparent from the implementations and embodiments described below, and will be described with reference to the following, and further with reference to the accompanying drawings, which serve only to illustrate a non-limiting specific figure of a more general concept, and in the accompanying drawings, dash symbols are used to indicate that a component is optional, and a component without a dash symbol is not necessarily essential. The dash symbol can be further used to indicate an element hidden inside an object although it is described as essential, or for intangible things (and how they are displayed on a display), such as, for example, the selection of an object / region.

[0083] The drawings will be described below.

Brief Description of the Drawings

[0084]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Best Mode for Carrying Out the Invention

[0085] FIG. 5 generally shows an integrated circuit or a similar unit constituting an improved display adjustment unit of the image pixel luminance adjustment device 500 of the present patent application.

[0086] As already described above, the novel image pixel luminance adjustment device 500 includes an alternative luma mapping function determination unit 502. Depending on the device in which it exists (e.g., depending on a set-top box that prepares an image for a specific TV or the TV itself, etc.), this luma mapping function determination unit can configure an alternative luma gradient function shape ALT_FL_50t1_1 within a range from a function that does not depend, or hardly depends, on either the content of the image or the SEI transmission luma adjustment function of the content creator in various ways, or this function can mostly follow the shape of the SEI transmission luma adjustment function. For example, in the v_input, v_output graph, it can be slightly higher, or have some shape disturbance in a small partial range of v_input. Only one aspect can be (slightly) changed.

[0087] This can be a fixed amount designed by the technology provider or device manufacturer, or can be a variable amount that may be determined during execution, such as for each image.

[0088] The alternative luma mapping function (ALT_FL_50t1_1) and the original luma mapping function (FL_50t1_1) received as is and determined by the content creator as a function indicating the need for re-grading (usually extracted by the video decoder 207, and in this case the video decoder is included in the device 500 or at least connected to the device 500 during operation so that the function can be received via the input 501) are combined in some way (in the combining unit 503) to generate a combined grading function CMB_FL_50t1_1. The combined grading function CMB_FL_50t1_1 performs a part of both functions according to the need for display adjustment, that is, usually according to a specific value of PB_D of the connected display. Note that the input 501 may be twice the input for the decoded image Im_RHDR, or may be an individual input for that image from the provided decoder or a connectable decoder.

[0089] In the above example, the original function is of a rather rough shape (e.g., pure Para, etc.), and it can be seen that it almost realizes an increase in the relative luminance of the darkest luma (for a low PB_C MDR image or, more precisely, for SDR grading). The above alternative function realizes some contrast intensity in the perceptually uniform input partial range MR because, for example, there are important objects that do not easily realize good re-grading on the darkest, i.e., the lowest PB_D, display. Therefore, since the standard display adjustment algorithm is that of the static and known original image, it can be seen that the above behavior leaks through the adjusted combined luma mapping function (ADJ_F_DA50t6_1), thus leading to the necessary adjustments as required.

[0090] Finally, the adjusted combined luma mapping function ADJ_F_DA50t6_1 is used (as an input) by the luma mapping unit 510, and the luma mapping unit 510 uses this specification to map the luma of the reconstructed / decoded HDR image Im_RHDR to the optimally corresponding, display-adjusted output luma of the display-adjusted image Im_DA, and the mapping result can be transmitted to any image or video output according to the specific technical implementation of a specific image or video processing device or system.

[0091] Figure 6 generally shows a useful method of weighting two luma functions for each luma. For example, for a specific PB_D value equal to PB_D1, the resulting weighting factor A for adjustment is A = 0.6.

[0092] For the perceptually uniform total luma (Vn) normalized to 1.0 that can occur in the input image, thus the combined function can be calculated as follows. CMB_FL_50t1_1(Vn;PB_D1)=0.6*FL_50t1_1(Vn)+0.4*ALT_FL_50t1_1(Vn)

[0093] This combined function can then be input into the standard algorithm for display adjustment, seemingly different but as if it were the original content creator's lum mapping function (i.e., as if performing standard display adjustment).

[0094] Finally, this function is applicable in the color processing core, such as those described in Figure 3 for example, whereby the output lum of the MDR image is obtained based on the input lum of the HDR image (or something that embodies calculations starting from an equivalent SDR image and then embodied by a combined lum mapping function with a different shape).

[0095] Figure 8 shows some examples where several luminance values overlap. Usually, for the determination of the PB_D value, the scale terminates at a PB_C value of the content, such as 4000 nits for movies in this example. Since some content has low importance and is appropriately graded on a large subset with a displayed PB_D, and other content starts to show some problems later, etc., the weight factor determination curve changes according to, for example, the type of image content being processed (e.g., the first weight determination curve WP1 and the second weight determination curve WP2). Although an example was described using the logarithmic definition of a perceptually uniform lum representation, the definition of the adjustment, especially the specification of the weight determination function, can also work with other non - linear representations.

[0096] The horizontal axis is also explained as the relative value of PB_D (i.e., PB_D / PB_H) in a logarithmic system defined by the maximum PB_H (i.e., using, for example, Equation 1 or a similar perceptually uniform equation).

[0097] When referring to the shape of a function or the function shape, it means the locus of output points for various input points controlled by shape control parameters such as the values a, b, and c in the equation y_out = a*x^2 + b*x + c, i.e., a parabolic convex shape for example.

[0098] FIG. 7 shows how the luminance behavior of the receiver's display adjustment behavior is appropriately and simply affected by using an alternative Para-based adjustment.

[0099] The original Para of the content creator received as metadata has a dark segment SD that terminates with Luma Lsd, a bright linear segment SB that starts with Lsb, and an intermediate (parabolic) segment SM that controls the gray scale of the intermediate range Luma.

[0100] For that alternative dark segment SDA (or, more generally, 1. times, or even 2. times, etc.), an alternative grayscale Para having a slope SL that is, for example, 1.3 times steeper than the slope SL of the creator's Para (known as Shadowgain) is generated by the alternative Luma mapping function determination unit 502. The rest of this function is different from the pure Para (e.g., the Para before any customized CC curve shape), but the alternative Luma mapping function determination unit 502 can determine, for example, the highlight gain of the alternative brightest segment SBA, which mostly corresponds to the slope of the SB region, while leaving room for the darkest output Luma in conjunction with the determination of the alternative intermediate region re-grayscale behavior of segment SMA. Other Para function shape control parameters, such as the highlight gain, may be equal for the alternative Luma mapping Para and the original Luma mapping Para or different Luma mapping Paras.

[0101] The components of the algorithms disclosed herein may be implemented (in whole or in part) in execution as software that operates on hardware (e.g., part of an application-specific IC) or a dedicated digital signal processor or a general-purpose processor.

[0102] From this description, a person with skills should be able to understand which components are optional improvements and can be implemented in combination with other components, and how each (optional) step of the method corresponds to each means of the device, or how each means corresponds to each (optional) step of the method. The term "device" in this application is used in the broadest sense, that is, a group of means enables the realization of a specific purpose, and thus, for example, it can be an IC (a small circuit part thereof), a dedicated device (such as a device with a display), or a part of a network-connected system. "Configuration" is also intended to be used in the broadest sense, so it may particularly include a single device, a part of a device, an aggregate of cooperating devices (parts thereof), etc.

[0103] It should be understood that the meaning of a computer program product encompasses any physical realization of a collection of commands that, after a series of loading steps (which may include intermediate conversion steps such as conversion to an intermediate language and a final processor language) for inputting commands to a processor and executing any of the characteristic functions of the present invention, enables the use of a general-purpose or dedicated processor. In particular, a computer program product may be realized as data on a carrier such as a disk or tape, data existing in memory, data moving through a wired or wireless network connection, or program code on paper. In addition to the program code, characteristic data necessary for the program may be further embodied as a computer program product.

[0104] Some of the steps necessary for the operation of this method may already exist in the function of the processor instead of being described in a computer program product, such as data input and output steps.

[0105] Note that the above-described embodiments are to be noted for explaining the present invention without limitation. If those with skills can easily realize the association of the presented examples with other areas of the claims, not all options are described in detail for the sake of brevity. In addition to the combinations of the elements of the present invention as combined in the claims, other combinations of the elements are possible. Any combination of the elements can be realized in a single dedicated element.

[0106] The reference signs in parentheses in the claims are not intended to limit the claims. The term "comprising" does not exclude the existence of elements or aspects not listed in the claims. The singular form of an element does not exclude the existence of a plurality of such elements.

Claims

Claim 1 A connection to a provided or connectable video decoder, wherein the video decoder receives a high-dynamic-range encoded image, the high-dynamic-range encoded image is encoded according to a first maximum encodable luminance, the video decoder receives metadata specifying at least one luma mapping function, the at least one luma mapping function specifies an offset of the luminance of a secondary image corresponding to the high-dynamic-range encoded image from the luminance at the same pixel position when encoded in the high-dynamic-range encoded image, the secondary image has a second maximum encodable luminance, and the video decoder outputs a decoded high-dynamic-range image decoded from the high-dynamic-range encoded image and the luma mapping function; a connection part A display adjustment unit that receives a value of a display maximum luminance that a specific display can display as the brightest pixel color and an input luma mapping function, the display adjustment unit applies an algorithm that calculates at least one adjusted luma mapping function based on the input luma mapping function and the display maximum luminance, and the at least one adjusted luma mapping function corresponds to the input luma mapping function in shape, but first, a ratio between the value of the display maximum luminance and the first maximum encodable luminance, and second, a difference between the second maximum encodable luminance and the first maximum encodable luminance, and the at least one adjusted luma mapping function exists close to a diagonal line that increases by 45 degrees of the graph of the input luma mapping function on a perceptually uniform axis; a display adjustment unit, an image pixel luminance adjustment device comprising The image pixel luminance adjustment device includes an alternative luma mapping function determination unit that determines an alternative luma mapping function The display adjustment unit includes a combination unit configured to combine the at least one luma mapping function and the alternative luma mapping function to obtain a combined luma mapping function, and the display adjustment unit applies the algorithm to the combined luma mapping function as the input luma mapping function to generate an adjusted combined luma mapping function as the output The image pixel luminance adjustment device includes a luma mapping unit that receives the pixel luma of the decoded high-dynamic range image and applies the adjusted combined luma mapping function to those pixel lumas to obtain the output luma of the output image. The image pixel luminance adjustment device includes an output image or a video transmission cable or a wireless channel, and a display can be connected to the output image or the video transmission cable or the wireless channel, and the image pixel luminance adjustment device is characterized by transmitting the output image. **Claim 2** The combined luma mapping function is determined by linear weighting for each luma value defined as combined luma mapping function (CMB_FL_50t1_1(Vn)) = (1 - A) * luma mapping function (FL_50t1_1(Vn)) + A * alternative luma mapping function (ALT_FL_50t1_1(Vn)), where Vn is a perceptually uniform luma representation of pixel luminance that can be applied by applying a logarithmic function to the luminance, A is a weight value between zero and one, A equal to zero is set below the low display maximum luminance, A equal to one is set above the high display maximum luminance, and a function is applied to set A equal to a value between zero and one when the display peak luminance is between the low display maximum luminance and the high display maximum luminance according to a predetermined weighting profile shape, and is derived based on the value of the display maximum luminance. The image pixel luminance adjustment device according to claim 1. **Claim 3** The at least one luma mapping function is defined at least in part using a luma mapping function comprising a first linear segment for the darkest portion range of the total input luma range, the linear form being satisfied with the perceptually uniform luma representation, a second linear segment for the brightest portion range of the total input luma range, and a non-linear non-decreasing segment for an intermediate portion range between the darkest portion range and the brightest portion range, connecting at both ends with the first linear segment and the second linear segment, wherein the alternative luma mapping function includes at least a first alternative linear segment of the darkest portion range having a slope different from the slope of the first linear segment of the darkest portion range of the at least one luma mapping function. The image pixel luminance adjustment device according to claim 1 or 2.

4. Receiving a high-dynamic-range encoded image, the high-dynamic-range encoded image being encoded according to a first maximum codable luminance, and receiving metadata specifying at least one luma mapping function, wherein the at least one luma mapping function specifies an offset of the luminance of a secondary image corresponding to the high-dynamic-range encoded image from the luminance at the same pixel position as the high-dynamic-range encoded image, and the secondary image has a second maximum codable luminance; Decoding the high-dynamic-range encoded image into a high-dynamic-range decoded image; In a display adjustment step, receiving a value of a display maximum luminance that a specific display can display as a brightest pixel color and the luma mapping function, wherein the display adjustment step applies an algorithm for calculating at least one adjusted luma mapping function based on the luma mapping function and the display maximum luminance, and this at least one adjusted luma mapping function corresponds in shape to an input luma mapping function, but the adjusted luma mapping function exists in proximity to a diagonal line that increases by 45 degrees of a graph of the input luma mapping function on an axis perceptually uniformized with the value of the display maximum luminance, and the proximity to the diagonal line depends first on a ratio between the display maximum luminance and a first maximum codable luminance and second on a difference between a second maximum codable luminance and the first maximum codable luminance, a method for adjusting image pixel luminance, comprising: The method for adjusting image pixel luminance, which is a step of determining an alternative luma mapping function, The display adjustment step combines the at least one luma mapping function and the alternative luma mapping function to obtain a combined luma mapping function, and the display adjustment step applies the algorithm to the combined luma mapping function as an input luma mapping function to generate an adjusted combined luma mapping function, Receiving pixel luma of the decoded high dynamic range image, and applying the adjusted combined luma mapping function to the pixel luma to obtain output luma of an output image, Outputting an output image obtained by applying the adjusted combined luma mapping function to the pixel luma of the decoded high dynamic range image on an image or video transmission cable or wireless channel connectable to a display. A method for adjusting image pixel luminance is characterized by having the above steps.

5. The combined luma mapping function is determined by linear weighting for each luma value defined as combined luma mapping function (CMB_FL_50t1_1(Vn)) = (1 - A) * luma mapping function (FL_50t1_1(Vn)) + A * alternative luma mapping function (ALT_FL_50t1_1(Vn)), where Vn is a perceptually uniform luma representation of pixel luminance that can be applied by applying a logarithmic function to the luminance, A is a weight value between zero and one, A equal to zero is set below the low display maximum luminance, A equal to one is set above the high display maximum luminance, and a function is applied to set A equal to a value between zero and one when the display peak luminance is between the low display maximum luminance and the high display maximum luminance according to a predetermined weighting profile shape, and is derived based on the value of the display maximum luminance. The method for adjusting the luminance of an image pixel according to claim 4.

6. The at least one luma mapping function is at least partially defined using a luma mapping function consisting of a first linear segment for the darkest part range of the total input luma range, where the linear form is satisfied by the perceptually uniform luma representation, a second linear segment for the brightest part range of the total input luma range, and a non-linear non-decreasing segment for the intermediate part range between the darkest part range and the brightest part range that connects at both ends to the inner ends of the first linear segment and the second linear segment. The alternative luma mapping function includes at least a first alternative linear segment of the darkest part range having a slope different from the slope of the first linear segment of the darkest part range of the at least one luma mapping function. The method for adjusting the luminance of an image pixel according to claim 4 or 5.

Images