Automatic Brightness Adjustment for HDR Video Coding
Patent Information
- Application Number
- JP2024543970
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-01-27
- Filing Date
- 2023-01-13
- Publication Date
- 2025-02-19
Smart Images
Figure 2025504906000001_ABST
Abstract
Description
[Technical field]
[0001] In particular, the present invention relates to an apparatus and method for creating a secondary dynamic range image for a primary (or master) dynamic range image that can be used in high dynamic range (HDR) video coding, where at least two different dynamic range images are communicated to a receiver (typically one as an actual pixelated image and the other as data for a computational method to derive the secondary dynamic range image from the actual received image).
[0002] The invention also relates to an apparatus and a method to be used in a device including a high dynamic range image receiver for calculating a secondary dynamic range image according to various technical requirements.
[0003] Such an apparatus or method may be included in consumer devices such as television displays, mobile phones, etc., but also in professional systems such as video communication in commercial applications in stores, etc. On the HDR video production and communication side, it is applied, for example, at the endpoint stations of the deployers of television communication networks that communicate (distribute) HDR video images to end customers, such as in satellite, or cable headends, or mobile phone communication networks, but it may also be used in post, where the video is relayed from a first professional (business) that creates the video, for example, to a second business that distributes the video via a medium, region, or client.
[0004] The present apparatus and method take advantage of advances in machine learning that can learn specific attributes of images, in this case the required luminance remapping function to create a high quality secondary dynamic range image given the specifications of any image or image category, a useful scenario being, for example, the creation of a low dynamic range image for an HDR image. [Background technology]
[0005] High dynamic range video processing (such as coding or display adaptation, which is the remapping of images of a first dynamic range to the specific displayable dynamic range capabilities of any display) is still a fairly recent technology field with some unsolved problems and challenges (in the television world, early versions of it have occurred since 2010). HDR televisions have been on the market for several years now (generally with a maximum displayable luminance, aka peak brightness, of around 1000 nits or Cd / m 2 , i.e., low end around 600nits, high end around 2000nits, and possibly 5000nits or more in the future), but the technology that precedes the display on improved displays (which is a physics and electronics driven problem), i.e., content creation, coding, or color processing, still has some solutions to be invented / improved and deployed. There are some videos that have been created and communicated, and there are some early broadcasts, and while the results are generally great, there is still room for further improvement in some respects, so the technology is not at a very stable stage at the moment.
[0006] High dynamic range images are defined as images that are compared to the status quo of legacy low dynamic range (LDR) images, also known as standard dynamic range (SDR) images, which were produced and displayed in the second half of the 20th century and still dominate television or video distribution, whether through most video technologies, e.g., from terrestrial broadcasting to youtube video feeds over the Internet, etc. The properties of SDR images and how to process them (e.g., to create good looking video images) are well understood. SDR luma codes, typically 8-bit codes ranging from 0 to 255, are suitable for defining various gray levels of objects in relatively uniformly lit environments. With a relative maximum value of 255 representing "white" (which in LDR had no real luminance associated with it, since it was rendered differently depending on the peak brightness of the display purchased, but in the HDR era it is associated with a maximum luminance of ML_LDR=100 nits), in a quadratic fashion, lower levels are progressively darker, i.e., a white percentage of, e.g., 5%, appears black.
[0007] An HDR image is an image that can represent (code) a number of additional luminances (generally additional colors, if also involving a wider range of primaries), in particular a number of additional gray steps above white (at 100 nits). Whether these additional steps involve a larger luma number represented by additional bits (although generally one would like to use 10 bits for the HDR color components) is ultimately determined by which Electro-optical Transfer Function (EOTF) defines which pixel luminance as the amount of luma code available, and since several 8-bit HDR decoders have been demonstrated, there is no absolute requirement that these additional steps involve a larger luma number represented by additional bits.
[0008] Formally, one can define the luminance dynamic range as the span of all luminances from minimum black (MB) to peak white or peak brightness (PB), also known as maximum brightness (ML), so one can have an HDR video with very deep blacks, and one can have an HDR video with very ordinary (LDR) blacks, but with brighter pixels (often called highlights). Thus, in practice, one can define and process, e.g. color process, HDR images primarily based on just one value, the higher peak brightness (usually what a user is most interested in is whether it's a bright explosion or just more realistic specular spots on metals and jewels, etc., and in practice one can argue that minimum black is the same for SDR and HDR images).
[0009] In fact, from legacy times, one can argue that the 1000:1 luminance dynamic range of an SDR image is below 100 nits (and above 0.1 nits), while an HDR image generally has a PB that is at least 5 times brighter, i.e. 500 nits or more (this is where the difference starts to become impressive, making e.g. shiny parts in the image look really beautiful to the user. Obviously, the use of a higher maximum is better and more desirable, so one can start to define an HDR as really good when it has a ML of 1000 nits or more, i.e. generally, in a video, at least some of the pixels are given that luminance to make them look most impressive compared to the rest of the scene).
[0010] Without getting bogged down in a lot of details that are not necessary here, it should be noted that it can be indicated what kind of HDR it has by associating a peak brightness number as metadata with the HDR image, which is seen as the luminance of the brightest pixel present in the image and is often defined formally by associating a reference representation with the image (i.e., associating a virtual representation with the image with the peak brightness of the image that corresponds to the brightest pixel present in the image or video, i.e. the brightest pixel that needs to be displayed, and then coding this PB_C (C stands for "coding") of that virtual representation as metadata that is added to the image pixel color matrix).
[0011] This way there is no need to "inelegantly" code HDR images with an excessive amount of bits, but just reuse existing technology with 10-bit word lengths for color components (12+ bits are already seen as rather heavy by several tech suppliers in the overall video processing chain, at least for the near future, for various reasons; PB_C metadata makes it possible to easily upgrade the HDR framework for future scenarios).
[0012] One aspect that is reused from classical video engineering is that pixel colors are generally always reported as YCbCr colors, where Y is the so-called luma component (which is the technical coding for luminance), and Cb and Cr are the blue and red chroma components that complete the trichromatic additive color definition.
[0013] Luma is defined in terms of the non-linear R'G'B' color components (the primes 'denote the non-linear nature, i.e., the difference from the linear RGB components of a color; in other words, the amount of red, green, and blue photons that emanate from a displayed pixel of a particular color) via the formula shown below: Y=a1 * R'+a2 * G'+a3 *B' [Formula 1]
[0014] The question is how the non-linear R'G'B' relates to (and is in fact defined in terms of) the linear RGB components.
[0015] The so-called electro-optical transfer function (EOTF) or the inverse of the EOTF, the opto-electrical transfer function, which calculates the nonlinear components from the corresponding linear components, e.g. R'=OETF_PQ(R) [Formula 2] You define a coding system by specifying
[0016] For example, in the days of LDR, there was only the standard Rec. 709 OETF, which was the inverse function of the EOTF (omitting some irrelevant details for this patent application), which, as a fairly good approximation, was a simple square root.
[0017] Then, when technical problems appeared for coding a large range of HDR luminance (e.g., 1 / 10,000nit to 10,000nit) with only 10 bits, which is not possible with square root functions, a new EOTF was invented, the so-called Perceptual Quantizer EOTF (US Pat. No. 9,077,994). This is sufficient to "specify" any required HDR pixel color (or its luminance), but for practical video communication systems, it is not always sufficient, since more information about the HDR scene or image (e.g., as defined by the video creator) is generally desired.
[0018] Thus, this different EOTF definition, and the input YCbCr color defined thereby, will obviously have different numerically normalized components depending on whether it is YCbCr_Rec709 or YCbCr_PQ (effectively, this can be seen by rotating the R'G'B' cube on its black vertex, with the achromatic gray y-axes of the different lumas now forming a vertical line; the various image pixel colors will then have different extents along that vertical line). In a technically elegantly simple formulation, assuming the various components are normalized between 0 and 1 (1 meaning peak brightness, i.e., different luminance for the various coding scenarios), computing the secondary image can be seen as offsetting the pixel luminance or luma along its normalized axis.
[0019] Figure 1 illustrates some of the issues involved in using HDR images in absolute luminance representation for enhanced clarity.
[0020] In contrast to LDR imaging, where all scenes looked more or less the same (for example, in this case too, a full spread between luma 0 and almost 255 was needed to make all image objects fully visible, so night scenes could not be really black, since it was necessary to simulate the scene by coloring it blue; also, there was only one kind of white, there was no super-white, for example), a comprehensive HDR video system allows many kinds of HDR images to be created and ultimately displayed, thereby giving many kinds of different visual impressions (i.e. looks). For example, in addition to a "normally lit" image, for example of a cloudy day, or of a uniformly lit room, a desert image (ImSCN1) could be created, in which strongly sunlit objects are somewhat brighter (for example, 500 nits on average instead of 100 nits or less), but also containing extremely bright pixels, such as a very bright sun at 5000 nits. But also night images, such as a city night scene ImSCN2 (still containing some very bright pixels, such as street lights), or a cave ImSCN3 could be defined. Thus, a pixel may fall across the luminance axis spanning, for example, between 1 / 5000 nit and 5000 nit.
[0021] At least some video content makers therefore want to define their original (aka master) images as beautiful as possible, i.e., with a brightness range of, say, PB_C=5000nit for a 5000nit quality master HDR image. This is to create a "universal" HDR image. So, for example, a future display capable of displaying pixel brightness up to 10,000nits will actually show a bright 5000nit sun (as intended by the video creator).
[0022] Of course, even in the distant future, the problem is that at least some viewers will have displays that do not display the full 5000 nits, but only a maximum of, say, 800 nits.
[0023] The question then is what to do with the master 5000 nit HDR image pixel intensities, i.e. how to display them (with conversion to a maximum value of 800 nits, even if no brightness mapping algorithm was applied). For example, if the TV simply displayed the pixel intensities at the exact brightness specified in the received image, it would clip the shape of the sun disk and the clouds around it, which have pixel intensities of, say, 900 or 1200 nits in the master 5000 nit HDR image, and merge them that way. This is reasonable in some scenarios, but less so in others. The TV could try to use a smarter algorithm as an internal automatic brightness mapping, but still has no say in what final brightness any brightness, precisely artistically created by the creator, will map to, and on what images on what displays this will look reasonable or not.
[0024] Thus, more advanced HDR video codecs allow creators to create and communicate not only the master HDR image itself (i.e., as a matrix of YCbCr pixel colors, e.g., as defined in the SMPTE 2084 EOTF), but also to specify at least one secondary reference grading. That is, the video creator can define exactly how he thinks the luminance of any image should be mapped to the PB_C_LDR secondary reference luminance range, e.g., 100 nits. The video creator typically does this by defining a function F_L, e.g., one for each time-sequential video image, to map any possible input HDR luma (Y_HDR_in, spanning 0-1023) into the corresponding LDR luma (Y_LDR_out, spanning the same span of values 0-1023, but with a different value, or position, along the luma axis).
[0025] Once the two reference images (aka gradings), or one of the two images and the corresponding data, which is the function F_L, are communicated, an algorithm is obtained to determine any intermediate grading, such as a grading of 800 nits.
[0026] The scenario to date assumes that there is as input image one secondary, e.g. LDR reference grading corresponding to a master HDR image (one fundamental truth). This should not be confused with the different technical scenarios with different possible flavours of secondary images involved, which will be explained together with the embodiments below!
[0027] FIG. 2 shows an exemplary HDR video coding and communication chain, which is one embodiment in which the new techniques described below may be used.
[0028] Details of possible coding can be found in ETSI standard TS103433-2 V1.1.1 "High performance Single Layer High Dynamic Range [SLHDR], Parts 1 & 2", which is incorporated herein by reference.
[0029] On the video coding side, a master HDR image (MAST_HDR), e.g. a high quality 5000nit image, is obtained. Two useful variations are described, without being limited thereto. In the first variation, a human color grader starts from an initial image (e.g. captured directly from a camera) and performs a precise grading (i.e., determination) of the luminance of various objects in the image so that the image gets a certain look. As a second example, an automaton determines the master HDR image from the initial camera captured image, e.g., some rough grading is used on the relative luminance image from the camera to create a sufficiently high quality 5000nit master HDR image, or an existing LDR image is converted into a pseudo-HDR image by inverse tone mapping.
[0030] Without being limiting, the description of the embodiment of FIG. 2 assumes that the master HDR image is actually communicated as a corresponding LDR image (Im_SDR). Therefore, to calculate the LDR image from the master HDR image, a suitable luminance or luma mapping function (F_L) must be determined. If the normalized input luma Yi (i.e., the luma code divided by the exponent (2, number of bits)) is processed into a normalized output luma using a luma mapping function consisting of, for example, two linear segments, and if the input coordinate of the connection point (yp) is fixed, for example, in the middle, this function can be specified by one parameter P1 that determines the slope of the first linear segment.
[0031] A color converter 202 in the video encoder 221 applies an F_L luminance mapping of the luminance of the master HDR image (MAST_HDR) pixels (in practice, the color converter 202 typically applies a 3D color transformation F_ct).
[0032] The LDR image is then typically compressed in a compressor 203 using any of the known video compression techniques, for example VVC, resulting in a coded version Im_COD of the LDR video image. The luma mapping function F_L (or actually the color mapping data F_ct, which typically also includes a function for varying pixel saturation, since the hue is typically kept constant between the input and output functions) is treated as metadata by the compressor, and supplemental enhancement information (SEI) is a good way to convey any data regarding the desired processing function.
[0033] After the operation of the Content Video Encoder 221, from the point of view of image communication technology, the rest of the communication chain behaves as if it got a "normal SDR" image as input. Thus, for example, the Transmission Formatter 204 applies all the necessary transformations to format the data to travel over some transmission medium 205 (e.g., channel coding for storage on a BD disc, or frequency coding for cable transmission, splitting the video into suitable data packets, etc.).
[0034] The image data then travels via some transmission medium 205, e.g., satellite or cable or Internet transmission, according to, e.g., ATSC 3.0, or DVB, or whatever video signal communication principle, to one or more recipients, which may be a consumer video device such as a television receiver, or a set-top box, or a specialized system such as a cinema receiving unit.
[0035] At the consumer or professional receiving end, a receiver unformatter 206, embedded in various physical devices such as a set-top box, a television or a computer, undoes the channel coding (if any) by applying unformatting and channel decoding. A video restorer 207 inside a video redetermination device 220 (e.g. a video decoder) then applies, for example, HEVC decoding and decompresses the color transformation function metadata F_ct to yield a decoded SDR image Im_RLDR. A color converter 208 is then configured to convert the SDR image luminance to obtain an output image luminance of some output image.
[0036] Depending on the type of the video redetermination device 220, two scenarios apply. If the device is a pure decoder, it applies the inverse of the (luminance or) luma mapping function F_L to obtain a close reconstruction of the master HDR image as the reconstructed HDR image Im_RHDR (i.e., the same peak brightness dynamic range grading and approximately the same pixel brightness except for some video compression errors of the communicated LDR image). The device can also determine an image of different peak brightness (i.e., different from the peak brightness of the master HDR image and the peak brightness of the second reference grading, i.e., 100 nit, for example), which in this embodiment is to be replicated as the communicated image. For example, a display adaptation algorithm of the display adaptation unit (e.g., electronic circuit) 209 determines a function for calculating a 900 nit display adaptation image Im_DA_MDR optimized for a connected 900 nit capacity display. Such algorithms, some variations of which are described in WO2017108906, generally apply a weaker version of the inverse of the function F_L.
[0037] The technical components (such as the innovative technical components according to the current teachings and / or the conventional technical components to which they are connected, cooperate and integrated) are implemented or realized as various technical systems that are common in image or video technology, i.e., for example, in various hardware devices. For example, the video re-determination device 220 has a technical video supply output, for example, an HDMI cable that can be connected to a television display, etc. (and also a network cable or even a wireless output, for example, to a storage device, etc., or to communicate the output image, Im_RHDR or Im_DA_MDR, to another potentially remote device or system, etc.). Depending on the physical variant chosen, there is an image or video output signal formatter that converts the image appropriately for any technical situation (for example, pixel colors have, for example, HLG formatted, expanded, R,G,B representation defined by the second OETF, etc.).
[0038] There are two embodiments for the determination of the luma mapping function F_L: on the one hand, it is defined by a human color expert, i.e., a color grader, who optimizes the produced video; on the other hand, however, in some scenarios, it relies on automata (potentially partially pre-configured by a human to bias them towards a certain colorimetric behavior) to determine the preferred curve shape of the luma mapping function F_L for each different type of HDR scene (e.g., potentially a different function per video time instant image).
[0039] The required curve shape depends on various factors. Ideally, the optimal curve depends on the semantic information that humans attach to various image objects. It may be desired to make a fireplace fire or a light bulb light brighter compared to the rest of the image, and thus the curve shape may be defined such that the function increases the brightness of pixels that fall within the range of brightness that various pixels in the fire have.
[0040] In practice, one can already define sufficient functions depending on such factors as not only the total dynamic range (i.e. the peak brightness PB_C of the image, e.g. 5000nit), but also what regions of different brightness values are present in the image (in more advanced methods, not only what percentile the brightest pixels are, e.g. 95%, but also whether they are, e.g. in the center or on the periphery of the image, or other geometric aspects).
[0041] The applicant has deployed several versions of the automaton that currently serve various video communication systems.
[0042] For example, an automaton was designed to determine the optimal shape, e.g., optimal slope, for the luminance of the tripartite curve of the above-mentioned SLHDR ETSI standard, using a first linear system to boost the darkest luma (see also WO2016119979). This curve (called Para [see inset in FIG. 3 of this application]) is defined by three parameters: a first slope for a linear segment starting from the darkest black (0 luma) that defines an intersection point, a second slope for a second linear segment that extends downward from the brightest white (i.e., normalized luma=1), and a third parameter that defines the width of a parabolic region around this intersection point that connects the shortened versions of the two linear segments. The automaton uses a number of calculations to analyze the statistics of the input image and arrive at the best regrading, i.e., the three optimal parameter values for the luma remapping curve for that image. A five parameter version also exists, which also includes clipping regions for the darkest black and brightest white, and similar automata can be created for further differently parameterized curves, e.g. customizable polylinear segment curves.
[0043] Automata will also prove important in the future as non-professionals, i.e. consumers, start creating their own HDR content. As opposed to a professional grader who might spend perhaps half a day grading a particular scene (depending on the budget the filmmaker has set aside or still available for grading), it is expected that at least a portion of consumers will only want to do as much tweaking as necessary.
[0044] A better curve that attempts to cover other image aspects may be desired.
[0045] However, what makes the automaton technique even more tricky is if one desires to have different flavors, for whatever reason. The applicant has recently been considering shifting the paradigm from wanting there to be only one "best" grading curve, to being able to create, for example, two reasonable regrading functions to define two flavors of LDR images that work well with the input master HDR image. For example, a first luma mapping curve F_L1, with a somewhat larger brightness boost in the darkest linear segment that makes the darkest pixels in the output LDR image somewhat brighter, and a second flavor of luma mapping curve F_L2 that keeps the darkest pixels in the scene somewhat darker. An artist creating a video may decide to get just one best function, for example, he wants to keep the darkest pixels appropriately dark, so that they retain an air of mystery. However, maybe a user wants a somewhat brighter or even more revealing version of the dark areas in an image. The embodiments described below make it possible to build much more powerful devices, for example mobile phones, that automatically optimize the appearance of images, for example when walking from indoors to outdoors.
[0046] US Patent Application Publication No. 2020 / 0211503 teaches a system for brightening HDR images to compensate for brighter viewing environments. Its features are summarized by FIG. 7 of this patent application. An input image (Im_inpr) comes in. If this happens to be an SDR image, an SDR to HDR conversion is performed by unit 701 to start the actual environment compensation process for a general level HDR image (Imstrt). To obtain a better displayed image under a brighter viewing environment, pixel brightness is increased (e.g., by multiplying all pixel luminances by a common factor k), in which case some clipping of the brightest pixels is taken for granted (again subject to some dynamic range for an end-user display with a fixed maximum luminance). It should be understood that the quality of such processing depends on the quality of the image (noise, compression artifacts, resolution). For example, if an image contains a lot of analog noise, using a strong gray level conversion function with a high k value may not result in the best image for display, as it will increase the noise and make the image look annoying, especially when the environment is not very bright. Therefore, the best one of the alternative gray level curves to apply depends on a combination of image quality values and a measurement of the amount of ambient illumination (I_amb) by the measurement sensor 760. One or more quality aspects of the various images are determined by the deep neural network (750) in this teaching. For example, the DNN can look at an image and classify it as a standard resolution image (SD) or a 4K image (UHD). This tends to be thought of as a simple way of counting the amount of horizontal pixels, but the UHD image may have been generated based on a poor quality SD original image by spatial upconversion, which the DNN finds out. Such a measurement of resolution (M_resol) is useful in the enhancement process. For example, if the enhancement does spatial upscaling from 4K to 8K, it will use different scaling algorithms depending on what the spatial quality of the image is, for example, trying to make the blurred edges somewhat sharper.Another measurement is the analog noise measure M_noi determined by the DNN. This value is useful both for noise removal (denoiser 702) and for determining the optimal gray level conversion function of the first converter 703. In fact, after a proper noise removal, a somewhat stronger first conversion is performed, for example for a standard definition starter image. Then, after the enhancer 704 (i.e. when the resolution is increased and therefore the spatial quality of the image is higher or the block artifacts of the MPEG compression are mitigated), a topping up gray level conversion can be performed by the second converter 705. It is important to note that this DNN only has an external role parallel to the standard processing pipeline, i.e. it is intended to give a meaningful quality measure of various (quite different) input images than a typical noise level estimator does. Thus, a proper environment-compensated version of the original image is obtained, i.e. ImHDRprout.
[0047] 8 illustrates a principle that can be used according to Japanese Patent No. 2907057. Exposure modification is a technique that multiplies an input value (e.g., luminance or brightness) according to a determined exposure value E, e.g., luminance_out A(E) *luminance_in. For example, if we have an indoor scene (figure A) with some outdoor objects in the sun visible through a window, we may want to focus on the indoor objects rather than the outdoor ones (e.g. when computing an LDR image from an input HDR image), which may be clipped to white, as was the case in classical LDR images generated in the past. Knowing the limited (e.g. LDR) output range, we can determine the optimal exposure and increase it by a multiplier B (E). If we want to focus on the outdoors, we can determine a different optimal exposure, which gives us a multiplier A (E2), which in this example is darker (figure 8B). Figure 8C shows an example of how this can be determined statistically, if we do not want to use a neural network (which may give better results). A simple heuristic is to look for lobes of significant luminance occurrences in the image, determine a representative luminance value LinX, and map it, for example, to the middle of the output luminance range (setting another multiplier value).
[0048] JP 2907057 improves on this by determining not only the general (grey) level of ambient lighting, but also red, green and blue measurements so that colored (e.g. bluish) environments can be compensated for. The average value of the processed image is the fourth input to a neural network which determines control signals to increase the drive values to the red, green and blue electron guns of the CRT display (thereby brightening the displayed image to compensate for the brighter environment). The fifth parameter is the duration for which the viewer is exposed to the (potentially overly) brightened image.
[0049] A paper by T. Bashford-Rogers et al., learning preferential perceptual exposure for HDR displays, IEEE Access, April 2019, teaches a statistical neural network model to determine optimal exposure for different classes of displays (e.g., from maximum displayable brightness of 500 nits to 10,000 nits) and different illuminance values (from darkness to typically lit indoors such as 400 lux to 4000 lux outdoors).
[0050] The optimal exposure depends on the image specifications by a constant factor, in a linear fashion on the amount of ambient lighting around the display, and logarithmically on the maximum brightness of the display. The following variables characterizing the image seem to be sufficient to characterize the image: the log10 average value of the brightness in the image, an image key that defines the ratio of the mid-brightness to the minimum value divided by the total span, and a dynamic range measure. Neural networks can learn such aspects about the image internally. They teach neural networks whose first layer does a convolutional summary, and then the final fully connected layer determines a single optimal exposure value based on all that summary information.
[0051] US Patent Application Publication No. 2016 / 0100183 teaches a playback device for sending images from a recording medium (e.g., a Blu-ray disc) to a television, where the recording medium includes the images and a brightness mapping function, and if the television conveys the information, the playback device sends the images and the function, and if the television does not convey such information, a regular SDR image is sent to the television. Summary of the Invention
[0052] The difficulty in obtaining a proper regrading of a video is illustrated in an apparatus (300) for luminance regrading of an input high dynamic range image (IM_HDR) of a first luminance dynamic range to a second image (IM_DR2) of a second luminance dynamic range, where the maximum luminance of the second image may be higher or lower than the maximum luminance of the input high dynamic range image, the apparatus comprising: a first neural network processing circuit (301) having as input a set of pixel intensities of an input high dynamic range image (IM_HDR), the first neural network processing circuit having at least two sets of outputs (S1, S2); The set of outputs has output nodes for supplying parameter values of a parametric regrading equation, the parametric regrading equation being composed of at least two partial functions, and the amounts of the at least two partial functions in the parametric regrading equation are controlled by at least two corresponding output parameter values (P11, P21); a first neural network processing circuit, the first set of outputs (S1) including at least two parameter values (P11, P21) determining a first shape of the regrading function, and the second set of outputs (S2) including at least two corresponding parameter values (P12, P22) determining an alternative second shape of the regrading function, the alternative second shapes of the regrading function being determined by the same parametric definition, but using different values for the parameters of the regrading function to obtain different shapes; a second neural network processing circuit (302) having as input at least one measurement (La) from at least one sensor (311) and having as output a node providing a set of weights corresponding in quantity to the number of sets of outputs of at least the first neural network processing circuit; a combiner (303) for adding a first parameter value (P11) of the first set of outputs of the first neural network processing circuit multiplied by a corresponding first weight (w11) from the second neural network processing circuit to a corresponding parameter value (P12) for the same regrading function defining a parameter of a second set of outputs of the first neural network processing circuit multiplied by a corresponding second weight (w12) from the second neural network processing circuit to result in a final parameter value (P1_f); a luma mapping circuit (320) for mapping an input luma of an input high dynamic range image (IM_HDR) using a luma mapping function (F_L) defined by at least two parameters, including a last parameter value (P1_f), to obtain an output luma of a second image (IM_DR2);
[0010] The present invention is addressed by an apparatus for luminance regrading, comprising:
[0053] The limits of both dynamic ranges are preset in some manner or set at processing time. For example, when the minimum black of both ranges is considered to be equal and fixed, e.g. 0.01 nit, the luminance dynamic range can be defined using only the maximum luminance (preferably of the image's associated target display). This can be built into the hardware in the device or input via a user interface, etc. It can be an additional input parameter of the neural network, or a condition for preloading an alternative set of internal coefficients of the neural network, etc. The preferred embodiment works with HDR input images that are uniquely defined according to an associated target display with a maximum luminance (a 4000 nit target display image or video should typically have no pixels with a luminance brighter than 4000 nit, and will generally have at least some image objects that contain pixels that adequately fill the range of the associated target display, i.e., those pixels have a luminance somewhat below 4000 nit and / or up to 4000 nit), although it should be noted that the images actually displayed do not have to be on that scale (e.g., in very bright daylight conditions, they may be shown with an output range of 0.01 to 8000). The neural network can also operate with relative (e.g., normalized to a maximum value of 1.0) luminance pixel values.
[0054] The second network can determine a blend of two different regrading functions for the input image, representing alternative flavors to be coded in the primary network, depending on, for example, relevant measurements of the display periphery, or relevant aspects on the coding side. Thus, the second network controls the first neural network to some extent in a certain manner, i.e., the second network adjusts parallel aspects already included in the first neural network as a whole. In some embodiments, the blending weight is equal to the amount of sets, and thus the grading curves can be blended as a whole. For example, there are three sets (to be blended), each set being weighted by two parameters (e.g., A * When defining a regrading function controlled by pixel_luma+B, the final value of the first parameter is (w1 * A1+w2 * A2+w3 * A3) / normalized value, and the second final parameter value is determined strictly by those same three weights from the second neural network, i.e., B_final = (w1 * B1+w2 * B2+w3 * B3) / normalized value. Typically, at least some of the parameters are mixed. Other embodiments output more weights, e.g., each contribution value has its own weight. For example, (w1 * A1+w2 * A2+w3 * A3) / normalized value, and (w4 * B1+w5 * B2+w63 * B3) / normalized value. On the other hand, not all parameters get a changed final value in the combiner. For example, the first parameter gets a weight value A_final=(w1 * A1+w2 * A2) / normalized value, and B_final = B1 (or even a fixed value, etc.).
[0055] Generally, all parameters that define the re-grading function are mixed, but a parametric definition (which can train a linear neural network to find suitable values) means adjustable numbers that determine the weighting of partial functions over a certain domain (i.e., define a part of the overall definition of the re-grading function). For example, when a function is defined over the entire domain of input values (e.g., for Luma code 0, with a multiplier (2, number of bits), or for luminance ranging between Luminance_min, e.g., 0 nit, and Luminance_max, e.g., 4000 nit), F_final(x)=parameter_1 * first-partial-function(x)+parameter_2 * A re-grading function can be constructed as an additive one, i.e., F_final(x)=parameter_1 * first-partial-polynomial(x)+parameter_2 * second-partial-polynomial(x), where x is any value in the input domain. When the partial functions are polynomials, of course, the functions input into the device's hardware or software are actually usable functions for re-grading between higher dynamic range images and lower dynamic range images, i.e., those functions are generally monotone increasing functions, also known as strictly monotone increasing functions, which those skilled in the art will understand. An example of a multiplicative definition is, for example, F_final(x)=parameter_1 * first-partial-function(x) * [parameter_2+second-partial-function(x)]. An example of a differential function, also known as a partial domain definition function, is, for example, when x<parameter_1, parameter_2 * applies partial_function_1, and in other cases, as a result for F_final(x>parameter_1), (parameter_3 *partial_function_2+parameter_4 * Those skilled in the art will understand how to define further parametric regrading functions and which values for those parameters work well under various circumstances, e.g., how a first order neural network can learn an image of a dark cave with many dark areas under both a first order situation requiring a first regrading function shape and a second order situation requiring a second shaped different regrading function shape.
[0056] Thus, if a first set of nodes yields first values for parameters A, B, C, then a second set may be generated using the same regrading function, e.g. A+B * luminance-in+C * This results in alternative, eg somewhat larger, values for those same parameters that define power(luminance-in; 3).
[0057] The system also knows from which dynamic range to map (e.g., its maximum luminance of the associated target display, i.e., its maximum pixel luminance expected to be in the image in general). This is pre-fixed, for example, in a system that always needs to get an input video of 1000 nits and downgrade to an output video of 200 nits, or even 1500 nits (if that is, for example, the capability of the display on which the output image is to be viewed, for example, under various circumstances). Alternatively, configurable values for the inputs and / or outputs are set at runtime, for example, by a user, and then, for example, the neural network loads different weights for different situations, or covers all situations inside the network, and gets these settings as additional inputs (or even derives it from within, if the image luminance is a set of pixel luminance values sampled from the input image to be luminance mapped, etc.). These functions are also pre-set in the device circuitry, or defined by the user before use, etc.
[0058] In the pre-set situation, commonly useful functions are used, for example the function of two outer linear segments connected by the applicant's parabolic intermediate segment is used in the simpler version. Complex versions use complex models that the neural network can optimize. Many systems are shipped pre-optimized from the factory, while others involve a training phase by the user of the device, e.g. a mobile phone. In that case the user derives, for example, a multi-segmented function and the neural network optimizes the segments based on the preferences expressed by the user regarding the final regraded look of the displayed image. Different neural networks can use different learning / optimization algorithms. In a general-purpose device, the measurements can be any values that affect the regrading as needed, measured, for example, surroundings, display characteristics, user characteristics, etc., as illustrated by the described example.
[0059] Parameters that determine the shape of a function means that the values of those parameters uniquely determine the shape of the function, i.e. on the plot where its output coordinate values are obtained from its input coordinate values. The neural network trains its internal weights differently so that if the neural network decides to lower the value of the output, the curve will be, for example, lower, closer to the diagonal, less curved, etc.
[0060] An example of a single-parameter controllable regrading function is a two-linear segment function defined on the luma axis normalized to 1, where the parameters control the height of the first segment at an input value of 0.5, i.e., P1=h, and Y_out(0.5)=h. *0.5. Although a three-parameter (black tilt, white tilt, midtone width) parametric function has been found to be quite suitable for regrading images in some applications (e.g., broadcasting), more advanced regrading functions have been developed and proposed (see, e.g., the SLHDR ETSI standard), and such functions can also be calculated by the following first neural network embodiment. If the number of weights is equal to the number of sets, the flavor of the function is simply weighted by itself, i.e., all parameters are weighted by the same value. If the number of output weights of the second neural network is equal to the number of sets times the number of parameters per set, i.e., the total number of output nodes of the first neural network, each parameter can be weighted differently (as before, each triplet of parameters defines the curve shape of a certain final parametric function).
[0061] The first neural network processing circuit (301) can be trained to obtain a useful set of regrading functions (flavors) for any kind of HDR scene image. Since it is necessary to learn some parameters that characterize the curve shape of one or more mathematically defined curves (illustrated with one-parameter and three-parameter curves (para), the system can be extended to cover parallel regrading curves, such as choosing para or multi-linear segment curves, or to learn to be continuously applied functions, such as fine-tuning multi-linear segment curves applied to HDR luma after initial para regrading, etc.), any neural network topology can be used by training such a neural network topology with some cost minimization function to yield output parameters close to those that a human grader would choose for various input images. The input images have fixed (e.g., all 1000 nits, e.g., if the device is to function in a fixed broadcast system) or variable coding peak brightness, and include a fairly large set of HDR images of various characteristics (all bright, such as the cowboy in the desert example, semi-dark with some bright areas, such as sunlight shining through the ceiling of an open-air market, larger and smaller dark and bright areas, a human actor's face appearing in different areas of the image with various lighting, such as indoors and outside a window, bright areas of light such as explosions or fireworks, etc.).
[0062] The second neural network (NN) processing circuit (302) can now learn how these functions should be combined; for example, if the lux meter 311 measures a certain ambient lighting level (e.g., watching television in the evening in low light, or watching on a train during the day), the second neural network processing circuit (302) learns that a certain amount of function 1 (e.g., a parametric function with a brighter linear slope for the darkest colors in the scene) needs to be mixed with function 2 (a parametric function with a smaller slope, which is the average situation). The second NN can then learn different settings for different peak brightness, or in particular minimum coded luminance (MB_C), anticipating that, on average, more correction is needed for better visibility.
[0063] This allows for extremely low user interaction, and the system then quickly switches to an appropriate regraded output image for any of several end viewing situations.
[0064] It is useful when the device for luminance regrading is coupled to a sensor coupled to the display. In this way, details of what is happening on the display can be measured. For example, a user may be viewing differently with a landscape mobile display (this is further combined with, for example, the type of content data extracted from the descriptive metadata, for example, whether the user is viewing a downloaded video or a linear broadcast program), and an orientation sensor can provide input for a second NN to move to a different set of output weights for that situation. Another example is a compass and / or 3D orientation sensor, which can be measured in combination with, for example, the time of day and the average influence of the sun in the direction in which the user is viewing the image. Another example is using a camera adjacent to the display.
[0065] It is useful when the apparatus for luminance regrading is coupled to a light meter (311) configured to provide a measure of the amount of light in the environment. This can in particular enable different weighting of the curves from the first NN that are still optimized to emerge for different kinds of HDR scene input images and some of those HDR scene input images, for example, having curve shapes that are better tuned for darker content in brighter viewing environments.
[0066] It is useful when the device for luminance regrading is coupled to an image summarization unit (310) configured to measure luminance aspects of the input high dynamic range image (IM_HDR). There are various algorithms for summarizing images, ranging from simple ones (e.g., subsampled averaged regional luma images) to complex ones, such as intra- and inter-object contrast measures (such as various derivatives) and texture measures. All of them can easily form inputs for the second NN to train its internal understanding. The neural network can then learn (based on preprocessed knowledge as input or natively on the image luma) whether, for example, luminance generally corresponds to indoor scenes or to indoor and outdoor scenes (e.g., views through windows). In principle, many embodiments work with high-level global parameters, but the second NN also works, at least in part, based on geometrically localized aspects of the image, such as centrally concentrated mid-gray areas, which are correlation factors for human faces. If further luma-derived parameters are also used, such as estimated motion, the device can take into account, for example, sports settings (e.g., no central motion and fast outer regions as the camera tracks a motorcycle, etc.).
[0067] It is useful when the apparatus for luminance regrading comprises a user interface unit (330) configured to determine a set of user-specified weights and a selector (331) for selecting at least one user-specified weight instead of a corresponding weight from the second neural network processing circuit (302) to enter the combiner (303). It is useful to have a manner in which, in addition to the automatic mode of regrading, the user can benefit from an insight of all learned regrading needs inside the first NN and can set some weights for at least some of the parameters himself. For example, the user can set the slope for the linear segment of the darkest pixel of the pararegrading curve and the automaton can handle the other two. For example, some embodiments include a feedback input into the second neural network of the user-selected first parameter, so that the network can determine the other two accordingly. In a simpler embodiment, instead of the selector 331, there may be a weight recalculator circuit or algorithm that uses a pre-designed method to determine the two other weights of the parabolic function given a chosen slope for darks (e.g., a fixed midtone width of the parabolic mid-region, and a slope for brights that depends on the user selected slopes for blacks and PB_C, or in more advanced versions, the amount of different pixel luma in the brightest region, etc.). In the simplest variant, the user sets the three final weights himself, for example by moving between the weights of the two flavors using one or three sliders.
[0068] It is useful when the device for luminance regrading is physically included in the device containing the display panel, which can be a useful device for optimizing, for example, a television display in a home viewing environment, or a mobile phone, etc. This allows various receiving devices to tweak the image with their own flavor in a system that separates image creation from image optimization for display.
[0069] It is useful when the apparatus for luminance regrading is included in a video coding and communication device for communicating the video to one or more receivers. In contrast to the receiver embodiment, it may also be useful in some situations when a creator uses the apparatus or method. For example, consumer-created videos can benefit from the technique, since at least some consumers do not have the sophisticated colorimetry acumen of professional graders.
[0070] In such a configuration, the regrading device is generally not used to calculate a secondary graded image that is suitable for (final) display, but also to calculate a secondary grading that is suitable for communication (e.g. an appropriate LDR image corresponding to an input HDR image for SLHDR coding).
[0071] The technique also provides a method (300) for luminance regrading of an input high dynamic range image (IM_HDR) of a first luminance dynamic range to a second image (IM_DR2) of a second luminance dynamic range, the second image having a maximum luminance that may be higher or lower than the maximum luminance of the input high dynamic range image, the method comprising: using a pre-trained first neural network processing circuit (301) having as input a set of pixel intensities of an input high dynamic range image (IM_HDR), the first neural network processing circuit having at least two sets of outputs (S1, S2); the set of outputs has output nodes for supplying parameter values of a parametric regrading equation, the parametric regrading equation being composed of at least two partial functions, the quantities of the at least two partial functions in the parametric regrading equation being controlled by at least two corresponding output parameter values (P11, P21); using a first neural network processing circuit (301) where a first set of outputs (S1) includes at least two parameter values that determine a shape of a regrading function and a second set of outputs (S2) includes at least two corresponding parameter values (P12) for parameters that are different from the parameter values in the first set of outputs, the second set of outputs determining the different shape of the regrading function; using a second neural network processing circuit (302) having as input at least one measurement (La) from at least one sensor (311) and having as output a set of weights having weight amounts at least equal to the amounts of the set of outputs of the first neural network processing circuit; adding a first one (P11) of the at least two parameter values of the first output set of the first neural network processing circuit after multiplication by a corresponding first weight (w11) from the second neural network processing circuit to a corresponding first one (P12) of the at least two parameter values of the second output set, the value of the corresponding parameter of the same regrading function from a second output set of the first neural network processing circuit, multiplied by a corresponding second weight (w12) from the second neural network processing circuit before the addition, to result in a final parameter value (P1_f); applying a luma mapping function having a curve shape defined by the last parameter value (P1_f) to map the input luma of the input high dynamic range image (IM_HDR) to obtain an output luma of the second image (IM_DR2); The method can be carried out by a method having the steps:
[0072] These and other aspects of the method and apparatus according to the invention will become apparent from and be elucidated with reference to the implementations and embodiments described below and the accompanying drawings, which merely serve as non-limiting specific examples illustrating more general concepts, in which dashed or dotted lines are used to indicate that an element is optional, but an element without dashed lines is not necessarily required. Dashed or dotted lines may also be used to indicate elements that are described as required but are hidden inside an object, or elements that are described in terms of intangibles, such as, for example, selection of an object / area (and how they are shown on the display). [Brief description of the drawings]
[0073] [Figure 1] FIG. 13 shows a schematic diagram of an aspect involving a more advanced HDR video coding system, i.e. a system that does not just code pixel colors of a master HDR image only, but rather, what is shown is how a function, represented by a mapping arrow between input luminance and output luminance, can be defined that defines at least one secondary regraded image of a different coding peak luminance (e.g. SDR) than the master HDR image. [Diagram 2] FIG. 2 illustrates a schematic diagram of an encoder and a decoder in a video communication system for communicating a master image (MAST_HDR) in a first dynamic range (DR_1) as a representative image in a different second dynamic range (DR_2). [Diagram 3] FIG. 3 shows a schematic diagram of the basic elements of an inventive apparatus (300) for luminance regrading, together with some optional embodiment elements (dotted lines). [Figure 4] FIG. 4 is a schematic illustrating a convolutional neural network, which is a suitable neural network topology for the first neural network processing circuit (301) of FIG. [Diagram 5]FIG. 4 is a schematic diagram illustrating how an exemplary video illustrating data values may be formed that may be input into the second neural network processing circuit (302) of FIG. [Figure 6] FIG. 13 illustrates generally an embodiment in which user weights are supplemented or replaced by weights from an external database. [Figure 7] FIG. 1 is a diagram summarizing the technical approach of the first prior art. [Figure 8] FIG. 11 summarizes the technical approach of the second prior art. [Figure 9] 1A-1C illustrate some example devices or applications in which at least some of the present embodiments may be used. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0074] Figure 3 shows the elements of an apparatus (300) for luminance regrading. It obtains an input HDR image IM_HDR (e.g. 4K, but often also subsampled, since regrading does not need to be strongly resolution dependent). This can be for example a master grading in a video coding situation, or directly from a camera image in a consumer situation where the video is captured with a mobile phone. So, in effect, when the apparatus is in operation receiving an input image to be luminance regraded, a number of pixels (horizontal *There can be as many input neurons (aka nodes) as the number of inputs (vertical)×3 (since there are three color components). Any of various possible internal structures (number of hidden layers, etc.) that give slightly better results but are more computationally intensive are selected by the device manufacturer, depending for example on whether the device has a dedicated neural processor or can advantageously use a GPU or circuitry for performing massive parallel computations. As an output layer, there is a set of parameter values, e.g., two flavors of parameters for the parametric function (e.g., P21 and P22 give the same mid-range width, but P11 gives a larger slope value for the dark straight line segment of the parametric function than P12, and P31 is, for example, a smaller slope than P32). Both of these are generally reasonable regradings for the image, but are not necessarily the final best function to be used in the luma mapping circuit 320. A better curve is determined based on further considerations, namely other parameters characterizing the situation (some of them correspond to the inputs of the first NN, i.e. parameters from or related to the input image IM_HDR).
[0075] The second neural network processing circuit (302) obtains one or more situation-related variables. In practice, it is all selected measurements input together to let the second NN sort everything out, but other embodiments are conceivable, for example with a concatenation of partial networks for different types of situation-related variables, which can then switch to an overall second NN configuration, for example depending on whether measurements are available or desired.
[0076] A luminance meter (311) configured to provide a measure of the amount of light in the environment is a useful sensor for providing measured contextual input into the second NN. Another example is parameters characterizing the input image provided by the image summarization unit 310 (FIG. 5 teaches about possible examples).
[0077] The input for the HDR image in the second NN does not necessarily have to be as detailed as that in the first NN, and therefore embodiments may use subsampled summaries or image characteristics that have been found to be useful in flavor selection. For example, texture measures are generally useful as one input for the HDR image to be used together with additional values that characterize the spread of the target luma, such as contrast measures. Although summed total micro-texture measures for the entire image may be used, it may also be calculated for each region block of the input image, thereby obtaining a subsampled matrix of such local texture measures. Furthermore, texture measures may be used to blend based only on the general mid-tone shape of the image, where, for example, an actor's face is expected to be, or the mid-tone width of an example parametric function, for example, but values characterizing various textures for different luma subranges, from dark to bright, may be input into the second NN.
[0078] The combiner 303 receives as input the weights (first weight w11, second weight w21, etc.) from the output layer of the second NN for the current situation (i.e., the input image and surroundings, and display-related aspects such as display orientation, etc.) The combiner makes an appropriate combination from at least two related (corresponding) parameters of the various sets of equivalent parameters that are output from the first NN.
[0079] The combiner performs the following calculations in the example to get two weights from the second NN (eg w11=0.6 and w21=0.4): P1_f=w11 * P11+w21 * P12 P2_f=w11 * P21+w21 * P22 P3_f=w11 * P31+w21 * P32 [Formula 3]
[0080] In the scenario where the second NN outputs 6 weights, the combiner calculates: P1_f=w11 * P11+w12 * P12 P2_f=w21 * P21+w22 * P22 P3_f=w31 * P31+w32 * P32 [Formula 4]
[0081] These final parameters (P1_f, P2_f, P3_f) uniquely characterize the final regrading function to be used, which is used by the luma mapping circuit 320, which maps all the input lumas of the input HDR image (IM_HDR) to obtain the output luma of the corresponding regraded output image in the second dynamic range.
[0082] In a basic configuration embodiment that operates completely automatically, there is no selector 331 or further configuration circuitry, however it is useful if the user of the device can also have his own opinions about useful combinations of regrading functions output from the first NN.
[0083] In addition, a user interface unit 330 is built into the user device that is connected to the rest of the device for regrading. It can have various configurations, such as sliders, to provide two or more weights (e.g., putting the slider in the 60 / 40 position as explained with Equation 3).
[0084] The first NN only needs to output some parameters of one or more functions, in at least two versions( / sets), and therefore there are several neural network topologies (which are equivalent computational engines for the problem) that can be trained to yield those parameters.
[0085] However, it is useful for a neural network to be able to extract spatial aspects of an image.
[0086] Therefore, a useful and preferred embodiment for the first NN is a Convolutional Neural Network (CNN).
[0087] This is further elucidated using an example of a CNN, which is briefly explained using FIG.
[0088] A "typical" neural network generally consists of multiple layers of networks, where each cell in the first layer connects to every cell in the following second layer via learnable weights (i.e., it has weight_N * previous_value_N and typically passes this sum as input to an activation function), but CNNs try to regularize by having different kinds of pre-processing stages (schematically illustrated in Figure 4).
[0089] The process of the NN starts by the first layer of nodes getting an input HDR image. If it was a 4K luma-only image (i.e., horizontal pixel resolution Wim = about 4000 pixels and vertical resolution Him = about 2000, where Nim indicates the number of images in the training set), we get about 8 million inputs for each training image, but in the case of color images, there are also three color components (generally YCbCr) per pixel. In the first stage, convolutions are performed to detect interesting spatially localized features. In general, it is not necessary to learn complex things like blending the top left pixel of the image with the bottom right pixel (which often leads to overmodeling, or problems where neurons get stuck in their state, not to mention an unnecessarily excessive amount of calculations that usually do not result in better learning and accuracy of the output results).
[0090] Thus, the convolution produces a number (Fim) of feature maps, starting from a first feature map f1 to, say, a third feature map f3.
[0091] The values of the pixels of a feature map (e.g., also a 4K image) are obtained by convolution with a learned kernel. We show what happens with the example kernel 401. Let us assume that the convolution values of this 3x3 kernel have been learned by a first feature map. Such a configuration, with values of -1 on the left and values of +1 on the right, is known from theoretical image processing as an edge detector for vertical edges. Convolution with this kernel at all locations of the image thus indicates in the feature map whether there is such an edge (or how strongly such an edge should be present is measured) at various image positions.
[0092] By doing this, more and more data is created, and therefore at a certain moment it is desired to reduce the data by using a so-called pooling layer. For example, a single value can be determined for a block of 10x10 values in the feature map, thereby reducing the resolution and obtaining, for example, a first pooled map Mp1 of 400x200 pixels as a subsequent layer of the first feature map (layer). The convolution kernels often start to correlate more and more, overlapping more and more with the localized pattern being detected, so it is only necessary to take the largest one of the results in the feature map. It is the whole of the NN that learns how to determine the correct regrading function, but since the network works by determining ever more complex and ultimately extremely interesting image features (for example, a square can be seen as a collection of two vertical lines and two horizontal lines), this is also probably a well-learned insight of that layer that will be useful later. In fact, in general, one does not use a single convolution and pooling layer as shown in the schematic, but several pooling layers for successive data reduction, each of which may have one or several convolution layers before it. Thus, one can examine not only the derivatives (i.e. contrast) of small local regions, but also coarser scale, i.e. more distant regions. For example, the cave example ImSCN3 in Figure 1 (and its optimally shaped pararegrading function, i.e. its three curve shape values that control the parameters) can be learned from the derivatives of the bright central region versus the very dark surroundings.
[0093] Instead of using fixed convolution kernels as in typical programmed machine vision, the NN automatically learns optimal kernels via backpropagation during the training phase, i.e., the first inner weight Winl1_fl of the first feature kernel converges to -1 over successive training iterations.
[0094] Finally, it is time to reduce to a fully connected network layer, since we eventually want to learn from the entire image data, e.g., a person is required to be at a location 2 / 3 of the way in the image from the left, but there is a lamp illuminating the cave in the upper right corner (e.g., a rectangle of 5% by 5% of the image size). To do this, we must flatten the data of the last convolutional layer into a 1-dimensional vector of NN cells and their values.
[0095] Thus, in general, there is one or more 1D fully connected layers, e.g., a fully connected layer FC1 followed by a final fully connected (output) layer FC2, resulting in the desired understood data, in this case e.g. 2×3 parameters of two flavors of useful regrading parametric functions for at least one presented image.
[0096] Finally, typically after the convolution an activation function is used to condition the convolution, e.g., in a preferred embodiment, a rectified linear unit activation function (ReLU), which zeroes out negative convolution results, if any.
[0097] An example embodiment code for the first NN written in Pytorch is as follows (Model Training): import torch from torch.utils.data import Dataset, DataLoader import torch.nn as nn import torch.nn.functional as F import torch.optim as optim import torchvision from torch.utils.data import Subset from sklearn.model_selection import train_test_split from torchvision import datasets, transforms import pandas as pd #pandas is a software library written for the Python programming language for data manipulation and analysis. import cv2 import numpy as np #NumPy is a library for the Python programming language that adds support for large multidimensional arrays and matrices. class hdrDataset(Dataset): def __init__(self, dataPath, imageWidth, imageHeight, outputSize): self.dataPath = dataPath self.imageWidth = imageWidth self.imageHeight = imageHeight self.outputSize = outputSize self.data = pd.read_csv('C: / Data / hdr / y.csv') #Mapping curve parameters (y) self.numberOfSamples = len(self.data.index) def __len__(self): return self.numberOfSamples def __getitem__(self, idx): if torch.is_tensor(idx): idx = idx.tolist() imageFile = 'C: / Data / hdr / x_'+str(idx+1)+'.png' #HDR input image sample(x) image = cv2.imread(imageFile,cv2.IMREAD_UNCHANGED).astype('float32') / 65535.0 x = torchvision.transforms.ToTensor()(image) parameters = self.data.iloc[idx].to_numpy().astype('float32') y = parameters[2:5] #Predict three parameters (y) sample = {'x': x, 'y': y} #The sample is a pair (x,y) return sample #Here, the neural network topology (layers, e.g. convolutional layers) are defined, followed by the processing of the layers (which yields new successive values of x). class hdrNet(nn.Module): def __init__(self,outputSize): super(hdrNet, self).__init__() #Network function self.conv1 = nn.Conv2d(3, 8, 3) #The meaning of the three parameters in the successive calls is to use a 3x3 filter in the convolution, from 3 to 8 channels (i.e. 3 input channels for the input image, and 8 feature maps from them). self.conv2 = nn.Conv2d(8, 32, 3) #8 to 32 channels (i.e., 8 feature maps in the first hidden layer multiplied together to give 32 more knowledgeable secondary feature maps), 3x3 filters self.conv3 = nn.Conv2d(32, 64, 3) #32 to 64 channels, 3x3 filter self.fc1 = nn.Linear(64 * 14 * 27, 64) #The image has a size of h×w=14×27. Now, after subsampling / pooling, we get a fully connected (i.e. each neuron in the layer connects with each neuron in the following layer) linear layer, but it is a smaller network because the image was downsampled to 14×27. The 64 comes from collecting together the 64 feature maps that were in the last convolutional layer. The number of output nodes in this first fully connected layer is 64. self.fc2 = nn.Linear(64, 128) # A fully connected linear layer that maps 64 inputs to 128 output nodes self.fc3 = nn.Linear(128, outputSize) # If the network should be trained on two common flavors of parametric regrading functions of the type illustrated, outputSize is, for example, 2x3 parameters def forward(self,x): #How the functions connect x = F.relu(self.conv1(x)) #Filter a 3-channel image using a 3x3 convolution kernel (learned by the NN during training) x = F.max_pool2d(x, (8, 8)) # 8 channel 8x8 downscale (amount of feature maps obtained from the first convolutional layer as defined in self.conv1 = nn.Conv2d(3, 8, 3)) x = F.relu(self.conv2(x)) #Filter the 8-channel low-resolution image using conv2 (relu is an activation function that sets negative numbers to 0 and keeps positive numbers as is) x = F.max_pool2d(x, (8, 8)) # 32 channel 64x64 downscale x = F.relu(self.conv3(x)) # Filter the 32 channel lower resolution image x = x.view(-1, self.num_flat_features(x)) #Convert from X, Y, Chan to a 1D tensor (view is a dimension selection operation) x = F.relu(self.fc1(x)) #Multilayer Perceptron (MLP) x = F.relu(self.fc2(x)) x = F.relu(self.fc3(x)) return x # Print the mapping parameters of the mapping (x is, for example, 2 x 3 parameters that define the two flavors of the parafunction) def num_flat_features(self, x): size = x.size()[1:] # Includes all dimensions except the batch dimension num_features = 1 for s in size: num_features *= s return num_features #This defines a training phase that iterates over the training data several times until the NN weights are reliable enough (the amount of iterations is controlled by the epochs variable, and batchsize defines how much training data is taken (generally depending on available training resources) (the less, the longer you have to train with smaller batches)) def train(epochs, batchsize): #Estimated three parameters outputSize = 3 #Create a dataset dataset = hdrDataset('dataPath',1920,1080,outputSize) device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") print(device) #Create an object to automatically load data during training dataloader = DataLoader(dataset, batchsize, shuffle=True, num_workers=0) #Define a model based on your network architecture model = hdrNet(outputSize) #Print the model architecture print(model) #Transfer the model to the GPU model.to(device) #Use the mean squared error between the predicted parameters and the ground truth parameters (i.e., the chosen error criterion for learning to match the data in the training set is MSE) criterion = torch.nn.MSELoss() #Set parameters related to the error minimization search (lr is the learning rate) optimizer = torch.optim.Adam(model.parameters(), lr=0.001, betas=(0.9,0.99)) # Run multiple times across the entire dataset (e.g., 50 epochs) for epoch in range(epochs): #Load a subset of all data (say 10%) for batch, sample in enumerate(dataloader): #x = HDR input image, y = ground truth mapping curve parameters x = sample['x'].to(device) y = sample['y'].to(device) #Set the parameter gradient to 0 optimizer.zero_grad() #fitting ypred = model(x) #Predict parameters from HDR image loss = criterion(ypred,y) #Calculate the error loss.backward() #Backpropagation (i.e. backpropagation is used to train the model to get the correct weights) optimizer.step() #The optimizer is ready for the next iteration # Print the error for the current epoch and batch print("epoch = ",epoch," batch = ",batch," loss = ", loss.item()) #Save the model weights torch.save(model.state_dict(),'C: / Data / hdr / state_dict_model.pt') def evaluate(): #Estimated 6 parameters outputSize = 6 #Get model architecture model = hdrNet(outputSize) #Load the model parameters model.load_state_dict(torch.load('C: / Data / hdr / state_dict_model.pt')) #Load the evaluation dataset dataset = hdrDataset('dataPath',1920,1080,outputSize) #Create a Data Loader dataloader = DataLoader(dataset) #Loop one sample across the dataset at that time for batch, sample in enumerate(dataloader): #Predict mapping curve parameters from input HDR image y = model(sample['x']) #Print the predicted parameters print("predicted = ", y) #Print ground truth parameters print("true = ", sample['y']) #end return 0
[0098] Another embodiment uses the above system to create one set of parafunction parameters in the third fully connected layer (i.e., outputsize=3), but then adds a fourth fully connected layer to derive different flavors of the "average" parafunction of the third layer, with the corresponding modified code fragment below. class hdrNet(nn.Module): def __init__(self,outputSize): super(hdrNet, self).__init__() self.conv1 = nn.Conv2d(3, 8, 3) self.conv2 = nn.Conv2d(8, 32, 3) self.conv3 = nn.Conv2d(32, 64, 3) self.fc1 = nn.Linear(64 * 14 * 27, 64) self.fc2 = nn.Linear(64, 128) self.fc3 = nn.Linear(128, 128) #Include additional layers self.fc4 = nn.Linear(128, outputSize) #For the four flavors of parafunctions, outputSize is 4×3=12 parameters. def forward(self, x): x = F.relu(self.conv1(x)) x = F.max_pool2d(x, (8, 8)) x = F.relu(self.conv2(x)) x = F.max_pool2d(x, (8, 8)) x = F.relu(self.conv3(x)) x = x.view(-1, self.num_flat_features(x)) x = F.relu(self.fc1(x)) x = F.relu(self.fc2(x)) x = F.relu(self.fc3(x)) x = F.relu(self.fc4(x)) return x
[0099] Since no such spatial preprocessing is necessary, the second NN can generally be a fully connected network such as a multi-layer perceptron, or a deep belief network, rather than a CNN.
[0100] An exemplary 4-element sensor vector (of size Nsensor) is: - The device direction angle (compass sensor in smartphone) in degrees relative to north, - device azimuth angle in degrees, - local time, measured in hours of the day, using values 0 to 23; - Weather code [0=sunny, 1=partly sunny, 2=cloudy, 3=rain] (e.g. obt, - Device ambient light sensor readings, coded on 100 levels It consists of:
[0101] These five variables are then used in one way to inform a second neural network model that learns to predict whether a smartphone screen is illuminated by bright light or viewed in the shade.
[0102] A possible code for the second neural network could be, for example: class hdrNet(nn.Module): def __init__(self, Nsensor, Nweights): super(hdrNet, self).__init__() self.fc1 = nn.Linear(Nsensor, 64) self.fc2 = nn.Linear(64, 128) self.fc3 = nn.Linear(128, Nweights) def forward(self, x2): #x2 = sensor data vector (in other embodiments, a different x12 input for the HDR input image IM_HDR can also be obtained) x = F.relu(self.fc1(x2)) x = F.relu(self.fc2(x)) x = F.relu(self.fc3(x)) return x It is.
[0103] Figure 5 shows an example of how a simple contrast intensity measure can be calculated, which may be a suitable image summary input for the second NN to determine which is the appropriate weighting. It determines how far bright objects are from the darker lobes of luma (in general, i.e. wherever they are located, which can be improved by making various measurements for different locations), weighted also by how many bright pixels there are in the luma histogram N(Y) of the input image IM_HDR. Thus, the measure is e.g. * DB * It could be F(NB), where F(NB) is a function that saturates its height increase for really large amounts of bright pixels. Any such useful metric that is an input can help the second NN arrive at a more appropriate weight output.
[0104] FIG. 6 shows a more advanced version of the device for luminance regrading (600). It is especially useful in applications where the user should have control over the grading of the video he / she captured (i.e., in consumer devices such as mobile phones that capture video). The difference with FIG. 3 is the HDR image grading situation analysis unit 601. This unit analyzes what the situation is with the user selected weights (w12u) and determines if different weights are in a normal state. In its most basic form, it operates based on the user weights and the weights of the second NN, and in more advanced embodiments, it may also check the features of the input image (instead of doing this by itself, it may take the input from what one or two of the two neural networks have on their output, if the two neural networks also generated some features of the image).
[0105] The HDR image grading situation analysis unit 601 is configured to check at least one value of at least one of the user weights and the second NN weights (when checking both, the HDR image grading situation analysis unit 601 may check the difference between their corresponding weights). The HDR image grading situation analysis unit 601 derives a situation summary signal (SITS) consisting of, for example, an identification signal of the required weights for a certain type of regrading curve (for example, the weight for the linear segment for the darkest luma of the parafunction). This signal requests the corresponding proposed weights w12su, for example, from an external database 612, typically connected via the Internet. In this database, for example, there are generally summarized weights that are appropriate for a specific regrading situation, collected from many regrading operations (for example, via a similar device 600). Preferably, a certain type of HDR image situation is also communicated in the situation summary signal SITS, so that a specific proposed weight for the currently viewed video scene can be retrieved. This can be as simple as, for example, the percentage of pixels below a first luma threshold (typically dark pixels, e.g., a threshold below 30% or 25% on the PQ scale) and / or the percentage of pixels above a second luma threshold (e.g., 75% or higher), but can also include a vector of other image description values. The management circuit of the database 612 can then provide the proposed weight w12su, provided that the proposed weight w12su corresponds to an image close to the communicated image statistics, or generally, the characteristics communicated in the SITS. The image grading situation analysis unit 601 can then communicate this proposed weight w12su to the combiner 303, instead of the user-selected weight, or the result of a formula that balances those two weights, e.g., the average weight.
[0106] Figure 9 shows some examples of devices (or applications) in which the present embodiment can be used. A mobile phone 900 regrades images captured by a camera 901 based on measured characteristics that determine the flavor before storing them in an internal or external memory for later use. A VR or AR goggle 950 shows an application in which a received HDR image is regraded, for example, via an image receiver circuit 951 by using a forward-looking camera 952. For example, in an AR application, the camera measures how bright the environment the viewer is looking at is, which is partially transmitted to the viewer's eyes, and the overlaid HDR image information can be optimized given the present embodiment.
[0107] The algorithmic components disclosed in this text may in fact be implemented (in whole or in part) as hardware (e.g., part of an application specific IC) or as software running on specialized digital signal processors, or general purpose processors, etc. They are semi-automatic in the sense that there is / are at least some user input (e.g., at the factory, or consumer input, or other human input).
[0108] Those skilled in the art will understand from this presentation that the elements of this presentation are optional improvements and can be realized in combination with other elements, how the (optional) steps of the method correspond to the respective means of the device, and how the means of the device correspond to the respective steps of the method. The fact that some elements are disclosed in the present invention in a certain relationship (e.g. in a single figure in a certain configuration) does not mean that other configurations are impossible as embodiments under the same inventive thinking as disclosed for patenting in this specification. Also, the fact that only a limited range of examples is described for practical reasons does not mean that other variations cannot fall within the scope of the claims. In fact, the elements of the present invention can be implemented in different variations along any usage chain, and for example, all variations on the production side, such as an encoder, are similar to or correspond to a corresponding device, such as a decoder, on the consumption side of the decomposed system, and vice versa. Some elements of the embodiments are encoded as specific signal data in a signal for transmission, or for further use, such as coordination, in any transmission technology between the encoder and the decoder. The word "device" in this application is used in its broadest sense, i.e. as a group of means enabling the realization of a particular purpose, and thus may be, for example, a (small part of) an IC, or a dedicated device (such as a device with a display), or a part of a networked system, etc. An "arrangement" or "system" is also used in its broadest sense, and thus it includes, inter alia, a single physical, purchaseable device, a piece of device, a collection of (parts of) devices working together, etc.
[0109] The explicit meaning of a computer program product should be understood to encompass the physical realization of a set of commands that, after a series of loading steps (including intermediate conversion steps, such as conversion to an intermediate language and to a final processor language), allows a general-purpose or special-purpose processor to input the commands into the processor in order to execute any of the characteristic functions of the present invention. In particular, a computer program product can be realized as data on a carrier, such as a disk or tape, data present in a memory, data transmitted via a wired or wireless network connection, or program code on paper. Besides the program code, characteristic data required for the program can also be embodied as a computer program product. Such data is (partially) supplied in some manner.
[0110] Any data usable according to the principles of the present invention, or the present embodiments, such as video data, may also be embodied as a signal on a data carrier, which may be a removable memory such as an optical disc, a flash memory, a removable hard disk, a portable device writable via wireless means, etc.
[0111] Some of the steps required for the implementation of any presented method, such as well-known commonly built-in processing steps such as data input and output steps, standard display driving, etc., may already be present in the processor or in the functionality of the device embodiment of the present invention, instead of being described in a computer program product or any unit, device or method described herein (using the specification of the present embodiment). It should be noted that the above-mentioned embodiments are illustrative and not limiting of the present invention. For the sake of brevity, not all these options have been described in detail, when a person skilled in the art can easily realize the mapping of the presented examples to other areas of the claims. Other combinations of the elements of the present invention are possible other than those combinations combined in the claims. Any combination of elements can be realized in a single dedicated element.
[0112] Any reference signs between parentheses in the claims or any particular signs in the drawings are not intended to limit the scope of the claims. The word "comprises" does not exclude the presence of elements or aspects not stated in the claims. The word "a" or "an" does not exclude the presence of a plurality of such elements.
Claims
1. 1. An apparatus for luminance regrading of an input high dynamic range image of a first luminance dynamic range to a second image of a second luminance dynamic range, the second image having a maximum luminance that may be higher or lower than the maximum luminance of the input high dynamic range image, the apparatus comprising: a first neural network processing circuit having as an input a set of pixel intensities of the input high dynamic range image, the first neural network processing circuit having at least two sets of outputs; the set of outputs having output nodes for providing parameter values of a parametric regrading equation, the parametric regrading equation being composed of at least two partial functions, the amounts of the at least two partial functions in the parametric regrading equation being controlled by at least two corresponding output parameter values; a first neural network processing circuit, a first set of outputs including at least two parameter values determining a first shape of a regrading function, and a second set of outputs including at least two corresponding parameter values determining an alternative second shape of the regrading function, the alternative second shapes of the regrading function being determined by the same parametric definition but using different values for the parameters of the regrading function to obtain different shapes; a second neural network processing circuit having as input at least one measurement from at least one sensor and having as output nodes providing a set of weights corresponding in quantity to at least a number of sets of outputs of said first neural network processing circuit; a combiner for adding first parameter values of the first set of outputs of the first neural network processing circuit multiplied by corresponding first weights from the second neural network processing circuit to corresponding parameter values for the same regrading function defining parameters of the second set of outputs of the first neural network processing circuit multiplied by corresponding second weights from the second neural network processing circuit to result in a final parameter value; a luma mapping circuit for mapping the input luma of the input high dynamic range image with a luma mapping function defined by at least two parameters including the last parameter value to obtain an output luma of the second image; 16. An apparatus for luminance regrading comprising:
2. 10. An apparatus for luminance regrading according to claim 1 specifically coupled to a sensor coupled to a display.
3. 3. An apparatus for luminance regrading according to claim 1 or 2, coupled to a luminance meter for providing a measure of the amount of light in the environment.
4. An apparatus for luminance regrading according to any one of claims 1 to 3, coupled to an image summarization unit for measuring luminance aspects of the input high dynamic range image.
5. 5. An apparatus for luminance regrading as claimed in any one of claims 1 to 4, comprising a user interface unit for determining a set of user specified weights, and a selector for selecting at least one user specified weight in place of a corresponding weight from the second neural network processing circuit for feeding into the combiner.
6. 6. An apparatus for luminance regrading according to any one of claims 1 to 5, characterized in that the apparatus for luminance regrading is included in a device including a display panel.
7. 7. An apparatus for luminance regrading according to any one of claims 1 to 6, characterized in that the apparatus for luminance regrading is included within a video coding and communication device for communicating video to one or more receivers.
8. 1. A method for luminance regrading of an input high dynamic range image of a first luminance dynamic range to a second image of a second luminance dynamic range, the second image having a maximum luminance that may be higher or lower than the maximum luminance of the input high dynamic range image, the method comprising: using a pre-trained first neural network processing circuit having as input a set of pixel intensities of the input high dynamic range image, the first neural network processing circuit having at least two sets of outputs; the set of outputs having output nodes for providing parameter values of a parametric regrading equation, the parametric regrading equation being composed of at least two partial functions, the amounts of the at least two partial functions in the parametric regrading equation being controlled by at least two corresponding output parameter values; using a first neural network processing circuit, a first set of outputs including at least two parameter values that determine a shape of a regrading function, and a second set of outputs including at least two corresponding parameter values for parameters that are different from the parameter values in the first set of outputs, the second set of outputs determining a different shape of the regrading function; using a second neural network processing circuit having as input at least one measurement from at least one sensor and having as output a set of weights having weight amounts at least equal to the amounts of the set of outputs of the first neural network processing circuit; adding a first one of the at least two parameter values of the first set of outputs of the first neural network processing circuit after multiplication by a corresponding first weight from the second neural network processing circuit to a corresponding first one of the at least two parameter values of the second set of outputs of the first neural network processing circuit, the value of a corresponding parameter of the same regrading function from the second set of outputs of the first neural network processing circuit, the value being multiplied by a corresponding second weight from the second neural network processing circuit before addition, to result in a final parameter value; applying a luma mapping function having a curve shape defined by the last parameter value to map the input luma of the input high dynamic range image to obtain an output luma of the second image; The method comprising:
9. A computer program comprising code which, when executed, performs the method of claim 8.