Backward compatible high dynamic range (HDR) images

Abstract

The present disclosure relates to backward compatible high dynamic range (HDR) images. The present disclosure discloses an apparatus, method and program storage device for creating and / or displaying backward compatible high dynamic range (HDR) images, the apparatus, method and program storage device comprising: obtaining two or more exposures of a scene; creating a gain map based on at least one of the two or more exposures, wherein the gain map comprises a plurality of pixels, each pixel corresponding to a portion of the scene, and wherein the value of the pixel in the gain map comprises an indication of a brightness level for the corresponding portion of the scene; combining the two or more exposures to form a first image; tone mapping the first image based on a standard dynamic range (SDR) format to generate a first SDR image of the scene; and storing the first SDR image and the created gain map in a first enhanced image file.

Description

Technical Field

[0001] The present disclosure relates generally to the field of digital image processing. More specifically, but not by way of limitation, the present disclosure relates to techniques for creating and displaying backward-compatible high dynamic range (HDR) images. Background Art

[0002] Conventional HDR photography techniques have been developed and refined to handle the capture and display of the huge dynamic range of the real world, versus the limited dynamic range available in typical imaging sensors and display devices. This is typically accomplished by capturing multiple so-called "brackets" of an image, i.e., images with different exposure times (also called "stops"); fusing these multiple brackets together into a single image; and then applying some form of "tone mapping" operation to bring the fused image into the dynamic range of a typical display. In the case of a standard dynamic range (SDR) display, the resulting image is typically stored as an 8-bit image, giving the image a final dynamic range of approximately 250:1.

[0003] As display technology continues to improve, there are now displays that are capable of producing much higher dynamic ranges than the 250:1 range mentioned above. The ecosystem for the film and video industry has evolved to take advantage of these HDR displays by creating a variety of competing HDR video file formats that can be processed by a variety of displays. However, digital still photography does not yet have this ubiquitous HDR ecosystem.

[0004] It may therefore be beneficial to provide devices, methods and non-transitory program storage devices (NPSDs) capable of capturing, storing and displaying HDR digital still images in an enhanced image file format that is both compatible with existing SDR image display workflows and seamlessly allows display of the image and its full range of vivid HDR content on a "smart" or "aware" HDR display. Summary of the invention

[0005] According to some embodiments described herein, at image capture time, for most HDR scenes, the image capture device may capture at least three brackets, which are generally referred to as "EV0," "EV-," and "EV+" (these terms are defined in more detail below). Generally speaking, an EV0 image corresponds to a "normal" or "ideal" exposure setting for the scene; an EV- image corresponds to an underexposed (e.g., 4 times darker) image of the scene; and an EV+ image corresponds to an overexposed (e.g., 4 times brighter) image of the scene. The EV0 image is sometimes also referred to as an "auto-exposure" capture of the scene, and has the optimal settings, as determined by the capabilities of the image sensor of the image capture device used to capture the scene.

[0006] If some pixels in the EV-image (e.g., an image that is expected to be 4 times darker on average than the EV0 image if the EV-image is underexposed by 2 exposure "stops") are still bright, this is a strong indication that such pixels are actually very bright pixels in the real scene (such pixels are also referred to herein as "HDR pixels" because their intensity values ​​may give the overall image a higher dynamic range than can be displayed on a typical SDR display). This set of "still bright" pixels or "HDR pixels" in the EV-image can be used to generate a "gain map" of all pixels in the image that may indicate true HDR pixels. This gain map can then be thresholded and / or downscaled as needed for a given specific implementation, and then stored alongside or with the final bracketed image of the scene (e.g., within a single enhanced image file format), which itself may have been formed by blending together two or more of the original bracketed captures, and may also have been tone mapped as needed (e.g., for a typical SDR display). For example, in modern HEIF / HEIC file formats, this gain map may be stored as an "auxiliary" image along with the final bracketed image of the scene. For the JPEG file format, the gain map may be stored, for example, by appending the gain map to the end of the standard image data in the file structure.

[0007] At this point, the enhanced image file format can be shared with any user and / or any display device. If the display device that is ultimately to display the enhanced image is not intelligent or aware of the presence of the gain map within the enhanced image file format, the enhanced image will simply be displayed at the display device, as any other SDR still image capture would be. However, if the display device is intelligent or aware (i.e., does know the presence of the gain map within the enhanced image file format and how to apply the gain map), the display device can utilize the gain map information by, for example, querying the display to request the amount of extended range headroom available (i.e., the display device's illumination range that is higher than the "normal range" of a conventional SDR display, e.g., sometimes referred to as a brightness level higher than so-called "diffuse white", and may also depend on ambient lighting conditions). Assuming that there is available extended range headroom, the display device can synthesize the final bracketed image, the gain map, and the knowledge of the available extended range headroom of the display to present a final enhanced image on the display that utilizes the available higher dynamic range of the display for those areas of the originally captured scene that correspond to true HDR content in the scene, as originally captured.

[0008] Thus, disclosed herein are devices, methods, and NPSDs for providing creation and display of backward-compatible HDR images. According to some embodiments, a method for digital image processing is provided, the method comprising: obtaining two or more exposures of a scene captured by one or more image capture devices; creating a gain map based on at least one of the two or more exposures, wherein the gain map comprises a plurality of pixels, each pixel corresponding to a portion of the scene, and wherein the values ​​of the pixels in the gain map comprise an indication of a brightness level for the corresponding portion of the scene; combining the two or more exposures to form a first image; tone mapping the first image based on a first SDR format to generate a first SDR image of the scene; and storing the first SDR image and the created gain map in a first enhanced image file. According to some such embodiments, the first enhanced image file may comprise one of the following: a HEIF file; a HEIC file; a PNG file; a GIF file; or a JPEG file.

[0009] According to some such embodiments, the first exposure of the two or more exposures comprises an EV0 image. According to other embodiments, the second exposure of the two or more exposures comprises an EV-image that is underexposed relative to the EV0 image. According to still other embodiments, when the EV-image is obtained, the EV-image has been registered with the EV0 image. According to still other embodiments, the first image comprises an HDR image. According to still other embodiments, the third exposure of the two or more exposures comprises an EV+ image that is overexposed relative to the EV0 image.

[0010] According to other such embodiments, creating the gain map further comprises creating a gain map having a lower resolution than the first image (e.g., a resolution of one quarter or less of the resolution of the first image). According to other embodiments, pixel values ​​in the gain map that are equal to or below a predetermined noise threshold may be set to zero.

[0011] According to still other such embodiments, the created gain map may be applied to the first SDR image to create a second enhanced image, wherein the second enhanced image has a higher dynamic range than the first SDR image. According to other embodiments, the second enhanced image may be displayed on a display device. According to still other embodiments, the method may further determine an amount of available headroom at the display device, and apply the created gain map to the first SDR image based at least in part on the determined amount of available headroom to create the second enhanced image. According to still other embodiments, a value (e.g., an average) representing a pixel in the created gain map may be determined, and then the determined amount of available headroom may be modulated, for example, based on the determined value representing a pixel in the created gain map.

[0012] In other embodiments, applying the created gain map to the first SDR image to create the second enhanced image may further include determining whether a human face is present in the first image, and then applying the created gain map to the first SDR image to further create the second enhanced image based at least in part on the determination of whether a human face is present in the first image.

[0013] According to yet other embodiments, a method for digital image processing is provided, the method comprising: obtaining a first enhanced image file, wherein the first enhanced image file comprises a tone-mapped first SDR image of a scene and a gain map corresponding to the scene; determining an amount of available headroom at a display device where the first SDR image is to be displayed; determining values ​​of pixels in a representative gain map; modulating the determined amount of available headroom based on the determined values ​​of the pixels in the representative gain map; applying the gain map to the first SDR image to create a second enhanced image based at least in part on the determined amount of available headroom and the determined values ​​of the pixels in the representative gain map; and causing the second enhanced image to be displayed at the display device.

[0014] Various NPSD implementations are also disclosed herein. Such NPSDs may be readable by one or more processors. Instructions may be stored on the NPSD for causing one or more processors to perform any of the digital image processing techniques for creating and / or displaying backward-compatible high dynamic range (HDR) images disclosed herein.

[0015] According to the embodiments of the NPSD and methods listed above, various programmable electronic devices are also disclosed herein. Such electronic devices may include: one or more image capture devices, such as optical image sensors / camera units; displays; user interfaces; one or more processors; and / or memory, which is coupled to the one or more processors. Instructions may be stored in the memory, which instructions cause the one or more processors to execute instructions according to the various techniques disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 An enhanced image creation workflow is shown according to one or more embodiments.

[0017] Figure 2 An enhanced image display workflow is shown according to one or more embodiments.

[0018] Figure 3 is a flow chart illustrating a method of creating an enhanced image according to various embodiments.

[0019] Figure 4 is a flow chart illustrating a method of enhancing image display according to various embodiments.

[0020] Figure 5 is a block diagram illustrating a programmable electronic computing device in which one or more of the techniques disclosed herein may be implemented. DETAILED DESCRIPTION

[0021] In the following description, for the purpose of explanation, a lot of specific details are set forth in order to provide a thorough understanding of the invention disclosed herein. However, it is obvious to those skilled in the art that the present disclosure can be practiced in the absence of these specific details. In other cases, structures and devices are shown in the form of block diagrams to avoid blurring the present disclosure. References to numbers without subscripts or suffixes should be understood as references to all subscripts and suffixes corresponding to the figure marks. In addition, the language used in the present disclosure has been selected primarily for readability and instructive purposes, and may not be selected to delimit or limit the subject matter of the present disclosure, and therefore may need to resort to the claims to determine such invention subject matter. References to "one embodiment" or "embodiment" (or similar expressions) in the specification refer to specific features, structures or characteristics described in conjunction with the embodiment included in at least one embodiment of one of the inventions, and multiple references to "one embodiment" or "embodiment" should not be understood as all necessarily referring to the same embodiment.

[0022] definition

[0023] Discussion will now turn to the nomenclature that will be used herein to refer to the various differently exposed images that may be captured by an image capture device. As with the traditional bracket notation, "EV" stands for Exposure Value and refers to a given exposure level of an image (which may be controlled by one or more settings of the device, such as a shutter speed setting and / or an aperture setting of the image capture device). Different images can be captured at different EVs, with a one EV difference (also referred to as a "stop") between the images being equal to a predefined power difference in the exposure. Typically, a "stop" is used to represent the power of two differences between exposures. Thus, changing the exposure value can change the amount of light received for a given image, depending on whether the EV is increasing or decreasing. For example, one stop doubles (or halves) the amount of light received for a given image, depending on whether the EV is increasing (or decreasing), respectively.

[0024] An "EV0" image in a traditional bracketing exposure refers to an image captured using an exposure value determined by the exposure algorithm of the image capture device, for example, as specified by the automatic exposure (AE) mechanism. Typically, an EV0 image is assumed to have an ideal exposure value (EV) given the current lighting conditions. It should be understood that the term "ideal" used in the context of an EV0 image in this article refers to an ideal exposure value, as calculated for a given image capture system. In other words, it is a system-dependent version of the ideal exposure. For given lighting conditions, different image capture systems may have different versions of ideal exposure values ​​and / or may utilize different constraints and analyses to determine the exposure settings for capturing EV0 images.

[0025] The term "EV-" image refers to an underexposed image captured at a stop lower than the stop that would be used to capture an EV0 image (e.g., 0.5, 1, 2, or 3 stops). For example, an "EV-1" image refers to an underexposed image captured at one stop below the EV0 image exposure, and an "EV-2" image refers to an underexposed image captured at two stops below the exposure value of the EV0 image. The term "EV+" image refers to an overexposed image captured at a stop higher than the EV0 image (e.g., 0.5, 1, 2, or 3 stops). For example, an "EV+1" image refers to an overexposed image captured at one stop above the EV0 image exposure, and an "EV+2" image refers to an overexposed image captured at two stops above the exposure value of the EV0 image.

[0026] According to some embodiments, some degree of overexposure (e.g., EV+2) may also be intentionally employed in bright scenes and scenes with moderate brightness in order to restore a desired amount of shadow detail in the captured images. In order to maintain consistent brightness levels across the various captured images, the gain may be reduced in proportion to the increase in exposure time for capture, since according to some embodiments, brightness may be defined as the product of gain and exposure time. According to some embodiments, even images captured as EV0 images may still be modified in one or more ways as desired for a given implementation, for example, in terms of gain and exposure time combinations.

[0027] In some embodiments, the specific exposure time (and / or gain) of a captured image may be further based at least in part on the ambient light level surrounding the image capture device, with brighter ambient conditions allowing for relatively shorter image exposure times, and darker ambient conditions allowing for the use of relatively longer image exposure times. In still other embodiments, the specific exposure time (and / or system gain) for a particular image may be further based at least in part on whether the image capture device is using an OIS system during the capture operation.

[0028] Example backwards-compatible HDR image creation and display workflow

[0029] Now go to Figure 1 , shows an enhanced image creation workflow 100 according to one or more embodiments. Exposure #1 (105) may represent a first exposure of the captured scene, such as an EV0 or "auto-exposed" image. Exposure #2 (110) may represent a second exposure of the captured scene, such as an EV- or "underexposed" image. In some embodiments, it may be preferred that the two exposures are as closely synchronized with each other in capture time as possible so that image registration (i.e., finding correspondence between the exposures) is as reliable and error-free as possible. In some preferred embodiments, the exposures may even arrive from the image sensor in a pre-registered manner, i.e., they may have been registered (e.g., by the sensor hardware) before any image processing firmware or software performs further image processing operations (such as fusion, tone mapping, etc.).

[0030] exist Figure 1 In an example where exposure #2 (110) represents an EV-image that is underexposed (e.g., 2x darker, 4x darker, 8x darker, etc.) relative to the EV0 image, regions in the EV-image that remain bright (e.g., as defined by one or more brightness or intensity thresholds) may indicate portions of the original captured scene that actually represent "real" HDR content. For example, in exposure #2 (110), pixels inside region 115 remain bright, even after intentionally "stopping down" and underexposing in the EV-image.

[0031] In some embodiments, as shown at block 130, the two or more exposures may be combined (e.g., fused) to form a first image. In some cases, for example, for a typical SDR display, the first image may be tone mapped to generate the first SDR image. Figure 1 The use of one EV0 exposure and one EV-exposure in the examples of are merely illustrative, and other types and numbers of image exposures may be used in a given implementation. For example, below certain scene lux levels, capturing an EV- image may not provide any usable or helpful image information and therefore may not be captured. In such cases, the image capture device may instead determine to capture multiple EV0 images, then blend those EV0 images together, and then apply an amount of digital gain or tone mapping to a portion (or all) of the range of the fused image signal.

[0032] In some embodiments, a gain map 120 may be created based on at least one of the two or more exposures (in Figure 11 ( 110 ), the gain map 120 is based on the values ​​of pixels in exposure #2 ( 110 ). The gain map 120 may include a plurality of pixels, wherein each pixel corresponds to some portion of the captured scene, and wherein the values ​​of the pixels in the gain map include an indication of the brightness level of the corresponding portion of the scene (e.g., higher values ​​in the gain map represent brighter portions of the original captured scene, and lower values ​​in the gain map represent darker portions of the original captured scene). For example, the gain map 120 pixel values ​​for the portion of the captured scene that corresponds to the aforementioned bright pixels within region 115 of exposure #2 ( 110 ) remain high in the gain map 120 (as indicated by the brighter pixel values ​​in the corresponding region 125 of the gain map 120 ), which will result in these pixels being effectively “boosted” the maximum amount if or when the enhanced image file is to be displayed on a perceptual HDR display, as will be explained in more detail below.

[0033] It should be understood that in some implementations, the gain map may be a single image plane (e.g., associated with the overall brightness / luminance value of the pixel), while in other implementations, multiple gain map image planes may be stored in the enhanced image file (e.g., with a separate gain map for each primary color channel). In some cases, for the purpose of memory efficiency, it may also be preferred to create and store the gain map 120 at a lower resolution than the first SDR image and / or to compress the information stored in the gain map 120 before storage. For example, in some implementations, the lower resolution gain map 120 may include dimensions that are each reduced to half the size of the first SDR image, resulting in a resolution that is one quarter of the first SDR image. It should be understood that the lower resolution gain map 120 may have other resolutions, such as 1 / 2 the resolution of the first SDR image, 1 / 8 the resolution of the first SDR image, 1 / 16 the resolution of the first SDR image, etc. It is also possible not to create a lower resolution gain map, for example, by creating a gain map with the same resolution as the first SDR image.

[0034] In some embodiments, the first SDR image from block 130 may be stored alongside or with the lower resolution gain map 120 (e.g., at a designated portion of the same image file), as indicated by enhanced image file 135. As will be explained in greater detail herein, enhanced image file 135 provides a backward compatible HDR digital still image that can be displayed on a current "unaware" SDR display using an existing display pipeline and utilizing the aforementioned blending and / or tone mapping operations to render an optimal image for the SDR display while allowing an "aware" HDR display to access the information stored in the lower resolution gain map 120 and use that information to intelligently enhance the first SDR image 130 by, for example, utilizing available display brightness headroom to increase the brightness of those portions of the captured scene indicated in the gain map 120 as indicating true HDR scene content.

[0035] Now go to Figure 2 , illustrating an enhanced image display workflow 200 according to one or more embodiments. Figure 2 Assuming the above reference has been executed Figure 1 The workflow described above, i.e., wherein an enhanced image file 135 has been received and is ready to be displayed by a "perceived" HDR display. Figure 1 Starting with the first SDR image 130 and the lower resolution gain map 120 created by the first SDR image file 135, the first SDR image 130 and the lower resolution gain map 120 can be extracted from the enhanced image file 135. Next, the lower resolution gain map 120 can be upscaled in the form of the full resolution gain map 205, for example to match the resolution of the first SDR image 130. In some embodiments, the full resolution gain map 205 can be modified or tuned in one or more ways as needed. For example, pixel values ​​in the gain map that are equal to or below a predetermined noise threshold can be set to zero to avoid inadvertently amplifying noise data in the final displayed enhanced image. In some embodiments, the gain map can be thresholded with a function (e.g., a smooth step function) that can be used to "crush" noise in shadows while also enhancing bright areas in the image a little. In some embodiments, the gain map can also be encoded with a gamma function (e.g., a sqrt or sRGB gamma function). In addition or alternatively, if it is desired to smooth the gain map data, for example, to remove "islands" of outliers in the gain map and / or fill "holes" of outliers with lower values ​​in the gain map, etc., various morphological operations can be performed on the gain map. If desired, other operations may also be performed on the gain map, such as operations based on other semantic features, such as using a person segmentation mask to ensure that human subjects are enhanced by a desired amount (e.g., perhaps more or less than otherwise indicated in the gain map).

[0036] At the same time or prior thereto, a display device (i.e., a computing device having or communicating with the aforementioned "aware" HDR display) may determine an amount of available headroom for displaying the first SDR image. A typical SDR display may be said to be a display having a so-called "normal range" (e.g., corresponding to pixel bit values ​​of 0 to 255) or a display having a normalized range of 0.0 to 1.0, while the available headroom or "extended range headroom" of an HDR display (e.g., having some larger normalized range (such as 0.0 to 8.0)) will allow certain areas of the image to expand, i.e., become brighter than the normal range that the SDR display is capable of. Typically, the amount of "extended range headroom" available at the display is also a function of the current ambient lighting level, e.g., higher ambient lighting levels result in less available extended range headroom to push the HDR content of the image into.

[0037] In some embodiments, at block 215, workflow 200 may also determine a value (e.g., an average) representing a pixel in full-resolution gain map 205. In some cases, the determined value representing a pixel in full-resolution gain map 205 may be used to modulate (e.g., alter in some manner based on the tuning preferences and / or desired look and feel of a given implementation) the amount of available headroom determined at block 210. For example, in some implementations, if the values ​​representing pixels in the full-resolution gain map 205 indicate that a relatively high percentage of HDR pixels are present in the first enhanced image (e.g., via a gain map of image pixels having a determined average value above a threshold and / or having a brightness that exceeds a threshold percentage sufficient (e.g., exceeds another threshold, such as a normalized luminance value of 0.75 on a scale of 0 to 1, etc.), it may be desirable to display the enhanced image file 135 using less than all (or less than a typical amount) of the available headroom for the display device determined at block 210, since displaying a large percentage of pixels in the enhanced image as being too bright (even if such pixels are indeed bright or “HDR pixels” in the captured scene) may be visually irritating. For similar reasons, in some embodiments, it may also be desirable to modulate the amount of available headroom based on the size of the image that is ultimately being displayed (e.g., in terms of the actual “real world” screen space encompassed by the displayed image). For example, when an HDR image is being displayed as a small square thumbnail on a mobile phone, using all available headroom may be visually acceptable, but when the same HDR image is displayed full screen on a 75-inch television monitor, using all available headroom may result in unacceptably bright images.

[0038] Finally, at block 220, the full-resolution gain map 205 may be applied to the first SDR image 130 based at least in part on the amount of available headroom determined at block 210 (subject to any modulation, e.g., as caused by the values ​​representing pixels in the gain map determined at block 215) to create a second enhanced image 225 for display (i.e., an image having a higher dynamic range than the first SDR image). In some embodiments, applying the full-resolution gain map 205 to the first SDR image 130 to create the second enhanced image 225 for display may further include determining whether there is a human face in the first SDR image. For example, if there is a human face, it may not be desirable to use the entire available headroom because an overly bright human face may have an unpleasant or unnatural appearance when displayed on an HDR display. Therefore, according to some embodiments, for images in which there is a human face, the amount of available headroom determined to be used may be reduced (e.g., divided in half). Furthermore, as mentioned above, the values ​​stored in the gain map itself may also be modified based on a semantic understanding of the image content (e.g., the presence of a human face).

[0039] In some embodiments, the value of a pixel in the second enhanced image 225 may be determined according to the following formula:

[0040] SecondEnhancedImageRGB=FirstSDRImageRGB*(1.0+(ModulatedAvailableHeadrooom–1.0)*GainMapValue) (Formula 1).

[0041] In other words, for any pixel in the second enhanced image 225, the workflow 200 can add anywhere from 0% to 100% of the additional available headroom based on the corresponding gain map value (and subject to any desired modulation of the amount of available headroom the display is permitted to utilize at a given time). [In this example, the gain map stores values ​​in a normalized range of 0 to 1.]

[0042] In some embodiments, only pixels in the first SDR image having values ​​above a certain threshold (e.g., 18% gray value) may be encoded into the extended range headroom available on a given HDR display (e.g., using a smooth step function), while values ​​below the certain threshold may remain unchanged. In other embodiments, the entire image range of the first SDR image may be encoded into the extended range headroom available on a given HDR display, thereby providing a smooth and natural appearance to the displayed enhanced image.

[0043] According to some embodiments, it may also be desirable to be able to "transcode" (i.e., convert) enhanced image files back and forth between a "true" HDR format (e.g., a still image format with 10-bit, 12-bit, 16-bit color depth, etc.) and the "enhanced" backward-compatible HDR image format described herein (including a tone-mapped SDR image and corresponding gain map) for use with a perceptual HDR display.

[0044] Exemplary method for creating and displaying backward compatible HDR images

[0045] Now go to Figure 3 , a flowchart 300 illustrating a method of creating an enhanced image according to various embodiments is shown. First, at step 302, the method 300 may obtain two or more exposures of a scene captured by one or more image capture devices. Next, at step 304, the method 300 may create a gain map based on at least one of the two or more exposures, wherein the gain map includes a plurality of pixels, each pixel corresponding to a portion of the scene, and wherein the values ​​of the pixels in the gain map include an indication of a brightness level of the corresponding portion of the scene.

[0046] Next, at step 306, method 300 may combine the two or more exposures to form a first image, if desired (e.g., if combining the two or more exposures in the exposure would produce a higher quality final bracketed image than using only a single exposure (such as an EV0 exposure), which may not always be the case, depending on the scene composition and / or the exposure settings used by the image capture device). At step 308, method 300 may tone map the first image based on a first SDR format (e.g., sRGB or P3) to generate a first SDR image of the scene. Finally, at step 310, method 300 may store the first SDR image and the created gain map in a first enhanced image file. As mentioned above, in some embodiments, the first enhanced image file may include one of the following: a HEIF file; a HEIC file; a PNG file; a GIF file; or a JPEG file

[0047] Figure 3 Optional steps 312 and 314 are included in the method 300 to further illustrate how the intelligent or perceptual display device can utilize the created enhanced image file to produce a higher dynamic range version of the image content. Specifically, at step 312, the method 300 (assuming that the method is being performed on a perceptual display device) can apply the created gain map to the first SDR image to create a second enhanced image, wherein the second enhanced image has a higher dynamic range than the first SDR image, and then at step 314, the second enhanced image is displayed on the perceptual display device.

[0048] Now go to Figure 4, a flowchart 400 showing an enhanced image display method according to various embodiments is shown. First, at step 402, method 400 may obtain a first enhanced image file, wherein the first enhanced image file includes a tone-mapped first SDR image of a scene and a gain map corresponding to the scene. In some specific implementations, the value of the gain map may be made available to the developer and / or calling application via an application programming interface (API). Next, at step 404, method 400 may determine an amount of available headroom at a display device on which the first SDR image is to be displayed. Next, at step 406, method 400 may determine a value (e.g., an average value) representing a pixel in the gain map. Next, at step 408, method 400 may modulate the determined amount of available headroom based on the determined value representing a pixel in the gain map, for example, as described above with reference to FIG. Figure 2 described.

[0049] Next, at step 410, method 400 may apply the gain map to the first SDR image based at least in part on the determined amount of available headroom and the determined values ​​representing pixels in the gain map to create a second enhanced image (i.e., having a higher dynamic range than the first SDR image). Finally, at step 412, method 400 may cause the second enhanced image to be displayed at a display device if, for example, desired by a user or if, for example, requested by a user.

[0050] While the techniques herein are largely described in the context of displaying SDR images with greater dynamic range on certain perceptual HDR displays, it should be understood that these techniques may be applicable and beneficial, such as in any application that would benefit from a more accurate image capture-time "ground truth" knowledge of the locations of true scene highlights in captured images. For example, various photo editing tools (e.g., highlight editors or bokeh effect generators) may benefit from knowledge of the exact locations, intensities, and colors of true scene highlights, rather than making educated inferences based on scene brightness in the resulting tone-mapped image being edited.

[0051] (Note: The use of a dashed box indicates Figure 3 and Figure 4 The optionality of performing some of the steps in the

[0052] Exemplary Electronic Computing Devices

[0053] See now Figure 5, a simplified functional block diagram of an exemplary programmable electronic computing device 500 according to one embodiment is shown. The electronic device 500 may be, for example, a mobile phone, a personal media device, a portable camera, or a system of a tablet, a laptop or a desktop computer. As shown, the electronic device 500 may include a processor 505, a display 510, a user interface 515, a graphics hardware 520, a device sensor 525 (e.g., a proximity sensor / ambient light sensor, an accelerometer, an inertial measurement unit and / or a gyroscope), a microphone 530, an audio codec 535, a speaker 540, a communication circuit 545, an image capture device 550 (e.g., it may include multiple camera units / optical image sensors with different characteristics or capabilities (e.g., still image stabilization (SIS), HDR, OIS systems, optical zoom and digital zoom, etc.), a video codec 555, a memory 560, a storage device 565, and a communication bus 570.

[0054] The processor 505 may execute instructions necessary for implementing or controlling the operation of various functions performed by the electronic device 500 (e.g., generation and / or processing of images according to various embodiments described herein). The processor 505 may, for example, drive the display 510 and may receive user input from the user interface 515. The user interface 515 may take a variety of forms, such as buttons, keypads, dials, click wheels, keyboards, display screens, and / or touch screens. The user interface 515 may, for example, be a conduit through which a user can view a captured video stream and / or indicate a specific image frame that the user wants to capture (e.g., by clicking a physical or virtual button at the moment when the desired image frame is being displayed on the display screen of the device). In one embodiment, the display 510 may display a video stream that is captured while the processor 505 and / or graphics hardware 520 and / or image capture circuitry are simultaneously generating the video stream and storing the video stream in the memory 560 and / or storage device 565. The processor 505 may be a system on a chip (SOC) such as those found in mobile devices, and may include one or more dedicated graphics processing units (GPUs). Processor 505 may be based on a reduced instruction set computer (RISC) or complex instruction set computer (CISC) architecture or any other suitable architecture, and may include one or more processing cores. Graphics hardware 520 may be specialized computing hardware for processing graphics and / or assisting processor 505 in performing computing tasks. In one embodiment, graphics hardware 520 may include one or more programmable graphics processing units (GPUs) and / or one or more specialized SOCs, for example, specifically designed to implement neural network and machine learning operations (e.g., convolutions) in a more energy-efficient manner than a host device central processing unit (CPU) or a typical GPU (such as Apple's Neural Engine processing core).

[0055] For example, according to the present disclosure, the image capture device 550 may include one or more camera units configured to capture images, for example, images that can be processed to generate a higher dynamic range version of the captured image. The output from the image capture device 550 may be processed at least in part by the following devices: a video codec 555 and / or a processor 505 and / or graphics hardware 520, and / or a dedicated image processing unit or image signal processor incorporated in the image capture device 550. The images captured in this way may be stored in a memory 560 and / or a storage device 565. The memory 560 may include one or more different types of media used by the processor 505, the graphics hardware 520, and the image capture device 550 to perform device functions. For example, the memory 560 may include a memory cache, a read-only memory (ROM), and / or a random access memory (RAM). The storage device 565 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. The storage device 565 may include one or more non-transitory storage media, including, for example, magnetic disks (fixed hard disks, floppy disks, and removable disks) and tapes, optical media such as CD-ROMs and digital video disks (DVDs), and semiconductor memory devices such as electrically programmable read-only memories (EPROMs), and electrically erasable programmable read-only memories (EEPROMs). The memory 560 and the storage device 565 may be used to maintain computer program instructions or codes organized into one or more modules and written in any desired computer programming language. For example, when executed by the processor 505, such computer program code may implement one or more of the methods or processes described herein. The power supply 575 may include a rechargeable battery (e.g., a lithium-ion battery, etc.) or other electrical connection to a power source (e.g., to a power source) for managing the electronic components and associated circuits of the electronic device 500 and / or providing power to the electronic components and associated circuits of the electronic device.

[0056] It should be understood that the above description is intended to be exemplary and not restrictive. For example, the above embodiments may be used in combination with each other. Upon reviewing the above description, many other embodiments will be apparent to those skilled in the art. Therefore, the scope of the present disclosure should be determined with reference to the attached claims and the full scope of equivalents to such claims.

Claims

1. A device, comprising: Display devices; Memory; and one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions that cause the one or more processors to: obtaining two or more exposures of a scene captured by one or more image capture devices; creating a gain map based on at least one of the two or more exposures, wherein the gain map comprises a plurality of pixels, each pixel corresponding to a portion of the scene, and wherein values ​​of the pixels in the gain map comprise an indication of a brightness level of the corresponding portion of the scene; combining the two or more exposures to form a first image, wherein the gain map has a lower resolution than the first image; Performing tone mapping on the first image based on a first standard dynamic range (SDR) format to generate a first SDR image of the scene; storing the first SDR image and the created gain map in a first enhanced image file; determining an amount of available headroom at the display device; Amplifying the gain map to obtain a full-resolution gain map to match the resolution of the first SDR image; determining a value representing the pixel in the full resolution gain map; modulating the determined amount of available headroom based on the determined value representing the pixel in the full resolution gain map; applying the full resolution gain map to the first SDR image to create a second enhanced image based at least in part on the determined amount of available headroom and the determined values ​​representing the pixels in the full resolution gain map; as well as The second enhanced image is caused to be displayed at the display device.

2. The apparatus of claim 1 , wherein the first exposure of the two or more exposures comprises an EV0 image.

3. The apparatus of claim 2, wherein a second exposure of the two or more exposures comprises an EV-image that is underexposed relative to the EV0 image.

4. The apparatus according to claim 3, wherein when the EV-image and the EV0 image are obtained, the EV-image is registered with the EV0 image. The apparatus of claim 3 , wherein the first image comprises a high dynamic range (HDR) image.

6. The apparatus of claim 3, wherein a third exposure of the two or more exposures comprises an EV+ image that is overexposed relative to the EV0 image.

7. The apparatus of claim 1, wherein the lower resolution of the gain map comprises a resolution that is one quarter of the resolution of the first image or less.

8. The apparatus of claim 1 , wherein the instructions causing the one or more processors to create a gain map further comprise instructions causing the one or more processors to: Pixel values ​​in the gain map that are equal to or below a predetermined noise threshold are set to zero.

9. The apparatus of claim 1, wherein the first enhanced image file comprises one of: a HEIF file; a HEIC file; a PNG file; a GIF file; or JPEG files.

10. The apparatus of claim 1, wherein the instructions causing the one or more processors to apply the created gain map to the first SDR image to create a second enhanced image further comprise instructions causing the one or more processors to: determining whether a face exists in the first image; and The created gain map is applied to the first SDR image to create a second enhanced image based at least in part on the determination of whether a face is present in the first image.

11. A non-transitory program storage device (NPSD), the NPSD comprising computer readable instructions executable by one or more processors to: obtaining two or more exposures of a scene captured by one or more image capture devices; creating a gain map based on at least one of the two or more exposures, wherein the gain map comprises a plurality of pixels, each pixel corresponding to a portion of the scene, and wherein values ​​of the pixels in the gain map comprise an indication of a brightness level of the corresponding portion of the scene; combining the two or more exposures to form a first image, wherein the gain map has a lower resolution than the first image; Performing tone mapping on the first image based on a first standard dynamic range (SDR) format to generate a first SDR image of the scene; storing the first SDR image and the created gain map in a first enhanced image file; determining an amount of available headroom at a display device on which the first SDR image is to be displayed; Amplifying the gain map to obtain a full-resolution gain map to match the resolution of the first SDR image; determining a value representing the pixel in the full resolution gain map; modulating the determined amount of available headroom based on the determined value representing the pixel in the full resolution gain map; applying the full resolution gain map to the first SDR image to create a second enhanced image based at least in part on the determined amount of available headroom and the determined values ​​representing the pixels in the full resolution gain map; as well as The second enhanced image is caused to be displayed at the display device.

12. A method comprising: obtaining a first enhanced image file, wherein the first enhanced image file comprises a tone-mapped first SDR image of a scene and a gain map corresponding to the scene, the gain map comprising a plurality of pixels, wherein the gain map has a lower resolution than the first SDR image; determining an amount of available headroom at a display device on which the first SDR image is to be displayed; Amplifying the gain map to obtain a full-resolution gain map to match the resolution of the first SDR image; determining a value representing the pixel in the full resolution gain map; modulating the determined amount of available headroom based on the determined value representing the pixel in the full resolution gain map; applying the full resolution gain map to the first SDR image to create a second enhanced image based at least in part on the determined amount of available headroom and the determined values ​​representing the pixels in the full resolution gain map; as well as The second enhanced image is caused to be displayed at the display device.

Images