Technique for generating a gain map based on acquired images - Patent Application 20070122997

Abstract

Various techniques for generating a gain map are disclosed. According to some embodiments, one technique for generating a gain map includes: (1) accessing a high dynamic range (HDR) image; (2) accessing a standard dynamic range (SDR) image; (3) for each pixel shared between the HDR image and the SDR image, (4) plotting the plurality of pixels on a graph, where the HDR value of the pixel is plotted on the Y-axis of the graph and the SDR value of the pixel is plotted on the X-axis of the graph; (5) establishing a first curve representing an approximation of the plotted pixels; (6) inverting the first curve to establish a second curve; (7) applying the second curve to the plotted pixels to establish replotted pixels; and generating a gain map based on the replotted pixels; and (8) embedding the gain map into the HDR image.

Description

[Technical Field]

[0001]

[0003] Embodiments described herein describe techniques for generating a gain map based on acquired images. In particular, the gain map may be generated by comparing a first image with a second image. The first image and the gain map can then be used to efficiently reconstruct the second image by embedding the gain map into the first image. [Background technology]

[0002] The dynamic range of an image refers to the range of pixel values ​​between the brightest and darkest parts of the image (often called "luminance"). In particular, conventional image sensors can only capture a limited range of luminance in a single exposure of a scene, at least relative to what the human eye can perceive from the same scene. In digital photography, this limited range is usually referred to as standard dynamic range (SDR).

[0003] Despite the aforementioned limitations of image sensors, improvements in photographic technology have made it possible to capture a wider range of light (referred to herein as high dynamic range (HDR)). This can be achieved by (1) capturing multiple "brackets" of images, i.e., images with different exposure times (also called "stops"), and then (2) fusing the bracketed images into a single image that incorporates different aspects of the different exposures. In this regard, a single HDR image has a wider dynamic range of luminance compared to what could otherwise be captured in each of the individual exposures. This makes HDR images superior to SDR images in some embodiments.

[0004] Display devices capable of displaying HDR images (in their true form) are becoming more accessible due to advances in design and manufacturing technologies. However, the majority of display devices currently in use (and continue to be manufactured) are only capable of displaying SDR images. Therefore, a device with an SDR-limited display that receives an HDR image must perform various tasks to convert (i.e., downgrade) the HDR image to an SDR image equivalent. Conversely, a device with an HDR-capable display that receives an SDR image may attempt to perform various tasks to convert (i.e., upgrade) the SDR image to an HDR image equivalent.

[0005] Unfortunately, the aforementioned conversion techniques typically produce inconsistent and / or undesirable results. In particular, downgrading an HDR image to an SDR image can introduce visual artifacts (e.g., banding) into the resulting image, which are often uncorrectable with additional image processing. Conversely, upgrading an SDR image to an HDR image involves applying various levels of inference, which may also introduce uncorrectable visual artifacts.

[0006] Therefore, what is needed is a technique that allows an image to be efficiently and accurately transitioned between different states. For example, it would be desirable to allow an HDR image to be downgraded to its true SDR counterpart (and vice versa) without relying on the aforementioned (and inadequate) conversion techniques. Summary of the Invention

[0007] Representative embodiments described herein disclose techniques for generating a gain map based on acquired images. In particular, the gain map may be generated by comparing a first image with a second image. The gain map may then be embedded into the first image, allowing the first image and the gain map to be used to efficiently reproduce the second image.

[0008] Another embodiment describes an alternative method for generating a gain map based on an HDR image and an SDR image. In particular, the method includes: (1) accessing an HDR image; (2) accessing an SDR image; (3) for each pixel shared between the HDR image and the SDR image, (4) plotting the plurality of pixels on a graph, where the HDR value of the pixel is plotted on the Y-axis of the graph and the SDR value of the pixel is plotted on the X-axis of the graph; (5) establishing a first curve representing an approximation of the plotted pixels; (6) inverting the first curve to establish a second curve; (7) applying the second curve to the plotted pixels to establish replotted pixels; and generating a gain map based on the replotted pixels; and (8) embedding the gain map into the HDR image.

[0009] Yet another embodiment illustrates an alternative method for generating a gain map based on a first version of an image and a second version of the image. In particular, the method includes the steps of: (1) accessing the first version of the image; (2) detecting at least one modification to the first version of the image that generates a second version of the image; (3) generating the gain map by comparing the first version of the image with the second version of the image; and (4) embedding the gain map into the first version of the image.

[0010] Other embodiments include a non-transitory computer-readable storage medium configured to store instructions that, when executed by a processor included in the computing device, cause the computing device to perform various steps of any of the aforementioned methods. Further embodiments include a computing device configured to perform various steps of any of the aforementioned methods.

[0011] Other aspects and advantages of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the described embodiments.

[0012] The present disclosure will be readily understood by the following detailed description in conjunction with the accompanying drawings, in which like reference numerals indicate like structural elements, and in which: [Brief explanation of the drawings]

[0013] [Figure 1] 1 illustrates an overview of a computing device that can be configured to perform various techniques described herein, according to some embodiments.

[0014] [Figure 2A] 1 shows a sequence of conceptual diagrams of a technique for generating a gain map based on an SDR image and an HDR image, according to some embodiments. [Figure 2B] 1 shows a sequence of conceptual diagrams of a technique for generating a gain map based on an SDR image and an HDR image, according to some embodiments. [Figure 2C] 1 shows a sequence of conceptual diagrams of a technique for generating a gain map based on an SDR image and an HDR image, according to some embodiments. [Figure 2D] 1 shows a sequence of conceptual diagrams of a technique for generating a gain map based on an SDR image and an HDR image, according to some embodiments. [Figure 2E] 1 shows a sequence of conceptual diagrams of a technique for generating a gain map based on an SDR image and an HDR image, according to some embodiments.

[0015] [Figure 3A] 10A-10C illustrate a sequence of conceptual diagrams of alternative techniques for generating gain maps based on SDR and HDR images, according to some embodiments. [Figure 3B] 10A-10C illustrate a sequence of conceptual diagrams of alternative techniques for generating gain maps based on SDR and HDR images, according to some embodiments. [Figure 3C] 10A-10C illustrate a sequence of conceptual diagrams of alternative techniques for generating gain maps based on SDR and HDR images, according to some embodiments. [Figure 3D] 10A-10C illustrate a sequence of conceptual diagrams of alternative techniques for generating gain maps based on SDR and HDR images, according to some embodiments. [Figure 3E] 10A-10C illustrate a sequence of conceptual diagrams of alternative techniques for generating gain maps based on SDR and HDR images, according to some embodiments. [Figure 3F] 10A-10C illustrate a sequence of conceptual diagrams of alternative techniques for generating gain maps based on SDR and HDR images, according to some embodiments. [Figure 3G] 10A-10C illustrate a sequence of conceptual diagrams of alternative techniques for generating gain maps based on SDR and HDR images, according to some embodiments.

[0016] [Figure 4A] 10A-10C show a sequence of conceptual diagrams of alternative techniques for generating gain maps based on two different versions of the same image, according to some embodiments. [Figure 4B] 10A-10C show a sequence of conceptual diagrams of alternative techniques for generating gain maps based on two different versions of the same image, according to some embodiments. [Figure 4C] 10A-10C show a sequence of conceptual diagrams of alternative techniques for generating gain maps based on two different versions of the same image, according to some embodiments. [Figure 4D] 10A-10C show a sequence of conceptual diagrams of alternative techniques for generating gain maps based on two different versions of the same image, according to some embodiments. [Figure 4E] 10A-10C show a sequence of conceptual diagrams of alternative techniques for generating gain maps based on two different versions of the same image, according to some embodiments.

[0017] [Figure 5]FIG. 1 shows a detailed diagram of a computing device that can be used to perform various techniques described herein, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0018] Representative examples of applications of the methods and apparatus according to the present application are described in this section. These examples are provided solely to add context and to aid in understanding the described embodiments. Thus, it will be apparent to one skilled in the art that the described embodiments may be practiced without some or all of these specific details. In other instances, well-known process steps have not been described in detail to avoid unnecessarily obscuring the described embodiments. Other applications are possible, and therefore the following examples should not be construed as limiting.

[0019] In the following Detailed Description, reference is made to the accompanying drawings which form a part of the description and in which is shown by way of illustration specific embodiments in accordance with the described embodiments. These embodiments have been described in sufficient detail to enable one skilled in the art to practice the described embodiments, but it is understood that these examples are not limiting, and that other embodiments may be used and changes may be made without departing from the spirit and scope of the described embodiments.

[0020] Representative embodiments described herein disclose techniques for generating a gain map based on acquired images. In particular, the gain map may be generated by comparing a first image with a second image. The first image and the gain map can then be used to efficiently reconstruct the second image by embedding the gain map into the first image. A more detailed description of these techniques is provided below in connection with FIGS. 1, 2A-2E, 3A-3G, 4A-4E, and 5.

[0021] FIG. 1 illustrates an overview 100 of a computing device 102 that can be configured to perform various techniques described herein. As shown in FIG. 1, the computing device 102 can include a processor 104, a volatile memory 106, and a non-volatile memory 124. A more detailed breakdown of exemplary hardware components that may be included in the computing device 102 is illustrated in FIG. 5; note that these components are omitted from the illustration of FIG. 1 solely for simplicity. For example, the computing device 102 can include additional non-volatile memory (e.g., a solid-state drive, a hard drive, etc.), other processors (e.g., a multi-core central processing unit (CPU)), a graphics processing unit (GPU), etc.). According to some embodiments, an operating system (OS) (not shown in FIG. 1) can be loaded into the volatile memory 106, and the OS can execute various applications that collectively enable the various techniques described herein to be implemented. For example, these applications can include an image analyzer 110 (and its internal components), a gain map generator 120 (and its internal components), one or more compressors (not shown in FIG. 1), etc.

[0022] As shown in FIG. 1 , the volatile memory 106 can be configured to receive a multi-channel image 108. The multi-channel image 108 can be provided, for example, by a digital imaging unit (not shown in FIG. 1 ) configured to capture and process digital images. According to some embodiments, the multi-channel image 108 can be comprised of a collection of pixels, with each pixel in the collection of pixels including a group of sub-pixels (e.g., red sub-pixels, green sub-pixels, blue sub-pixels, etc.). It should be noted that the term “sub-pixel” as used herein can be synonymous with the term “channel.” It should also be noted that the multi-channel image 108 can have different resolutions, layouts, bit depths, etc. without departing from the scope of this disclosure.

[0023] According to some embodiments, a given multi-channel image 108 can represent a standard dynamic range (SDR) image constituting a single exposure of a scene collected and processed by the digital imaging unit. A given multi-channel image 108 can also represent a high dynamic range (HDR) image constituting multiple exposures of a scene collected and processed by the digital imaging unit. To generate an HDR image, the digital imaging unit may capture the scene under different exposure brackets, for example, three exposure brackets often referred to as “EV0,” “EV−,” and “EV+.” Generally, an EVO image corresponds to a normal / ideal exposure of the scene (typically captured using the digital imaging unit's auto-exposure setting). An EV− image corresponds to an underexposed image of the scene (e.g., four times darker than EV0), and an EV+ image corresponds to an overexposed image of the scene (e.g., four times brighter than EV0). The digital imaging unit can combine different exposures to generate a resultant image incorporating a wider range of brightness relative to an SDR image. Note that the multi-channel image 108 described herein is not limited to SDR / HDR images. Conversely, multi-channel image 108 may represent any form of digital image (eg, scanned image, computer-generated image, etc.) without departing from the scope of this disclosure.

[0024] 1 , the multi-channel image 108 may (optionally) be provided to an image analyzer 110. According to some embodiments, the image analyzer 110 may include various components configured to process / modify the multi-channel image 108 as needed. For example, the image analyzer 110 may include a tone mapping unit 112 (e.g., configured to perform global / local tone mapping operations, inverse tone mapping operations, etc.), a noise reduction unit 114 (e.g., configured to reduce global / local noise in the multi-channel image), a color correction unit 116 (e.g., configured to perform global / local color corrections on the multi-channel image), and a sharpening unit 118 (e.g., configured to perform global / local sharpening corrections on the multi-channel image). It should be noted that the image analyzer 110 is not limited to the aforementioned processing units, and the image analyzer 110 may incorporate any number of processing units configured to perform any processing / modifications on the multi-channel image 108 without departing from the scope of the present disclosure.

[0025] As shown in FIG. 1 , the multi-channel image 108 may be provided to the gain map generator 120 after being processed by the image analyzer 110. However, it should be noted that the multi-channel image 108 may bypass the image analyzer 110 and be provided to the gain map generator 120, if desired, without departing from the scope of this disclosure. It should also be noted that the multi-channel image 108 may bypass one or more of the processing units of the image analyzer 110 without departing from the scope of this disclosure. For example, two given multi-channel images may pass through the tone mapping unit 112 to receive local tone mapping modifications and then bypass the remaining processing units in the image analyzer 110. In this regard, the two multi-channel images that have undergone local tone mapping operations may be utilized to generate a gain map 123 that reflects the performed local tone mapping operations. In either case, as described in more detail herein, upon receiving the two multi-channel images 108, the gain map generator 120 may generate the gain map 123 based on the two multi-channel images 108. Gain map generator 120 can then store gain map 123 in one of the two multi-channel images 108 to generate enhanced multi-channel image 122. It is further noted that the gain map generation technique can be performed at any time relative to receipt of the multi-channel image on which the gain map is based. For example, gain map generator 120 can be configured to postpone generation of the gain map when the digital imaging unit is actively being used to ensure adequate processing resources are available so that no slowdown is imposed on the user. A more detailed breakdown of how gain map generator 120 can generate gain map 123 is provided below in connection with FIGS. 2A-2E, 3A-3G, and 4A-4E.

[0026] Additionally, although not shown in FIG. 1 , one or more compressors may be implemented on the computing device 102 to compress the enhanced multi-channel image 122. For example, the compressor may implement a Lempel-Ziv-Welch (LZW)-based compressor, other types of compressors, a combination of compressors, etc. Furthermore, the compressor may be implemented in any manner to establish the most efficient environment for compressing the enhanced multi-channel image 122. For example, multiple buffers may be instantiated (pixels may be pre-processed in parallel), each buffer may be coupled to a separate compressor, and the buffers may be compressed simultaneously in parallel. Furthermore, compressors of the same or different types may be coupled to each buffer based on the format of the enhanced multi-channel image 122.

[0027] 1 thus provides a high-level overview of different hardware / software architectures that may be implemented by computing device 102 to perform the various techniques described herein. A more detailed breakdown of these techniques is described below in conjunction with FIGS. 2A-2E, 3A-3G, and 4A-4E.

[0028] 2A-2E show a sequence of conceptual diagrams of a technique for generating a gain map based on an SDR image and an HDR image, according to some embodiments. As shown in FIG. 2A, step 210 may include a computing device 102 accessing a multi-channel HDR image 211 composed of pixels 212 (each denoted as "P"). As shown in FIG. 2A, the pixels 212 may be arranged according to a row / column layout, with the subscript "P" (e.g., "1,1") of each pixel 212 indicating the location of the pixel 212 according to row and column. In the example shown in FIG. 2A, the pixels 214 of the multi-channel image 108 are arranged in an equal number of rows and columns such that the multi-channel image 108 is a square image. However, it should be noted that the techniques described herein may be applied to a multi-channel image 108 having a different layout (e.g., an unequal row / column count). 2A, each pixel 212 may be composed of three subpixels 214: a red subpixel 214 (designated "R"), a green subpixel 214 (designated "G"), and a blue subpixel 214 (designated "B"). Note, however, that each pixel 212 may be composed of any number of subpixels without departing from the scope of the present disclosure.

[0029] 2B illustrates step 220, which involves computing device 102 accessing multi-channel SDR image 221. As shown in FIG. 2B, multi-channel SDR image 221 is composed of pixels 222 (and sub-pixels 224) similar to pixels 212 (and sub-pixels 214) of multi-channel HDR image 211 shown in FIG. 2A. According to some embodiments, multi-channel SDR image 221 is a single-exposure capture of the same scene captured by multi-channel HDR image 211; thus, multi-channel SDR image 221 and multi-channel HDR image 211 are substantially related to one another. For example, if multi-channel HDR image 211 was generated using the EV−, EV0, and EV+ techniques described herein, multi-channel SDR image 221 can be based on the EV0 exposure (e.g., before the EV0 exposure is merged with the EV− and EV+ exposures to generate multi-channel HDR image 211). This approach can ensure that both multi-channel HDR image 211 and multi-channel SDR image 221 correspond to the same scene at the same moment in time. In this way, pixels in multi-channel HDR image 211 and multi-channel SDR image 221 can differ only in luminous intensity collected from the same point in the same scene (as opposed to differing in scene content due to motion resulting from the passage of time that occurs through successively captured exposures).

[0030] 2C illustrates step 230, which involves the computing device 102 generating a multi-channel gain map 231 (composed of pixels 232) by comparing the multi-channel HDR image 211 and the multi-channel SDR image 221 (shown as comparison 234 in FIG. 2C). Here, if it is desired to enable the multi-channel SDR image 221 to be reproduced using the multi-channel HDR image 211, a first approach may be utilized. In particular, the first approach involves dividing the value of each pixel of the multi-channel SDR image 221 by the value of the corresponding pixel of the multi-channel HDR image 211 to generate a quotient. Each quotient can then be assigned to the value of the corresponding pixel 232 in the multi-channel gain map 231. For example, the "P" of the multi-channel HDR image 211 may be 1,1 ” has a value of “5”, and the pixel indicated by “P 1,1 If the pixel indicated by "P" has a value of "1", the quotient will be "0.2" and the "P" of the multi-channel gain map 231 1,1 In this way, and as will be explained in more detail herein, the "P" of the multi-channel SDR image 221 is assigned a value of 1,1 The pixels indicated by "P" are the pixels of the multi-channel HDR image 211. 1,1 ” in the multi-channel gain map 231. 1,1 In particular, the multiplication is performed by multiplying the pixel indicated by "P" (which has a value of "0.2") in the multi-channel SDR image 221. 1,1 2D , which produces a product "1" that corresponds to a value of "1" for the pixel indicated by "." Storing the multi-channel gain map 231 along with the multi-channel HDR image 211 can therefore allow the multi-channel SDR image 221 to be reproduced independently of the multi-channel SDR image 221 itself. A more detailed description of various ways in which the multi-channel gain map 231 can be stored along with the corresponding multi-channel image is provided below in connection with FIG. 2D .

[0031] Alternatively, if it is desired to enable the multi-channel HDR image 211 to be reproduced using the multi-channel SDR image 221, a second (different) approach may be utilized. In particular, the second approach involves dividing the value of each pixel of the multi-channel HDR image 211 by the value of the corresponding pixel of the multi-channel SDR image 221 to generate a quotient. Each quotient can then be assigned to the value of the corresponding pixel 232 in the multi-channel gain map 231. For example, 1,1 The pixel indicated by "P" in the multi-channel HDR image 211 has a value of "3". 1,1 If the pixel indicated by "P" has a value of "6", the quotient will be "2" and the "P" in the multi-channel gain map 231 1,1 In this way, and as will be explained in more detail herein, the value of the pixel indicated by " " in the multi-channel HDR image 211 is assigned. P1,1 The pixels indicated by " " are the pixels of the multi-channel SDR image 221 P1,1 " in the multi-channel gain map 231. P1,1 In particular, the multiplication is performed by multiplying the pixels indicated by "P" (which have a value of "2") in the multi-channel SDR image 221. 1,1 2D , which corresponds to the value of the pixel indicated by "6." Storing the multi-channel gain map 231 with the multi-channel SDR image 221 can therefore allow the multi-channel HDR image 211 to be reproduced independently of the multi-channel HDR image 211 itself. Again, a more detailed description of various ways in which the multi-channel gain map 231 may be stored with the corresponding multi-channel image is described below in connection with FIG. 2D .

[0032] 2C (and described herein) constitute a pixel-level comparison, it should be noted that embodiments are not so limited. Conversely, pixels of images may be compared to one another at any level of granularity without departing from the scope of this disclosure. For example, sub-pixels of multi-channel HDR image 211 and multi-channel SDR image 221 may be compared to one another (instead of, or in addition to, pixel-level comparison) such that multiple gain maps are generated under different comparison techniques (e.g., separate gain maps for each color channel).

[0033] Additionally, it should be noted that various optimizations may be employed when generating the gain map without departing from the scope of the present disclosure. For example, if two values ​​are identical to each other, a comparison operation may be skipped, and a single-bit value (e.g., “0”) may be assigned to the corresponding value in the gain map to minimize the size (i.e., storage requirements) of the gain map. Furthermore, the resolution of the gain map may be smaller than the resolution of the images being compared to generate the gain map. For example, an approximation of every fourth pixel in a first image may be compared with a corresponding approximation of every fourth pixel in a second image to generate a gain map that is one-quarter the resolution of the first and second images. This approach substantially reduces the size of the gain map, but reduces the overall accuracy with which the first image can be reproduced from the second image and gain map (or vice versa). Furthermore, the first and second images may be resampled in any conceivable manner before generating the gain map. For example, the first and second images may undergo a local tone mapping operation before generating the gain map.

[0034] 2D illustrates step 240, in accordance with some embodiments, in which the computing device 102 embeds a multi-channel gain map 231 into the multi-channel HDR image 211 or the multi-channel SDR image 221. In particular, when the first technique described above in connection with FIG. 2C is utilized, which enables the multi-channel HDR image 211 and the multi-channel gain map 231 to be used to reproduce the multi-channel SDR image 221, the computing device 102 embeds the multi-channel gain map 231 into the multi-channel HDR image 211 (thereby generating the enhanced multi-channel image 122). As illustrated in FIG. 2D , one technique for embedding the multi-channel gain map 231 into the multi-channel HDR image 211 includes interleaving each pixel 232 (of the multi-channel gain map 231) with respect to its corresponding pixel 212 (of the multi-channel HDR image 211). An alternative approach may include embedding each pixel 232 (of the multi-channel gain map 231) into its corresponding pixel 212 (of the multi-channel gain map 231) as an additional channel of the pixel 212. Yet another approach may include embedding the multi-channel gain map 231 as metadata stored with the multi-channel HDR image 211. It should be noted that the aforementioned approaches are exemplary and not intended to be limiting, and that the multi-channel gain map 231 (as well as other supplemental gain maps, if generated) may be stored with the multi-channel HDR image 211 using any conceivable approach without departing from the scope of the present disclosure.

[0035] 2E illustrates a method 250 for generating a gain map based on an SDR image and an HDR image, according to some embodiments. As shown in FIG. 2E, method 250 begins at step 252, where computing device 102 accesses an HDR image (e.g., as described above in connection with FIG. 2A). At step 254, computing device 102 accesses an SDR image (e.g., as described above in connection with FIG. 2B). At step 256, computing device 102 generates a gain map by comparing the HDR image with the SDR image, or vice versa (e.g., as described above in connection with FIG. 2C). At step 258, computing device 102 embeds the gain map into the HDR image or SDR image (thereby generating enhanced multi-channel image 122, e.g., as described above in connection with FIG. 2D).

[0036] 3A-3G show a sequence of conceptual diagrams of an alternative technique for generating a gain map based on an SDR image and an HDR image, according to some embodiments. As shown in FIG. 3A, step 310 may include computing device 102 accessing multi-channel HDR image 311 and multi-channel SDR image 313 (e.g., in a manner similar to that described above in connection with FIGS. 2A-2B). Next, computing device 102 plots shared pixels (i.e., pixels 312 / 314) of multi-channel HDR image 311 and multi-channel SDR image 313 as plotted pixel 316 on HDR / SDR graph 315. As shown in FIG. 3A, the y-axis of HDR / SDR graph 315 corresponds to HDR pixel values, and the x-axis of HDR / SDR graph 315 corresponds to SDR pixel values. In this regard, the y-value of a given plotted pixel 316 is assigned based on the corresponding pixel 312 in the multi-channel HDR image 311, and the x-value of the plotted pixel 316 is assigned based on the corresponding pixel 314 in the multi-channel SDR image 313. It should be noted that other approaches can be utilized to plot pixels without departing from the scope of this disclosure, such as assigning HDR values ​​to the x-axis and SDR values ​​to the y-axis, basing the axes on different values ​​(e.g., channels) of the pixel, etc.

[0037] FIG. 3B illustrates step 320, which includes establishing an approximate curve 322 for the plotted pixels 316. According to some embodiments, the approximate curve 322 can be established using any technique for generating a curve based on the plotted points of a graph. For example, the approximate curve can represent a moving average of the values ​​of the plotted pixels 316. Regardless of how the approximate curve 322 is generated, the approximate curve 322 can be utilized as a global tone map for downgrading the multi-channel HDR image 311 to an SDR image equivalent. However, because the approximate curve 322 constitutes an average of the differences between the multi-channel HDR image 311 and the multi-channel SDR image 313, the SDR image equivalent (generated by applying the global tone map to the multi-channel HDR image 311) also constitutes an approximation of the multi-channel SDR image 313 (as opposed to an exact reproduction of the multi-channel SDR image 313, which could otherwise be generated using the techniques described above in connection with FIGS. 2A-2E). In either case, data for the fitted curve 322 (e.g., any number of coordinates to effectively recreate the fitted curve 322, any formula to recreate the fitted curve 322, etc.) can be stored along with the multi-channel HDR image 311 (e.g., as metadata accompanying the multi-channel HDR image 311). In this way, the multi-channel HDR image 311 can be downgraded to an SDR equivalent with some accuracy, which can be beneficial when performance / efficiency outweighs the importance of accuracy.

[0038] FIG. 3C illustrates step 330, which includes inverting fit curve 322 to establish modified curve 332. Fit curve 322 can be inverted using any approach for inverting an existing curve. For example, fit curve 322 may be replotted on inverted x- and y-axes of HDR / SDR graph 315 to generate modified curve 332. FIG. 3D illustrates step 340, which includes applying modified curve 332 to plotted pixels 316 to establish replotted pixels 342. This may include, for example, adjusting the value of each of plotted pixels 316 based on its corresponding area (i.e., value) within modified curve 332. As shown in FIG. 3D, replotted pixels 342 converge across HDR / SDR graph 315 in a tighter formation relative to plotted pixels 316. In this regard, a gain map generated based on replotted pixels 342 has desirable characteristics relative to a gain map generated based on plotted pixels 316, such as reduced overall variance, which improves the overall compressibility of the gain map.

[0039] FIG. 3E illustrates step 350, which includes generating a multi-channel gain map 351 based on the replotted pixels 342. According to some embodiments, generating the multi-channel gain map 351 may include, for each replotted pixel 342, comparing the y-axis HDR value of the replotted pixel 342 with the x-axis SDR value of the replotted pixel 342 to generate a quotient (e.g., using the comparison technique described above in connection with FIG. 2C ). The quotient value may then be assigned to the corresponding / respective pixel 352 of the multi-channel gain map 351 (as shown in FIG. 3E ). Furthermore, FIG. 3F illustrates step 360, which includes embedding the multi-channel gain map 351 into the multi-channel HDR image 311, thereby generating the enhanced multi-channel image 122. As shown in FIG. 3F , the embedding step may be performed using any of the embedding techniques described above in connection with FIG. 2D .

[0040] 3G illustrates a method 370 for an alternative technique for generating a gain map based on an SDR image and an HDR image, according to some embodiments. As shown in FIG. 3G, the method 370 begins at step 372, where the computing device 102 accesses an HDR image (e.g., as described above in connection with FIG. 3A). At step 374, the computing device 102 accesses an SDR image (e.g., as described above in connection with FIG. 3A).

[0041] In step 376, for each pixel shared between the HDR image and the SDR image, computing device 102 performs steps to plot a plurality of pixels on a graph (e.g., as described above in connection with FIG. 3A), where the pixel's HDR value is plotted on the graph's y-axis and the pixel's SDR value is plotted on the graph's x-axis. In step 378, computing device 102 establishes a first curve representing an approximation of the plotted pixels (e.g., as described above in connection with FIG. 3B). In step 380, computing device 102 inverts the first curve to establish a second curve (e.g., as described above in connection with FIG. 3C). In step 382, ​​computing device 102 applies the second curve to the plotted pixels to establish a replotted pixel (e.g., as described above in connection with FIG. 3D).

[0042] At step 384, the computing device 102 generates a gain map based on the replotted pixels (e.g., as described above in connection with FIG. 3E). At step 386, the computing device 102 embeds the gain map into the HDR image (thereby generating the enhanced multi-channel image 122, e.g., as described above in connection with FIG. 3F).

[0043] 4A-4E illustrate a conceptual diagram sequence of an alternative technique for generating a gain map based on two different versions of the same image, according to some embodiments. As shown in FIG. 4A, step 410 involves computing device 102 accessing a first version of a multi-channel image 411 (e.g., in a manner similar to that described above in connection with FIG. 2A), where multi-channel image 411 includes pixels 412 and sub-pixels 414. Multi-channel image 411 may represent, for example, an imported image (e.g., a scanned image, a computer-generated image, a received image, etc.).

[0044] 4B illustrates step 420, which includes computing device 102 detecting at least one modification (shown as modification 426) to first version 411 of the multi-channel image, which generates a second version of the multi-channel image (shown as multi-channel image 422 in FIG. 4B). This may include, for example, applying a markup to multi-channel image 411, applying a filter to multi-channel image 411, applying a photographic style to multi-channel image 411, applying a destination display device profile to multi-channel image 411, applying a color correction profile to multi-channel image 411, etc. The foregoing examples are not meant to be limiting, and it should be noted that modification 426 may represent any conceivable change that may be made to multi-channel image 411 without departing from the scope of the present disclosure. In either case, as shown in FIG. 4B, modification 426 may be a "P R,2 " ~ "P R,C This includes changes to pixels indicated by ".

[0045] FIG. 4C illustrates step 430, which involves the computing device 102 generating a multi-channel gain map 432 by comparing the multi-channel image 411 (i.e., a first version of the image) with the multi-channel image 422 (i.e., a second version of the image). Here, the comparison can be performed using any of the techniques described herein, such as those described above in connection with FIG. 2C. Furthermore, as shown in FIG. 4C, the comparison can be optimized by narrowing the comparison to only pixels 424 of the multi-channel image 422 that have changed relative to pixels 412 of the multi-channel image 411. This optimization is beneficial in that it allows pixels 412 that have not been modified relative to pixels 424 to be ignored. This provides the advantage of reducing the size (i.e., storage requirements) of the multi-channel gain map 432 while saving comparison operations that may be resource-intensive.

[0046] 4C , multi-channel gain map 432 may be constrained to include only pixels 434 that store the difference between pixels 424 of multi-channel image 422 that actually changed relative to pixels 412 of multi-channel image 411. In this regard, the overall size / storage space requirements of multi-channel gain map 432 are substantially reduced. However, by omitting pixels 434 that would otherwise correspond to pixels 424 and 412, coordinate information for pixels 434 is retained as needed to effectively embed in multi-channel image 411 the information necessary to reconstruct multi-channel image 411 using multi-channel gain map 432.

[0047] FIG. 4D illustrates step 440, which includes the computing device 102 embedding the multi-channel gain map 432 into the multi-channel image 411. This may include, for example, implementing any of the embedding techniques described above in connection with FIG. 2D to generate the enhanced multi-channel image 122. However, unlike FIG. 2D , there are fewer pixels 424 in the multi-channel gain map 432 compared to the number of pixels 412 in the multi-channel image 411. In this regard, the multi-channel gain map 432 may be stored as metadata, with each pixel 434 supplemented with coordinate information that effectively maps each pixel 434 to a corresponding pixel 412 in the multi-channel image 411. Alternatively, the value of each pixel 434 may be injected into its corresponding pixel 412 (e.g., as an adjacent pixel, as a supplemental channel to the corresponding pixel 412, etc.), such that such coordinate information may be omitted. It should be noted that the above-described techniques are not meant to be limiting, and any technique may be utilized to incorporate the multi-channel gain map 432 into the multi-channel image 411 in an optimal manner.

[0048] In either case, the foregoing techniques utilize the multi-channel image 411 and the multi-channel gain map 432 to enable the generation of the multi-channel image 422 independently of the multi-channel image 422 itself. This can provide various benefits, such as the ability for a given user to undo changes that occurred between a first version of an image and a second version of the image. Furthermore, it should be noted that additional gain maps can be generated as additional modifications are made to the multi-channel image 411. As this occurs, temporal information regarding each gain map can also be incorporated into the multi-channel image 411, allowing different states of the multi-channel image 411 to be progressively applied / inverted.

[0049] 4E illustrates a method 450 for an alternative technique for generating a gain map based on a first version of an image and a second version of the image, according to some embodiments. As shown in FIG. 4E, method 450 begins at step 452, where computing device 102 accesses the first version of the image (e.g., as described above in connection with FIG. 4A). In step 454, computing device 102 detects at least one modification to the first version of the image that generates a second version of the image (e.g., as described above in connection with FIG. 4B). In step 456, computing device 102 generates the gain map by comparing the first version of the image with the second version of the image (e.g., as described above in connection with FIG. 4C). In step 458, computing device 102 embeds the gain map into the first version of the image (thereby generating enhanced multi-channel image 122, e.g., as described above in connection with FIG. 4D).

[0050] FIG. 5 shows a detailed diagram of a computing device 500 that can be used to implement the various techniques described herein, according to some embodiments. In particular, the detailed diagram illustrates various components that may be included in the computing device 102 described in connection with FIG. 1. As shown in FIG. 5, the computing device 500 may include a processor 502, which represents a microprocessor or controller for controlling the overall operation of the computing device 500. The computing device 500 may also include a user input device 508 that allows a user of the computing device 500 to interact with the computing device 500. For example, the user input device 508 may take various forms, such as buttons, a keypad, a dial, a touchscreen, an audio input interface, a visual / image capture input interface, input in the form of sensor data, etc. Additionally, the computing device 500 may include a display 510 that may be controlled by the processor 502 (e.g., via a graphics component) to display information to the user. A data bus 516 may facilitate data transfer between at least the storage device 540, the processor 502, and the controller 513. The controller 513 may be used to interface with and control different devices through an device control bus 514. The computing device 500 may also include a network / bus interface 511 that couples to a data link 512. In the case of a wireless connection, the network / bus interface 511 may include a wireless transceiver.

[0051] As mentioned above, the computing device 500 also includes a storage device 540, which may include a single disk or a collection of disks (e.g., a hard drive). In some embodiments, the storage device 540 may include flash memory, semiconductor (solid-state) memory, or the like. The computing device 500 may also include a random access memory (RAM) 520 and a read-only memory (ROM) 522. The ROM 522 may store executed programs, utilities, or processes in a non-volatile manner. The RAM 520 may provide volatile data storage and stores instructions related to the operation of applications executing on the computing device 500, for example, the image analyzer 110 / gain map generator 120.

[0052] The techniques described herein include a first technique for generating a gain map. According to some embodiments, the first technique can be implemented by a computing device and includes: (1) accessing a high dynamic range (HDR) image; (2) accessing a standard dynamic range (SDR) image; (3) for each pixel shared between the HDR image and the SDR image, (4) plotting the plurality of pixels on a graph, where the HDR value of the pixel is plotted on the Y-axis of the graph and the SDR value of the pixel is plotted on the X-axis of the graph; (5) establishing a first curve representing an approximation of the plotted pixels; (6) inverting the first curve to establish a second curve; (7) applying the second curve to the plotted pixels to establish replotted pixels; generating a gain map based on the replotted pixels; and (8) embedding the gain map into the HDR image.

[0053] According to some embodiments, the first technique may further include, following establishing the first curve, storing the first curve in the HDR image. According to some embodiments, the first curve is stored as metadata associated with the HDR image. According to some embodiments, the first curve includes a global tone map of the HDR image.

[0054] According to some embodiments, the first technique may further include, before accessing the HDR image, (1) receiving at least a first exposure of the scene and a second exposure of the scene, the first and second exposures being captured at a bit depth for storing the HDR image, and (2) processing the first and second exposures to generate the HDR image.

[0055] According to some embodiments, the first technique may further include, before accessing the SDR image, (1) receiving at least a first exposure of the scene and a second exposure of the scene, the first and second exposures being captured at a bit depth for storing the SDR image, and (2) processing the first and second exposures to generate the SDR image.

[0056] The techniques described herein also include a second technique for generating a gain map. According to some embodiments, the second technique may be implemented by a computing device and includes (1) accessing a first version of the image, (2) detecting at least one modification to the first version of the image that generates a second version of the image, (3) generating the gain map by comparing the first version of the image with the second version of the image, and (4) embedding the gain map into the first version of the image.

[0057] According to some embodiments, the first version of the image comprises a high dynamic range (HDR) image or a standard dynamic range (SDR) image. According to some embodiments, the at least one modification is detected in conjunction with applying at least one photographic style to the first version of the image, such that a second version of the image represents the first version of the image after the at least one photographic style has been applied to the first version of the image. According to some embodiments, the at least one modification is detected in conjunction with applying a destination display device profile to the first version of the image, such that the second version of the image represents the first version of the image after the destination display device profile has been applied to the first version of the image. According to some embodiments, comparing the first version of the image with the second version of the image includes, for each pixel of the second version of the image modified relative to the first version of the image, (i) identifying a corresponding pixel in the first version of the image, (ii) dividing the pixel by the corresponding pixel to generate a quotient, and (iii) storing the quotient as the corresponding pixel in a gain map.

[0058] According to some embodiments, embedding the gain map in the first version of the image includes, for each pixel of the gain map, (i) identifying a corresponding pixel in the first version of the image, and (ii) storing the value of the pixel as supplemental information for the corresponding pixel. According to some embodiments, embedding the gain map in the first version of the image includes storing the gain map as metadata associated with the first version of the image.

[0059] Various aspects, embodiments, implementations, or features of the described embodiments can be used individually or in any combination. Various aspects of the described embodiments can be implemented by software, hardware, or a combination of hardware and software. The described embodiments can also be embodied as computer-readable code on a computer-readable medium. The computer-readable medium is any data storage device that can store data which can thereafter be read by a computer system. Examples of computer-readable media include read-only memory, random-access memory, CD-ROMs, DVDs, magnetic tape, hard disk drives, solid-state drives, and optical data storage devices. The computer-readable medium can also be distributed over network-coupled computer systems so that the computer-readable code is stored and executed in a distributed fashion.

[0060] In the foregoing description, for purposes of explanation, specific terminology was used to provide a thorough understanding of the described embodiments. However, it will be apparent to those skilled in the art that specific details are not required to practice the described embodiments. Thus, the descriptions of the specific embodiments set forth above are presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the described embodiments to the precise forms disclosed. It will be apparent to those skilled in the art that numerous modifications and variations are possible in light of the above teachings.

Claims

1. 1. A method for generating a gain map, the method comprising, in a computing device, Accessing high dynamic range (HDR) images; Accessing a standard dynamic range (SDR) image; For each pixel shared between the HDR image and the SDR image, plotting the plurality of pixels on the graph, the HDR value of the pixel being plotted on the Y-axis of the graph and the SDR value of the pixel being plotted on the X-axis of the graph; establishing a first curve representing an approximation of the plotted pixels; inverting the first curve to establish a second curve; applying the second curve to the plotted pixels to establish replotted pixels; generating the gain map based on the replotted pixels; Embedding the gain map into the HDR image; A method comprising:

2. 1. A non-transitory computer-readable storage medium configured to store instructions that, when executed by at least one processor included in a computing device, cause the computing device to: Accessing high dynamic range (HDR) images; Accessing a standard dynamic range (SDR) image; For each pixel shared between the HDR image and the SDR image, plotting the plurality of pixels on the graph, the HDR value of the pixel being plotted on the Y-axis of the graph and the SDR value of the pixel being plotted on the X-axis of the graph; establishing a first curve representing an approximation of the plotted pixels; inverting the first curve to establish a second curve; applying the second curve to the plotted pixels to establish replotted pixels; generating the gain map based on the replotted pixels; Embedding the gain map into the HDR image; a non-transitory computer-readable storage medium for generating the gain map by performing steps including:

3. 1. A computing device configured to generate a gain map, the computing device comprising: at least one processor; at least one memory that stores instructions that, when executed by the at least one processor, cause the computing device to: Accessing high dynamic range (HDR) images; Accessing a standard dynamic range (SDR) image; For each pixel shared between the HDR image and the SDR image, plotting the plurality of pixels on the graph, the HDR value of the pixel being plotted on the Y-axis of the graph and the SDR value of the pixel being plotted on the X-axis of the graph; establishing a first curve representing an approximation of the plotted pixels; inverting the first curve to establish a second curve; applying the second curve to the plotted pixels to establish replotted pixels; generating the gain map based on the replotted pixels; Embedding the gain map into the HDR image; A computing device that causes steps to be performed, including:

4. 1. A computing device configured to generate a gain map, the computing device comprising: means for accessing high dynamic range (HDR) images; means for accessing a standard dynamic range (SDR) image; For each pixel shared between the HDR image and the SDR image, means for plotting the plurality of pixels on the graph, the HDR values ​​of the pixels being plotted on the Y-axis of the graph and the SDR values ​​of the pixels being plotted on the X-axis of the graph; means for establishing a first curve representing an approximation of the plotted pixels; means for inverting the first curve to establish a second curve; means for applying the second curve to the plotted pixels to establish replotted pixels; means for generating the gain map based on the replotted pixels; means for embedding the gain map into the HDR image; A computing device comprising:

Images