Techniques for generating gain maps based on acquired images

Abstract

Various techniques for generating gain maps are disclosed. In accordance with some embodiments, a technique for generating a gain map includes the steps of: (1) accessing a high dynamic range (HDR) image, (2) accessing a standard dynamic range (SDR) image, (3) for each pixel shared between the HDR image and the SDR image: rendering the pixel onto a graph wherein the HDR value of the pixel is rendered on the Y-axis of the graph, and (4) generating a gain map for each pixel, where the HDR value of the pixel is rendered on the Y-axis of the graph. In one embodiment, the method includes (1) generating a gain map in an HDR image, (2) generating an HDR value for a pixel, and drawing the SDR value for the pixel on an X-axis of the graph, (4) establishing a first curve representing an approximation of the drawn pixel, (5) inverting the first curve to establish a second curve, (6) applying the second curve for the drawn pixel to establish a redrawn pixel, (7) generating the gain map based on the redrawn pixel, and (8) embedding the gain map into the HDR image.

Description

Technical Field

[0001] The embodiments described herein set forth techniques for generating a gain map based on an acquired image. In particular, a gain map can be generated by comparing a first image with a second image. Then, the gain map can be embedded into the first image such that the second image can be efficiently reproduced using the first image and the gain map. Background Art

[0002] The dynamic range of an image refers to the range of pixel values (commonly referred to as "luminance") between the brightest and darkest parts of the image. Notably, conventional image sensors can only capture a limited range of luminance in a single exposure of a scene, at least relative to the luminance that the human eye can perceive from the same scene. This limited range is commonly referred to as the standard dynamic range (SDR) in the field of digital photography.

[0003] Despite the foregoing image sensor limitations, improvements in photographic techniques have enabled the capture of a wider range of light (referred to herein as high dynamic range (HDR)). This can be achieved by: (1) capturing multiple "bracketed" images, i.e., images with different exposure times (also referred to as "stops"), and then (2) fusing the bracketed images into a single image that incorporates different aspects of the different exposures. In this regard, a single HDR image has a wider luminance dynamic range compared to the luminance that can be captured in each of the individual exposures. This makes HDR images superior to SDR images in several respects.

[0004] Due to advancements in design and manufacturing techniques, display devices capable of displaying HDR images (in their true form) are becoming increasingly accessible. However, most display devices currently in use (and that continue to be manufactured) are only capable of displaying SDR images. Thus, a device with an SDR - limited display that receives an HDR image must perform various tasks to convert (i.e., degrade) the HDR image into an SDR - image equivalent. Conversely, a device with an HDR - capable display that receives an SDR image may attempt to perform various tasks to convert (i.e., upscale) the SDR image into an HDR - image equivalent.

[0005] Unfortunately, the foregoing conversion techniques typically produce inconsistent and / or undesirable results. In particular, degrading an HDR image to an SDR image can introduce visual artifacts (e.g., banding) into the resulting image, which are generally not correctable by additional image processing. Conversely, upscaling an SDR image to an HDR image involves applying varying levels of guesswork, which can also introduce uncorrectable visual artifacts.

[0006] Accordingly, there is a need for a technique that enables an image to transition effectively and precisely between different states. For example, it is desirable to be able to degrade an HDR image to its true SDR counterpart (and vice versa) without relying on the aforementioned (and flawed) conversion techniques. Summary of the Invention

[0007] The representative embodiments described herein disclose techniques for generating a gain map based on an acquired image. In particular, a gain map can be generated by comparing a first image with a second image. Then, the gain map can be embedded into the first image such that the second image can be efficiently reproduced using the first image and the gain map.

[0008] Another embodiment describes an alternative method for generating a gain map based on an HDR image and an SDR image. In particular, the method includes the following steps: (1) accessing the HDR image, (2) accessing the SDR image, (3) for each pixel shared between the HDR image and the SDR image: plotting the pixel on a graph, where the HDR value of the pixel is plotted on the Y-axis of the graph and the SDR value of the pixel is plotted on the X-axis of the graph, (4) establishing a first curve representing an approximation of the plotted pixels; (5) inverting the first curve to establish a second curve, (6) applying the second curve to the plotted pixels to establish re-plotted pixels, (7) generating a gain map based on the re-plotted pixels, and (8) embedding the gain map into the HDR image.

[0009] Yet another embodiment describes an alternative method for generating a gain map based on a first version of an image and a second version of the image. In particular, the method includes the following steps: (1) accessing the first version of the image, (2) detecting at least one modification to the first version of the image that results in the second version of the image, (3) generating a gain map by comparing the first version of the image with the second version of the image, and (4) embedding the gain map into the first version of the image.

[0010] Other embodiments include a non-transitory computer-readable storage medium configured to store instructions that, when executed by a processor included in a computing device, cause the computing device to perform the steps of any of the methods described above. Additional embodiments include a computing device configured to perform the various steps of any of the methods described above.

[0011] Based on the following detailed description taken in conjunction with the drawings that illustrate the principles of the described embodiments by way of example, other aspects and advantages of the present invention will become apparent. Brief Description of the Drawings

[0012] The present disclosure will be more readily understood by reference to the following detailed description taken in conjunction with the drawings, in which like reference numerals refer to like structural elements.

[0013] Figure 1 Illustrates an overview of a computing device that can be configured to perform various techniques described herein according to some embodiments.

[0014] Figures 2A to 2E Illustrates a series of conceptual diagrams of techniques for generating a gain map based on an SDR image and an HDR image according to some embodiments.

[0015] Figures 3A to 3G Illustrates a series of conceptual diagrams of alternative techniques for generating a gain map based on an SDR image and an HDR image according to some embodiments.

[0016] Figures 4A to 4E Illustrates a series of conceptual diagrams of alternative techniques for generating a gain map based on two different versions of the same image according to some embodiments.

[0017] Figure 5 Illustrates a detailed view of a computing device that can be used to implement various techniques described herein according to some embodiments. Detailed Description

[0018] Representative applications of the methods and apparatuses according to the present application are described in this section. These examples are provided only to add context and facilitate understanding of the described embodiments. Thus, it will be apparent to those skilled in the art that the described embodiments can be practiced without some or all of these specific details. In other instances, well-known processing steps are not described in detail to avoid unnecessarily obscuring the described embodiments. Other applications are possible, so the following examples should not be considered restrictive.

[0019] In the following detailed description, reference is made to the accompanying drawings that form a part of the specification, and specific embodiments in accordance with the described embodiments are shown by way of illustration. Although these embodiments are described in sufficient detail to enable those skilled in the art to practice the described embodiments, it should be understood that these examples are not restrictive, and thus other embodiments may be used and changes may be made without departing from the spirit and scope of the described embodiments.

[0020] The representative embodiments set forth herein disclose techniques for generating a gain map based on an acquired image. In particular, a gain map can be generated by comparing a first image with a second image. Then, the gain map can be embedded into the first image such that the second image can be efficiently reproduced using the first image and the gain map. The following is described in conjunction with Figure 1 , Figures 2A to 2E , Figures 3A to 3G , Figures 4A to 4E andFigure 5 Provides a more detailed description of these technologies.

[0021] Figure 1 Illustrates an overview 100 of a computing device 102 that can be configured to perform the various technologies described herein. As Figure 1 shown, the computing device 102 can include a processor 104, volatile memory 106, and non-volatile memory 124. Note that a more detailed breakdown of example hardware components that can be included in the computing device 102 is illustrated in Figure 5 and these components are omitted from the illustration in Figure 1 for simplicity purposes only. For example, the computing device 102 can include additional non-volatile memory (e.g., solid state drive, hard disk drive, etc.), other processors (e.g., multi-core central processing unit (CPU), graphics processing unit (GPU), etc.). According to some embodiments, an operating system (OS) ( Figure 1 not illustrated in Figure 1 ) can be loaded into the volatile memory 106, where the OS can execute various applications that together enable the various technologies described herein. For example, these applications can include an image analyzer 110 (and its internal components), a gain map generator 120 (and its internal components), one or more compressors (

[0022] As Figure 1 shown, the volatile memory 106 can be configured to receive a multi-channel image 108. The multi-channel image 108 can be provided, for example, by a digital imaging unit ( Figure 1 not illustrated in

[0023] According to some embodiments, a given multi-channel image 108 can represent a standard dynamic range (SDR) image, which is a single exposure of a scene collected and processed by a digital imaging unit. The given multi-channel image 108 can also represent a high dynamic range (HDR) image, which is multiple exposures of a scene collected and processed by a digital imaging unit. To generate an HDR image, the digital imaging unit can capture the scene at different bracketed exposure phases (e.g., three bracketed exposure phases commonly referred to as "EV0", "EV-", and "EV+"). Typically, the EV0 image corresponds to the normal / desired exposure of the scene (typically captured using the automatic exposure settings of the digital imaging unit); the EV- image corresponds to an underexposed image of the scene (e.g., four times darker than EV0), and the EV+ image corresponds to an overexposed image of the scene (e.g., four times brighter than EV0). The digital imaging unit can combine the different exposures to produce a resulting image that incorporates a greater range of luminance relative to the SDR image. It should be noted that the multi-channel image 108 discussed herein is not limited to SDR / HDR images. Instead, without departing from the scope of the present disclosure, the multi-channel image 108 can represent any form of digital image (e.g., scanned image, computer-generated image, etc.).

[0024] As Figure 1 shown, the multi-channel image 108 can be (optionally) provided to an image analyzer 110. According to some embodiments, the image analyzer 110 can include various components configured to process / modify the multi-channel image 108 as needed. For example, the image analyzer 110 can include a tone mapping unit 112 (e.g., configured to perform global / local tone mapping operations, inverse tone mapping operations, etc.), a noise reduction unit 114 (e.g., configured to reduce global / local noise in the multi-channel image), a color correction unit 116 (e.g., configured to perform global / local color correction in the multi-channel image), and a sharpening unit 118 (e.g., configured to perform global / local sharpening correction in the multi-channel image). It should be noted that without departing from the scope of the present disclosure, the image analyzer 110 is not limited to the foregoing processing units, and the image analyzer 110 can incorporate any number of processing units configured to perform any processing / modification on the multi-channel image 108.

[0025] As Figure 1As shown, after being processed by the image analyzer 110, the multi-channel image 108 can be provided to the gain map generator 120. However, it should be noted that, without departing from the scope of the present disclosure, if desired, the multi-channel image 108 can bypass the image analyzer 110 and be provided to the gain map generator 120. It should also be noted that, without departing from the scope of the present disclosure, the multi-channel image 108 can bypass one or more of the processing units in the processing unit of the image analyzer 110. For example, two given multi-channel images can pass through the tone mapping unit 112 to receive local tone mapping modification, and then bypass the remaining processing units in the image analyzer 110. In this regard, two multi-channel images that have undergone local tone mapping operations can be used to generate a gain map 123 that reflects the local tone mapping operations performed. In any case, and as described in more detail herein, the gain map generator 120 can generate the gain map 123 based on the two multi-channel images 108 when receiving the two multi-channel images 108. Subsequently, the gain map generator 120 can store the gain map 123 in one of the two multi-channel images 108 to produce an enhanced multi-channel image 122. It should also be noted that the gain map generation technique can be performed at any time relative to the reception of the multi-channel image on which the gain map will be based. For example, the gain map generator 120 can be configured to defer the generation of the gain map when the digital imaging unit is in an active use state to ensure that there are sufficient processing resources so as not to cause deceleration to the user. The following is described in conjunction with Figures 2A to 2E 、 Figures 3A to 3G and Figures 4A to 4E provides a more detailed breakdown of the ways in which the gain map generator 120 can generate the gain map 123.

[0026] In addition, and although not illustrated in Figure 1 , one or more compressors can be implemented on the computing device 102 for compressing the enhanced multi-channel image 122. For example, the compressor can implement a Lempel–Ziv–Welch (LZW)-based compressor, other types of compressors, a combination of compressors, etc. In addition, the compressor can be implemented in any way to establish the most efficient environment for compressing the enhanced multi-channel image 122. For example, multiple buffers (where pixels can be preprocessed in parallel) can be instantiated, and each buffer can be connected to a corresponding compressor such that these buffers can also be compressed simultaneously in parallel. In addition, depending on the formatting of the enhanced multi-channel image 122, the same or different types of compressors can be connected to each of the buffers.

[0027] Therefore, Figure 1 provides a high-level overview of different hardware / software architectures that can be implemented by the computing device 102 to perform the various techniques described herein. Now the following will be described in conjunction with Figures 2A to 2E 、Figures 3A to 3G and Figures 4A to 4E provide a more detailed breakdown diagram of these technologies.

[0028] Figures 2A to 2E Illustrates a series of conceptual diagrams of techniques for generating a gain map based on SDR images and HDR images according to some embodiments. As Figure 2A shown, step 210 may involve computing device 102 accessing a multi-channel HDR image 211, which is composed of pixels 212 (each pixel is represented as "P"). As Figure 2A shown, the pixels 212 may be arranged according to a row / column layout, where the subscript "P" of each pixel 212 (e.g., "1,1") indicates the position of the pixel 212 according to the row and column. In Figure 2A the example illustrated, the pixels 214 of the multi-channel image 108 are arranged in an equal number of rows and columns such that the multi-channel image 108 is a square image. However, it should be noted that the techniques described herein may be applied to multi-channel images 108 having different layouts (e.g., disproportionate row / column counts). In any case, and as Figure 2A additionally shown, each pixel 212 may be composed of three sub-pixels 214, namely a red sub-pixel 214 (represented as "R"), a green sub-pixel 214 (represented as "G"), and a blue sub-pixel 214 (represented as "B"). However, it should be noted that, without departing from the scope of the present disclosure, each pixel 212 may be composed of any number of sub-pixels.

[0029] Figure 2B Illustrates step 220 involving computing device 102 accessing a multi-channel SDR image 221. As Figure 2B shown, the multi-channel SDR image 221 is composed of Figure 2AThe pixels 222 (and sub - pixels 224) similar to the pixels 212 (and sub - pixels 214) of the illustrated multi - channel HDR image 211 are formed. According to some embodiments, the multi - channel SDR image 221 is a single - exposure capture of the same scene captured by the multi - channel HDR image 211, such that the multi - channel SDR image 221 and the multi - channel HDR image 211 are substantially related to each other. For example, if the multi - channel HDR image 211 is generated using the EV -, EV0, and EV + methods described herein, the multi - channel SDR image 221 can be based on the EV0 exposure (e.g., before the EV0 exposure is combined with the EV - and EV + exposures to generate the multi - channel HDR image 211). This method ensures that both the multi - channel HDR image 211 and the multi - channel SDR image 221 correspond to the same scene at the same moment. In this way, the pixels of the multi - channel HDR image 211 and the multi - channel SDR image 221 can differ only in terms of the photometric values collected from the same point of the same scene (rather than differing in scene content, which is caused by movement resulting from the passage of time due to sequentially captured exposures).

[0030] Figure 2C Step 230 is illustrated, which involves the computing device 102 generating a multi - channel gain map 231 (formed by pixels 232) by comparing the multi - channel HDR image 211 and the multi - channel SDR image 221 (illustrated as comparison 234 in Figure 2C ). Here, if it is desired to be able to reproduce the multi - channel SDR image 221 using the multi - channel HDR image 211, the first method can be utilized. In particular, the first method involves dividing the value of each pixel of the multi - channel SDR image 221 by the value of the corresponding pixel of the multi - channel HDR image 211 to produce a quotient. Subsequently, the corresponding quotient can be assigned to the value of the corresponding pixel 232 in the multi - channel gain map 231. For example, if the pixel of the multi - channel HDR image 211 represented as "P 1,1 " has a value of "5", and the pixel of the multi - channel SDR image 221 represented as "P 1,1 " has a value of "1", then the quotient will be "0.2", and it will be assigned to the value of the pixel of the multi - channel gain map 231 represented as "P 1,1 ". In this way, and as described in more detail herein, the pixel of the multi - channel SDR image 221 represented as "P 1,1 " can be reproduced by multiplying the pixel of the multi - channel HDR image 211 represented as "P 1,1 " (with a value of "5") by the pixel of the multi - channel gain map 231 represented as "P 1,1 " (with a value of "0.2"). In particular, this multiplication will generate a product of "1", which is the same as the value of the pixel of the multi - channel SDR image 221 represented as "P 1,1Therefore, storing the multi-channel gain map 231 together with the multi-channel HDR image 211 can enable the multi-channel SDR image 221 to be reproduced independently of the multi-channel SDR image 221 itself. Figure 2D A more detailed description of various ways in which the multi-channel gain map 231 may be stored with the corresponding multi-channel image is described.

[0031] Alternatively, if it is desired to be able to reproduce the multi-channel HDR image 211 using the multi-channel SDR image 221, a second (different) method may be utilized. In particular, the second method involves dividing the value of each pixel of the multi-channel HDR image 211 by the value of the corresponding pixel of the multi-channel SDR image 221 to produce a quotient. The corresponding quotient may then be assigned to the value of the corresponding pixel 232 in the multi-channel gain map 231. For example, if the representation of the multi-channel SDR image 221 is “P 1,1 ” has a value of “3”, and the representation of the multi-channel HDR image 211 is “P 1,1 ” has a value of “6”, the quotient will be “2” and will be assigned to the representation of the multi-channel gain map 231 as “P 1,1 In this manner, and as described in more detail herein, the representation of the multi-channel SDR image 221 as “P 1,1" ” (having a value of “3”) is multiplied by the representation of the multi-channel gain map 231 as “P 1,1 ” (having a value of “2”) to reproduce the multi-channel HDR image 211 is represented as “P 1,1 In particular, this multiplication will generate a product "6" which is the same as the representation of the multi-channel SDR image 221 as "P 1,1 ” matches the value “6” of the pixel of ”. Therefore, storing the multi-channel gain map 231 together with the multi-channel SDR image 221 can enable the multi-channel HDR image 211 to be reproduced independently of the multi-channel HDR image 211 itself. Similarly, the following Figure 2D A more detailed description of various ways in which the multi-channel gain map 231 may be stored with the corresponding multi-channel image is described.

[0032] Briefly, it is important to note that although Figure 2C The comparisons illustrated in (and described herein) are pixel-level comparisons, but the embodiments are not limited thereto. Rather, pixels of the images may be compared to one another at any level of granularity without departing from the scope of the present disclosure. For example, sub-pixels of the multi-channel HDR image 211 and the multi-channel SDR image 221 may be compared to one another (instead of or in addition to pixel-level comparisons) such that multiple gain maps (e.g., corresponding gain maps for each color channel) are generated under different comparison methods.

[0033] In addition, it should be noted that various optimizations may be employed when generating the gain map without departing from the scope of the present disclosure. For example, when two values ​​are identical to each other, the comparison operation may be skipped, and a single bit value (e.g., "0") may be assigned to the corresponding value in the gain map to minimize the size of the gain map (i.e., storage requirements). In addition, the resolution of the gain map may be less than the resolution of the image being compared to generate the gain map. For example, an approximation of every four pixels in the first image may be compared to an approximation of every four corresponding pixels in the second image to generate a gain map that is one-quarter the resolution of the first image and the second image. This approach will significantly reduce the size of the gain map, but will reduce the overall accuracy with which the first image can be reproduced from the second image and the gain map (or vice versa). In addition, the first image and the second image may be resampled in any conceivable manner before generating the gain map. For example, the first image and the second image may undergo a local tone mapping operation before generating the gain map.

[0034] Figure 2D Step 240 according to some embodiments is illustrated, which involves the computing device 102 embedding the multi-channel gain map 231 into the multi-channel HDR image 211 or the multi-channel SDR image 221. In particular, if the above is combined with Figure 2C In the first method discussed (which enables the use of the multi-channel HDR image 211 and the multi-channel gain map 231 to reproduce the multi-channel SDR image 221), the computing device 102 embeds the multi-channel gain map 231 into the multi-channel HDR image 211 (thereby producing the enhanced multi-channel image 122). Figure 2D As shown, one method for embedding a multi-channel gain map 231 into a multi-channel HDR image 211 involves interleaving each pixel 232 (of the multi-channel gain map 231) relative to its corresponding pixel 212 (of the multi-channel HDR image 211). An alternative method may involve embedding each pixel 232 (of the multi-channel gain map 231) into its corresponding pixel 212 (of the multi-channel gain map 231) as an additional channel to the pixel 212. Yet another method may involve embedding the multi-channel gain map 231 as metadata stored with the multi-channel HDR image 211. It should be noted that the foregoing methods are exemplary and not intended to be limiting, and that any conceivable method may be used to store the multi-channel gain map 231 (and other supplemental gain maps, if generated) with the multi-channel HDR image 211 without departing from the scope of the present disclosure.

[0035] Figure 2E A method 250 for generating a gain map based on an SDR image and an HDR image according to some embodiments is illustrated. Figure 2EAs shown, method 250 begins at step 252, where computing device 102 accesses an HDR image (e.g., as described above in connection with Figure 2A ). At step 254, computing device 102 accesses an SDR image (e.g., as described above in connection with Figure 2B ). At step 256, computing device 102 generates a gain map by comparing the HDR image with the SDR image, or vice versa (e.g., as described above in connection with Figure 2C ). At step 258, computing device 102 embeds the gain map into the HDR image or the SDR image (e.g., as described above in connection with Figure 2D ), thereby producing enhanced multi-channel image 122).

[0036] Figures 3A to 3G Illustrates a series of conceptual diagrams of alternative techniques for generating a gain map based on an SDR image and an HDR image according to some embodiments. As Figure 3A shown, step 310 may involve computing device 102 accessing multi-channel HDR image 311 and multi-channel SDR image 313 (e.g., in a manner similar to those described above in connection with Figures 2A to 2B ). Subsequently, computing device 102 plots the shared pixels (i.e., pixels 312 / 314) of multi-channel HDR image 311 and multi-channel SDR image 313 onto HDR / SDR graph 315 as plotted pixel 316. As Figure 3A shown, the y-axis of HDR / SDR graph 315 corresponds to HDR pixel values, while the x-axis of HDR / SDR graph 315 corresponds to SDR pixel values. In this regard, the y-value of a given plotted pixel 316 is assigned based on the corresponding pixel 312 in multi-channel HDR image 311, while the x-value of plotted pixel 316 is assigned based on the corresponding pixel 314 in multi-channel SDR image 313. It should be noted that other methods may be utilized to plot pixels without departing from the scope of the present disclosure, such as assigning HDR values to the x-axis and SDR values to the y-axis, having the axes based on different values of the pixels (e.g., channels), etc.

[0037] Figure 3BIllustrates step 320, which involves establishing an approximation curve 322 for rendering pixel 316. According to some embodiments, any method for generating a curve based on graphical rendering points may be used to establish the approximation curve 322. For example, the approximation curve may represent a moving average of the values of rendering pixel 316. Regardless of how the approximation curve 322 is generated, the approximation curve 322 may be used as a global tone mapping for downgrading the multi-channel HDR image 311 to an SDR image equivalent. However, since the approximation curve 322 is an average of the differences between the multi-channel HDR image 311 and the multi-channel SDR image 313, the SDR image equivalent (generated by applying the global tone mapping to the multi-channel HDR image 311) will also be an approximation of the multi-channel SDR image 313 (rather than an exact reproduction of the multi-channel SDR image 313, which could be generated using the techniques described above in conjunction with Figures 2A to 2E ). In any case, the data of the approximation curve 322 (e.g., any number of coordinates for effectively regenerating the approximation curve 322, any formula for regenerating the approximation curve 322, etc.) may be stored together with the multi-channel HDR image 311 (e.g., as metadata accompanying the multi-channel HDR image 311). In this way, the multi-channel HDR image 311 may be semi-precisely downgraded to an SDR equivalent, which may be beneficial when the importance of performance / efficiency outweighs accuracy.

[0038] Figure 3C Illustrates step 330, which involves inverting the approximation curve 322 to establish a modified curve 332. Any method for inverting an existing curve may be used to invert the approximation curve 322. For example, the approximation curve 322 may be redrawn on the reverse x-axis and y-axis of the HDR / SDR graph 315 to generate the modified curve 332. Figure 3D Illustrates step 340, which involves applying the modified curve 332 to the rendering pixel 316 to establish a redrawn pixel 342. This may involve, for example, adjusting the value of each rendering pixel in the rendering pixel 316 based on the corresponding region (i.e., value) in the modified curve 332. As Figure 3D shown, the redrawn pixels 342 converge on the HDR / SDR graph 315 in a form that is closer relative to the rendering pixels 316. In this regard, the gain map generated based on the redrawn pixels 342 will have desirable properties, such as a reduction in the overall variance that improves the overall compressibility of the gain map, as compared to the gain map generated based on the rendering pixels 316.

[0039] Figure 3EIllustrates step 350, which involves generating a multi-channel gain map 351 based on the redrawn pixels 342. According to some embodiments, generating the multi-channel gain map 351 may involve: for each redrawn pixel 342, comparing the y-axis HDR value of the redrawn pixel 342 with the x-axis SDR value of the redrawn pixel 342 to generate a quotient (e.g., using the comparison techniques described above in connection with Figure 2C ). Subsequently, the value of the quotient can be assigned to the corresponding pixel 352 in the multi-channel gain map 351 (as Figure 3E illustrated). Additionally, Figure 3F Illustrates step 360, which involves embedding the multi-channel gain map 351 into the multi-channel HDR image 311, thereby producing an enhanced multi-channel image 122. As Figure 3F shown, the embedding step can be performed using any of the embedding techniques described above in connection with Figure 2D .

[0040] Figure 3G Illustrates method 370 for an alternative technique for generating a gain map based on an SDR image and an HDR image according to some embodiments. As Figure 3G shown, method 370 begins at step 372, where computing device 102 accesses the HDR image (e.g., as described above in connection with Figure 3A ). At step 374, computing device 102 accesses the SDR image (e.g., also as described above in connection with Figure 3A ).

[0041] At step 376, computing device 102 performs the following steps for each pixel shared between the HDR image and the SDR image: draw the pixel onto a graph, where the HDR value of the pixel is drawn on the y-axis of the graph and the SDR value of the pixel is drawn on the x-axis of the graph (e.g., also as described above in connection with Figure 3A ). At step 378, computing device 102 establishes a first curve representing an approximation of the drawn pixels (e.g., as described above in connection with Figure 3B ). At step 380, computing device 102 reverses the first curve to establish a second curve (e.g., as described above in connection with Figure 3C ). At step 382, computing device 102 applies the second curve to the drawn pixels to establish redrawn pixels (e.g., as described above in connection with Figure 3D ).

[0042] At step 384, computing device 102 generates a gain map based on the redrawn pixels (e.g., as described above in connection with Figure 3E ). At step 386, computing device 102 embeds the gain map into the HDR image (e.g., as described above in connection with Figure 3Fas described, thereby generating an enhanced multi-channel image 122).

[0043] Additionally, Figures 4A to 4E illustrates a series of conceptual diagrams of alternative techniques for generating a gain map based on two different versions of the same image according to some embodiments. As Figure 4A shown, step 410 involves computing device 102 accessing a first version of multi-channel image 411 (e.g., in a manner similar to that described above in connection with Figure 2A ), where multi-channel image 411 includes pixels 412 and sub-pixels 414. The multi-channel image 411 may represent, for example, an imported image (e.g., a scanned image, a computer-generated image, a received image, etc.).

[0044] Figure 4B Illustrates step 420, which involves computing device 102 detecting at least one modification (illustrated as modification 426) to the first version of multi-channel image 411, which results in a second version of the multi-channel image (illustrated as multi-channel image 422 in Figure 4B ). This may involve, for example, applying a marker to multi-channel image 411, applying a filter to multi-channel image 411, applying a photographic style to multi-channel image 411, applying a destination display device profile to multi-channel image 411, applying a color correction profile to multi-channel image 411, etc. It should be noted that the foregoing examples are not intended to be limiting, and without departing from the scope of the present disclosure, modification 426 may represent any conceivable change that can be made to multi-channel image 411. In any case, and as Figure 4B shown, modification 426 involves a change to the pixels represented as "P R,2 " to "P R,C ".

[0045] Figure 4C Illustrates step 430, which involves computing device 102 generating a multi-channel gain map 432 by comparing multi-channel image 411 (i.e., the first version of the image) with multi-channel image 422 (i.e., the second version of the image). Here, any of the techniques described herein (such as those described above in connection with Figure 2C ) may be used to perform the comparison. Additionally, and as Figure 4C shown, the comparison can be optimized by only narrowing the comparison to the pixels 424 of multi-channel image 422 that have been changed relative to the pixels 412 of multi-channel image 411. This optimization is beneficial because pixels 412 that have not been modified relative to pixels 424 can be ignored. The benefit provided by this is to save comparison operations that may consume a large amount of resources and to reduce the size (i.e., storage requirements) of the multi-channel gain map 432.

[0046] Therefore, and asFigure 4C As shown, the multi-channel gain map 432 can be constrained to include only pixels 434 that store the differences between pixels 424 of the multi-channel image 422 that have actually changed relative to the pixels 412 of the multi-channel image 411. In this regard, the overall size / storage space requirements of the multi-channel gain mapper 432 are significantly reduced. However, since the pixels 434 that otherwise correspond to the pixels 424 and 412 are omitted, the coordinate information of the pixels 434 will be retained as needed to effectively embed the necessary information into the multi-channel image 411 to reproduce the multi-channel image 411 using the multi-channel gain map 432.

[0047] Figure 4D Step 440 is illustrated, which involves the computing device 102 embedding the multi-channel gain map 432 into the multi-channel image 411. This can involve, for example, implementing any of the embedding techniques described above in connection with Figure 2D to produce an enhanced multi-channel image 122. However, Figure 2D Unlike, there are fewer pixels 424 in the multi-channel gain map 432 compared to the number of pixels 412 in the multi-channel image 411. In this regard, the multi-channel gain map 432 can be stored as metadata, where each pixel 434 is supplemented with coordinate information that effectively maps each pixel 434 to the corresponding pixel 412 in the multi-channel image 411. Alternatively, the value of each pixel 434 can be injected into its corresponding pixel 412 (e.g., as a neighboring pixel, as a supplementary channel of the corresponding pixel 412, etc.), such that the foregoing coordinate information can be omitted. It should be noted that the foregoing methods are not intended to be limiting, and any method can be utilized to optimally incorporate the multi-channel gain map 432 into the multi-channel image 411.

[0048] In any case, the foregoing methods enable the use of the multi-channel image 411 and the multi-channel gain map 432 to generate the multi-channel image 422 independently of the multi-channel image 422 itself. This can provide various benefits, such as the ability for a given user to undo changes that occur between a first version of an image and a second version of the image. Additionally, it should be noted that when additional modifications are made to the multi-channel image 411, additional gain maps can be generated. When this occurs, time information regarding each gain map can also be incorporated into the multi-channel image 411 to enable progressive application / restoration of different states of the multi-channel image 411.

[0049] Figure 4E Method 450 is illustrated for an alternative technique for generating a gain map based on a first version of an image and a second version of the image according to some embodiments. As Figure 4E shown, method 450 begins at step 452, where the computing device 102 accesses a first version of an image (e.g., as described above in connection withFigure 4A (as described). At step 454, computing device 102 detects at least one modification to the first version of the image that results in a second version of the image (e.g., as described above in connection with Figure 4B (as described). At step 456, computing device 102 generates a gain map by comparing the first version of the image with the second version of the image (e.g., as described above in connection with Figure 4C (as described). At step 458, computing device 102 embeds the gain map into the first version of the image (e.g., as described above in connection with Figure 4D (as described, thereby producing enhanced multi-channel image 122).

[0050] Figure 5 Illustrates a detailed view of computing device 500 that can be used to implement various techniques described herein according to some embodiments. In particular, this detailed view illustrates various components that can be included in computing device 102 described in connection with Figure 1 (described). As Figure 5 shown, computing device 500 can include a processor 502 that represents a microprocessor or controller for controlling the overall operation of computing device 500. Computing device 500 can also include a user input device 508 that allows a user of computing device 500 to interact with computing device 500. For example, user input device 508 can take various forms, such as buttons, keypads, dials, touchscreens, audio input interfaces, visual / image capture input interfaces, inputs in the form of sensor data, etc. Additionally, computing device 500 can include a display 510 that can be controlled by processor 502 (e.g., via a graphics component) to display information to the user. Data bus 516 can facilitate data transfer between at least storage device 540, processor 502, and controller 513. Controller 513 can be used to interact with and control different equipment via equipment control bus 514. Computing device 500 can also include a network / bus interface 511 coupled to data link 512. In the case of a wireless connection, network / bus interface 511 can include a wireless transceiver.

[0051] As described above, computing device 500 also includes a storage device 540, which can include a single disk or a collection of disks (e.g., a hard disk drive). In some embodiments, storage device 540 can include flash memory, semiconductor (solid-state) memory, etc. This computing device 500 can also include a random access memory (RAM) 520 and a read-only memory (ROM) 522. ROM 522 can store programs, utilities, or processes that will be executed in a non-volatile manner. RAM 520 can provide volatile data storage and store instructions related to the operation of applications (e.g., image analyzer 110 / gain map generator 120) executed on computing device 500.

[0052] The techniques described herein include a first technique for generating a gain map. According to some embodiments, the first technique may be implemented by a computing device and includes the following steps: (1) accessing a high dynamic range (HDR) image; (2) accessing a standard dynamic range (SDR) image; (3) for each pixel shared between the HDR image and the SDR image: plotting the pixel on a graph, where the HDR value of the pixel is plotted on the Y-axis of the graph and the SDR value of the pixel is plotted on the X-axis of the graph; (4) establishing a first curve representing an approximation of the plotted pixels; (5) inverting the first curve to establish a second curve; (6) applying the second curve to the plotted pixels to establish re-plotted pixels; (7) generating a gain map based on the re-plotted pixels; and (8) embedding the gain map into the HDR image.

[0053] According to some embodiments, the first technique may further include the following steps after establishing the first curve: storing the first curve into the HDR image. According to some embodiments, the first curve is stored as metadata accompanying the HDR image. According to some embodiments, the first curve includes a global tone mapping of the HDR image.

[0054] According to some embodiments, the first technique may further include the following steps before accessing the HDR image: (1) receiving at least a first exposure of a scene and a second exposure of the scene, where the first exposure and the second exposure are captured at the bit depth used to store the HDR image; and (2) processing the first exposure and the second exposure to generate the HDR image.

[0055] According to some embodiments, the first technique may further include the following steps before accessing the SDR image: (1) receiving at least a first exposure of a scene and a second exposure of the scene, where the first exposure and the second exposure are captured at the bit depth used to store the SDR image; and (2) processing the first exposure and the second exposure to generate the SDR image.

[0056] The techniques described herein also include a second technique for generating a gain map. According to some embodiments, the second technique may be implemented by a computing device and includes the following steps: (1) accessing a first version of an image; (2) detecting at least one modification to the first version of the image that results in a second version of the image; (3) generating a gain map by comparing the first version of the image with the second version of the image; and (4) embedding the gain map into the first version of the image.

[0057] According to some embodiments, the first version of the image includes a high dynamic range (HDR) image or a standard dynamic range (SDR) image. According to some embodiments, at least one modification is detected in conjunction with applying at least one photographic style to the first version of the image such that, after applying the at least one photographic style to the first version of the image, the second version of the image represents the first version of the image. According to some embodiments, at least one modification is detected in conjunction with applying a destination display device profile to the first version of the image such that, after applying the destination display device profile to the first version of the image, the second version of the image represents the first version of the image. According to some embodiments, comparing the first version of the image with the second version of the image includes: for each pixel of the second version of the image that is modified relative to the first version of the image: (i) identifying the corresponding pixel in the first version of the image, (ii) dividing the pixel by the corresponding pixel to generate a quotient, and (iii) storing the quotient as the corresponding pixel in a gain map.

[0058] According to some embodiments, embedding the gain map into the first version of the image includes: for each pixel of the gain map: (i) identifying the corresponding pixel in the first version of the image, and (ii) storing the value of the pixel as supplementary information in the corresponding pixel. According to some embodiments, embedding the gain map into the first version of the image includes: storing the gain map as metadata accompanying the first version of the image.

[0059] Aspects, embodiments, implementations, or features of the described embodiments may be used singly or in any combination. Aspects of the described embodiments may be implemented by software, hardware, or a combination of hardware and software. The described embodiments may also be embodied as computer-readable code on a computer-readable medium. A computer-readable medium is any data storage device that can store data, which can then be read by a computer system. Examples of the computer-readable medium include read-only memory, random access memory, CD-ROM, DVD, magnetic tape, hard disk drive, solid state drive, and optical data storage device. The computer-readable medium may also be distributed over network-coupled computer systems such that the computer-readable code is stored and executed in a distributed fashion.

[0060] For purposes of explanation, the foregoing description uses specific names to provide a thorough understanding of the described embodiments. However, it will be apparent to those skilled in the art that no specific details are required in order to practice the described embodiments. Accordingly, the foregoing description of specific embodiments is presented for purposes of illustration and description. The foregoing description is not intended to be exhaustive or to limit the described embodiments to the precise forms disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art in light of the above teachings.

Claims

1. A method for generating a gain map, the method comprises: at a computing device: access a high dynamic range (HDR) image; access a standard dynamic range (SDR) image; for each pixel shared between the HDR image and the SDR image: plot the pixel on a graph, where the HDR value of the pixel is plotted on the Y-axis of the graph and the SDR value of the pixel is plotted on the X-axis of the graph; establish a first curve approximating the plotted pixels; invert the first curve to establish a second curve; apply the second curve to the plotted pixels to establish re-plotted pixels; generate the gain map based on the re-plotted pixels; and embed the gain map into the HDR image.

2. A non-transitory computer-readable storage medium configured to store instructions that, when executed by at least one processor included in a computing device, cause the computing device to generate a gain map by performing steps including the following: access a high dynamic range (HDR) image; access a standard dynamic range (SDR) image; for each pixel shared between the HDR image and the SDR image: plot the pixel on a graph, where the HDR value of the pixel is plotted on the Y-axis of the graph and the SDR value of the pixel is plotted on the X-axis of the graph; establish a first curve approximating the plotted pixels; invert the first curve to establish a second curve; apply the second curve to the plotted pixels to establish re-plotted pixels; generate the gain map based on the re-plotted pixels; and embed the gain map into the HDR image.

3. A computing device configured to generate a gain map, the computing device comprises: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the computing device to perform steps including the following: access a high dynamic range (HDR) image; access a standard dynamic range (SDR) image; for each pixel shared between the HDR image and the SDR image: plot the pixel on a graph, where the HDR value of the pixel is plotted on the Y-axis of the graph and the SDR value of the pixel is plotted on the X-axis of the graph; establish a first curve approximating the plotted pixels; invert the first curve to establish a second curve; apply the second curve to the plotted pixels to establish re-plotted pixels; generate the gain map based on the re-plotted pixels; and embed the gain map into the HDR image.

4. A computing device configured to generate a gain map, the computing device comprises: means for accessing a high dynamic range (HDR) image; means for accessing a standard dynamic range (SDR) image; means for performing the following operations for each pixel shared between the HDR image and the SDR image: Draw the pixel onto a graph, where the HDR value of the pixel is drawn on the Y-axis of the graph and the SDR value of the pixel is drawn on the X-axis of the graph; Apparatus for establishing a first curve approximating the drawn pixel; Apparatus for inverting the first curve to establish a second curve; Apparatus for applying the second curve to the drawn pixel to establish a redrawn pixel; Apparatus for generating the gain map based on the redrawn pixel; and Apparatus for embedding the gain map into the HDR image.

Images