Adaptive Image Data Linearization for HDR Image Sensors

Abstract

A high dynamic range (HDR) camera module with adaptive image data linearization includes: (i) an HDR image sensor, the HDR image sensor being configured to generate a tone-compressed HDR image as a corresponding frame including active pixel data and metadata; (ii) a processor, the processor being external to the HDR image sensor; and (iii) a memory, the memory being external to the HDR image sensor and storing machine-readable instructions, which, when executed by the processor, control the processor to: (a) extract a histogram of tone-compressed pixel intensities from the active pixel data and pre-tone-compressed pixel intensities from the metadata from a frame of a first tone-compressed HDR image; (b) derive a correspondence between the tone-compressed pixel intensities and the pre-tone-compressed pixel intensities from the tone-compressed pixel intensities and the histogram; and (c) linearize at least a portion of the active pixel data of the first tone-compressed HDR image or a subsequent tone-compressed HDR image according to the correspondence to produce a linearized HDR image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 885,458, filed on August 12, 2019, and European Patent Application No. 19191257.5, filed on August 12, 2019, both of which are incorporated herein by reference. Technical Field

[0003] The present application relates to high dynamic range (HDR) image sensors, processing of image data generated by HDR image sensors, and implementation of HDR image sensors in mobile devices and related processing. Background Art

[0004] Over the past few decades, many methods for high dynamic range (HDR) image capture have been developed. High-end camera systems, primarily found in the professional and scientific fields, typically employ large sensor pixels combined with high bit depth analog-to-digital conversion circuitry. Since each sensor pixel is large, each sensor pixel has an inherent high dynamic range. Therefore, the raw images captured by such high-end HDR camera systems can provide an HDR image in a single exposure of the sensor. To further improve the performance of such systems, the readout noise level has been significantly reduced in both CCD and CMOS designs, allowing the capture of dynamic ranges exceeding 14 f-stops in a single exposure in high-end HDR camera systems.

[0005] However, the most widely used HDR cameras are limited by both cost and space, which prevents the use of image sensors with large sensor pixels and high bit depth analog-to-digital conversion circuits. Most HDR cameras (such as those on-board mobile devices) instead use time or space multiplexing to capture two or more images of the same scene at two or more different exposure settings, respectively. With two different exposure settings, the low-exposure image provides the best image data for the bright part of the scene, while the high-exposure image provides the best image data for the dim part of the scene. The low-exposure image and the high-exposure image are merged after capture to generate an HDR image that spans a higher dynamic range than either of the low-exposure image and the high-exposure image.

[0006] In time-multiplexed HDR imaging, the full pixel array is exposed twice, once with a long exposure time and / or high gain, and once with a short exposure time and / or low gain. Time-multiplexed HDR imaging can utilize more than two different exposure settings to further extend the dynamic range. Time-multiplexed HDR imaging can be performed with conventional non-HDR image sensors.

[0007] On the other hand, spatially multiplexed HDR imaging utilizes HDR-specific pixel arrays. Different subsets of the pixel array are configured to operate at different corresponding exposure settings, so that each frame captured by the pixel array includes an image captured at a different exposure setting. For example, in a "zigzag HDR" configuration, the pixel array layout consists of zigzag pixel lines, where the layout alternates between low-exposure zigzag lines and high-exposure zigzag lines. This layout is superimposed on a traditional Bayer color filter arrangement. In a "quadruple HDR" configuration, the traditional Bayer color filter arrangement is extended so that each color filter is located on a 2×2 pixel group (rather than on a single pixel), where each 2×2 pixel group consists of a low-exposure pixel, two medium-exposure pixels, and one high-exposure pixel. Summary of the invention

[0008] In an embodiment, a high dynamic range (HDR) camera module with adaptive image data linearization includes an HDR image sensor for generating a tone-compressed HDR image of a scene and configured to output the tone-compressed HDR image as a corresponding frame including active pixel data and metadata. The HDR camera module further includes a processor external to the HDR image sensor and a memory external to the HDR image sensor. The memory stores machine-readable instructions that, when executed by the processor, control the processor to perform the following steps: (a) extracting from a frame of a first tone-compressed HDR image generated by the HDR image sensor (i) a histogram of tone-compressed pixel intensities from the active pixel data of the frame and (ii) pre-tone-compressed pixel intensities from the metadata of the frame; (b) deriving a correspondence between tone-compressed pixel intensities and pre-tone-compressed pixel intensities from the tone-compressed pixel intensities and the histogram; and (c) linearizing at least a portion of the active pixel data of a second tone-compressed HDR image according to the correspondence to produce a linearized HDR image, the second tone-compressed HDR image being selected from the group consisting of the first tone-compressed HDR image and a subsequent tone-compressed HDR image.

[0009] In an embodiment, an adaptive method for linearizing a tone-compressed HDR image comprises: (i) receiving a frame of a first tone-compressed HDR image from an HDR image sensor; (ii) extracting from the frame (a) a histogram of tone-compressed pixel intensities from active pixel data of the frame and (b) pre-tone-compressed pixel intensities from metadata of the frame; (iii) deriving a correspondence between tone-compressed pixel intensities and pre-tone-compressed pixel intensities from the tone-compressed pixel intensities and the histogram; (iv) receiving a frame of a second tone-compressed HDR image from the HDR image sensor; and (v) linearizing at least a portion of the active pixel data of the frame of the second tone-compressed HDR image according to the correspondence to produce a linearized HDR image. The second tone-compressed HDR image is selected from the group consisting of the first tone-compressed HDR image and a subsequent tone-compressed HDR image. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A high dynamic range (HDR) camera module with adaptive image data linearization according to an embodiment is illustrated in an example usage scenario.

[0011] Figure 2 Further details are shown Figure 1 HDR camera module.

[0012] Figure 3 A frame of a tone compressed HDR image is shown according to an embodiment.

[0013] Figure 4 An adaptive method for linearizing tone compressed HDR images according to an embodiment is illustrated.

[0014] Figure 5 A linearizer for adaptive linearization of a tone compressed HDR video stream according to an embodiment is illustrated.

[0015] Figure 6 Another adaptive method for linearizing tone compressed HDR images according to an embodiment is illustrated.

[0016] Figure 7 An HDR camera module is illustrated according to an embodiment configured to linearize each tone compressed HDR image based on the tone compressed HDR image itself, wherein each tone compressed HDR image is composed of Figure 7 The HDR camera module's image sensor generates

[0017] Figure 8 An HDR camera module with machine-readable instructions for adaptive image data linearization is illustrated according to an embodiment.

[0018] Fig. 9Yet another adaptive method for linearizing tone compressed HDR images according to an embodiment is illustrated.

[0019] Fig.10 is a flow chart of an adaptive method for linearizing a tone compressed HDR image based on a cumulative distribution function according to an embodiment.

[0020] Figures 11 to 13 yes Fig.10 An example diagram of some elements of the method.

[0021] Fig.14 and Fig.15 yes Fig.10 A visualization diagram of another example of the correspondence acquisition step of the method.

[0022] Fig.16 An adaptive method for linearizing tone compressed HDR color images based on luminance according to an embodiment is illustrated.

[0023] Fig.17 An adaptive method for linearizing a tone compressed HDR color image based on individual color components of pre-tone compressed luminance and tone compressed HDR color images according to an embodiment is illustrated.

[0024] Fig.18 An adaptive method for linearizing a tone compressed HDR color image based on individual consideration of each color component according to an embodiment is illustrated.

[0025] Fig.19 An adaptive method for linearizing a tone compressed HDR color image based in part on pre-tone compressed pixel intensities of a single color component according to an embodiment is illustrated.

[0026] Fig. 20 is a flowchart of an adaptive method for determining correspondence between tone-compressed pixel intensities and pre-tone-compressed pixel intensities by deriving a partial correspondence of a direct match between the tone-compressed pixel intensities and the pre-tone-compressed pixel intensities and then extrapolating the direct match correspondence according to an embodiment.

[0027] Fig.21 It's a picture. Fig. 20 An example diagram of some elements of the method.

[0028] Fig. 22 A method for determining a range of partial correspondences that achieve a direct match between tone-compressed pixel intensities and pre-tone-compressed pixel intensities according to an embodiment is illustrated.

[0029] Fig.23An adaptive method for linearizing a tone compressed HDR image according to a spatially varying correspondence between tone compressed pixel intensities and pre-tone compressed pixel intensities according to an embodiment is illustrated. DETAILED DESCRIPTION

[0030] Figure 1 A high dynamic range (HDR) camera module 100 with adaptive image data linearization is illustrated in an example scenario. The HDR camera module 100 can be implemented in a mobile device 102 such as a mobile phone or a tablet computer. More generally, the camera module 100 can be implemented in an electronic device. Example electronic devices in which the HDR camera module 100 can be implemented include (in addition to mobile devices such as mobile phones and tablet computers): camera devices, display devices, media players, gaming devices, laptop computers, and desktop computers. The electronic device may include a housing, wherein the HDR camera module 100 is arranged in the housing. The HDR camera module 100 includes an HDR image sensor 110 and a linearizer 120. The HDR camera module 100 is configured to generate an HDR picture of a scene. The HDR picture may be an HDR video stream or one or more HDR still images. The HDR image sensor 110 may be a monochrome image sensor or a color image sensor. When the HDR image sensor 110 is a color image sensor, each HDR image captured by the HDR image sensor 110 includes multiple color components, such as red, green, and blue.

[0031] exist Figure 1 , HDR camera module 100 generates an HDR picture of scene 190, which includes bright areas (e.g., a landscape illuminated by sun 196) and dark areas (e.g., face 194 of person 192 backlit by sun 196, and writing on sign 198 also backlit by sun 196). The dynamic range of a conventional non-HDR camera, such as having a bit depth of 8 or 10 bits, may be insufficient to capture both the bright and dark areas of scene 190. As a result, the non-HDR image of scene 190 is likely to suffer from underexposure of face 194 and writing on sign 198 and / or overexposure of the landscape of scene 190. HDR camera module 100 is configured to capture HDR images with an extended dynamic range to better capture HDR scenes such as scene 190.

[0032] Figure 2The HDR camera module 100 is further shown in detail. The HDR image sensor 110 is configured to generate an HDR image 282 at an initial HDR bit depth. However, the output of the HDR image sensor 110 is limited to an output bit depth that is less than the initial HDR bit depth, for example, to be compatible with the device communication protocol. Therefore, the HDR image sensor 110 outputs a tone-compressed HDR image 284. The pixel intensity of each tone-compressed HDR image 284 is the pixel intensity of the HDR image 282 mapped to the output bit depth. Typically, the pixel intensity of the tone-compressed HDR image 284 is related to the actual scene intensity in a nonlinear manner so as to best preserve the image information through the tone compression process. The linearizer 120 is configured to re-linearize each tone-compressed HDR image 284 to produce a linearized HDR image 286 having a pixel intensity that is related to the actual scene intensity in an at least approximately linear manner. The linearizer 120 can also map the pixel intensity of the tone-compressed HDR image 284 to a higher bit depth, such as the first bit depth, so that the bit depth of the linearized HDR image 286 exceeds the bit depth of the tone-compressed HDR image 284. The bit depth of the linearized HDR image 286 may be equal to the bit depth of the pre-tone compressed HDR image 282 .

[0033] The linearizer 120 does not require a priori knowledge of the tone compression function used by the HDR image sensor 110 to generate the tone compressed HDR image 284 from the pre-tone compressed HDR image 282. Such a priori knowledge is generally not available. Instead, the linearizer 120 obtains the correspondence between the tone compressed pixel intensities and the pre-tone compressed pixel intensities from the frames of the tone compressed HDR image 284. Therefore, the linearizer 120 can adapt to changes in the tone compression function used by the HDR image sensor 110. In a scene, the HDR image sensor 110 dynamically adjusts the tone compression function according to the attributes of the scene and / or the capture settings. The linearizer 120 may include a processor and a non-transitory memory encoding machine-readable instructions that, when executed by the processor, control the processor to linearize the tone compressed HDR image 284, as discussed herein. The linearizer 120 may be implemented in a digital signal processor (DSP).

[0034] Figure 3A frame 300 of a tone compressed HDR image 284 is shown. The frame 300 includes active pixel data 310 and metadata 320. The active pixel data 310 may include rows 312 of pixel intensities, where each row 312 corresponds to a row of pixels of the tone compressed HDR image 284. The metadata 320 includes pre-tone compression information 322, i.e., information about the pixel intensities of the pre-tone compressed image 282. The pre-tone compression information 322 may be statistical information, such as one or more full or partial histograms of the pixel intensities of the pre-tone compressed HDR image 282. The type of information included in the metadata 320 may be configurable via a function call to the HDR image sensor 110.

[0035] As used herein, the term "histogram" of an image refers to the distribution of pixel values ​​of one or more color components in the image across a number of "bins." For example, the number of bins N b can be equal to all possible values ​​of a pixel (e.g., N b =2 bitdepth ), where bitdepth represents pixel precision (e.g., 8 bits, 10 bits, etc.). In some embodiments, the number of bins may be less than N. b For example and without limitation, given N b bins and an image with P s(i) pixel values, the histogram hist[] of the image can be calculated as follows:

[0036]

[0037] Among them, fN b (x) means mapping the value of x to N b For example, when N b =2 bitdepth ,fN b (x) = x, or for any N b :

[0038]

[0039] Given an image histogram hist[], in an embodiment, the cumulative distribution function (CDF) of the image may be calculated as:

[0040] CDF[0] = hist[0];

[0041] for(b=1;b<N b ; b++)

[0042] CDF[h]=CDF[b-1]+hist[b];

[0043] Reference again Figure 2, the linearizer 120 processes a frame 300 of a tone compressed HDR image 284 and compares the pre-tone compression information 322 with the pixel intensities of the active pixel data 310 to determine a correspondence between the tone compressed pixel intensities and the pre-tone compressed pixel intensities. The linearizer 120 then applies the correspondence to the tone compressed pixel intensities of the active pixel data 310 of the same frame 300 or another frame 300 associated with a subsequent tone compressed HDR image 284 to linearize the tone compressed pixel intensities of the active pixel data 310. The resulting linearized pixel intensities are output in the linearized HDR image 286.

[0044] The HDR camera module 100 may include a main controller 240 that defines the type of information included in the metadata 320 such that the metadata 320 includes the pre-tone compression information 322. The main controller 240 may further control images captured by the HDR image sensor 110.

[0045] Figure 2 An example implementation of an HDR image sensor 110 is depicted. In this implementation, the HDR image sensor 110 includes a pixel array 210, an analog-to-digital converter (ADC) 212, an HDR merger 214, a tone compressor 216, and an interface 218. For each exposure of the pixel array 210, the ADC 212 converts the analog signal from the pixel array 210 into digital image data 281. In one example, the pixel array 210 is spatially multiplexed, and each exposure of the pixel array 210 generates two or more images of a scene captured at two or more different exposure settings, respectively. In this article, "exposure setting" refers to exposure time, gain, or a combination thereof. In this example, the pixel array 210 can be a zigzag HDR pixel array or a quadruple HDR pixel array. In another example, the HDR image sensor 110 is time-multiplexed and is configured to expose the pixel array 210 twice or more times with two or more different exposure settings, respectively. Regardless of whether the HDR image sensor 110 utilizes spatial multiplexing or temporal multiplexing, the HDR merger 214 generates each pre-tone compressed HDR image 282 from the digital image data 281 by merging two or more images of a scene captured at two or more different exposure settings, respectively. The tone compressor 216 performs tone compression on each HDR image 282 to produce a tone compressed HDR image 284. The interface 218 outputs the tone compressed HDR image 284.

[0046] In one embodiment, the bit depth of the digital image data 281 is 8 or 10 bits, the bit depth of each HDR image 282 is 14 or 16 bits, and the interface 218 complies with the Mobile Industry Parallel Interface (MIPI) standard that limits the output bit depth to 10 bits. In this embodiment, the tone compressor 216 compresses the bit depth from 14 or 16 bits to 10 bits.

[0047] The HDR camera module 100 may further include an image signal processor (ISP) 230. The ISP 230 processes the linearized HDR image 286 to produce a processed HDR image 288. The ISP 230 may apply one or more procedures selected from the group consisting of white balance, color correction, gamma correction, demosaicing, noise reduction, and image sharpening to the linearized HDR image 286.

[0048] Due to the generally non-linear relationship between tone compressed pixel intensities and scene intensities, the tone compressed HDR image 284 is not suitable for display, especially where the tone compression is scene dependent. The linearized HDR image 286 generated by the linearizer 120 is display-ready or at least ready for processing by the ISP 230. Although many camera modules include an ISP for processing images output by an image sensor, these ISPs: (a) are configured to process scene-linear pixel intensities (i.e., pixel intensities that are related to the scene intensity in an at least approximately linear manner) and (b) are not configured to linearize tone compressed pixel intensities, where the tone compression is scene dependent. In embodiments of the HDR camera module 100 that include the ISP 230, the linearizer 120 serves as a necessary pre-processor to linearize the pixel intensities prior to processing by the ISP 230.

[0049] In a potential alternative solution, the HDR image sensor 110 would be configured to output the digital image data 281 prior to HDR combining by the HDR combiner 214 to perform the HDR combining external to the HDR image sensor 110. However, except for very high-end systems, typical mobile devices lack such off-sensor HDR combining capabilities and must therefore rely on tone compression on the HDR image sensor. Therefore, this potential alternative solution is generally not a viable solution, at least for mobile devices.

[0050] To summarize certain advantages, the linearizer 120 is capable of linearizing tone compressed pixel intensities in a camera module that: (a) lacks off-sensor HDR merging capabilities and therefore must rely on HDR merging on the HDR image sensor, and (b) imposes a restriction on the bit depth of the output of the HDR image sensor to be less than the HDR bit depth, such that tone compression is required. The linearization performed by the linearizer 120 does not require a priori knowledge of the tone compression function applied on the HDR image sensor. Instead, the linearizer 120 derives the necessary information from the tone compressed HDR image frames output by the HDR image sensor. This feature of the linearizer 120 enables the linearizer 120 to adapt to dynamic changes in the tone compression function applied on the HDR image sensor.

[0051] Without departing from the scope hereof, the linearizer 120 may be provided as a standalone product configured for implementation in a camera module together with the HDR image sensor 110 provided by a third party, and optionally together with the ISP 230 and / or the main controller 240 provided by a third party.

[0052] Figure 4 An adaptive method 400 for linearizing a tone compressed HDR image is illustrated. The method 400 may be performed by the linearizer 120. The method 400 includes steps 410, 420, 430, and 440.

[0053] Step 410 receives a frame of a first tone compressed HDR image from an HDR image sensor. In one example of step 410 , the linearizer 120 receives the frame 300 of the first tone compressed image 284 from the HDR image sensor 110 .

[0054] Step 420 derives a correspondence between tone compressed pixel intensities and pre-tone compressed pixel intensities from a comparison between the active pixel data of the frame and the pre-tone compressed pixel intensity information in the metadata of the frame. In one example of step 420, the linearizer 120 compares the active pixel data 310 of the frame 300 of the first tone compressed image 284 with the pre-tone compressed information 322. Based on the comparison, the linearizer 120 derives a correspondence between the tone compressed pixel intensities of the first tone compressed image 284 and the pre-tone compressed pixel intensities of the pre-tone compressed HDR image 282. The comparison performed in step 420 may take into account all tone compressed pixel intensities of the first tone compressed image.

[0055] Step 430 receives a frame of a subsequent tone compressed HDR image from the HDR image sensor. The subsequent tone compressed HDR image is captured after the first tone compressed HDR image is received in step 410. In one example of step 430, the linearizer 120 receives a frame 300 of another tone compressed image 284 that is based on a pre-tone compressed HDR image 282 captured by the HDR image sensor 110 after the pre-tone compressed HDR image 282 associated with the tone compressed image 284 received in step 410.

[0056] Step 440 linearizes the subsequent tone-compressed HDR image received in step 430 according to the correspondence relationship obtained in step 420. In one example of step 440, the linearizer 120 linearizes the tone-compressed pixel intensities of the subsequent tone-compressed HDR image 284 received in step 430 according to the correspondence relationship obtained in step 420 to generate the linearized HDR image 286. Step 440 may include a step 442 of mapping the second tone-compressed HDR image to the pre-tone-compressed bit depth (or to another bit depth exceeding the bit depth of the first tone-compressed HDR image).

[0057] The method 400 is particularly suitable for linearization of tone compressed HDR video streams, where the time and / or memory available to process each frame is limited. When used to linearize a tone compressed HDR video stream, the method 400 can be applied to each tone compressed HDR image of the video stream and linearized according to a correspondence determined for a previous tone compressed HDR image of the video stream. Video processing is often subject to time constraints, where all processing must be done at a frame rate that matches the capture frame rate.

[0058] In one example scenario, the linearizer 120 linearizes the frame 300 of each tone compressed image 284 of the tone compressed HDR video stream "on the fly". For each frame 300 in the scenario, the linearizer 120: (a) serially receives different subsets of the frames 300 and linearizes the different subsets of the frames 300 according to the correspondence relationship obtained from the immediately previous frame 300 of the tone compressed HDR video stream, and synchronously therewith: (b) serially extracts the tone compressed pixel intensity of the active pixel data 310 from each of the different subsets. After completing this serial processing, the linearizer has collected data for performing step 420 and proceeds to determine the correspondence relationship to be applied to the next frame 300 of the tone compressed HDR video stream. It should be understood that the tone compressed pixel intensities collected for performing step 420 can be collected at a reduced resolution. In one example, the tone compressed pixel intensities are binned in predefined histogram bins so that the linearizer 120 only needs to calculate the number of pixels in each bin instead of storing the exact tone compressed pixel intensity of each pixel.

[0059] Figure 5 One linearizer 520 for adaptive linearization of a tone compressed HDR video stream is illustrated. The linearizer 520 is configured to perform an embodiment of the method 400. The linearizer 520 includes a buffer 524 capable of storing rows 312 of tone compressed pixel intensities of a frame 300 of tone compressed HDR images 284. For each tone compressed HDR image 284, the linearizer 520 stores all N rows 312 serially to the buffer 524, one row 312 after another. Each row 312 is processed according to step 440 based on a correspondence 522 obtained from a previous tone compressed HDR image 284, and data is collected for step 420 in an iteration of the method 400 for linearizing subsequent tone compressed HDR images 284. After processing all rows 312, the linearizer 520 extracts the pre-tone compression information 322 from the metadata 320 and completes step 420 to update the correspondence 522.

[0060] Without departing from the scope hereof, the buffer 524 may be capable of storing blocks of several rows 312 such that the linearizer 520 processes blocks of rows 312 serially rather than processing individual rows 312 serially.

[0061] Figure 6 Another adaptive method 600 for linearizing a tone compressed HDR image is illustrated. The method 600 may be performed by the linearizer 120. The method 600 first performs steps 410 and 420 as described above with respect to Figure 4as discussed. After determining the correspondence in step 420 based on the first tone compressed image, step 640 linearizes the first tone compressed image according to the correspondence. Step 640 is similar to step 440, except that step 640 applies the correspondence to the linearization of the same tone compressed HDR image from which the correspondence has been derived. Step 640 may include a step 642 of mapping the first tone compressed HDR image to a pre-tone compressed bit depth (or to another bit depth that exceeds the bit depth of the first tone compressed HDR image). Method 600 is particularly suitable for processing tone compressed HDR still images. Still image processing is typically less time-constrained than video stream processing, thus allowing full frames to be processed rather than subsets (e.g., lines) of each frame serially processed on the fly.

[0062] Figure 7 One HDR camera module 700 is illustrated that is configured to linearize each tone compressed HDR image generated by an image sensor of the HDR camera module 700 based on the tone compressed HDR image itself. The HDR camera module 700 includes an HDR image sensor 110 and a linearizer 720. The HDR camera module 700 may further include one or both of an ISP 230 and a main controller 240. The HDR camera module 700 is an embodiment of the HDR camera module 100, and the linearizer 720 is an embodiment of the linearizer 120.

[0063] The linearizer 720 is configured to perform an embodiment of the method 600. The linearizer 720 is communicatively coupled to an image storage device 770 capable of storing a full frame 724 of active pixel data 310 of the frame 300 of the tone compressed HDR image 284. In operation, the linearizer 720 receives the frame 300 in step 410 of the method 600 and stores all active pixel data 310 of the frame 300 as the full frame 724 in the image storage device 770. In step 420 of the method 600, the linearizer 720 obtains the correspondence 522 from the frame 300. Next, in step 640 of the method 600, the linearizer 720 obtains the full frame 724 from the image storage device 770 and linearizes its tone compressed pixel data according to the correspondence 522. The linearizer 720 may utilize the image storage device 770 to store a portion of the entirety of the linearized HDR image 286 before outputting the linearized HDR image 286.

[0064] The HDR camera module 700 can be implemented in a mobile device 702 along with the image storage 770. In one example, the mobile device 702 is a mobile phone or tablet computer having a memory external to the HDR camera module 700, wherein the memory has several functions, including serving as the image storage 770. The memory can be a random access memory (RAM) or a solid state drive (SSD). Without departing from the scope hereof, the image storage 770 can alternatively be implemented on the HDR camera module 700. The HDR camera module 700 can more generally be implemented in an electronic device (e.g., in conjunction with the image storage 770) along with the image storage 770. Figure 1 implemented in any type of electronic device discussed).

[0065] Figure 8 One HDR camera module 800 with machine readable instructions for adaptive image data linearization is illustrated. The HDR camera module 800 includes a processor 810, a non-transitory memory 820, an interface 890, and an HDR image sensor 110. The HDR camera module 800 is an embodiment of the HDR camera module 100 and may include one or both of the ISP 230 and the main controller 240. The processor 810 and the memory 820 cooperate to form an embodiment of the linearizer 120. The memory 820 includes machine readable instructions 830. The processor 810 and the memory 820 may be implemented in a digital signal processor. The machine readable instructions 830 include data extraction instructions 832, correspondence acquisition instructions 834, and linearization instructions 836.

[0066] When executed by the processor 810, the data extraction instructions 832 control the processor 810 to extract data from the frame 300 of the tone compressed HDR image 284 received from the HDR image sensor 110 via the interface 890. Specifically, the processor 810 extracts (a) tone compressed pixel intensities from the active pixel data 310 and extracts pre-tone compression information 322 from the metadata 320. In one embodiment, the memory 820 includes a buffer 524, and the processor 810 utilizes the buffer 524 to store the rows 312 of the active pixel data 310, such as described above with reference to FIG. Figure 5 discussed. Figure 8 In another embodiment not shown in FIG. 8 , the HDR camera module 800 is communicatively coupled to the image storage device 770 via the interface 890, and the processor 810 utilizes the image storage device 770 to store the active pixel data 310, such as described above with reference to FIG. Figure 7 In yet another embodiment, the memory 820 includes a dynamic data storage device 860 , and the processor 810 utilizes the dynamic data storage device 860 to store the active pixel data 310 and / or the pre-tone compression information 322 .

[0067] When executed by the processor 810, the correspondence relationship acquisition instruction 834 controls the processor 810 to obtain the correspondence relationship 522 from the comparison between the active pixel data 310 and the pre-tone compression information 322. The processor 810 may store the correspondence relationship 522 in the dynamic data storage device 860.

[0068] When executed by the processor 810, the linearization instructions 836 control the processor 810 to linearize the tone compressed HDR image 284 to generate the linearized HDR image 286. The linearization instructions 836 can control the processor 810 to output the linearized HDR image 286 via the interface 890. During the execution of the linearization instructions 863, the processor 810 can utilize the buffer 524, the dynamic data storage device 860 and / or the image storage device 770 ( Figure 8 ) to store tone compressed pixel intensities and / or linearized pixel intensities.

[0069] Fig. 9 Yet another adaptive method 900 for linearizing tone compressed HDR images is illustrated. The method 900 includes steps 910, 920, 930, and 950. The method 900 may be performed by the HDR camera module 800.

[0070] Step 910 receives a frame of a first tone compressed HDR image from an HDR image sensor. In one example of step 910 , the processor 810 receives the frame 300 of the tone compressed HDR image 284 from the HDR image sensor 110 via the interface 890 .

[0071] Step 920 extracts from the frame received in step 410 (a) tone-compressed pixel intensities from the active pixel data of the frame and (b) a histogram of pre-tone-compressed pixel intensities provided in the metadata of the frame. In one example of step 920, processor 810 executes data extraction instructions 832 to (a) extract tone-compressed pixels from active pixel data 310 and (b) extract pre-tone-compressed histogram 864 (see FIG. 1 ) from metadata 320. Figure 8 ). The pre-tone compression histogram 864 is an example of the pre-tone compression information 322, or is included in the pre-tone compression information 322. The pre-tone compression histogram 864 is a histogram of pixel intensities of the pre-tone compressed HDR image 282. The pre-tone compression histogram 864 may represent all pixels of the pre-tone compressed HDR image 282, or a subset of the pixels of the pre-tone compressed HDR image 282. For an HDR color image, the pre-tone compression histogram 864 may be a luminance histogram.

[0072] Step 930 obtains the correspondence between the tone compression pixel intensity and the pre-tone compression pixel intensity from the tone compression pixel intensity and the pre-tone compression histogram. In one example of step 930 , the processor 810 executes the correspondence obtaining instruction 834 to determine the correspondence 522 .

[0073] Step 950 linearizes at least a portion of active pixel data of a frame of a second tone compressed HDR image according to the correspondence relationship obtained in step 930, thereby generating a linearized HDR image. In one example of step 940, the processor 810 executes the linearization instructions 836 to linearize the tone compressed HDR image 284 according to the correspondence relationship 522, thereby generating a linearized HDR image 286. The second tone compressed HDR image is (a) the first tone compressed HDR image or (b) a subsequent tone compressed HDR image, which is captured by the HDR image sensor after the capture of the first tone compressed HDR image received in step 910. The first tone compressed HDR image and the subsequent tone compressed HDR image may be consecutive images of the tone compressed HDR video stream.

[0074] When used to linearize a tone compressed HDR video stream, method 900 may further include the step of limiting temporal variations across the video stream in the correspondence determined in step 930. Fig. 9 ). Such a step may be used to prevent visually significant discrete changes between frames in a linearized video stream. In one embodiment performed between steps 930 and 950 for each tone compressed HDR image of the video stream, the method 900 evaluates the deviation between (a) the correspondence determined in step 930 for the currently processed first tone compressed HDR image and (b) the correspondence determined in step 930 for the immediately previous first tone compressed HDR image. If the deviation exceeds a threshold, the correspondence of the currently processed first tone compressed HDR image is modified to be closer to the correspondence of the immediately previous first tone compressed HDR image so that the deviation does not exceed the threshold. For each value of tone compressed pixel intensity, the threshold may be defined as the maximum allowable relative difference between (a) the pre-tone compressed pixel intensity according to the correspondence of the currently processed first tone compressed HDR image and (b) the pre-tone compressed pixel intensity according to the correspondence of the immediately previous first tone compressed HDR image. The maximum allowable relative difference is, for example, in the range between one percent and ten percent.

[0075] In one embodiment, step 950 linearizes all tone-compressed pixel intensities of the second tone-compressed HDR image according to the correspondence relationship obtained from consideration of all tone-compressed pixel intensities of the first tone-compressed HDR image in step 930. In another embodiment, step 950 linearizes pixel intensities in a spatial sub-region of the second tone-compressed HDR image according to the correspondence relationship obtained from consideration of the tone-compressed pixel intensities of the same spatial sub-region of the first tone-compressed HDR image in step 930.

[0076] In an embodiment where step 950 linearizes a first tone compressed HDR image, steps 910, 920, 930, and 950 cooperate to form an embodiment of method 600. In an embodiment where step 950 linearizes a subsequent tone compressed HDR image, method 900 further includes step 940, and steps 910, 920, 930, 940, and 950 cooperate to form an embodiment of method 400. Step 940 receives a frame of a subsequent tone compressed HDR image from the HDR image sensor. In one example of step 940, processor 810 receives frame 300 of a subsequent tone compressed HDR image 284 from HDR image sensor 110 via interface 890, wherein the subsequent tone compressed HDR image 284 has been captured after the first tone compressed HDR image 284.

[0077] In some embodiments, method 900 further includes steps 902, 904, and 906, each of which can be performed by an HDR image sensor such as HDR image sensor 110. Step 902 captures raw image data at a capture bit depth. In one example of step 902, pixel array 210 (in cooperation with ADC 212) captures digital image data 281. Step 904 merges the raw image data to form an HDR image at a pre-tone compression bit depth. In one example of step 904, HDR merger 214 merges digital image data 281 obtained at two or more different exposure settings to form a pre-tone compression HDR image 282. Step 906 tone compresses the HDR image formed in step 904 to a compression bit depth less than the pre-tone compression bit depth. In one example of step 906, tone compressor 216 tone compresses the pre-tone compression HDR image 282 to generate a tone compression HDR image 284.

[0078] The method 900 may further include a step 960 of post-processing the linearized HDR image generated in step 950. Step 960 may apply one or more processes to the linearized HDR image, such as one or more of white balance, color correction, gamma correction, demosaicing, noise reduction, and image sharpening. In one example of step 960, the ISP 230 processes the linearized HDR image 286.

[0079] Embodiments of method 900 that do not include any of steps 902 , 904 , 906 , and 960 may be performed by linearizer 120 , implemented as processor 810 and memory 820 , for example.

[0080] Figures 10 to 13 An adaptive method 1000 for linearizing a tone compressed HDR image based on a cumulative distribution function (CDF) is illustrated. Fig.10 is a flow chart of method 1000, and Figures 11 to 13 is an example diagram of certain elements of method 1000. It is best viewed together with the following description Figures 10 to 13 .

[0081] Method 1000 is a derivative of method 900 (see Fig. 9 ). The method 1000 may be performed by the linearizer 120, which may be implemented as a processor 810 and a memory 820, for example. The method 1000 includes steps 1010, 1020, 1030, and 1050. Steps 1010, 1020, and 1030 cooperate to form an embodiment of step 930 of the method 900, which may be encoded in the correspondence acquisition instruction 834. Step 1050 is an embodiment of step 950 of the method 900 and may be encoded in the linearization instruction 836.

[0082] Step 1010 generates a first CDF of the tone-compressed pixel intensities obtained in step 920. In one example of step 1010, step 1010 generates a CDF 1210 of the tone-compressed pixel intensities of the active pixel data 310 (see Fig.12 ). The CDF 1210 may be based on all tone-compressed pixel intensities of the active pixel data 310, or a subset of the tone-compressed pixel intensities of the active pixel data 310. The first CDF may be generated by: (a) binning the tone-compressed pixel intensities into N C Warehouse B C,i where i = 0, 1, ..., N C -1 and each bin B C,i Compressed pixel intensity V with hue C,i Corresponding and having count f(V C,i ), and (b) calculate the first CDF as:

[0083]

[0084] Where j = 0, 1, ..., N′ C -1, and where N′ C With N CSame or N' C with the highest bin B having non-zero count C,i Corresponding. C,i Can instruct to allocate to warehouse B C,i The middle value of the range of hue-compressed pixels is either assigned to bin B C,i In one embodiment, first, according to the equation CDF C (V C,0 )=f(V C,0 ) For the first warehouse B c,o Calculating CDF C (V C,j ) and then taking the CDF C (V C,j ) is calculated as:

[0085] CDF C (V C，j )=CDF C (V C，j-1 )+f(V C，j ).

[0086] Step 1020 generates a second CDF of the histogram of pre-tone compressed pixel intensities obtained in step 920 . Fig.11 An example histogram 1100 is shown in which the HDR image sensor 110 has binned the pre-tone compressed pixel intensities of the pre-tone compressed HDR image 282 in bins 1110 . Fig.12 A CDF 1220 of the histogram 1100 is shown (schematically). Step 1020 may calculate a second CDF based on the histogram of pre-tone compressed pixel intensities in a manner similar to the calculation of the first CDF based on the histogram of tone compressed pixel intensities (as discussed above with reference to step 1010), CDF P .

[0087] Step 1030 determines a correspondence between tone-compressed pixel intensities and pre-tone-compressed pixel intensities such that for each of a plurality of corresponding pairs of tone-compressed pixel intensities and pre-tone-compressed pixel intensities, a first CDF evaluated at the tone-compressed pixel intensity is equal to a second CDF evaluated at the pre-tone-compressed pixel intensity. In other words, for any instance of tone-compressed pixel intensity u C (i), the corresponding pre-tone compressed pixel intensity v P The pre-tone compressed pixel intensity is determined to satisfy the following equation:

[0088] CDF C (u C (i))=CDF P (v P).

[0089] The correspondence relationship determined by step 1030 is an example of the correspondence relationship 522 .

[0090] Fig.12 An example visualization of this correspondence determined from the CDFs 1210 and 1220 is provided in graph 1200. For each of the plurality of tone-compressed pixel intensities, step 1030 attempts to determine the pre-tone-compressed pixel intensity associated with the same CDF value. Graph 1200 considers a tone-compressed pixel intensity 1212 of the tone-compressed HDR image 284. The value of the CDF 1210 at the tone-compressed pixel intensity 1212 is the CDF value 1224. Knowing the CDF value 1224, the pre-tone-compressed pixel intensity 1222 of the pre-tone-compressed HDR image 282 corresponding to the CDF 1220 that reaches the CDF value 1224 can be read from graph 1200 by following the dashed arrow. As an example, if the tone-compressed pixel intensity 1212 is the 574th brightest tone-compressed pixel intensity, then the pre-tone-compressed pixel intensity 1222 is at least approximately the 574th brightest pre-tone-compressed pixel intensity. The limited binning resolution in the histogram 1100 may result in deviations from an exact match between the tone-compressed pixel intensities 1212 and the pre-tone-compressed pixel intensities 1222. It should be understood that the graph 1200 is intended for explanation purposes and that step 1030 may be performed numerically without visually reading the graph. It should be further understood that the CDF 1210 and / or the CDF 1220 may be incremented in a stepped manner as opposed to a smoothed manner. For example, the CDF 1220 may have one step for each bin 1110. In this case, interpolation between adjacent steps of the CDF 1220 may be used to determine the pre-tone-compressed pixel intensities 1222, such as described below with reference to Fig.15 Continuing with diagram 1200 as a visualization example, step 1030 considers a number of different tone compressed pixel intensities 1212 and finds corresponding pre-tone compressed pixel intensities 1222 .

[0091] When method 1000 is used for monochrome image processing, the pixel intensities throughout method 1000 are grayscale intensities. When method 1000 is used for color image processing, the pixel intensities processed in method 1000 can be (a) color-specific pixel intensities (e.g., red, green, and blue pixel intensities), (b) brightness formed by combining different types of color-specific pixel intensities, or (c) a combination of color-specific pixel intensities and brightness. In one example, both CDFs 1210 and 1220 represent brightness. In another example, CDFs 1210 and 1220 represent color-specific pixel intensities of the same single color component (e.g., green). In yet another example, CDF 1210 represents color-specific pixel intensities of a single color component, while CDF 1220 represents brightness. Figures 16 to 19 Certain embodiments of method 1000 are discussed that are specifically tailored to color image processing.

[0092] Fig.13 13 is a graph 1300 of an example correspondence 1310 determined in step 1030 showing pre-tone compressed pixel intensity as a function of tone compressed pixel intensity. In this example, the pre-tone compressed pixel intensity is generally greater than the tone compressed pixel intensity, where the correspondence 1310 is above the 1:1 correspondence indicated by the dashed line 1330.

[0093] Step 1030 may include one of steps 1032, 1034, and 1036. Step 1032 generates the correspondence as a lookup table. The lookup table lists a plurality of pairs of corresponding tone-compressed pixel intensities and pre-tone-compressed pixel intensities, for example, pairs of tone-compressed pixel intensities 1212 and pre-tone-compressed pixel intensities 1222 corresponding to the same CDF value. Step 1034 generates the correspondence as a polynomial function. Step 1034 may determine the polynomial function by fitting a plurality of pairs of corresponding tone-compressed pixel intensities and pre-tone-compressed pixel intensities. Step 1036 generates the correspondence as a spline representation having a plurality of polynomial segments. The polynomial segments may be quadratic or cubic. Step 1036 may determine the spline representation by spline interpolation between a plurality of pairs of corresponding tone-compressed pixel intensities and pre-tone-compressed pixel intensities.

[0094] Step 1050 linearizes at least a portion of active pixel data of a frame of the first tone compressed HDR image or a subsequent tone compressed HDR image according to the correspondence determined in step 1030. Step 1050 may include a step 1052 of mapping the first tone compressed HDR image or the subsequent tone compressed HDR image to a pre-tone compressed bit depth (or to another bit depth exceeding the pre-tone compressed bit depth).

[0095] In an embodiment of the method 1000 including step 1032, step 1050 includes step 1054. Step 1054 searches a lookup table for each tone-compressed pixel intensity of the first tone-compressed HDR image or the subsequent tone-compressed HDR image and reads the corresponding pre-tone-compressed pixel intensity, thereby at least approximately re-linearizing the pixel intensity. Step 1054 may include step 1056 of interpolating between entries in the lookup table when the tone-compressed pixel intensity of the active pixel data of the first subsequent tone-compressed HDR image is not equal to the tone-compressed pixel intensity listed in the lookup table.

[0096] In an embodiment of the method 1000 that includes step 1034, step 1050 includes a step 1058 of applying a polynomial function to the tone-compressed pixel intensities. Step 1050 thereby determines the corresponding pre-tone-compressed pixel intensities, thereby at least approximately re-linearizing the pixel intensities. In an embodiment of the method 1000 that includes step 1036, step 1050 includes a step 1059 of applying a spline representation to the tone-compressed pixel intensities. Step 1050 thereby determines the corresponding pre-tone-compressed pixel intensities, thereby at least approximately re-linearizing the pixel intensities.

[0097] Fig.14 and Fig.15 is another example visualization of step 1030 of method 1000. It is best viewed together with Fig.14 and Fig.15 In this example, the histogram of pre-tone compressed pixel intensities obtained in step 920 is logarithmically binned such that each bin corresponds to log2(v), where v is the pre-tone compressed pixel intensity. Fig.14 An example histogram 1400 is shown having logarithmic bins 1410. The histogram 1400 has 14 bins corresponding to an HDR image 282 having a bit depth of 14 bits. Fig.15 1200, except that it is based in part on a logarithmic binned histogram 1400. Graph 1500 plots (a) a CDF 1510 of tone-compressed pixel intensities of the active pixel data 310 of the tone-compressed HDR image 284 and (b) a CDF 1520 of the histogram 1400. Due to the logarithmic binning of the histogram 1400, the CDF 1520 has a noticeable stepping behavior. The corresponding pairs of tone-compressed pixel intensities and pre-tone-compressed pixel intensities can be plotted in a manner similar to that described above with reference to FIG. Fig.12 A given tone-compressed pixel intensity 1512 corresponds to a CDF value 1524 of the CDF 1510 .

[0098] The corresponding pre-tone compressed pixel intensity 1522 may be determined as the pre-tone compressed pixel intensity within a range 1526 of steps of the CDF 1520 across the CDF value 1524. For example, the corresponding pre-tone compressed pixel intensity 1522 is set to the middle value of the range 1526. In an alternative example, the distribution of the pre-tone compressed pixel intensity within the range 1526 is assumed to be linear, as indicated by line 1528, and the corresponding pre-tone compressed pixel intensity 1522 is determined as the pre-tone compressed pixel intensity where the line 1528 matches the CDF value 1524. In one embodiment, using this linear interpolation, the nearest nodes 1530 and 1532 of the CDF 1520 that surround the CDF value 1524 are found. The coordinates of the lower node 1530 are (v P，0 , CDF P，0 ), and the coordinates of the upper node 1532 are (v P，1 , CDF P，1 ). With hue compressed pixel intensity 1512u C (i) The corresponding pre-tone compressed pixel intensity v P Then interpolation can be performed according to the following expression:

[0099]

[0100] Fig.16 An adaptive method 1600 for linearizing tone compressed HDR color images based on brightness is illustrated. Method 1600 is an embodiment of method 1000. Method 1600 includes steps 1610, 1612, 1620, 1630, and 1650. Steps 1610, 1612, 1620, and 1630 cooperate to form an embodiment of step 930 of method 900, which can be encoded in correspondence acquisition instructions 834. Step 1650 is an embodiment of step 950 of method 900 and can be encoded in linearization instructions 836.

[0101] Step 1610 combines the tone compressed color specific pixel intensities of the tone compressed HDR color image to form tone compressed pixel intensities representing brightness. In one example of step 1610, the red, green, and blue pixel intensities of the tone compressed HDR image 284 are combined to form brightness values ​​corresponding to the tone compressed HDR brightness image.

[0102] Step 1612 performs step 1010 of method 1000 using the tone-compressed pixel intensities representing brightness as input to form a first CDF. In one example of step 1612, the brightness pixel values ​​generated in step 1610 are processed to generate a brightness-based version of CDF 1210.

[0103] Step 1620 generates a second CDF of the luminance histogram. In one example of step 1620, the HDR image sensor 110 is configured to output the pre-tone compressed histogram 864 as the luminance histogram, and step 1620 forms the CDF 1220 of the luminance histogram. In another example of step 1620, the HDR image sensor 110 is configured to output the pre-tone compressed histogram 864 for each of several color components of the HDR image sensor 110, and step 1620 combines these color-specific histograms to form the luminance histogram before forming the CDF 1220 of the luminance histogram.

[0104] Step 1630 performs step 1030 of method 1000 using the CDF generated in steps 1612 and 1620 as input and using the pixel intensities representing brightness. Step 1630 thereby generates a correspondence between tone-compressed brightness and pre-tone-compressed brightness (e.g., correspondence 522). Step 1650 performs step 1050 of method 1000 separately for each color using the same correspondence for each color. In other words, the correspondence between tone-compressed brightness and pre-tone-compressed brightness is applied to each color-specific pixel intensity of the tone-compressed HDR color image. In one example of step 1650, (a) the tone-compressed red pixel intensity u of the tone-compressed HDR image 284 is R is scaled to produce the linearized red pixel intensity v according to the following expression R :

[0105] v R =u R ×(L P ×L C ),

[0106] Among them, L C is the hue compression brightness, and L P is the corresponding pre-tone compressed brightness according to the correspondence determined in step 1630, (b) the tone compressed green pixel intensity u of the tone compressed HDR image 284 G is scaled to produce the linearized green pixel intensity v according to the following expression G :

[0107] v G =u G ×(L P / L C ),

[0108] and (c) the tone compressed blue pixel intensity u of the tone compressed HDR image 284 B is scaled to produce the linearized blue pixel intensity v according to the following expressionB :

[0109] v B =u B ×(L P / L C ).

[0110] refer to Fig.10 The lookup table (eg, correspondence 522) generated in step 1032 can be based on L C The ratio L P / L C Make a table.

[0111] Fig.17 An adaptive method 1700 for linearizing a tone compressed HDR color image based on individual color components of pre-tone compressed luminance and tone compressed HDR color images is illustrated. Method 1700 is an embodiment of method 1000. Method 1700 includes steps 1710, 1620, 1730, and 1650. Method 1700 is similar to method 1600, except for step 1710 replacing steps 1610 and 1612, and step 1730 replacing step 1630.

[0112] Step 1710 performs step 1010 of method 1000 using the tone compressed pixel intensities as color specific pixel intensities representing a single color component of the tone compressed HDR color image to form a first CDF. In one example of step 1710, the green pixel intensities of the active pixel data 310 of the tone compressed HDR image 284 are processed to generate a single color version of the CDF 1210.

[0113] Step 1730 uses the CDF generated in steps 1710 and 1620 as input and performs step 1030 of method 1000 using the tone-compressed pixel intensities representing individual colors and the pre-tone-compressed pixel intensities representing brightness. Step 1730 thereby generates a correspondence (e.g., correspondence 522) between (a) color-specific tone-compressed pixel intensities for individual color components and (b) pre-tone-compressed brightness. In method 1700, step 1650 applies the same correspondence to all color components of the tone-compressed HDR color image, even though the correspondence has been determined based on a single color component of the tone-compressed HDR color image.

[0114] Fig.18An adaptive method 1800 for linearizing a tone compressed HDR color image based on separate consideration of each color component is illustrated. Method 1800 is an embodiment of method 1000. Method 1800 may be encoded in machine readable instructions 830. Method 1800 includes three instances of step 1810. Each instance of step 1810 linearizes a corresponding color component of the tone compressed HDR color image according to a correspondence relationship derived only from image data associated with that color component.

[0115] Step 1810(1) performs method 1000 for a first color (e.g., red) based on a first color hue-compressed pixel intensity and a first color histogram of pre-tone compressed pixel intensities. Step 1810(2) performs method 1000 for a second color (e.g., green) based on a second color hue-compressed pixel intensity and a second color histogram of pre-tone compressed pixel intensities. Step 1810(3) performs method 1000 for a third color (e.g., blue) based on a third color hue-compressed pixel intensity and a third color histogram of pre-tone compressed pixel intensities. Method 1800 relies on an HDR image sensor that provides a pre-tone compressed histogram 864 for each color component.

[0116] Fig.19 An adaptive method 1900 for linearizing a tone compressed HDR color image based in part on pre-tone compressed pixel intensities of a single color component is illustrated. Method 1900 is an embodiment of method 1000. Method 1900 includes steps 1710, 1920, 1930, and 1650. Method 1900 is similar to method 1700, except for step 1920 replacing step 1620 and step 1930 replacing step 1730. Step 1920 is similar to step 1620, except for being based on a pre-tone compressed histogram specific to a single color component (e.g., green). Step 1930 takes as input the CDF generated in steps 1710 and 1920 and performs step 1030 of method 1000 using the tone compressed pixel intensities and the pre-tone compressed pixel intensities representing the single color.

[0117] Fig. 20 and Fig.21 An adaptive method 2000 is illustrated for determining correspondence between tone compressed pixel intensities and pre-tone compressed pixel intensities by deriving a partial correspondence of a direct match between the tone compressed pixel intensities and the pre-tone compressed pixel intensities and then extrapolating the direct match correspondence. The method 2000 is an embodiment of step 1030 of the method 1000 and may be encoded in the linearization instruction 836. Fig. 20 is a flow chart of method 2000, and Fig.21 is an example diagram illustrating certain elements of method 2000. It is best viewed together with Fig. 20 and Fig.21 .

[0118] Method 2000 may be advantageously employed, for example, in situations where an HDR image sensor provides only a partial histogram of pre-tone compressed pixel intensities and / or in situations where higher pixel intensities are affected by saturation and are therefore not suitable for correspondence retrieval. Method 2000 includes steps 2010 and 2020. Step 2010 determines a directly matched partial correspondence between tone compressed pixel intensities and pre-tone compressed pixel intensities, and step 2020 extrapolates the directly matched partial correspondence.

[0119] For a partial range of tone compressed pixel intensities of the tone compressed HDR image that does not include the highest range of tone compressed pixel intensities, step 2010 determines a direct matching correspondence between tone compressed pixel intensities and pre-tone compressed pixel intensities. Fig.21 In one example illustrated in , step 2010 determines a directly matched correspondence 2112 between a CDF (e.g., CDF 1210) of tone compressed pixel intensities of the active pixel data 310 of the tone compressed HDR image 284 and a CDF (e.g., CDF 1220) of the pre-tone compressed histogram 864. The directly matched correspondence 2112 is determined only for tone compressed pixel intensities below a cutoff pixel intensity 2120.

[0120] Step 2020 extrapolates the directly matched correspondences determined in step 2010 via the highest range to generate all correspondences between tone-compressed pixel intensities and pre-tone-compressed pixel intensities. Fig.21 In one example of step 2020 illustrated in , step 2020 extrapolates the directly matched correspondence 2112 beyond the cutoff pixel intensity 2120 to determine the extrapolated correspondence 2114. The directly matched correspondence 2112 and the extrapolated correspondence 2114 together form the correspondence 2110. The correspondence 2110 covers the entire range of tone compressed pixel intensities. Step 2020 may include any of steps 1032 and 1034 of method 1000. Step 2020 may determine the extrapolated correspondence 2114 using a polynomial fit or spline representation of the directly matched correspondence 2112 or a portion of the directly matched correspondence 2112 near the cutoff pixel intensity 2120.

[0121] In one embodiment, step 2010 includes steps 2012 and 2013. Step 2012 evaluates at least one of the first tone compression image (e.g., tone compression HDR image 284) and the first cumulative distribution function (e.g., CDF 1210) to determine a cutoff tone compression pixel intensity (e.g., cutoff pixel intensity 2120) that does not include saturated tone compression pixel intensities. Step 2013 defines the portion of the range as a range of tone compression pixel intensities that are not greater than the cutoff tone compression pixel intensity (e.g., cutoff pixel intensity 2120).

[0122] In another embodiment, step 2010 includes steps 2014 and 2018. Step 2014 evaluates a histogram of pre-tone compressed pixel intensities representing only a lower subset of the full dynamic range (e.g., pre-tone compressed histogram 864) to determine a cutoff pre-tone compressed pixel intensity. Step 2018 defines the portion of the range of tone compressed pixel intensities as the number of lowest tone compressed pixel intensities that is equal to the number of pre-tone compressed pixel intensities that are not greater than the cutoff pre-tone compressed pixel intensity determined in step 2014.

[0123] Step 2014 may include step 2015. Step 2015 sets the cutoff pre-tone compression pixel intensity to the maximum pixel intensity of the histogram (e.g., the highest bin of the pre-tone compression histogram 864 with a non-zero count, or the highest bin of the histogram 1100 or the logarithmic histogram 1400 with a non-zero count). In one example, the histogram is logarithmic and has N bins B. i , i=0,1,...,N-1. Each bin B i Compress pixel intensity with pre-tone 2 i Correspondingly, the highest bin with non-zero count is Among them, N max is a positive integer not greater than N-1. In this example, the cutoff pre-tone compression pixel intensity is set to

[0124] Fig. 22 A method 2200 is illustrated for determining a range of partial correspondences that achieve a direct match between tone compressed pixel intensities and pre-tone compressed pixel intensities. The method 2200 may be implemented in step 2010 of the method 2000. The method 2200 utilizes a logarithmic histogram of pre-tone compressed pixel intensities, such as the histogram 1400. The method 2200 includes steps 2210, 2220, 2230, and 2240.

[0125] Step 2210 counts the number N of saturated pixels in the tone compressed HDR image (eg, tone compressed HDR image 284). SATStep 2220 identifies the number of cells containing N SAT The bin of the logarithmic pre-tone compression histogram (e.g., histogram 1400) with the highest pre-tone compressed pixel intensity is selected. For example, starting with the highest bin of the logarithmic pre-tone compression histogram, step 2220 may descend through the bins in the following manner. For each bin of the logarithmic pre-tone compression histogram, a count variable is set to N. SAT，i =min(N SAT , N i ), where N i is the count of the ith bin, and N SAT Reset to N SAT =N SAT -N i This process continues until N SAT Less than or equal to zero.

[0126] Step 2230 takes as input the bins of the logarithmic pre-tone compression histogram and calculates its average bin. Step 2230 may calculate the geometric mean of the bins identified in step 2220. In one example, the average bin is calculated as:

[0127]

[0128] Among them, Bin i is the bin number of the ith bin. Then, step 2240 limits the range of obtaining the partial correspondence of direct matches between the tone compressed pixel intensities and the pre-tone compressed pixel intensities to the bins of the logarithmic pre-tone compression histogram that are no greater than the average bin.

[0129] Fig.23 An adaptive method 2300 for linearizing a tone compressed HDR image based on a spatially varying correspondence between tone compressed pixel intensities and pre-tone compressed pixel intensities is illustrated. The method 2300 is an embodiment of steps 920, 930, and 950 of the method 900, and may be employed when the HDR image sensor is configured to output a separate pre-tone compression histogram for each of a plurality of non-overlapping spatial image regions of the pre-tone compressed HDR image 282. The method 2300 may be encoded in the machine-readable instructions 830.

[0130] Method 2300 refers to the above Fig. 9 Each of the first tone-compressed image and the second tone-compressed image in question is divided into a plurality of non-overlapping spatial image regions 2310 . Fig.23 A 6×6 grid of aerial image regions 2310 is shown. It should be understood that the shape and / or number of aerial image regions 2310 may vary. Fig.23The method 2300 then repeats steps 920, 930, and 950 of the method 900 for each spatial image region 2310 to determine a separate correspondence between the tone-compressed pixel intensity and the pre-tone-compressed pixel intensity for each spatial image region 2310. The method 2300 may implement the method 1000 when performing steps 930 and 950.

[0131] In some embodiments, the method 2300 is configured to smooth the transition between correspondences determined for adjacent spatial image regions 2310. In one such embodiment, the method 2300, when performing step 950, includes, at least for each pixel within a threshold distance of a boundary between adjacent spatial image regions 2310, spatially interpolating the correspondences determined for adjacent spatial image regions based on the position of the pixel. For example, for a pixel within an area of ​​the spatial image region 2310(1,3) that is less than a distance 2330 from the spatial image region 2310(1,4), a correspondence applicable to the pixel may be found by spatially interpolating between the two correspondences determined for the spatial image regions 2310(1,3) and 2310(1,4) based on the precise position of the pixel.

[0132] In another embodiment, configured to smooth the transition between correspondences determined for adjacent spatial image regions 2310, spatial interpolation is applied to all pixels. In this embodiment, the correspondence determined for each spatial image region 2310 is assigned to its center position 2316. For each pixel of the tone compressed HDR image, when method 2300 performs step 950, method 2300 spatially interpolates the correspondences determined for the four spatial image regions having corresponding center positions closest to the pixel. For example, for pixel 2320, method 2300 spatially interpolates the four correspondences determined for spatial image regions 2310 (5, 5), 2310 (5, 6), 2310 (6, 5), and 2310 (6, 6) from center position 2316 to the position of pixel 2320.

[0133] Feature combination

[0134] The features described above and the features claimed below may be combined in various ways without departing from the scope thereof. For example, it should be understood that aspects of one system having adaptive image data linearization or method for adaptive image data linearization described herein may be combined or exchanged for features of another system having adaptive image data linearization or method for adaptive image data linearization described herein. The following examples illustrate some possible non-limiting combinations of the embodiments described above. It should be clear that many other changes and modifications may be made to the methods, products, and systems herein without departing from the spirit and scope of the invention:

[0135] (A1) A high dynamic range (HDR) camera module with adaptive image data linearization includes an HDR image sensor for generating a tone-compressed HDR image of a scene and configured to output the tone-compressed HDR image as a corresponding frame including active pixel data and metadata. The HDR camera module also includes a processor external to the HDR image sensor and a memory external to the HDR image sensor. The memory stores machine-readable instructions that, when executed by the processor, control the processor to perform the following steps: (a) extracting from a frame of a first tone-compressed HDR image generated by the HDR image sensor (i) a histogram of tone-compressed pixel intensities from the active pixel data of the frame and (ii) pre-tone-compressed pixel intensities from the metadata of the frame; (b) deriving a correspondence between tone-compressed pixel intensities and pre-tone-compressed pixel intensities from the tone-compressed pixel intensities and the histogram; and (c) linearizing at least a portion of the active pixel data of a second tone-compressed HDR image according to the correspondence to produce a linearized HDR image, the second tone-compressed HDR image being selected from the group consisting of the first tone-compressed HDR image and a subsequent tone-compressed HDR image.

[0136] (A2) In the HDR camera module described in (A1), the processor and the memory can be implemented in a digital signal processor, and the HDR camera module can further include an image signal processor, which is communicatively coupled to the digital signal processor for further processing the linearized HDR image by the digital signal processor.

[0137] (A3) In the HDR camera module described in (A2), the image signal processor can be configured to apply one or more procedures selected from the group consisting of: white balance, color correction, gamma correction, demosaicing, noise reduction and image sharpening to the linearized HDR image.

[0138] (A4) In any one of the HDR camera modules described in (A1) to (A3), the first tone compressed HDR image and the second tone compressed HDR image can be two consecutive images of an HDR video stream output by the HDR image sensor.

[0139] (A5) In the HDR camera module described in (A4), the processor and the memory can be implemented in a digital signal processor, the digital signal processor includes a buffer, and the machine-readable instructions can be further configured to control the processor to perform a linearization step on a subset of active pixel data of the second tone compressed image continuously stored in the buffer.

[0140] (A6) In any one of the HDR camera modules described in (A1) to (A3), the first tone compressed HDR image and the second tone compressed HDR image can be the same still image, and the machine-readable instructions can be further configured to control the processor to store active pixel data of the still image in an image storage device, at least until the linearization step is completed.

[0141] (A7) In any one of the HDR camera modules described in (A1) to (A6), the HDR image sensor may include an output interface, which is limited by an output bit depth, and the HDR image sensor may be configured to (i) generate a linear HDR image at a first bit depth exceeding the output bit depth, and (ii) perform tone compression on the linear HDR image to generate the tone compressed HDR image at the output bit depth, and the machine-readable instructions may be configured to control the processor to map the active pixel data from the output bit depth to the first bit depth in the linearization step.

[0142] (A8) In any one of the HDR camera modules described in (A1) to (A7), the machine-readable instructions can be configured so that the obtaining step includes: generating a first cumulative distribution function of the tone-compressed pixel intensity; generating a second cumulative distribution function of the histogram; and determining the corresponding relationship so that for each of a plurality of corresponding pairs of tone-compressed pixel intensity and pre-tone-compressed pixel intensity, the first cumulative distribution function evaluated at the tone-compressed pixel intensity is equal to the second cumulative distribution function evaluated at the pre-tone-compressed pixel intensity.

[0143] (A9) In the HDR camera module described in (A8), the machine-readable instructions can be configured such that the determining step includes storing a lookup table listing the corresponding pairs in the memory, and such that the linearizing step includes interpolating between entries in the lookup table when the tone-compressed pixel intensity of the active pixel data of the second tone-compressed HDR image is not equal to the tone-compressed pixel intensity listed in the lookup table.

[0144] (A10) In the HDR camera module as described in (A8), the machine-readable instructions can be configured so that the determining step includes determining the correspondence as a polynomial function, which specifies the pre-tone compressed pixel intensity as a function of the tone compressed pixel intensity, and storing the parameters of the polynomial function in the memory.

[0145] (A11) In the HDR camera module as described in (A8), the machine-readable instructions can be configured so that the determining step includes determining the correspondence as a spline representation, which specifies the pre-tone compressed pixel intensity as a function of the tone compressed pixel intensity, and storing the parameters of the spline representation in the memory.

[0146] (A12) In any one of the HDR camera modules described in (A8) to (A11), the HDR image sensor may be a color image sensor, the histogram may be a brightness histogram, and the machine-readable instructions may be configured such that (I) the step of extracting the tone-compressed pixel intensities includes: (a) extracting multiple sets of tone-compressed color-specific pixel intensities, each of the sets being specific to a corresponding color, and (b) combining the sets to form the tone-compressed pixel intensities, such that each of the tone-compressed pixel intensities represents the corresponding pixel brightness; and (II) the linearization step includes separately linearizing each color of the active pixel data of the second tone-compressed HDR image according to the corresponding relationship.

[0147] (A13) In any one of the HDR camera modules as described in (A8) to (A12), the machine-readable instructions can be configured so that the step of determining the correspondence includes: (a) for a partial range of tone compression pixel intensity that does not include the highest range of the tone compression pixel intensity, determining a directly matched correspondence between the tone compression pixel intensity and the pre-tone compression pixel intensity, so that for corresponding pairs of the tone compression pixel intensity and the pre-tone compression pixel intensity associated with the partial range, the first cumulative distribution function evaluated at the tone compression pixel intensity is equal to the second cumulative distribution function evaluated at the pre-tone compression pixel intensity, and (b) extrapolating the directly matched correspondence via the highest range to generate the correspondence.

[0148] (A14) In the HDR camera module as described in (A13), the machine-readable instructions can be configured so that the step of determining the corresponding relationship further includes: (a) evaluating at least one of the first tone compression image and the first cumulative distribution function to determine a cutoff tone compression pixel intensity, wherein the cutoff tone compression pixel intensity does not include a saturated tone compression pixel intensity, and (b) defining the partial range as a range of tone compression pixel intensities that are not greater than the cutoff tone compression pixel intensity.

[0149] (A15) In the HDR camera module as described in (A13), the histogram can be a partial histogram representing only a portion of the dynamic range, the partial dynamic range forming a lower subset of the full dynamic range of the first tone compressed HDR image before tone compression, and the machine-readable instructions can be configured so that the step of determining the correspondence further includes evaluating the histogram to determine a cutoff pre-tone compression pixel intensity, and defining the partial range as the number of lowest tone compression pixel intensities, the number of lowest tone compression pixel intensities being equal to the number of pre-tone compression pixel intensities that are not greater than the cutoff pre-tone compression pixel intensity.

[0150] (A16) In any one of the HDR camera modules described in (A1) to (A15), the machine-readable instructions can be configured to control the processor when executed by the processor to: (a) divide each of the first tone compressed HDR image and the second tone compressed HDR image into a plurality of non-overlapping spatial image regions, and (b) for each of the plurality of spatial image regions, repeatedly perform the steps of extracting the tone compressed pixel intensity, generating the first cumulative distribution function, extracting the histogram, generating the second cumulative distribution function, determining the correspondence, and linearizing.

[0151] (A17) In the HDR camera module as described in (A16), the machine-readable instructions can be configured so that the linearization step includes: at least for each pixel within a threshold distance of a boundary between different spatial image regions in the spatial image region, spatially interpolating the correspondence determined for different spatial image regions in the spatial image region based on the position of the pixel.

[0152] (A18) In the HDR camera module as described in (A16), each of the spatial image areas may have a corresponding center position, and the machine-readable instructions may be configured so that the linearization step includes: for each pixel and based on the position of the pixel, spatially interpolating the correspondence determined for the four spatial image areas having the corresponding center positions closest to the pixel.

[0153] (B1) An adaptive method for linearizing a tone-compressed HDR image comprises: (i) receiving a frame of a first tone-compressed HDR image from an HDR image sensor; (ii) extracting from the frame (a) a histogram of tone-compressed pixel intensities from active pixel data of the frame and (b) pre-tone-compressed pixel intensities from metadata of the frame; (iii) obtaining a correspondence between tone-compressed pixel intensities and pre-tone-compressed pixel intensities from the tone-compressed pixel intensities and the histogram; (iv) receiving a frame of a second tone-compressed HDR image from the HDR image sensor, the second tone-compressed HDR image being selected from a group consisting of the first tone-compressed HDR image and subsequent tone-compressed HDR images; and (v) linearizing at least a portion of the active pixel data of the frame of the second tone-compressed HDR image according to the correspondence to produce a linearized HDR image.

[0154] (B2) In the method described in (B1), the first tone compressed HDR image and the second tone compressed HDR image can be consecutive images of an HDR video stream.

[0155] (B3) The method as described in (B2) may further include: before the linearization step, limiting the deviation between the correspondence and a previous correspondence obtained for a previously captured image pair of the HDR video stream.

[0156] (B3) In the method described in (B1), the first tone compressed HDR image and the second tone compressed HDR image may be the same still image.

[0157] (B4) In any of the methods described in (B1) to (B3), the tone compression pixel intensity can be limited to a compression bit depth, which is less than the pre-tone compression bit depth, and the linearization step includes mapping the active pixel data to the pre-tone compression bit depth.

[0158] (B5) In any one of the methods described in (B1) to (B4), the obtaining step may include: generating a first cumulative distribution function of the tone-compressed pixel intensity; generating a second cumulative distribution function of the histogram; and determining the corresponding relationship so that for each of a plurality of corresponding pairs of tone-compressed pixel intensity and pre-tone-compressed pixel intensity, the first cumulative distribution function evaluated at the tone-compressed pixel intensity is equal to the second cumulative distribution function evaluated at the pre-tone-compressed pixel intensity.

[0159] (B6) The method as described in (B5) may include: (a) in the determining step, generating the correspondence as a lookup table listing the corresponding pairs, and (b) in the linearizing step, when the tone compressed pixel intensity of the active pixel data of the second tone compressed HDR image is not equal to the tone compressed pixel intensity listed in the lookup table, interpolating between the entries in the lookup table.

[0160] (B7) The method described in (B5) may include: in the determining step, generating the corresponding relationship as a polynomial function, and the polynomial function specifies the pre-tone compression pixel intensity as a function of the tone compression pixel intensity.

[0161] (B8) The method described in (B5) may include: in the determining step, generating the correspondence as a spline representation, wherein the spline representation specifies the pre-tone compressed pixel intensity as a function of the tone compressed pixel intensity.

[0162] (B9) In any of the methods described in (B5) to (B8), the first tone compressed HDR image and the second tone compressed HDR image may be color images, and the histogram may be a brightness histogram.

[0163] (B10) The method as described in (B9) may further include: (a) in the step of extracting the tone-compressed pixel intensities, extracting multiple sets of tone-compressed color-specific pixel intensities, each of the sets being specific to a corresponding color; (b) combining the sets to form the tone-compressed pixel intensities, so that each of the tone-compressed pixel intensities represents the corresponding pixel brightness; and (c) in the linearization step, individually linearizing each color of the active pixel data of the second tone-compressed HDR image according to the corresponding relationship.

[0164] (B11) The method as described in (B9) may further include: (a) in the step of extracting the tone compressed pixel intensity, extracting the tone compressed color specific pixel intensity specific to a single color, wherein the single color is one of the multiple colors constituting the first tone compressed HDR image; (b) in the generating step, generating the first cumulative distribution function regarding the single color; and (c) in the linearizing step, linearizing each color of the active pixel data of the second tone compressed HDR image separately according to the corresponding relationship.

[0165] (B12) In any one of the methods described in (B5) to (B8), the first tone compressed HDR image and the second tone compressed HDR image may be color images, and the method may include separately performing the steps of extracting the tone compressed pixel intensity, generating the first cumulative distribution function, extracting the histogram, generating the second cumulative distribution function, determining the correspondence, and linearizing at least a portion of the active pixel data of the second tone compressed HDR image for each color of the first tone compressed HDR image and the second tone compressed HDR image.

[0166] (B13) In any one of the methods described in (B5) to (B8), the first tone-compressed HDR image and the second tone-compressed HDR image may be color images, and the method may further include: (a) in the step of extracting the tone-compressed pixel intensity, extracting the tone-compressed color-specific pixel intensity specific to a single color, wherein the single color is one of the multiple colors constituting the first tone-compressed HDR image; (b) in the generating step, generating the first cumulative distribution function with respect to the single color; (c) in the step of extracting the histogram, extracting the color-specific histogram specific to the single color; (d) in the generating step, generating the second cumulative distribution function with respect to the color-specific histogram; and (e) in the linearizing step, linearizing each color of the active pixel data of the second tone-compressed HDR image separately according to the corresponding relationship.

[0167] (B14) In any one of the methods described in (B5) to (B13), the determining step may include: (a) for a partial range of tone compression pixel intensities that does not include the highest range of the tone compression pixel intensities, determining a directly matched correspondence between the tone compression pixel intensities and the pre-tone compression pixel intensities, so that for corresponding pairs of the tone compression pixel intensities and the pre-tone compression pixel intensities associated with the partial range, the first cumulative distribution function evaluated at the tone compression pixel intensities is equal to the second cumulative distribution function evaluated at the pre-tone compression pixel intensities, and (b) extrapolating the directly matched correspondence via the highest range to generate the correspondence.

[0168] (B15) In the method described in (B14), the step of determining the corresponding relationship may further include: evaluating at least one of the first tone compression image and the first cumulative distribution function to determine a cutoff tone compression pixel intensity, wherein the cutoff tone compression pixel intensity does not include a saturated tone compression pixel intensity, and defining the partial range as a range of tone compression pixel intensities that are not greater than the cutoff tone compression pixel intensity.

[0169] (B16) In the method described in (B14), the histogram may be a partial histogram representing only a portion of the dynamic range, the portion of the dynamic range forming a lower subset of the full dynamic range of the first tone compressed HDR image before tone compression, and the step of determining the correspondence may further include evaluating the histogram to determine a cutoff pre-tone compression pixel intensity, and defining the partial range as the number of lowest tone compression pixel intensities, the number of lowest tone compression pixel intensities being equal to the number of pre-tone compression pixel intensities that are not greater than the cutoff pre-tone compression pixel intensity.

[0170] (B17) In the method described in (B16), the cutoff pre-tone compression pixel intensity can be a maximum pixel intensity of the histogram.

[0171] (B18) In any of the methods described in (B1) to (B17), the bins of the histogram may be on a logarithmic scale.

[0172] (B19) Any of the methods described in (B1) to (B18) may further include: (a) dividing each of the first tone compressed HDR image and the second tone compressed HDR image into a plurality of non-overlapping spatial image areas; and (b) for each of the plurality of spatial image areas, repeatedly performing the steps of extracting the tone compressed pixel intensity, generating the first cumulative distribution function, extracting the histogram, generating the second cumulative distribution function, determining the correspondence, and linearizing.

[0173] (B20) In the method described in (B19), the linearization step may include: at least for each pixel within a threshold distance of a boundary between different spatial image regions in the spatial image region, spatially interpolating the correspondence determined for different spatial image regions in the spatial image region based on the position of the pixel.

[0174] (B21) In the method described in (B19), each of the spatial image areas may have a corresponding center position, and the linearization step may include: for each pixel and based on the position of the pixel, spatially interpolating the correspondence determined for the four spatial image areas having the corresponding center positions closest to the pixel.

[0175] Changes may be made to the above systems and methods without departing from the scope thereof. It should therefore be noted that the contents contained in the above description and shown in the accompanying drawings should be interpreted in an illustrative rather than a restrictive sense. The following claims are intended to cover all statements of the general and specific features described herein and the scope of the present systems and methods that can be said to fall therein in a language.

[0176] Various aspects of the present invention may be understood from the following enumerated example embodiments (EEE):

[0177] EEE 1. A high dynamic range (HDR) camera module with adaptive image data linearization, the high dynamic range camera module comprising:

[0178] an HDR image sensor for generating a tone-compressed HDR image of a scene and configured to output the tone-compressed HDR image as a corresponding frame including active pixel data and metadata;

[0179] a processor, the processor being external to the HDR image sensor; and

[0180] a memory external to the HDR image sensor and storing machine-readable instructions that, when executed by the processor, control the processor to perform the following steps:

[0181] (a) extracting from a frame of a first tone-compressed HDR image generated by the HDR image sensor (i) a histogram of tone-compressed pixel intensities from active pixel data of the frame and (ii) pre-tone-compressed pixel intensities from metadata of the frame;

[0182] (b) obtaining a correspondence between tone-compressed pixel intensities and pre-tone-compressed pixel intensities from the tone-compressed pixel intensities and the histogram; and

[0183] (c) linearizing at least a portion of the active pixel data of a second tone compressed HDR image according to the correspondence to produce a linearized HDR image, the second tone compressed HDR image being selected from the group consisting of the first tone compressed HDR image and a subsequent tone compressed HDR image.

[0184] EEE 2. The HDR camera module of EEE 1, wherein the processor and the memory are implemented in a digital signal processor, and the HDR camera module further comprises an image signal processor, which is communicatively coupled to the digital signal processor for further processing the linearized HDR image by the digital signal processor. EEE 2.

[0185] EEE 3. The HDR camera module of EEE 2, wherein the image signal processor is configured to apply one or more procedures selected from the group consisting of: white balance, color correction, gamma correction, demosaicing, noise reduction, and image sharpening to the linearized HDR image. ...

[0186] EEE 4. The HDR camera module as described in EEE 1, wherein the first tone compressed HDR image and the second tone compressed HDR image are two consecutive images of an HDR video stream output by the HDR image sensor. EEE 4.

[0187] EEE 5. The HDR camera module of EEE 4, wherein the processor and the memory are implemented in a digital signal processor, the digital signal processor includes a buffer, and the machine-readable instructions are further configured to control the processor to perform a linearization step on a subset of active pixel data of the second tone compressed image continuously stored in the buffer.

[0188] EEE 6. An HDR camera module as described in any one of EEE 1 to 5, wherein the first tone compressed HDR image and the second tone compressed HDR image are the same still image, and the machine-readable instructions are further configured to control the processor to store active pixel data of the still image in an image storage device, at least until the linearization step is completed.

[0189] EEE 7. An HDR camera module as described in any one of EEE 1 to 6, wherein the HDR image sensor includes an output interface, the output interface is limited by an output bit depth, the HDR image sensor is configured to (i) generate a linear HDR image at a first bit depth exceeding the output bit depth, and (ii) perform tone compression on the linear HDR image to generate the tone compressed HDR image at the output bit depth, and the machine-readable instructions are configured to control the processor to map the active pixel data from the output bit depth to the first bit depth in the linearization step.

[0190] EEE 8. The HDR camera module according to any one of EEE 1 to 7, wherein the machine-readable instructions are configured such that the obtaining step comprises:

[0191] generating a first cumulative distribution function of the tone-compressed pixel intensity;

[0192] generating a second cumulative distribution function of the histogram;

[0193] The correspondence is determined such that for each corresponding pair of a plurality of corresponding pairs of tone-compressed pixel intensities and pre-tone-compressed pixel intensities, the first cumulative distribution function evaluated at the tone-compressed pixel intensities is equal to the second cumulative distribution function evaluated at the pre-tone-compressed pixel intensities.

[0194] EEE 9. The HDR camera module of EEE 8, wherein the machine-readable instructions are configured to:

[0195] The determining step includes storing a lookup table listing the corresponding pairs in the memory; and

[0196] The linearizing step includes interpolating between entries in the lookup table when the tone-compressed pixel intensities of the active pixel data of the second tone-compressed HDR image are not equal to the tone-compressed pixel intensities listed in the lookup table.

[0197] EEE 10. The HDR camera module of EEE 8 or EEE 9, wherein the machine-readable instructions are configured such that the determining step comprises:

[0198] determining the correspondence as a polynomial function that specifies the pre-tone-compressed pixel intensity as a function of the tone-compressed pixel intensity; and

[0199] Parameters of the polynomial function are stored in the memory.

[0200] EEE 11. The HDR camera module of EEE 8 or EEE 9, wherein the machine-readable instructions are configured such that the determining step comprises:

[0201] determining the correspondence as a spline representation specifying the pre-tone compressed pixel intensity as a function of the tone compressed pixel intensity; and

[0202] Parameters of the spline representation are stored in the memory.

[0203] EEE 12. The HDR camera module according to any one of EEEs 1 to 11, wherein the HDR image sensor is a color image sensor, the histogram is a brightness histogram, and the machine-readable instructions are configured such that:

[0204] The step of extracting the tone compressed pixel intensity comprises:

[0205] (a) extracting a plurality of sets of hue-compressed color-specific pixel intensities, each of the sets being specific to a corresponding color, and

[0206] (b) combining the sets to form the tone-compressed pixel intensities such that each of the tone-compressed pixel intensities represents a corresponding pixel brightness; and

[0207] The linearizing step includes individually linearizing each color of active pixel data of the second tone-compressed HDR image according to the corresponding relationship.

[0208] EEE 13. The HDR camera module as described in any one of EEE 8 to 12, wherein the machine-readable instructions are configured such that the step of determining the corresponding relationship comprises:

[0209] determining, for a partial range of tone-compressed pixel intensities excluding a highest range of the tone-compressed pixel intensities, a directly matched correspondence between the tone-compressed pixel intensities and the pre-tone-compressed pixel intensities, such that, for corresponding pairs of the tone-compressed pixel intensities and the pre-tone-compressed pixel intensities associated with the partial range, the first cumulative distribution function evaluated at the tone-compressed pixel intensities is equal to the second cumulative distribution function evaluated at the pre-tone-compressed pixel intensities; and

[0210] The directly matched correspondence is extrapolated via the highest range to generate the correspondence.

[0211] EEE 14. The HDR camera module according to EEE 13, wherein the machine-readable instructions are configured such that the step of determining the corresponding relationship further comprises:

[0212] evaluating at least one of the first tone-compressed image and the first cumulative distribution function to determine cutoff tone-compressed pixel intensities that exclude saturated tone-compressed pixel intensities; and

[0213] The partial range is defined as a range of tone-compressed pixel intensities not greater than the cutoff tone-compressed pixel intensity.

[0214] EEE 15. The HDR camera module of EEE 13, wherein the histogram is a partial histogram representing only a partial dynamic range, the partial dynamic range forming a lower subset of the full dynamic range of the first tone compressed HDR image before tone compression, and the machine-readable instructions are configured such that the step of determining the correspondence further comprises:

[0215] evaluating the histogram to determine a cutoff pre-tone compressed pixel intensity; and

[0216] The partial range is defined as a number of lowest tone compression pixel intensities that is equal to a number of pre-tone compression pixel intensities that are not greater than the cutoff pre-tone compression pixel intensity.

[0217] EEE 16. The HDR camera module of any one of EEE 1 to 15, wherein the machine-readable instructions are configured to, when executed by the processor, control the processor to:

[0218] dividing each of the first tone compressed HDR image and the second tone compressed HDR image into a plurality of non-overlapping spatial image regions; and

[0219] For each of the plurality of spatial image areas, the steps of extracting the tone-compressed pixel intensity, generating the first cumulative distribution function, extracting the histogram, generating the second cumulative distribution function, determining the corresponding relationship, and linearizing are repeatedly performed.

[0220] EEE 17. In the HDR camera module described in EEE 16, the machine-readable instructions are configured so that the linearization step includes: at least for each pixel within a threshold distance of a boundary between different spatial image regions in the spatial image region, spatially interpolating the correspondence determined for different spatial image regions in the spatial image region based on the position of the pixel.

[0221] EEE 18. An HDR camera module as described in EEE 16 or 17, wherein each of the spatial image areas has a corresponding center position, and the machine-readable instructions are configured so that the linearization step includes: for each pixel and based on the position of the pixel, spatially interpolating the correspondence determined for the four spatial image areas having the corresponding center positions closest to the pixel.

[0222] EEE 19. An adaptive method for linearizing tone compressed high dynamic range (HDR) images, the adaptive method comprising:

[0223] receiving a frame of a first tone compressed HDR image from an HDR image sensor;

[0224] extracting from the frame a histogram of (a) tone-compressed pixel intensities from active pixel data of the frame and (b) pre-tone-compressed pixel intensities from metadata of the frame;

[0225] Obtaining a correspondence between tone compression pixel intensity and pre-tone compression pixel intensity from the tone compression pixel intensity and the histogram;

[0226] receiving a frame of a second tone compressed HDR image from the HDR image sensor, the second tone compressed HDR image selected from the group consisting of the first tone compressed HDR image and a subsequent tone compressed HDR image; and

[0227] At least a portion of active pixel data of a frame of the second tone compressed HDR image is linearized according to the correspondence to produce a linearized HDR image.

[0228] EEE 20. The method of EEE 19, wherein the first tone compressed HDR image and the second tone compressed HDR image are consecutive images of an HDR video stream. EEE 21.

[0229] EEE 21. The method of EEE 20, further comprising: before the linearization step, limiting the deviation between the correspondence and a previous correspondence obtained for a previously captured image pair of the HDR video stream. EEE 22. The method of EEE 22, further comprising: before the linearization step, limiting the deviation between the correspondence and a previous correspondence obtained for a previously captured image pair of the HDR video stream.

[0230] EEE 22. The method as described in any one of EEE 19 to 21, wherein the first tone compressed HDR image and the second tone compressed HDR image are the same still image. EEE 23. The method as described in any one of EEE 19 to 21, wherein the first tone compressed HDR image and the second tone compressed HDR image are the same still image.

[0231] EEE 23. A method as described in any one of EEEs 19 to 22, wherein the tone compression pixel intensity is limited to a compression bit depth, the compression bit depth is less than the pre-tone compression bit depth, and the linearization step includes mapping the active pixel data to the pre-tone compression bit depth.

[0232] EEE 24. The method according to any one of EEE 19 to 23, wherein the obtaining step comprises:

[0233] generating a first cumulative distribution function of the tone-compressed pixel intensity;

[0234] generating a second cumulative distribution function of the histogram; and

[0235] The correspondence is determined such that for each corresponding pair of a plurality of corresponding pairs of tone-compressed pixel intensities and pre-tone-compressed pixel intensities, the first cumulative distribution function evaluated at the tone-compressed pixel intensities is equal to the second cumulative distribution function evaluated at the pre-tone-compressed pixel intensities.

[0236] EEE 25. The method of EEE 24, comprising:

[0237] In the determining step, the corresponding relationship is generated as a lookup table listing the corresponding pairs; and

[0238] In the linearisation step, when the tone-compressed pixel intensities of the active pixel data of the second tone-compressed HDR image are not equal to the tone-compressed pixel intensities listed in the lookup table, interpolation is performed between entries in the lookup table.

[0239] EEE 26. The method as described in EEE 24 or EEE 25 includes: in the determining step, generating the corresponding relationship as a polynomial function, and the polynomial function specifies the pre-tone compressed pixel intensity as a function of the tone compressed pixel intensity.

[0240] EEE 27. The method as described in EEE 24 or EEE 25 includes: in the determining step, generating the correspondence as a spline representation, which specifies the pre-tone compressed pixel intensity as a function of the tone compressed pixel intensity.

[0241] EEE 28. The method as described in any one of EEE 24 to 27, wherein the first tone compressed HDR image and the second tone compressed HDR image are color images, and the histogram is a brightness histogram.

[0242] EEE 29. The method according to any one of EEE 19 to EEE 28, further comprising:

[0243] In the step of extracting the tone-compressed pixel intensities, a plurality of sets of tone-compressed color-specific pixel intensities are extracted, each of the sets being specific to a respective color;

[0244] combining the sets to form the tone-compressed pixel intensities such that each of the tone-compressed pixel intensities represents a corresponding pixel brightness; and

[0245] In the linearization step, each color of the active pixel data of the second tone-compressed HDR image is individually linearized according to the corresponding relationship.

[0246] EEE 30. The method according to any one of EEE 19 to EEE 28, further comprising:

[0247] In the step of extracting the tone compressed pixel intensity, a tone compressed color specific pixel intensity specific to a single color is extracted, the single color being one of a plurality of colors constituting the first tone compressed HDR image;

[0248] In the generating step, the first cumulative distribution function about the single color is generated; and

[0249] In the linearization step, each color of the active pixel data of the second tone-compressed HDR image is individually linearized according to the corresponding relationship.

[0250] EEE 31. A method as described in any one of EEE 24 to EEE 30, wherein the first tone compressed HDR image and the second tone compressed HDR image are color images, and the method includes separately performing the steps of extracting the tone compressed pixel intensity, generating the first cumulative distribution function, extracting the histogram, generating the second cumulative distribution function, determining the correspondence, and linearizing at least a portion of the active pixel data of the second tone compressed HDR image for each color of the first tone compressed HDR image and the second tone compressed HDR image.

[0251] EEE 32. The method according to any one of EEE 24 to EEE 31, wherein the first tone compressed HDR image and the second tone compressed HDR image are color images, and the method further comprises:

[0252] In the step of extracting the tone compressed pixel intensity, a tone compressed color specific pixel intensity specific to a single color is extracted, the single color being one of a plurality of colors constituting the first tone compressed HDR image;

[0253] In the generating step, the first cumulative distribution function about the single color is generated;

[0254] In the step of extracting the histogram, a color-specific histogram specific to the single color is extracted;

[0255] In the generating step, the second cumulative distribution function is generated with respect to the color-specific histogram; and

[0256] In the linearization step, each color of the active pixel data of the second tone-compressed HDR image is individually linearized according to the corresponding relationship.

[0257] EEE 33. The method according to EEE 24 to EEE 32, wherein the determining step comprises:

[0258] determining, for a partial range of tone-compressed pixel intensities excluding a highest range of the tone-compressed pixel intensities, a directly matched correspondence between the tone-compressed pixel intensities and the pre-tone-compressed pixel intensities, such that, for corresponding pairs of the tone-compressed pixel intensities and the pre-tone-compressed pixel intensities associated with the partial range, the first cumulative distribution function evaluated at the tone-compressed pixel intensities is equal to the second cumulative distribution function evaluated at the pre-tone-compressed pixel intensities; and

[0259] The directly matched correspondence is extrapolated through the highest range to generate the correspondence.

[0260] EEE 34. As described in EEE 33, the step of determining the corresponding relationship further comprises:

[0261] evaluating at least one of the first tone-compressed image and the first cumulative distribution function to determine cutoff tone-compressed pixel intensities that exclude saturated tone-compressed pixel intensities; and

[0262] The partial range is defined as a range of tone-compressed pixel intensities not greater than the cutoff tone-compressed pixel intensity.

[0263] EEE 35. The method as described in EEE 33 or EEE 34, wherein the histogram is a partial histogram representing only a partial dynamic range, the partial dynamic range forming a lower subset of the full dynamic range of the first tone compressed HDR image before tone compression, and the step of determining the corresponding relationship further comprises:

[0264] evaluating the histogram to determine a cutoff pre-tone compressed pixel intensity; and

[0265] The partial range is defined as a number of lowest tone compression pixel intensities that is equal to a number of pre-tone compression pixel intensities that are not greater than the cutoff pre-tone compression pixel intensity.

[0266] EEE 36. The method of EEE 35, wherein the cutoff pre-tone compression pixel intensity is the maximum pixel intensity of the histogram.

[0267] EEE 37. The method of any one of EEE 33 to EEE 36, wherein the bins of the histogram are on a logarithmic scale.

[0268] EEE 38. The method according to any one of EEE 19 to EEE 37, further comprising:

[0269] dividing each of the first tone compressed HDR image and the second tone compressed HDR image into a plurality of non-overlapping spatial image regions; and

[0270] For each of the plurality of spatial image areas, the steps of extracting the tone-compressed pixel intensity, generating the first cumulative distribution function, extracting the histogram, generating the second cumulative distribution function, determining the corresponding relationship, and linearizing are repeatedly performed.

[0271] EEE 39. In the method described in EEE 38, the linearization step includes: at least for each pixel within a threshold distance of a boundary between different spatial image regions in the spatial image region, spatially interpolating the correspondence determined for different spatial image regions in the spatial image region based on the position of the pixel.

[0272] EEE 40. A method as described in EEE 38 or EEE 39, wherein each of the spatial image areas has a corresponding center position, and the linearization step includes: for each pixel and based on the position of the pixel, spatially interpolating the correspondence determined for the four spatial image areas having the corresponding center positions closest to the pixel.

[0273] EEE 41. An HDR camera module as described in any one of EEE 1 to EEE 18, wherein the HDR image sensor is configured to output the tone compressed HDR image as a corresponding frame including active pixel data and metadata, the metadata including a histogram of pre-tone compressed pixel intensities. EEE 42.

[0274] EEE 42. An HDR camera module as described in any one of EEE 1 to EEE 18 or EEE 41, wherein a histogram of pre-tone compressed pixel intensities extracted from the first tone compressed HDR image generated by the HDR image sensor is provided in metadata of a frame of the first tone compressed HDR image.

[0275] EEE 43. An electronic device comprising the HDR camera module as described in any one of EEE 1 to EEE 18 or EEE 41 to EEE 42.

[0276] EEE 44. A mobile device comprising the HDR camera module as described in any one of EEE 1 to EEE 18 or EEE 41 to EEE 42.

[0277] EEE 45. The electronic device as described in EEE 43 or the mobile device as described in EEE 44 further comprises a housing, wherein the HDR camera module is arranged in the housing.

[0278] EEE 46. A method as described in any of EEE 19 to EEE 40, wherein a histogram of pre-tone compressed pixel intensities extracted from frames of the first tone compressed HDR image is provided in metadata of the frames of the first tone compressed HDR image.

[0279] EEE 47. A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed by a processor, control the processor to perform the method as described in any one of EEE 19 to EEE 40 or EEE 46.

Claims

1. A high dynamic range (HDR) camera module with adaptive image data linearization, comprising: an HDR image sensor for generating a tone-compressed HDR image of a scene and configured to output the tone-compressed HDR image as a corresponding frame including active pixel data and metadata; a processor, the processor being external to the HDR image sensor; and a memory external to the HDR image sensor and storing machine-readable instructions that, when executed by the processor, control the processor to perform the following steps: extracting from a frame of a first tone-compressed HDR image generated by the HDR image sensor a histogram of tone-compressed pixel intensities from active pixel data of the frame and pre-tone-compressed pixel intensities from metadata of the frame; Obtaining a correspondence between tone compression pixel intensity and pre-tone compression pixel intensity from the tone compression pixel intensity and the histogram; as well as linearizing at least a portion of the active pixel data of a second tone compressed HDR image according to the correspondence to produce a linearized HDR image, the second tone compressed HDR image being selected from the group consisting of the first tone compressed HDR image and a subsequent tone compressed HDR image; The machine readable instructions are configured such that the obtaining step comprises: generating a first cumulative distribution function of the tone-compressed pixel intensity; generating a second cumulative distribution function of the histogram; determining the correspondence relationship such that, for each corresponding pair of a plurality of corresponding pairs of tone-compressed pixel intensities and pre-tone-compressed pixel intensities, a value of the first cumulative distribution function evaluated at the tone-compressed pixel intensities is equal to a value of the second cumulative distribution function evaluated at the pre-tone-compressed pixel intensities, wherein the correspondence relationship is determined as a polynomial function or a spline representation that specifies the pre-tone-compressed pixel intensities as a function of the tone-compressed pixel intensities; and Parameters of the polynomial function or the spline representation are stored in the memory.

2. The HDR camera module of claim 1 , the processor and the memory being implemented in a digital signal processor, the HDR camera module further comprising an image signal processor communicatively coupled to the digital signal processor for further processing of the linearized HDR image by the digital signal processor.

3. The HDR camera module of claim 2, the image signal processor being configured to apply one or more procedures selected from the group consisting of: white balance, color correction, gamma correction, demosaicing, noise reduction, and image sharpening to the linearized HDR image.

4. The HDR camera module of claim 1, wherein the first tone compressed HDR image and the second tone compressed HDR image are two consecutive images of an HDR video stream output by the HDR image sensor.

5. The HDR camera module of claim 4, the processor and the memory being implemented in a digital signal processor, the digital signal processor comprising a buffer, the machine-readable instructions being further configured to control the processor to perform a linearization step on a subset of the active pixel data of the second tone compressed HDR image that is continuously stored in the buffer.

6. The HDR camera module of any one of claims 1 to 5, wherein the first tone compressed HDR image and the second tone compressed HDR image are the same still image, and the machine-readable instructions are further configured to control the processor to store the active pixel data of the still image in an image storage device, at least until the linearization step is completed.

7. The HDR camera module of any one of claims 1 to 5, the HDR image sensor comprising an output interface, the output interface being limited to an output bit depth, the HDR image sensor being configured to generate a linear HDR image at a first bit depth exceeding the output bit depth, and to perform tone compression on the linear HDR image to generate the tone compressed HDR image at the output bit depth, the machine-readable instructions being configured to control the processor to map the active pixel data from the output bit depth to the first bit depth in a linearization step.

8. The HDR camera module of claim 1 , wherein the machine-readable instructions are configured to: The determining step includes storing a lookup table listing the corresponding pairs in the memory; and The linearizing step includes interpolating between entries in the lookup table when the tone-compressed pixel intensities of the active pixel data of the second tone-compressed HDR image are not equal to the tone-compressed pixel intensities listed in the lookup table.

9. The HDR camera module of any one of claims 1 to 5, wherein the HDR image sensor is a color image sensor, the histogram is a brightness histogram, and the machine-readable instructions are configured such that: The step of extracting the tone compressed pixel intensity comprises: extracting a plurality of sets of hue-compressed color-specific pixel intensities, each of the sets being specific to a corresponding color, and combining the sets to form the tone-compressed pixel intensities, such that each of the tone-compressed pixel intensities represents a corresponding pixel brightness; as well as The linearizing step includes individually linearizing each color of the active pixel data of the second tone-compressed HDR image according to the corresponding relationship.

10. The HDR camera module of claim 1, wherein the machine-readable instructions are configured such that the step of determining the corresponding relationship comprises: determining, for a partial range of tone-compressed pixel intensities excluding a highest range of the tone-compressed pixel intensities, a directly matched correspondence between the tone-compressed pixel intensities and the pre-tone-compressed pixel intensities, such that, for the corresponding pair of the tone-compressed pixel intensities and the pre-tone-compressed pixel intensities associated with the partial range, the first cumulative distribution function evaluated at the tone-compressed pixel intensities is equal to the second cumulative distribution function evaluated at the pre-tone-compressed pixel intensities; and The directly matched correspondence is extrapolated via the highest range to generate the correspondence.

11. The HDR camera module of claim 10, wherein the machine-readable instructions are configured such that the step of determining the corresponding relationship further comprises: evaluating at least one of the first tone-compressed HDR image and the first cumulative distribution function to determine cutoff tone-compressed pixel intensities, the cutoff tone-compressed pixel intensities excluding saturated tone-compressed pixel intensities; as well as The partial range is defined as a range of tone-compressed pixel intensities not greater than the cutoff tone-compressed pixel intensity.

12. The HDR camera module of claim 10, wherein the histogram is a partial histogram representing only a partial dynamic range, the partial dynamic range forming a lower subset of the full dynamic range of the first tone compressed HDR image before tone compression, the machine readable instructions being configured such that the step of determining the correspondence further comprises: evaluating the histogram to determine a cutoff pre-tone compressed pixel intensity; as well as The partial range is defined as a number of lowest tone compression pixel intensities that is equal to a number of pre-tone compression pixel intensities that are not greater than the cutoff pre-tone compression pixel intensity.

13. The HDR camera module of claim 1 , wherein the machine-readable instructions are configured to, when executed by the processor, control the processor to: dividing each of the first tone compressed HDR image and the second tone compressed HDR image into a plurality of non-overlapping spatial image regions; and For each of the multiple non-overlapping spatial image areas, the following steps are repeatedly performed: extracting the tone compressed pixel intensity, generating the first cumulative distribution function, extracting the histogram, generating the second cumulative distribution function, determining the corresponding relationship, and linearizing.

14. The HDR camera module of claim 13, wherein the machine-readable instructions are configured such that the step of linearizing comprises: At least for each pixel within a threshold distance of a boundary between different ones of the aerial image regions, the correspondences determined for the different ones of the aerial image regions are spatially interpolated based on the position of the pixel.

15. The HDR camera module of claim 13, wherein each of the spatial image regions has a corresponding center position, and wherein the machine-readable instructions are configured such that the linearizing step comprises: For each pixel and based on the position of the pixel, the correspondences determined for the four spatial image regions having respective centre positions closest to the pixel are spatially interpolated.

16. The HDR camera module according to any one of claims 1 to 5, wherein: The HDR image sensor is configured to output the tone compressed HDR image as a respective frame comprising active pixel data and metadata, the metadata comprising a histogram of pre-tone compressed pixel intensities.

17. The HDR camera module according to any one of claims 1 to 5, wherein: The histogram of pre-tone compressed pixel intensities extracted from the first tone compressed HDR image generated by the HDR image sensor is provided in metadata of the frame of the first tone compressed HDR image.

18. An electronic device comprising the HDR camera module according to any one of claims 1 to 17.

19. An adaptive method for linearizing tone compressed high dynamic range (HDR) images, comprising: receiving a frame of a first tone compressed HDR image from an HDR image sensor; extracting from the frame a histogram of tone-compressed pixel intensities from active pixel data of the frame and pre-tone-compressed pixel intensities from metadata of the frame; Obtaining a correspondence between tone compression pixel intensity and pre-tone compression pixel intensity from the tone compression pixel intensity and the histogram; receiving a frame of a second tone compressed HDR image from the HDR image sensor, the second tone compressed HDR image selected from the group consisting of the first tone compressed HDR image and a subsequent tone compressed HDR image; as well as linearizing at least a portion of active pixel data of the frame of the second tone compressed HDR image according to the correspondence to produce a linearized HDR image, The step of obtaining the corresponding relationship includes: generating a first cumulative distribution function of the tone-compressed pixel intensity; generating a second cumulative distribution function of the histogram; determining the correspondence relationship such that, for each corresponding pair of a plurality of corresponding pairs of tone-compressed pixel intensities and pre-tone-compressed pixel intensities, a value of the first cumulative distribution function evaluated at the tone-compressed pixel intensities is equal to a value of the second cumulative distribution function evaluated at the pre-tone-compressed pixel intensities, wherein the correspondence relationship is determined as a polynomial function or a spline representation that specifies the pre-tone-compressed pixel intensities as a function of the tone-compressed pixel intensities; and The parameters of the polynomial function or the spline representation are stored in a memory.

20. A non-transitory computer-readable storage medium having instructions stored thereon, which when executed by a processor, control the processor to perform the method of claim 19.

21. A computer program product comprising a computer program which, when executed by a processor, causes the method of claim 19 to be performed.

Images