Temporal error mitigation in multi-exposure image fusion

Abstract

Stitched images compiled from base images with non-simultaneous exposures may exhibit unwanted aliasing, despite correction for exposure, particularly when the illumination on or present in the scene changes. Herein, a correction is made by first identifying comparison pixels in an image with greater exposure that have a high light metric. After correction for exposure, the light metrics of the same pixels in the lesser exposed image are determined. A ratio between the light metrics of these comparison pixels in the two images is determined and used to scale one of the images. With the comparison pixels now having the same light metrics, the images are stitched. More than two base images may be used, they may be 1, 2 or 3 dimensional, and may be obtained with a global or rolling shutter.

Description

TEMPORAL ERROR MITIGATION IN MULTI-EXPOSURE IMAGE FUSIONTECHNICAL FIELD

[0001] This application relates to image data processing, in particular to high dynamic range (HDR) imaging.BACKGROUND

[0002] HDR imaging is widely used in both photography and light measurement applications. US Patent 5,818,977 to Tansley describes an apparatus that creates a series of digital images of the same scene over a range of different exposures, and assembles data from the series of images to create a single composite image representing an intensity dynamic range greater than that obtainable from that which can be measured from any single image.

[0003] Images are scaled by taking their exposure time into account. In the simplest case, the exposure is scaled by the inverse of the time taken in the exposure. If the exposure is doubled, the scaling factor is one half. If the imaging is not linear with exposure time, more sophisticated linearity scaling factors may be required.Additionally, at any single exposure, the measured output may not have a linear relationship with the signal being measured. It is common to characterize the linearity of the sensor and apply a polynomial correction to the data values from each pixel to correct for the detector’s non-linearity at one exposure time. As an example, Yi=aixi2+bixi+ci, and Y2=a2X22+b2X2+C2, where Yi and Y2 are the values for a particular pixel in the images, at different exposure times, and xi, and X2 are the digitized raw pixel data values at the two exposures. Depending on the application, the linearity scaling may include a conversion from counts to absolute units such as luminance in cd / m2, radiance in W / cm2 / sr, or any other absolute or relative quantity or scale.

[0004] When using two images, the valid data from the longer exposure replaces the data from the shorter exposure, and the stitching of these two images is complete. In Tansley’s method, the raw pixel response before scaling or linearity need not be saved because the valid data from the longer exposure always overwrites the data from the shorter exposure. Stitching of further shorter and / or longer exposures, if used,continues until all pixels in the scene have a satisfactory signal to noise ratio (SNR). Stitching stops if there are no longer or shorter exposure times possible to acquire. The method has worked well and has been commercially available since 1997 in IQCAM® imaging photometers among others.

[0005] Traditional multi-exposure fusion (MEF) involves capturing multiple images at different exposure levels and combining them to create a single HDR image. Advanced algorithms in video applications have been developed to align and merge these images, minimizing artifacts such as ghosting caused by moving objects.

[0006] When light levels change from frame to frame in a non-HDR video sequence, temporal smoothing filters can be applied to average out brightness variations over several frames, thereby reducing the perception of flicker. However, if light levels vary between stitched exposures in an HDR frame, a more granular scaling method is required.

[0007] When a one-dimensional (1 D) or two dimensional (2D) light measurement with a CCD (charge-coupled device), CMOS (complementary metal-oxide-semiconductor) or other type of sensor array repeatedly images a modulated light source at a sufficiently short integration time, pixel values can become inconsistent, which is a phenomenon known as temporal measurement error. Even if the light source is not modulated, temporal measurement error may be caused by exposure time imprecision.

[0008] In existing HDR light measurement applications, temporal measurement errors can be mitigated in several ways. However, there are drawbacks with each of them. For example, when performing an averaging of sequential measurements, a disadvantage is speed. If the aliasing is significant, hundreds of samples or more may need to be acquired to achieve a precise average. This method also fails if any input data from a location has saturated data, which may be considered to be invalid data. This concern becomes more probable if the light source has a high modulation depth.

[0009] Another way is to set the exposure time base to be an integer multiple of the light period for scenes illuminated with modulated light. However, this method requires an additional measurement or prior knowledge of the light modulation period, which is not always possible. The matching of the measurement array integration time to the modulated light period may be compromised if either the integration period cannot be set with sufficient precision, or if the light modulation period is not known with sufficient accuracy. Furthermore, if the light saturates the detector when the exposure time isequal to one period of the light, no shorter exposures are possible without the risk of aliasing.

[0010] It is also possible to add a density filter or a neutral density (ND) filter to attenuate the light and thereby engage longer image acquisition times to capture the images. This method has several disadvantages, as follows. Firstly, the lower light signals will require much longer exposures if they are to be acquired with a good SNR. As a result, the dynamic range will be reduced, or the measurement times will get longer if an ND filter is deployed. Secondly, the solution requires mechanical automation or manual intervention to add the ND filter when needed. The added optics, mechanics, or user intervention have measurement speed, cost, and reliability implications. Thirdly, the added optical element may change the relative spatial and spectral transmission of the optical system, which, for some applications, may add new dimensions of error and uncertainty. Finally, the added optical element may cause unwanted reflections or distortions in the image.

[0011] Another way to mitigate temporal measurement errors is to change the iris to attenuate the light, enabling image acquisition at longer exposure times. This method has several disadvantages, as follows. For automation, a mechanical solution needs to be created to automate the iris manipulation. This may add cost and complexity to the system and also reduce reliability. In optical setups to image Near Eye Displays (NEDs), the iris is chosen to match a typical human pupil, so changing the iris size is not an option. Setting the iris may not be reproducible in many systems, such as consumer grade lenses, especially for minimal iris positions.

[0012] HDR imaging described previously by Tansley and others follows the general principles: (a) low SNR data in a short exposure is replaced with higher SNR data from a longer exposure, (b) saturated, i.e. over-exposed data in a long exposure is replaced with valid data from a shorter exposure, (c) data that is not over-exposed in the longest possible exposure is the best data, and (d) algorithms track the sub-optimal data and replace it as the imaged data sets are stitched (fused) into an HDR image.

[0013] Also, a detector that drifts during long exposures is problematic for the HDR method provided by Tansley. If the drifting is significant over the time period of the HDR capture, then there may be discontinuities in the captured data.

[0014] Referring to FIG. 1 , this depicts unmodulated and modulated signals graphed versus time. The dashed line 10 is unmodulated light. The solid line 12 is 100% pulsemodulated. FIG. 2 depicts a modulated signal 14 with an approximately 50% modulation depth and FIG. 3 depicts a modulated signal 16 with instability from period to period.

[0015] FIG. 4 depicts a measuring instrument integration time 18 and a multitude of signal pulses 20. In this example, the measurements should have good reproducibility. Even if the measurement base time is not an integer multiple of the pulse period, the number of pulses during one integration might vary only from 29 to 30. In this case, the precision would be about 3% for repeat measurements.

[0016] FIG. 5 depicts a camera integration time 22 that is matched to have the same interval as one period of the signal modulation 24. Even if sampled asynchronously, the measurement will always be for a single period and the measurements will have high precision.

[0017] FIG. 6 depicts a measurement integration time 26 that is shorter than the period of modulation of the signal 28. The measurement occurs when the signal is at a minimum. In this example, there is no signal when the camera is integrating. Other asynchronous measurements 30 as shown in FIG. 7 may measure the pulse at its peak 32 for the duration of the measurement. Other asynchronous measurements 34 as shown in FIG. 8 may measure the pulse 36 as being partly on for the duration of the measurement. With such a short integration time, the asynchronous measurement of the waveform will result in values of zero, maximum and any value in between, given enough sampling attempts.

[0018] FIG. 9 depicts a series of HDR spectra acquired of a modulated source. The long exposure 38 captures the data up to about 515 nm and beyond 565 nm. The long exposure is consistently saturated in the region from about 515 to 565 nm. However, the 515 to 565 nm region can be measured at a shorter exposure time. Because this light source has a high depth of modulation and the short exposure time is not too dissimilar from the period of the light source, asynchronous sampling of the short exposure yields irreproducible results 39, 40, 41. The measurement is therefore susceptible to aliasing.

[0019] Rolling shutter sensors are commonly used in CMOS cameras, including many smartphones and DSLR (digital single-lens reflex) cameras, due to their lower cost and faster readout speeds compared to global shutter sensors. A rolling shutter sensor captures an image by scanning across the sensor, typically from top to bottom,rather than capturing the entire image at once. Side-to-side capture is also possible. The pertinent details are that the sensor has a line-by-line exposure rather than exposing the entire array at once. The sensor starts exposing the top row of pixels first, then moves down to the next row, and so on, until it reaches the bottom. This process happens very quickly, but not instantaneously. As the sensor scans down, each row of pixels is exposed with a delay relative to the previous row. The exposure time can be chosen depending on the camera settings.

[0020] FIG. 10 depicts the timing in a rolling shutter camera. As each row is captured, there is a slight delay before the next row is read. This means that the top row is captured slightly earlier than the next row, which is slightly earlier than the next row, and so on. For each row captured there is a line reset time 42 followed by an exposure duration 44, which is then followed by a line readout 46. In this example, no two rows are captured simultaneously but there is some overlap of the time at which neighboring rows are captured. The exposure and readout happen in a rolling fashion, and as a result, fast-moving objects or rapid camera movements can cause distortions. For example, vertical lines might appear slanted, or fast-moving objects might look stretched or skewed. This effect is known as the "rolling shutter effect". In the case of modulated light sources, a uniform field may appear in a captured image to have horizontal lines with successive lighter and darker shading. This example of temporal aliasing is depicted in FIG. 11 , which is an HDR image of a modulated light source using a rolling shutter camera. The bands of light and dark are not reproducible from measurement to measurement.

[0021] In a rolling shutter imager there is a triggering delay for each successive row in the detector. If the delay is, for example, 35 microseconds and there are 6000 rows of pixels, the last row will start exposing 0.21 seconds after the first row. If one wanted to use exposure times where this triggering error is small, measurements would perhaps be limited to 10 s or longer. For an HDR image, this may mean exposure times of 10 s and 100 s for the bright and dim unstitched frames respectively. Density filters or very small iris settings would likely be required to provide the necessary attenuation. These attenuation methods are subject to the issues described above.SUMMARY OF INVENTION

[0022] Pixel values are tracked in unstitched frames that are used for forming an HDR image. Differences in their values between the frames, after adjusting for normal exposure differences, are monitored to allow for the calculation of scaling factors for adjusting all but one of the base images.

[0023] Specifically, after the usual adjustment for exposure, one image is scaled in its light metric based on comparing valid data from it and a reference image selected from the base images. The scaled image is then stitched to the reference image. To stitch two images, exposure times are chosen or images selected such that there is a subset of data locations in the two images with valid data that can be compared. If the data quality in these locations in the images is valid and satisfactory, then a scaling factor or ratio is created to adjust one of the image’s light metrics so that the data from the compared locations in the two images have the same values of the light metric or the same average values of the light metric. Following this, HDR stitching continues as normal.

[0024] The method applies to 1 D sensors such as line-scan cameras and spectrometers, 2D sensors and three-dimensional (3D) sensors, such as multispectral and hyperspectral imagers. It is applicable to both global shutter and rolling shutter imagers. After creating the scalars necessary to properly stitch images acquired with a rolling shutter camera, methods are described to improve the precision of the scalars and to compute the frequency of the modulation.

[0025] The method may be used to detect if there is modulation of the signal and if aliasing is a concern. For rolling shutter imagers, a method to determine the modulation frequency from an HDR exposure set is described. If the monitored pixels are not comparable, it may mean that there is modulation. In the case of a rolling shutter camera, the rolling shutter is used as a timed capture that may compute a scene’s modulation.

[0026] Disclosed is a method for forming a high dynamic range (HDR) image, comprising the steps of: (a) obtaining two corresponding images of a scene wherein: each image has pixels and a light metric for each pixel; the images are obtained with a difference in exposure, one exposure being greater and the other exposure being lesser; and one of the images is compensated for the difference in exposure; (b)selecting, from the pixels, comparison pixels that correspond in location to each other, wherein the light metric of each comparison pixel is, in the image with the greater exposure, between a lower boundary and an upper boundary; (c) determining a first statistical measure, of the light metrics of the comparison pixels in one of the images; (d) determining a second statistical measure, of the light metrics of the comparison pixels in the other of the images; (e) calculating a ratio of the first statistical measure to the second statistical measure; (f) multiplying the light metric of at least some of the pixels in the other of the images by the ratio; and (g) stitching the images to form the HDR image.

[0027] As further disclosed, another aspect of the present invention is a device for forming a high dynamic range (HDR) image, comprising a detector, a processor, and a computer readable memory storing computer-readable instructions that are executed by the processor to cause the processor to: obtain, via the detector, two corresponding images of a scene wherein each image has pixels and a light metric for each pixel, the images are obtained with a difference in exposure, one exposure being greater and the other exposure being lesser, and one of the images is compensated for the difference in exposure; select, from the pixels, comparison pixels that correspond in location to each other, wherein the light metric of each comparison pixel is, in the image with the greater exposure, between a lower boundary and an upper boundary; determine a first statistical measure, of the light metrics of the comparison pixels in one of the images; determine a second statistical measure, of the light metrics of the comparison pixels in the other of the images; calculate a ratio of the first statistical measure to the second statistical measure; multiply the light metric of at least some of the pixels in the other of the images by the ratio; and stitch the images to form the HDR image.

[0028] As still further disclosed, another aspect of the present invention is a computer readable memory storing computer-readable instructions, which, when executed by a processor, cause the processor to: obtain, via a detector, two corresponding images of a scene wherein each image has pixels and a light metric for each pixel, the images are obtained with a difference in exposure, one exposure being greater and the other exposure being lesser, and one of the images is compensated for the difference in exposure; select, from the pixels, comparison pixels that correspond in location to each other, wherein the light metric of each comparison pixel is, in the image with the greater exposure, between a lower boundary and an upper boundary; determine a firststatistical measure, of the light metrics of the comparison pixels in one of the images; determine a second statistical measure, of the light metrics of the comparison pixels in the other of the images; calculate a ratio of the first statistical measure to the second statistical measure; multiply the light metric of at least some of the pixels in the other of the images by the ratio; and stitch the images to form a high dynamic range (HDR) image.

[0029] This summary provides a simplified, non-exhaustive introduction to some aspects of the invention, without delineating the scope of the invention.BRIEF DESCRIPTION OF DRAWINGS

[0030] The following drawings illustrate embodiments of the invention and should not be construed as restricting the scope of the invention in any way.

[0031] FIG. 1 shows prior art unmodulated and modulated signals.

[0032] FIG. 2 shows a prior art 50% modulated signal.

[0033] FIG. 3 shows a prior art 50% modulated signal with instability.

[0034] FIG. 4 shows a prior art measurement integration signal spanning multiple signal pulses.

[0035] FIG. 5 shows a prior art camera integration time matched to the period of signal modulation.

[0036] FIG. 6 shows a prior art camera integration time shorter than the period of signal modulation and coinciding with the signal minimum.

[0037] FIG. 7 shows a prior art camera integration time shorter than the period of signal modulation and coinciding with the signal maximum.

[0038] FIG. 8 shows a prior art camera integration time shorter than the period of signal modulation and coinciding with a signal transition.

[0039] FIG. 9 depicts a prior art series of HDR spectra acquired from a modulated source.

[0040] FIG. 10 depicts timing in a prior art rolling shutter camera.

[0041] FIG. 11 is an HDR image of a modulated light source taken with a prior art rolling shutter camera.

[0042] FIG. 12 is an image of an array of light-emitting diodes (LEDs).

[0043] FIG. 13 is a histogram of properly exposed pixels in an image of the array in FIG. 12, exposed with a 20 ms exposure, according to an embodiment of the present invention.

[0044] FIG. 14 is a histogram of properly exposed pixels in an image of the array in FIG. 12, exposed with an 0.5 ms exposure, according to an embodiment of the present invention.

[0045] FIG. 15 is the image of FIG. 12 with comparison pixels marked, according to an embodiment of the present invention.

[0046] FIG. 16 shows two differently exposed spectral measurements after corrections for exposure that are based on the light source being unmodulated, according to an embodiment of the present invention.

[0047] FIG. 17 is a zoomed-in portion of FIG. 16.

[0048] FIG. 18 is a flowchart for removing aliasing, in accordance with some implementations of the present invention.

[0049] FIG. 19 is a flowchart for removing more severe aliasing, in accordance with some implementations of the present invention.

[0050] FIG. 20 depicts grouped rows in a rolling shutter camera, according to an embodiment of the present invention.

[0051] FIG. 21 is an HDR image of the source in FIG. 11 , corrected according to an embodiment of the present invention.

[0052] FIG. 22 is a plot of scalars used when stitching aliased short exposures to a stabilized longer exposure in a rolling shutter camera, according to an embodiment of the present invention.

[0053] FIG. 23 is a flowchart for setting exposure times in a rolling shutter camera, according to an embodiment of the present invention.

[0054] FIG. 24 is a schematic diagram of a system for mitigating temporal errors in HDR images, according to an embodiment of the present invention.DETAILED DESCRIPTIONA. Glossary

[0055] Aliasing: The presence of features in an image or video perceived as errors due to a timing mismatch between illumination and / or capture of a scene.

[0056] Firmware: may include, but is not limited to, program code and data used to control and manage the capture, analysis, modification, storage and stitching of images.

[0057] Hardware: includes, but is not limited to, a camera, sensor, a display screen, connectors, wiring, circuit boards and physical components of a computer, for example.

[0058] HDR: High dynamic range, which refers to compiled image data in which the number of discernable light levels is greater than that in any single constituent image.

[0059] MEF: Multi-exposure fusion

[0060] Light metric, or metric: Used herein to refer to a measure of the amount light in a pixel of an image, and may be an absolute or relative physical quantity. It may refer, for example, to luminance in cd / m2, radiance in W / cm2 / sr, a ratio expressed as a percentage, or a count on an arbitrary scale. The light may include any wavelengths of light from X-ray to infrared.

[0061] ND: Neutral density

[0062] Processor or processing circuitry: refers to any electronic circuit or group of circuits that perform calculations. The processor performs at least some of the steps in the flowcharts.

[0063] SNR: Signal to Noise Ratio

[0064] Software: may include, but is not limited to, program code and data used to control and manage the capture, analysis, modification and stitching of images.B. Exemplary Embodiments

[0065] For a first example and referring to FIG. 12, an image of an array of emissive LEDs is shown, with dark areas between the LEDs. This represents a scene that has a high dynamic range. The image was taken with an exposure time of 20 ms. Once a first image such as this is acquired, at a first exposure time, the image may have some pixels with light metrics with a satisfactory SNR; other pixels with light metrics with a poor SNR, or still other pixels with over-exposed values of the light metric. Pixels with a poor SNR are considered to be under-exposed, although the data is still valid. Pixels that are over-exposed are saturated and have data that is invalid. If there is underexposed data, pixel values from a longer exposure are stitched in using HDR methodsto improve the SNR of the values at those pixels. If there are over-exposed (invalid) pixel values, pixel data from a shorter exposure is stitched in using HDR methods to replace the invalid values with good values and thereby improve the range of the valid data in the ultimate HDR image.

[0066] A satisfactory SNR is, for example, one that is between a lower threshold of 5% of the saturation value up to, but not including, the saturation value. The saturation value, or upper threshold, is set, for example, to be slightly lower than the maximum detectable value. For example, it may be set to 3% less than the maximum detectable value, or it may be set to say 3950 counts on a 12-bit analog-to-digital converter, which has a maximum of 4095 counts. In other embodiments, different thresholds are chosen, for example within 5% of the maximum. Pixels with an SNR below the lower threshold may be marked as under-exposed or their values could be converted to whatever the light metric is for the image.

[0067] To select the exposures, in an exemplary embodiment, a moderate exposure time such as 65 ms is used to obtain a first image. If there are saturated pixels present in the first image then successively shorter exposure images are captured and stitched until there are no over-exposed pixels in the image. Then, if there were pixels with values below 5% of saturation in the 65 ms exposure, images with successively longer exposures are captured and stitched until all pixels in the scene have been acquired with values of at least 5% of saturation, or until the longest allowed capture is executed. In this way an HDR image set may contain multiple base images each partially stitched into one final, complete image with up to a 1 ,000,000:1 dynamic range in the light metric.

[0068] In other embodiments, the sequence of capture may be different. One of the exposures is selected as the anchor image (i.e. reference image), to which pixels from the other exposures are stitched. For example, the image selected as the anchor image is the one with the most properly exposed pixels, i.e. pixels that are not overexposed and above the SNR threshold. However, for convenience or efficiency, another image may be selected as the anchor image, such as the first image, the last image, the one with the longest exposure or in unlikely cases the one with the shortest exposure. Normally, when stitching images, priority is given to the pixels in the image with the longest exposure, which is assumed to have no temporal measurement error.

[0069] Taking the image of FIG. 12 as the first image captured, it is analyzed to determine which of its pixels have non-saturated data, i.e. values that are not overexposed but which may be above or below the SNR threshold, which is the lower threshold. Now, referring to FIG. 13, a histogram of the non-saturated data in the first image is shown. The histogram shows the number of pixels per bin at each particular light metric, which in this example is luminance in cd / m2, having been scaled from the raw data that was captured. A range 47 of light metric values near the top of the luminance range is selected in order to identify comparison pixels to be compared with a second image, to be captured with a shorter exposure. These comparison pixels have values of the light metric between a lower boundary 48 and an upper boundary 49. For example, the values of the light metric for the identified pixels range between just below the upper threshold and about 5% down from the upper threshold. As another example, the values of the light metric for the identified pixels are between the upper threshold and about halfway between the upper threshold and the lower threshold. The comparison pixels in the range 47 have values between 3656 cd / m2and 4250 cd / m2after scaling to luminance. This aggregate of the selected pixels has an average light metric of 3786 cd / m2. In some circumstances, the median would also provide reasonable results, or another statistical measure or statistical calculation may be used.

[0070] An algorithm may be used for the selection of suitable pixels for the comparison. Such an algorithm counts the pixels with values of the light metric in a range of 95% to 99.999% of the saturation threshold, the saturation threshold being the upper threshold, which may be set to a few percent below the actual saturation level of the detector. For one example of detector, if there are 1000 pixels or more in the range, then there is a sufficiently large population of comparative pixels. If there are not enough pixels for comparison, the range is successively enlarged at its lower limit in 5% increments down to 40%, or until the range is found to encompass 1000 pixels. If this quantity is not found, then the criteria for the pixel population is relaxed to 200 and the process is started again at a light metric range of 95 to 99.999%, and repeated as necessary. If still unsuccessful, then the population requirement is reduced to 25 pixels and the process is repeated. If not enough pixels are found, the process ends without stitching. Other thresholds and pixel populations may be used in other algorithms.

[0071] A second image of the LED array is then taken, at a shorter exposure of 0.50 ms. Referring to FIG. 14, a histogram of the non-saturated pixels in the second image is shown, after scaling to account for the different exposure time and to adjust to the same units of cd / m2. The range 50 of light metric values is marked for the same pixel locations that were identified for comparison in FIG. 13 for the first image. Note that the axes on FIGS. 13 and 14 are logarithmic and that the bin sizes are different. In the second image, the comparison pixels have values between a lower value 51 of 3248 cd / m2and an upper value 52 of 3717 cd / m2, with an average value of 3483 cd / m2.Note that the average value of the light metric for the selected pixels is different in the second image compared to the first image, despite having corrected for the exposure time difference between the images. FIG. 15 depicts an image of the LED array now including the distribution 54 of the pixels 47, 50 being compared.

[0072] The ratio between the average values of the light metrics for the comparison set of pixels, found by dividing the average for the first image by the average for the second image, is 1.0870. This ratio may be referred to as a scalar. To join these two images, the first, longer exposure image is “stitch scaled” by dividing it by 1 .0870. Alternatively, the shorter exposure image is multiplied by the same number, 1.0870. By scaling one of the images by this value, any difference in the lighting level between the two images is compensated for. This is a further correction that goes beyond the normal correction for exposure time. The two images are then stitched together using known techniques. The stitching results in valid data from the long exposure replacing any corresponding data in the shorter exposure. The valid data from the most reliable exposure takes precedence over the valid or invalid data from the other exposure or exposures. In the example described, the longer exposure has less or no aliasing and is therefore the more reliable. In the general case, the longest exposure in a set of images has the highest precision and lowest susceptibility to aliasing, and for that reason it is good practice to scale the valid light metrics of all shorter exposure images to the light metrics of the longest exposure image.

[0073] To work, the method of stitching requires the analysis of light metrics from the pixels in images with two different exposures, after compensating for exposure time. It therefore requires that the exposure times between the first and second images are close enough such that there is a set of pixel locations common to both images in which the light metrics in both images has a satisfactory SNR. If the exposure timeswere too different, it may not be possible to find sufficient or even any pixels that can be properly compared.

[0074] As an alternate to the above method, the second image may be taken with a longer exposure than the first image, as the order in which the images are taken is not critical. To each newly stitched HDR image, additional shorter and / or longer exposures may be successively stitched using different sets of comparison pixels. As such, more than two images may be captured and used to stitch together the final HDR image.

[0075] In another example, consider a scenario in which an LED is energized and synchronously triggered with the first, short exposure of the measurement set. Then after a time, the next measurement with a longer exposure is acquired. However, at this point the LED has heated up more and its output has dropped. In this case, the first measurement light metrics are made invariant, and the longer exposure is scaled so that the values of its comparative pixels match those of the shorter exposure, which is considered to be the reference image.

[0076] The HDR imaging disclosed herein is also used in one dimensional imaging. Array spectrometers or line scan cameras are some examples of the devices that are used for capturing data. The principle of how the stitching is achieved is the same as for two-dimensional imaging. As an example, consider FIG. 16, which depicts spectral measurements of a source that is modulated. The wavelength is on the horizontal axis and absolute or relative light level is on the vertical axis, which has a logarithmic scale. The solid curve 59 and the dotted curve 60 represent two differently exposed spectral measurements after correcting for exposure time, linearity and other calibration factors that are necessary for traditional HDR measurement of unmodulated light sources. The solid curve 59 is the shorter, aliased exposure, and the dotted curve 60 is the longer, stable data. The longer exposure is very long compared to the light modulation period, and as a result is a reproducible measurement. The dotted curve has valid data overlapping with valid data from the solid curve, in the overlap region 61. However, there is a slight mismatch between the two curves in the overlap region.

[0077] FIG. 17 is a zoomed representation of the overlap region 61 described in FIG. 16. This data is linear mapped. The solid curve 59 is the shorter, aliased exposure, and the dotted curve 60 is the longer, stable data. The ratio between the two data values at location 441 nm on the x-axis is 1.042. The aliased, short exposure data is therefore increased by a factor of 1.042 in order for it to match the long exposure curveat location 441 nm. After scaling in this way, the scaled short exposure data outside of the overlap region 61 is stitched to the long exposure curve 60 to form an HDR spectrum. Two or more exposures may be stitched using this method. In this example only one pixel light metric is compared between the two exposures. It may also be appropriate to take an aggregate of pixels in the overlap region 61 and compare their light metrics to derive the scalar.

[0078] It should also be noted that for long exposures, the methods described above may be employed in some embodiments to stabilize measurements for other reasons, such as detector drift. For example, very long exposures are less reliable if the temperature changes during the exposure. In that case it is helpful to apply the stitching scalar to the longer exposure and choose a shorter exposure as the anchor exposure, i.e. without a stitching scalar. The disclosed method applies a stitching scalar to all but one image and eliminates the discontinuities that may occur as a result of a drifting detector. This may be very useful if the HDR images are used to quantify luminance or chromaticity for example.

[0079] In the case of thermal drift of LEDs the spectrum may shift left or right on the wavelength scale. In this case, it may be appropriate to have a unique scalar for each segment being stitched. For example, a green LED has overlap regions around 500 nm and around 620 nm in one example. The segments being stitched at the two regions may have different scalars applied to the segments being stitched. By extension, there can be more than two uniquely scaled segments in a stitching scenario.

[0080] The disclosed method may also be used for 3D imaging, i.e. hyperspectral imaging, which has spatial and spectral dimensions. If there are overlaps in the spatial images and spectral images, image scaling can be performed as described above for 1 D and 2D imaging.

[0081] FIG. 18 depicts an exemplary workflow to illustrate the added steps to remove aliasing when performing HDR stitching of 2D images, or 1 D images such as spectra. In step 70, two images are taken, one at a longer exposure time and one at a shorter exposure time. More generally, one image has a greater exposure and the other has a lesser exposure, there being a difference between the two exposures. Each one of the two images is made up of pixels and there is a light metric for each pixel. One of the images is scaled to compensate for the difference in exposure time in step 72. In step74, a group of comparison pixels are selected that have optimal SNR at the longer exposure. The comparison pixels are chosen to be those with light metrics in the upper range of the detector’s range of detection, or a statistically significant group of them. The comparison pixels lie between an upper boundary and a lower boundary. The upper boundary is close to the maximum of the detector’s maximum sensitivity. The lower boundary is, for example, anything from 40% to 95% of the upper threshold, or in other embodiments, 95% of the maximum of the detector’s sensitivity.

[0082] Having a good sampling with enough pixels is important. In some circumstances a single pixel may be adequate, while in other situations, a large aggregate of pixels is desired to further improve precision. As explained above, at the long exposure the non-saturated pixels with the highest SNR, or almost highest SNR, are selected to fulfill the quantity requirement. These are then the pixels used for the comparison.

[0083] In step 76, the values of the comparison pixels in both images are compared. For example, the average value of the light metric of the comparison pixels for each of the two images is compared, after correcting for exposure time. If, in step 78, after scaling for exposure time, the selected comparison pixels have the same light metrics, then the two images are stitched using a traditional HDR stitching method. However, if in step 78 the sets of comparison pixels have different light metrics, then an additional step needs to be performed. In this case, a scalar is calculated that represents the ratio between the light metrics of the selected comparison pixels in the longer exposure image compared to the light metrics of the selected comparison pixels in the shorter exposure image. The scalar is then applied in step 80 to the shorter exposure image, by multiplying the values for all the pixels in the shorter exposure image by the ratio, so that the comparison pixels for the two images have equal values, on average, or median, or some other statistic. Then, in step 82, the two images are stitched using a traditional HDR stitching method.

[0084] FIG. 19 depicts the workflow to illustrate the added steps to remove aliasing when the aliasing is severe, as in the examples shown in FIGS. 6, 7 and 8. Starting from step 78 in FIG. 18, with the determination that the selected pixels do not have the same values after scaling for exposure, the method moves to step 90. In this step, it is determined whether the selected pixels in the shorter exposure have a suitable SNR. If they do have a suitable SNR, then in step 98, a scalar is applied to the shorterexposure image so that the selected pixels have equal values, at least on average, to the corresponding pixels in the longer exposure image. The scalar is also applied to all the other pixels in the shorter exposure image, or all of the ones that have a valid light metric. The images are then stitched in step 100.

[0085] If, in step 90, the selected pixels in the shorter exposure do not have an SNR that is suitable for stitching, then the method moves to step 92, in which N measurements are captured at the shorter exposure or until an image is acquired with a suitable SNR at the selected pixels. The number of measurements or exposures N may be any number. If the aliasing is anticipated to be significant, a large number of samples may need to be acquired before a suitable measurement with adequate SNR is acquired. The number of measurement attempts needed to acquire one with suitable SNR is defined as N in step 92, and N could be any number such as 3, 10 or 100. A multitude of asynchronous measurement samples, for example N, may need to be acquired before a measurement time samples the transition area appropriately and the SNR of the short and long exposure measurements are appropriate for stitching to continue. The scenario may be that the shorter exposure time is close to the modulation period of the light source, and the measurements are not very repeatable. By taking a large enough number of asynchronous captures, some measurements will eventually give a good enough image for stitching.

[0086] Trials were performed on ranges of scalars for acceptable short exposures. For example, assuming a 4x difference in exposure time, over 80% of the pixels may be saturated in the long exposure and about 20% in the short exposure. The length of the shorter exposure is set so that the percentage of saturated pixels in the comparison range in the image is about 20%. A range of 5% to 50% of pixels that are saturated is suitable to qualify for a suitable stitch in some embodiments. If the value is higher, 80% for example, few if any pixels would be fused. If the value is lower, e.g. 1%, the SNR of the comparative pixels may be poor. Other strategies are to take a series of measurements to obtain the counts of the comparative pixels, and use the short exposure image in which the comparative pixels have an average light metric that is closest to 20% of saturation. Other percentages may be used in other embodiments. Also, other strategies may be employed to choose a set of suitable exposure times.

[0087] In step 94, it is then determined whether a suitable SNR has been obtained for the selected pixels in the shorter exposure. If not, then the method ends in step 96, with an invalid data message, for example labeling the remaining pixels as invalid. If, in step 94, a suitable SNR has been obtained, then in step 98 a scalar is applied to all valid pixels in the shorter exposure image so that the selected pixels have equal values in both the longer exposure and the shorter exposure. Following this, the images are stitched in step 100 using known HDR stitching techniques.

[0088] It is possible that the data values when measuring a modulated source at a particular exposure time vary greatly. In that case, a multitude of images at the same exposure may be stitched, each with a unique stitch scaling factor to create an HDR image. For example, if the light being imaged has a low enough duty cycle and a high depth of modulation, and the exposure time is shorter than the period of the modulation, some asynchronously acquired images could measure no light, be saturated, or measure anywhere between saturated and under-exposed. The imager may continuously sample asynchronously at this short exposure time until by chance an image is acquired with valid data and better SNR at some pixels than in the current state of the HDR image.

[0089] For example, the scenario in FIG. 8 may represent an exposure that provides a good SNR for pixels that are over exposed in a longer exposure. As described above, an algorithm for the shorter exposure to be fused with a longer exposure at a 4x exposure stepping could be based on the light metrics of the comparative pixels of the shorter exposure being between 5% and 50% of saturation.

[0090] In one row of a rolling shutter imager all of the pixels are exposed simultaneously. Each row may be considered to form a separate image, and when a pair of such separate images is obtained and stitched, the stitched row of pixels represents a portion of the final HDR image. If a rolling shutter is to be used with the present invention, it is recommended to compare pixels from one row or a limited number of rows of the sensor at one time. FIG. 20 depicts how a multitude of consecutive rows 102 is split into groups 104, 106, 108, each group constituting a selection of rows in a rolling shutter camera. Within each group of rows there is a limited difference in exposure time delay. The rows of pixels in a group may be considered to be sufficiently simultaneous to be stitched using the same scalar.

[0091] A segment is any subset of pixels on the sensor array that are captured simultaneously, or within some acceptable window of simultaneity. A segment may include one row, a number of rows, or in general, any set of rows and columns of pixels. Each segment is evaluated for a stitching scalar in isolation of the other segments in the image. Once all of the segment scalars are computed, and the segments stitched, the entire image is stitched. As described above, shorter or longer exposure images may be stitched to each measurement.

[0092] The method does not limit the application of averaging a set of images before or after stitching. If any pixels in an averaged set are over-exposed, their inclusion in any average result are as “invalid” or “undefined”. In one embodiment a fixed number of segments is chosen for the HDR stitching no matter the time base of the image being stitched. In another embodiment, the HDR stitching method uses larger segments, comprising more rows when the exposure time is longer.

[0093] In another embodiment, the segments are not evaluated for stitching scalars in isolation of each other. The scalars are derived for each segment and smoothed before they are used for HDR stitching purposes. In this way, each segment is compared to its nearest neighbours and stabilized to reduce noise. One example method to achieve this is to plot the scalars and perform a smoothing function on the plot. The new, smoothed scalars for each segment are then used for the HDR stitching.

[0094] In another embodiment, the scalars are plotted versus time, where the increment of the time from scalar to scalar is the average delay time for the triggering of each segment. In this scenario, one segment of pixels is anchored to the long exposure measurement and considered free from temporal sampling errors and the others are modulated and exhibit sampling errors. The plot of the scalar data provides a waveform with an identical period and duty cycle as the light being imaged. The amplitude of the modulation is affected by how many periods of modulation are in one exposure and by how close the integration time is to being an integer multiple of the modulation period. If there are many periods in the exposure, the depth of modulation is smaller. If the integration time is very close to the modulation period, the depth of modulation is smaller. Adjustment of the exposure times in the images can affect the dynamic range and SNR in the derived scalar waveform. The waveform may be analyzed using a Fast Fourier Transform (FFT) computational tool to transform thetime domain data into the frequency domain. This transformation allows for the analysis of the signal’s frequency components, which is essential for tasks such as computing the fundamental frequency and flicker analysis of the light being measured

[0095] The measured fundamental frequency may then be used to adjust the default exposure time base of the measurement system. This minimizes temporal measurement errors for exposures equal to, or longer than the light source modulation period.

[0096] FIG. 21 is an HDR image of the same light source as in FIG. 11 , with the same exposure times and with the same camera. However, in FIG. 21 , the antialiasing method described above is invoked. In this example, each line in the rolling shutter camera has been adjusted by its own scalar.

[0097] FIG. 22 is a plot 110 of the scalars against time when stitching an aliased short exposure to a stabilized longer exposure image using HDR stitching of images from a rolling shutter camera. It is readily apparent that the waveform is a bit noisy. The noisiness is due, for example, to the fact that there are not many pixels being compared to create the scalar in any particular row. The curve may be smoothed using a boxcar average, median filtering or other similar filters to create a new set of scalars for the stitch. Furthermore, this waveform may be processed using standard FFT algorithms such as Cooley-Tukey or Radix-2 to determine the fundamental period of the modulation. Then, with that knowledge, the time base of a subsequent set of measurements may be set to be an integer multiple of the modulation period. Having such a selection of integration times may greatly diminish the aliasing effect with discrete and array detectors.

[0098] FIG. 23 shows a method for altering the time base for a subsequent set of measurements. In step 120, multiple images are taken with a rolling shutter camera. Each of these images is a portion of a complete image, such as a row or segment of the complete image. In step 122, the scalars that are calculated for each of the image portions are plotted against time or otherwise represented as a function of time. In step 124, the plot of the scalars is smoothed. In step 126, an FFT is used to determine the period of the smoothed plot. In step 128, the exposures of a subsequent set of portional image captures using the rolling shutter camera are set to be integral multiples of the calculated period of the smoothed plot.

[0099] Temporal aliasing may be suspected if HDR 1 D and 2D images exhibit discontinuities as exhibited in Fig 9, 11 , 16 and 17. Temporal aliasing may also be suspected if, when comparing pixel sets across two or more images, the comparative pixels from two successive exposures require a scalar to have no variation or a minimal variation. Temporal aliasing may also be suspected if two or more successive exposures at the same integration time exhibit unexpectedly large variations at comparable pixels.

[0100] If temporal aliasing is suspected, strategies to mitigate potential errors include (a) informing the user of the HDR fusion issue, (b) applying the anti-aliasing HDR corrective scalars outlined above to ensure proper alignment of comparative pixels, (c) using the anti-aliasing HDR corrective scalars, as described above, and capturing one or more longer exposure images for HDR fusion to address temporal aliasing, (d) reacquiring the HDR image with updated parameters better suited for stitching images of modulated scenes, (e) acquiring and averaging multiple images, (f) configuring exposure times to be integer multiples of the modulation period and reacquiring the image, or (g) adjusting the iris or inserting neutral density filters into the optical path to enable longer image acquisition times. Each of these options may be carried out automatically by the system that captures and analyzes the images.

[0101] Referring to FIG. 24, a system is shown for forming HDR images which mitigates temporal error. It include a camera 130, which may have one detector or an array of detectors in one or two dimensions. The camera includes or is connected to a processor 132 which in turn is connected to a non-transient computer-readable memory 136. The memory stores computer-readable instructions, which when executed by the processor, result in the implementation of one or more of the steps leading to the formation of an HDR image in which temporal errors are mitigated. The instructions may be in the form of a program 138, which may access or include an algorithm 140, the function of which is described above. The memory also stores computer-readable data 142, in the form of captured images such as a shorter or lesser-exposed image 144 and a longer or greater-exposed image 146. The data also includes the final HDR image 148. Optionally connected to the processor is a display device 150.

[0102] Several image sensors have been proposed or are currently available on the market that utilize groups of adjacent pixels with varying sensitivities or exposuretimes. One such example is the Sony IMX900™ sensor, which features a quad HDR arrangement. In this configuration, the sensor is logically divided into 2x2 pixel (quad) regions, where each pixel within a quad can have a unique exposure time. These types of sensors may generate up to four images simultaneously, each captured at a different exposure time. In this example, the image is de-mosaiced to create four images, typically, but not necessarily, at % the native resolution, where each image is imaged at a unique exposure. These images can then be combined using the HDR fusion methods outlined in the presently disclosed invention. Imaging systems described in patent application US2024 / 0264454, for example, may be used for the HDR sampling methods described herein.

[0103] Another approach involves image sensors incorporating attenuating filters, which modify the sensitivity of pixels within a quad. In these sensors, all pixels can be exposed for the same duration but still undergo HDR fusion based on sensitivity variations. For instance, if one pixel has an ND1 .0 filter while an adjacent pixel has no attenuation (ND0.0), the first pixel’s value must be scaled by a factor of 10 before HDR pixel fusion is performed. The pixel sub-region configurations may follow a 2x2, 1 x3, or similar arrangement.

[0104] Additionally, a third type of image sensor employs dual ADC (analog to digital converter) technology, in which two images, one captured with high gain and the other with low gain, are acquired in parallel and combined within the sensor. This method, utilized in Sony Pregius S® sensors, including the IMX530™, enables enhanced dynamic range and improved imaging performance. The images can then be fused using the HDR techniques described herein.C. Variations

[0105] HDR requires two or more exposures at different sensitivities. It has been described above using exposure time as the variable to change the imager sensitivity. It will be apparent to those skilled in the art that there are other ways to change the imager sensitivity. For example the imager sensitivity may be changed by the application of density filters, changing the iris of the lens, adjusting the electrical properties of the array sensor, and so on. These or similar methods may be used to extend the imager’s dynamic range, similar to the changing of exposure time in anHDR image set. Where this document describes using longer or shorter exposure times, one could equally well exchange one density filter for another, instead of changing the exposure time, or in augmentation to adjusting the exposure time. To cover all methods of changing the imager sensitivity, we may use the more general terms “greater” and “lesser” in relation to the different exposures of two images.

[0106] Furthermore, although the HDR imaging descriptions provided herein often specify that the longest exposure is captured first, the disclosed method can accommodate any sequence of exposures. As such, the valid data from the longer exposure does not necessarily overwrite the data from the shorter exposure, and it could be the other way around. However, it remains important to obtain an exposure duration long enough to prevent temporal aliasing.

[0107] Those skilled in the art will recognize that, although the invention describes stitching and converting quantized pixel data to tristimulus values, these conversions may also involve other types of scaling or no scaling at all. This includes radiometric values, counts, or other values determined by the user’s calibration.

[0108] Those skilled in the art will also understand that in an HDR image fusion process, scaling one image to achieve a smooth, unaliased result is independent of which image is scaled. What matters is that, ultimately, the comparable pixels from both images exhibit minimal variation i.e. within an acceptable tolerance.

[0109] Where rows of sensor pixels are described in the methods for rolling shutter imagers, it will be understood that if the sensor is rotated, the columns can replace rows in the described methods. Furthermore, if another sensor technology has a different ordering or timing of measurement synchronization, then any reference to rows or columns is equivalent to any aggregate of pixels in a 1 D, 2D or 3D detector wherein the aggregate pixel integration time is simultaneous or considered simultaneous. Where pixels are described to have “simultaneous” integration times in the method(s), users should understand that it is meant to mean that the aggregate of pixels have close enough integration times as to be considered simultaneous in the intended application.

[0110] Although the methods described above are useful to take more precise measurements of unstable or modulated light sources, the methods also have utility if the exposures of the imaging system have poor precision with their sensitivity at any particular exposure time. For example a mechanical shutter may have imprecision atshort exposure times. CMOS or CCD or other solid state sensors may have exposure timing limitations due to limitations or imprecisions with the timing mechanisms, such as system clocks and dividers or other timing circuits, to control exposure intervals. Additionally, the process of reading out and digitizing the signal from the sensor can introduce delays or inconsistencies, particularly when operating at high speeds. Additionally, a liquid crystal polarizer shutter may have timing errors related to the temperature of the liquid crystal or the circuits driving the shutter.

[0111] In another example, an imager may have one or more exposure times suitable for generating a light metric, while other exposure times may be poorly calibrated, estimated, extrapolated, or non-existent. Using the methods described here, comparative scalars from two exposed images can be used to adjust and improve the light metric from one of the images before fusing them into an HDR image.

[0112] For instance, if a light metric can be accurately computed for an exposure time of 100 ms, established mathematical methods can be used to estimate the light metric for shorter exposure times, such as 50 ms, 25 ms, 12.5 ms, and so on. As a first approximation, one could assume that halving the exposure time would result in half the response in the detector for the same stimulus. This estimation allows for the calculation of light metrics for shorter exposures. When performing the shorter exposures, the temporal anti-aliasing methods described above can be employed to apply scalars as needed to adjust the shorter exposures, anchoring their values to the trusted values obtained at the 100 ms exposure.

[0113] In another embodiment, the method described in the two preceding paragraphs is used to generate scaling factors to relate the detector’s response at one exposure to another and still to a third exposure and so on. In this way one can generate scalars for a multitude of exposure times that are anchored to a response function at a reference exposure time. These scalars may be incorporated into the response function for the compared exposures and subsequently saved as correction factors to be applied in later images, whether HDR or single exposure.

[0114] In other embodiments, such as where one is measuring fluorescence, transmission or reflection, the intensity of the illumination and / or the exposure time is varied between successive images. The methods described above can still be used to anchor all of the stitched exposures to a reference exposure’s light metric.

[0115] Embodiments, depending on their configuration, may exhibit all or fewer than all of the advantages described herein. Other advantages not mentioned may be present in one or more of the embodiments. Features from any of the embodiments may be combined with features from any of the other embodiments to form another embodiment within the scope of the invention.

[0116] In general, unless otherwise indicated, singular elements may be in the plural and vice versa with no loss of generality. All parameters, quantities, percentages, thresholds, boundaries and configurations described herein are examples only and may be changed depending on the specific embodiment implemented. Mathematical steps may be replaced with equivalent mathematical steps.

[0117] The detailed description has been presented partly in terms of methods or processes, symbolic representations of operations, functionalities and features of the invention. These method descriptions and representations are the means used by those skilled in the art to most effectively convey the substance of their work to others skilled in the art. A software implemented method or process is here, and generally, understood to be a self-consistent sequence of steps leading to a desired result. These steps require physical manipulations of physical quantities. Often, but not necessarily, these quantities take the form of electrical or magnetic signals or values capable of being stored, transferred, combined, compared, and otherwise manipulated. It will be further appreciated that the line between hardware and software is not always sharp, it being understood by those skilled in the art that the software implemented processes described herein may be embodied in hardware, firmware, software, or any combination thereof. Such processes may be controlled by coded instructions such as microcode and / or by stored programming instructions in one or more tangible or nontransient media readable by a computer or processor. The code modules may be stored in any computer storage system or device, such as hard disk drives, optical drives, solid state memories, etc. The methods may alternatively be embodied partly or wholly in specialized computer hardware, such as ASIC or FPGA circuitry.

[0118] It will be clear to one having skill in the art that further variations to the specific details disclosed herein can be made, resulting in other embodiments that are within the scope of the invention disclosed. Two or more steps in the flowcharts may be performed in a different order, other steps may be added, or one or more may be removed without altering the main outcome of the process or function of the invention.

[0119] Throughout the description, specific details have been set forth in order to provide a more thorough understanding of embodiments of the invention. However, the invention may be practiced without these specific details. In other instances, well known elements have not been shown or described in detail and repetitions of steps and features have been omitted to avoid unnecessarily obscuring the invention.Accordingly, the specification and drawings are to be regarded in an illustrative, rather than a restrictive, sense. It will be clear to one having skill in the art that variations to the details disclosed herein can be made, resulting in other embodiments that are within the scope of the invention disclosed. Accordingly, the scope of the invention is to be construed in accordance with the substance defined by the claims.

Claims

CLAIMS1. A method for forming a high dynamic range (HDR) image, comprising the steps of:(a) obtaining two corresponding images of a scene wherein: each image has pixels and a light metric for each pixel; the images are obtained with a difference in exposure, one exposure being greater and the other exposure being lesser; and one of the images is compensated for the difference in exposure;(b) selecting, from the pixels, comparison pixels that correspond in location to each other, wherein the light metric of each comparison pixel is, in the image with the greater exposure, between a lower boundary and an upper boundary;(c) determining a first statistical measure, of the light metrics of the comparison pixels in one of the images;(d) determining a second statistical measure, of the light metrics of the comparison pixels in the other of the images;(e) calculating a ratio of the first statistical measure to the second statistical measure;(f) multiplying the light metric of at least some of the pixels in the other of the images by the ratio; and(g) stitching the images to form the HDR image.

2. The method of claim 1 , wherein: the upper boundary is within 5% of a saturation level for the image with the greater exposure; and the lower boundary is 40% or more of the saturation level.

3. The method of claim 1 , wherein the light metric of the comparison pixels is within a range of 95% to 99.999% of a saturation level for the image with the greater exposure.

4. The method of claim 3, wherein the saturation level is within 3% of a maximum possible value of the light metric.

5. The method of claim 1 , wherein: the first statistical measure and the second statistical measure are averages or medians;the light metric is luminance or radiance in arbitrary or absolute units; the difference in exposure is based on time, iris size or attenuation.

6. The method of claim 1 , wherein stitching the images comprises including in the HDR image the pixels from the image having the greater exposure in preference to the pixels from the image having the lesser exposure.

7. The method of claim 1 , wherein the images are obtained using a one dimensional array detector or a two dimensional camera.

8. The method of claim 1 , wherein the images are obtained using a two dimensional, rolling shutter camera and the images correspond to one row of pixels, the method further comprising obtaining further pairs of images each further pair with its own difference in exposure, wherein each further pair corresponds to a different, other row of pixels and has its own ratio.

9. The method of claim 1 , wherein: the images are obtained using a pixelated image sensor; the images correspond to a group of one or more of the pixels that have a simultaneous integration time; further pairs of images are obtained; each further pair corresponds to a further group of one or more others of the pixels that have an integration time non-simultaneous with said simultaneous integration time; and each further group has its own ratio of which at least one is different to another.

10. The method of claim 1 , wherein to obtain the image with the lesser exposure, multiple prior exposures are acquired until one of the prior exposures has light metrics that are above a threshold signal to noise ratio; and said one of the exposures is selected as the image with the lesser exposure.

11. The method of claim 1 , wherein the lesser exposure has an imprecise exposure timing caused by a mechanical shutter, liquid crystal shutter or an electronic timing circuit.

12. The method of claim 1 comprising: incorporating the ratio, for the lesser exposure or the greater exposure, into a response function of a device used in step (a) for the obtaining of the two corresponding images; and using the response function to correct a later image obtained by the device.

13. The method of claim 1 , wherein: the images are obtained using a two dimensional, rolling shutter camera; multiple rows of pixels are grouped into multiple segments; and each segment has its own ratio of which at least one is different to another.

14. The method of claim 1 , wherein the images are of a portion of the scene, the method comprising: obtaining further pairs of images each further pair of a different portion of the scene; calculating a further ratio for each of the further pairs of images; comparing the ratio and the further ratios depending on a position in the scene to which they correspond; and filtering the ratio and the further ratios to reduce noise therein.

15. The method of claim 1 , wherein the images are of a portion of a scene, the method comprising: obtaining further pairs of images each further pair of a different portion of the scene; calculating a further ratio for each of the further pairs of images; comparing the ratio and the further ratios depending on time; and determining a fundamental frequency of illumination of the scene.

16. The method of claim 15, further comprising determining flicker in the scene.

17. The method of claim 15, further comprising: determining a period corresponding to the fundamental frequency; and reperforming the steps in claim 1 to obtain two subsequent corresponding images of the scene that are obtained with exposure times each having a duration of a different integral multiple of the period and to form another HDR image therefrom.

18. The method of claim 1 , wherein: the ratio is beyond a predetermined threshold; and the multiplying and stitching steps are performed automatically without user intervention.

19. The method of claim 1 , wherein the ratio is beyond a predetermined threshold, the method further comprising initiating an action to address aliasing.

20. The method of claim 19, wherein the action comprises generation of a notification that advises how to mitigate the aliasing including:(h) adjusting a time base of the exposures;(i) averaging measurements; and / or(j) applying attenuation to allow for longer exposure times.

21. The method of claim 20 comprising: before steps (f) and (g), automatically implementing (h), (i) and / or (j); repeating steps (a) to (e) to update at least some of the light metrics, at least one of said statistical measures and the ratio; and continuing with steps (f) and (g).

22. The method of claim 1 , further comprising repeating steps (a) to (g) using the HDR image as one of the images obtained in repeated step (a).

23. The method of claim 1 , wherein at least a predetermined quantity of comparison pixels is required and the comparison pixels are all those that have a light metric, in the image with the greater exposure, between the upper boundary and the lower boundary, the method further comprising: when the quantity of comparison pixels is below the predetermined quantity, lowering the lower boundary in steps to increase the quantity; and if the lower boundary reaches a predetermined limit without the quantity reaching the predetermined quantity, resetting the lower boundary and reducing the predetermined quantity.

24. A device for forming a high dynamic range (HDR) image, comprising: a detector; a processor; and a computer readable memory storing computer-readable instructions that are executed by the processor to cause the processor to: obtain, via the detector, two corresponding images of a scene wherein: each image has pixels and a light metric for each pixel; the images are obtained with a difference in exposure, one exposure being greater and the other exposure being lesser; and one of the images is compensated for the difference in exposure; select, from the pixels, comparison pixels that correspond in location to each other, wherein the light metric of each comparison pixel is, in the image with the greater exposure, between a lower boundary and an upper boundary;determine a first statistical measure, of the light metrics of the comparison pixels in one of the images; determine a second statistical measure, of the light metrics of the comparison pixels in the other of the images; calculate a ratio of the first statistical measure to the second statistical measure; multiply the light metric of at least some of the pixels in the other of the images by the ratio; and stitch the images to form the HDR image.

25. A computer readable memory storing computer-readable instructions, which, when executed by a processor, cause the processor to: obtain, via a detector, two corresponding images of a scene wherein: each image has pixels and a light metric for each pixel; the images are obtained with a difference in exposure, one exposure being greater and the other exposure being lesser; and one of the images is compensated for the difference in exposure; select, from the pixels, comparison pixels that correspond in location to each other, wherein the light metric of each comparison pixel is, in the image with the greater exposure, between a lower boundary and an upper boundary; determine a first statistical measure, of the light metrics of the comparison pixels in one of the images; determine a second statistical measure, of the light metrics of the comparison pixels in the other of the images; calculate a ratio of the first statistical measure to the second statistical measure; multiply the light metric of at least some of the pixels in the other of the images by the ratio; and stitch the images to form a high dynamic range (HDR) image.

Images