System for performing ambient light image correction

Abstract

A system (100) is provided, comprising a light source (110), an imaging unit (120), and a control unit (130), the imaging unit being configured to capture a plurality of images of an object, wherein each image of the plurality of images is captured with an exposure time shorter than a wave period of pulsed illumination, wherein a pulse frequency of the illumination is not an integer multiple of a frame rate at which the images are captured, and wherein a total time during which the plurality of images is captured is at least half of the wave period of the pulsed illumination, the control unit being configured to obtain a predetermined number n of candidate images, generate sorted lists of pixels by sorting respective pixels, apply a set of weights to respective sorted lists of pixels, and generate an estimated ambient light corrected image based on the plurality of weighted and sorted lists of pixels.

Description

Technical Field

[0001] This disclosure relates to an apparatus and method for optical imaging, and more particularly to a non-contact skin imaging system and method for performing ambient light image correction. Background Technology

[0002] In the realm of personal care and health applications, particularly in skin sensing, there is a growing demand for inconspicuous measurement and monitoring devices. Currently available skin measurement systems offer skin quantification and skin feature monitoring capabilities, providing consumer information that may be too small to detect, too faint to notice, or too slow to follow. For these types of systems to be perceived as usable by the average consumer, the embedded sensing capabilities must be sensitive and specific. Furthermore, robustness of the measurements is essential to building consumer trust. Summary of the Invention

[0003] A key issue with such imaging measurement systems is the inconsistency in measurements when used in uncontrolled environments, such as at home, due to undefined and potentially variable ambient lighting. Currently available ambient light removal techniques exist, such as active lighting methods that incorporate controlled illumination to detect and infer scene attributes. Furthermore, some currently available techniques include controlled capture settings, such as illumination stages, for capturing images of people or scenes in all lighting directions (multiplexed illumination), and for re-presenting realistic images under arbitrary lighting. Other techniques rely on projector-camera systems to project structured patterns for subsequent image analysis. While active techniques enable high-quality lighting analysis and editing, these systems are typically complex and expensive. On the other hand, flash photography provides ambient light removal by capturing two images of a scene, one with flash illumination and one without. These techniques have been used for image denoising, deblurring, artifact removal, non-photorealistic rendering, foreground segmentation, and gloss reduction.

[0004] It can be observed that, in cases such as Figure 1 In the case of the rolling shutter shown, pulsed or flashing illumination at a frequency higher than the camera's frame rate produces an image with dark-to-light horizontal bands. Figure 1 In the image, dark bands are labeled D, and bright bands are labeled B. Furthermore, Figure 2 This illustrates the generation of bright and dark bands using pulsed light at a rolling shutter camera. For a rolling shutter camera, the exposure time t... exposure Typically longer than a single horizontal line t line The series (t) exposure >tl ine =1 / (N*f framerate(where N is the number of horizontal lines). Therefore, under normal operating conditions, Figure 1 The illumination patterns shown appear. In subsequent images, these patterns undergo phase changes.

[0005] To generate ambient light corrected images, one proposed technique involves determining the minimum pixel value (corresponding to the low phase of the pattern, i.e., where only ambient light is present) and the maximum pixel value (corresponding to the high phase of the pattern, i.e., where both ambient and artificial light are present) for each pixel in a sequence of images (e.g., 10 frames), and calculating the difference between the minimum and maximum values. However, since signals derived from imaging sensors are typically noisy, especially under low-light conditions, this technique can lead to unwanted noise pickup—under ideal conditions, the technique for calculating the difference between the maximum and minimum values ​​actually uses the noisiest values, as these are outliers in the observed sequence by definition. Therefore, it would be advantageous to provide an improved system and method for estimating ambient light corrected images based on the same input image data.

[0006] To better address one or more of the aforementioned problems, in a first aspect, a system for performing ambient light image correction is provided. The system includes: a light source configured to provide pulsed illumination to an object; an imaging unit configured to capture multiple images of the object while it is illuminated by the pulsed illumination from the light source, wherein each of the multiple images is captured with an exposure time shorter than the wavelength of the pulsed illumination, wherein the pulse frequency of the illumination provided by the light source is not an integer multiple of the frame rate, the multiple images are captured by the imaging unit at an integer multiple of the frame rate, and wherein the total time during which the multiple images are captured is at least half the wavelength of the pulsed illumination; and a control unit configured to: obtain a predetermined number n of captured images, and candidate images, wherein the candidate images are n consecutive captured images. The process generates a sorted list of pixels for each pixel location in the multiple candidate images by sorting the corresponding pixels corresponding to their respective pixel locations in the multiple candidate images, where the sorting is based on the pixel value of the corresponding pixel; for each pixel location in the multiple candidate images, a set of weights is applied to the corresponding sorted list of pixels, where the set of weights is associated with a least-squares approximation of the probability density function of the pixel value over time; and an estimated ambient light corrected image is generated by summing the multiple weighted and sorted lists of pixels.

[0007] In some embodiments, n can be predetermined based at least on the pulse frequency of the pulsed illumination and the frame rate at which multiple images are captured.

[0008] In some embodiments, the corresponding pixels, each corresponding to a corresponding pixel position in a plurality of candidate images, may be sorted in ascending order.

[0009] In some embodiments, the control unit may be configured to determine a set of weights to be applied to a sorted list of pixels based on the signal-to-noise ratio of a plurality of candidate images.

[0010] In some embodiments, the control unit may be configured to determine a set of weights to be applied to a sorted list of pixels based on at least one of the following: the exposure time of multiple images being captured, the sensitivity level of the imaging unit, and the detection light level of the multiple captured images.

[0011] In some embodiments, determining a set of weights to be applied to the sorted list of pixels may include selecting a set of weights to be applied from a plurality of predetermined weight reassemblies.

[0012] In some embodiments, a set of weights to be applied to the sorted list of pixels can be selected from a predetermined coefficient table.

[0013] In some embodiments, the imaging unit may be configured to employ a rolling shutter such that each of the plurality of captured images includes a plurality of bright bands and dark bands, wherein the bright bands correspond to the high state of pulsed illumination and the dark bands correspond to the low state of pulsed illumination.

[0014] In a second aspect, a method for performing ambient light image correction is provided. The method includes: providing pulsed illumination to an object; providing pulsed illumination to the object; capturing a plurality of images of the object while the object is illuminated by the pulsed illumination, wherein each of the plurality of images is captured with an exposure time shorter than the wavelength of the pulsed illumination, wherein the pulse frequency of the provided illumination is not an integer multiple of the frame rate, the plurality of images are captured at an integer multiple of the frame rate, and wherein the total time during which the plurality of images are captured is at least half of the wavelength of the pulsed illumination; obtaining a predetermined number n of captured images, candidate images, wherein the candidate images are n consecutive captured images; and performing image processing on the corresponding pixel positions of each of the candidate images. The corresponding pixels are sorted to generate a sorted list of pixels for each pixel location in multiple candidate images, where the sorting is based on the pixel value of the corresponding pixel; for each pixel location in multiple candidate images, a set of weights (210) is applied to the corresponding sorted list of pixels, where the set of weights is associated with a least-squares approximation of the probability density function of the pixel value over time; and based on multiple weighted and sorted lists of pixels, an estimated ambient light corrected image (212) is generated by summing the multiple weighted and sorted lists of pixels.

[0015] In some embodiments, n can be predetermined based at least on the pulse frequency of the pulsed illumination and the frame rate at which multiple images are captured.

[0016] In some embodiments, the corresponding pixels corresponding to their respective pixel positions in the multiple candidate images can be sorted in ascending order.

[0017] In some embodiments, the method may further include determining a set of weights to be applied to a sorted list of pixels based on the signal-to-noise ratio of a plurality of candidate images.

[0018] In some embodiments, the method may further include determining a set of weights to be applied to the sorted list of pixels based on at least one of the following: the exposure time of the multiple images being captured, the sensitivity level of the imaging unit, and the detection light level of the multiple captured images.

[0019] In some embodiments, determining a set of weights to be applied to a sorted list of pixels may include selecting a set of weights to be applied from a plurality of predetermined weighting sets.

[0020] In a third aspect, a computer program product including a computer-readable medium is provided, the computer-readable medium having computer-readable code implemented therein, the computer-readable code being configured to cause the computer or processor to perform the methods as described herein when executed by a suitable computer or processor.

[0021] Based on the above aspects and embodiments, the limitations of the prior art are overcome. In particular, the above aspects and embodiments enable accurate estimation of ambient light corrected images at video rates, especially for low-cost imaging units that often exhibit significant jitter in their timing. The above embodiments provide a stateless technique for ambient light image correction that does not require determining the phase of the pattern caused by pulsed illumination. Therefore, an improved system and method for ambient light image correction are provided. These and other aspects of this disclosure will become apparent from the embodiments described below and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0022] To better understand these embodiments and to more clearly illustrate how they are implemented, reference will now be made to the accompanying drawings by way of example only, in which:

[0023] Figure 1 An example of an image of an object when it is illuminated by pulsed lighting is shown;

[0024] Figure 2 This demonstrates the generation of bright and dark bands using pulsed light at the rolling shutter speed;

[0025] Figure 3This is a block diagram of a system for ambient light image correction according to one embodiment;

[0026] Figure 4 The illustration shows a method for performing ambient light image correction according to one embodiment;

[0027] Figure 5A It is a graph showing the pixel values ​​as a function of time under the theoretically noise-free condition and a graph showing the corresponding probability density function;

[0028] Figure 5B It is a graph showing the pixel values ​​as a function of time and a graph showing the corresponding probability density function in actual situations.

[0029] Figure 6 It is a graph illustrating the standard deviation of a pixel sequence across multiple frames for multiple pre-calculated weights; and

[0030] Figure 7 It is a graph illustrating the results of a series of simulated measurements using a maximum-minimum estimation and estimation technique according to an embodiment of the present disclosure. Detailed Implementation

[0031] As described above, an improved system and method for solving existing problems are provided.

[0032] Figure 3 A block diagram of an apparatus 100 according to an embodiment is shown, which can be used to perform ambient light image correction. The system 100 includes a light source 110, an imaging unit 120, and a control unit 130.

[0033] Light source 110 is configured to provide pulsed illumination to an object. In the context of this disclosure, "pulsed illumination" refers to illumination provided at a sufficiently high frequency (i.e., referred to herein as pulse frequency) in both high and low states. In some embodiments, light source 110 may be configured to provide illumination having a pulse frequency of at least 70 Hz.

[0034] Imaging unit 120 is configured to capture multiple images of an object when the object is illuminated by pulsed illumination from light source 110. It should be understood that, in the context of this disclosure, since the illumination is pulsed (having a high state and a low state), at least one or more of the multiple captured images can be captured during the low state, i.e., when the object is not (fully) illuminated.

[0035] Each of the multiple images is captured with an exposure time shorter than the wave period of the pulsed illumination. The pulse frequency of the illumination provided by light source 110 is not an integer multiple of the frame rate at which the multiple images are captured by imaging unit 120. The total time during which the multiple images are captured is at least half the wave period of the pulsed illumination. It should be understood that although the theoretical minimum for the total time for capturing the multiple images is half a wave period, in practice this can actually be, for example, ten wave periods, because it is generally more difficult for imaging unit 120 to achieve a frame rate higher than that of light source 110. In some embodiments, the upper limit of the total time during which the multiple images are captured can be determined by any anticipated motion of imaging unit 120 and / or the object.

[0036] In some embodiments, the capture of multiple images of an object may be triggered by a user, for example via a user interface at system 100.

[0037] In some practical implementations, to obtain a sufficient number of data points representing the distribution of pixel values ​​in order to estimate the difference between light-on and light-off conditions, the multiple images captured by the imaging unit 120 can be on the order of at least 8-10 frames. For example, if the frame rate of the imaging unit is 100 Hz and the number of captured images is 10, the total time during which the multiple images are captured is 0.1 s. Similarly, the pulse frequency can be at a very high rate, such as 117 Hz, preferably to avoid flickering of light perceived by the user. Using the configuration presented in this example, the total time during which the images are captured will cover a large number of wave periods. By ensuring that the pulse frequency of the illumination is not an integer multiple of the frame rate, the light pulses can be effectively sampled at different phases of the pulsating light, which ensures sufficient data with a similar distribution.

[0038] In some embodiments, the imaging unit 120 may be configured to employ a rolling shutter such that each of the multiple captured images includes multiple bright bands and dark bands. The bright bands correspond to the high state of pulsed illumination, and the dark bands correspond to the low state of pulsed illumination.

[0039] Alternatively, in some embodiments, the imaging unit 120 may be configured to employ a global shutter. In these embodiments, the global shutter may be configured to be fast enough to accommodate the requirements of the imaging unit 120 (e.g., regarding frame rate and exposure time) while maintaining a sufficient signal-to-noise ratio (SNR).

[0040] Control unit 130 is configured to obtain a predetermined number n (e.g., 10) of candidate images from a plurality of images captured by imaging unit 120. The candidate images are n consecutive images among the plurality of captured images. In some embodiments, n may be predetermined based at least on the pulse frequency of the pulsed illumination and the frame rate at which the plurality of images are captured. In some embodiments, n may be further predetermined based on at least one of the following: illumination conditions and detected movement of the imaging unit and / or the object.

[0041] More specifically, the control unit 130 can be configured to obtain candidate images by selecting a subset of the images captured by the imaging unit 120. In some embodiments, to reduce motion artifacts, the predetermined number n of candidate images obtained can be minimized. Furthermore, in some embodiments, the imaging unit 120 can be configured to capture only the predetermined number n of candidate images. In this case, the candidate images can be the same as the captured images.

[0042] As described above, in some embodiments, the imaging unit 120 may be configured to employ a global shutter. In these embodiments, the control unit 130 may be configured to acquire candidate images by selecting images corresponding to the same phase of the pulsed illumination. By selecting images corresponding to the same phase of the pulsed illumination, the sorting operation (as described in more detail below) can be improved.

[0043] The control unit 130 is also configured to generate a sorted list of pixels for each pixel location in the plurality of candidate images by sorting the corresponding pixels, each corresponding to a corresponding pixel location in the plurality of candidate images. The sorting is based on the pixel value of the corresponding pixel. For example, in the case where the imaging unit 120 is a monochrome camera, the pixel value is the pixel value captured by the imaging unit 120 as is. As another example, in the case where the imaging unit 120 is an RGB camera, the pixel value may be an intensity value obtained by processing the R, G, and B values ​​of the corresponding pixel in the candidate image (e.g., using the "Lab" color space and obtaining the L value). The sorting of the corresponding pixels, each corresponding to a corresponding location in the plurality of candidate images, may be based on intensity. Alternatively, in some embodiments, the R, G, and B values ​​may be processed independently. In some embodiments, the corresponding pixels, each corresponding to a corresponding pixel location in the plurality of candidate images, may be sorted in ascending order.

[0044] The control unit 130 is configured to apply a set of weights to a corresponding sorted list of pixels for each pixel location in a plurality of candidate images, and to generate an estimated ambient light corrected image by summing the weighted and sorted lists of pixels based on the plurality of weighted and sorted lists. The set of weights is associated with a least-squares approximation of the probability density function of the pixel values ​​over time, which will be referenced below. Figure 5A and Figure 5B To explain in more detail.

[0045] In some embodiments, the control unit 130 may be configured to determine a set of weights to be applied to a sorted list of pixels based on the estimated signal-to-noise ratio (SNR) of a plurality of candidate images. In some embodiments, the control unit 130 may be configured to determine the set of weights to be applied to the sorted list of pixels based on at least one of the following: exposure time of the plurality of images captured, sensitivity level of the imaging unit (ISO sensitivity), and detection light level of the plurality of captured images. Determining the set of weights to be applied to the sorted list of pixels may include selecting a set of weights to be applied from a plurality of predetermined weight sets. In some embodiments, the set of weights to be applied to the sorted list of pixels may be selected from a predetermined coefficient table.

[0046] In general, the control unit 130 can control the operation of the system 100 and can implement the methods described herein. The control unit 130 may include one or more processors, processing units, multi-core processors, or modules configured or programmed to control the system 100 in the manner described herein. In a particular implementation, the control unit 130 may include multiple software modules and / or hardware modules, each configured to perform or be used to perform individual steps or multiple steps of the methods described herein.

[0047] In some embodiments, system 100 may further include at least one user interface. Alternatively or additionally, at least one user interface may be external to system 100 (i.e., separate from or remote from system 100). For example, at least one user interface may be part of another device. The user interface may be used to provide information generated by the methods described herein to a user of system 100. Alternatively or additionally, the user interface may be configured to receive user input. For example, the user interface may allow a user of system 100 to manually input instructions, data, or information. In these embodiments, control unit 130 may be configured to obtain user input from one or more user interfaces.

[0048] The user interface can be any user interface capable of presenting (or outputting or displaying) information to the user of system 100. Alternatively or additionally, the user interface can be any user interface that enables the user of system 100 to provide user input, interact with system 100, and / or control system 100. For example, the user interface may include one or more switches, one or more buttons, keypads, keyboards, touchscreens or applications (e.g., on a tablet or smartphone), displays, graphical user interfaces (GUIs) or other visual presentation components, one or more speakers, one or more microphones or any other audio components, one or more lights, components for providing haptic feedback (e.g., vibration functionality), or any other user interface, or a combination of user interfaces.

[0049] In some embodiments, system 100 may include memory. Alternatively or additionally, one or more memories may be external to system 100 (i.e., separate from or remote from system 100). For example, one or more memories may be part of another device. Memory may be configured to store program code that can be executed by control unit 130 to run the methods described herein. Memory may be used to store information, data, signals, and measurements acquired or generated by control unit 130 of system 100. For example, memory may be used to store multiple captured images, multiple candidate images, and / or estimated ambient light corrected images. Control unit 130 may be configured to control memory to store multiple captured images, multiple candidate images, and / or estimated ambient light corrected images.

[0050] In some embodiments, system 100 may include a communication interface (or circuitry) for enabling system 100 to communicate with any interface, memory, and / or device, either internal or external to system 100. The communication interface may communicate wirelessly or via a wired connection with any interface, memory, and / or device. For example, the communication interface may communicate wirelessly or via a wired connection with one or more user interfaces. Similarly, the communication interface may communicate wirelessly or via a wired connection with one or more memories.

[0051] It should be understood that Figure 1 Only the components necessary to illustrate aspects of system 100 are shown, and in actual implementation, system 100 may include alternative or additional components of the shown components.

[0052] Figure 4 A method for performing ambient light image correction according to one embodiment is illustrated. The illustrated method can generally be performed by system 100, and specifically, in some embodiments by or under the control of control unit 130 of system 100. For illustrative purposes, reference will be made to… Figure 3The various components of system 100 are described Figure 4 At least some of the boxes in the box.

[0053] refer to Figure 4 At box 202, pulsed illumination is provided to the object. Specifically, the pulsed illumination can be provided by the light source 110 of system 100. The pulse frequency of the illumination provided at box 202 can be at least 70 Hz.

[0054] Back Figure 4 At box 204, when the object is illuminated by pulsed illumination (as described with reference to box 202), multiple images of the object are captured. It should be understood that, in the context of this disclosure, since the illumination is pulsed illumination (having a high state and a low state), at least one or more of the multiple captured images can be captured during the low state, i.e., when the object is not (fully) illuminated.

[0055] At box 204, each of the multiple images is captured with an exposure time shorter than the wave period of the pulsed illumination. The pulse frequency of the illumination provided at box 202 is not an integer multiple of the frame rate at which the multiple images are captured. At box 204, the total time during which the multiple images are captured is at least half the wave period of the pulsed illumination. It should be understood that although the theoretical minimum for the total time during which the multiple images are captured is half a wave period, in practice, this can actually be, for example, ten wave periods, because frame rates with imaging units higher than the light source are generally more difficult. In some embodiments, the upper limit of the total time during which the multiple images are captured can be determined by any expected movement of the imaging unit and / or the object.

[0056] In some practical implementations, to obtain a sufficient number of data points representing the pixel value distribution in order to estimate the difference between lamp-on and lamp-off conditions, the captured multiple images can be on the order of at least 8-10 frames. For example, if the frame rate of the imaging unit is 100 Hz and the number of captured images is 10, the total time during which multiple images are captured is 0.1 s. Similarly, to preferably avoid flickering of light perceived by the user, the pulse frequency can be at a very high rate, such as 117 Hz. Using the configuration presented in this example, the total time during which images are captured will cover a large number of wave periods. By ensuring that the pulse frequency of the illumination is not an integer multiple of the frame rate, the light pulses can be effectively sampled at different phases of the pulsed light, which ensures sufficient data with a similar distribution.

[0057] In some embodiments, an imaging unit employing a rolling shutter can be used at frame 204. In this case, each of the plurality of captured images at frame 204 may include a plurality of bright bands and dark bands. Bright bands correspond to the high state of pulsed illumination, and dark bands correspond to the low state of pulsed illumination.

[0058] Alternatively, in some embodiments, an imaging unit employing a global shutter can be used at block 204. In these embodiments, the global shutter can be configured to be fast enough to accommodate the requirements of imaging unit 120 (e.g., with respect to frame rate and exposure time) while maintaining a sufficient signal-to-noise ratio (SNR).

[0059] Back Figure 4 At box 206, a predetermined number n (e.g., 10) of candidate images are obtained from the plurality of images captured at box 204. Specifically, the candidate images may be obtained by the control unit 130 of system 100. The candidate images are n consecutive images among the plurality of captured images. In some embodiments, n may be predetermined based at least on the pulse frequency of the pulsed illumination and the frame rate at which the plurality of images are captured. In some embodiments, n may be further predetermined based on at least one of the following: illumination conditions and the detection of movement of the imaging unit and / or the object.

[0060] The predetermined number of candidate images, n, can be a subset selected from a plurality of images captured at frame 204. In some embodiments, the predetermined number of candidate images, n, can be minimized to reduce motion artifacts. Similarly, in some embodiments, the plurality of images captured at frame 204 can be the same number as the predetermined number. In this case, the candidate images can be the same as the captured images.

[0061] As described above, in some embodiments, an imaging unit employing a global shutter can be used at box 204. In these embodiments, obtaining candidate images at box 206 may include selecting images corresponding to the same phase of the pulsed illumination. By selecting images corresponding to the same phase of the pulsed illumination, the sorting operation can be improved (as described in more detail below with respect to box 208).

[0062] Return to Figure 4 At box 208, a sorted list of pixels is generated for each pixel position in the multiple candidate images, corresponding to each pixel position in each of the multiple candidate images obtained at box 206. The sorting at box 208 is based on the pixel value of the corresponding pixel. The generation of the sorted list of pixels at box 208 can be performed by the control unit 130 of system 100.

[0063] As per the above reference Figure 3In the example of capturing multiple images using a monochrome camera, the pixel values ​​are the unaltered pixel values. As another example, in the case of capturing multiple images using an RGB camera, the pixel values ​​can be intensity values ​​obtained by processing the R, G, and B values ​​of the corresponding pixels in the candidate images (e.g., using "Lab" processing and obtaining the L value). The ordering of each corresponding pixel at a corresponding position in the multiple candidate images can be based on intensity. Specifically, in some embodiments, the ordering of corresponding pixels can be based on the average intensity of a group of pixels to enhance the robustness of the technique. In some embodiments, the corresponding pixels, each corresponding to a corresponding pixel position in the multiple candidate images, can be ordered in ascending order.

[0064] Return to Figure 4 At box 210, for each pixel location in the plurality of candidate images, a set of weights is applied to the corresponding sorted list of pixels. This application of weights can be performed by the control unit 130 of system 100. The set of weights is associated with a least-squares approximation of the probability density function of the pixel value over time. In some embodiments, a set of weights to be applied to the sorted list of pixels can be selected from a predetermined coefficient table.

[0065] although Figure 4 Not shown, but in some embodiments, the method may further include determining a set of weights to be applied to a sorted list of pixels based on the estimated signal-to-noise ratio of a plurality of candidate images. Furthermore, the method may further include determining the set of weights to be applied to the sorted list of pixels based on at least one of the following: exposure times of the plurality of images captured, sensitivity levels of the imaging unit, and detection light levels of the plurality of captured images. The step of determining the set of weights to be applied can be performed at any time before block 210. In some embodiments of these embodiments, determining the set of weights to be applied to the sorted list of pixels may include selecting a set of weights to be applied from a plurality of predetermined weight reassemblies.

[0066] Return to Figure 4 At box 212, an estimated ambient light corrected image is generated by summing multiple weighted and sorted lists of pixels. The generation of the estimated ambient light corrected image can be performed by the control unit 130 of system 100.

[0067] Figure 5A The graphs show the pixel values ​​as a function of time under theoretically noise-free conditions and the corresponding probability density functions under theoretically noise-free conditions. Figure 5BThe graphs show the pixel values ​​as a function of time in real-world scenarios and the corresponding probability density functions. (See above for reference.) Figure 3 and Figure 4 The generation of the ambient light-corrected image described is based on the theory of maximum likelihood estimation (MLE). In MLE, given a set of observations, the parameters of the model are estimated. A prototype example of MLE is determining the best possible estimates of the bias and standard deviation for a Gaussian process given multiple observations. Figure 5A This represents an exemplary pixel value over time (multiple frames or candidate images) under theoretically noise-free conditions, such as the red pixel value at a specific image location y, x, and the corresponding probability density function. In the context of MLE, to estimate the ambient light corrected image, it is necessary to estimate high and low values ​​(i.e., a) based on multiple observations (i.e., samples). hi -a lo The distance between them. For example... Figure 5A The parameter d shown is a relative portion of the time during which the pattern is high (or low) within that time period. This parameter depends on the flicker frequency of the modulated light provided by the imaging unit, the frame rate or readout rate at which the image is captured, the number of horizontal video lines, and the exposure time. For a given configuration or setting, these variables are fixed, and therefore d remains the same.

[0068] Compare Figure 5A and Figure 5B ,exist Figure 5B The actual situation shown is particularly concerning noise pickup under low-light conditions. Under the assumption of Gaussian noise with standard deviation, the probability distribution function can be given as follows:

[0069]

[0070] in:

[0071]

[0072]

[0073] as well as:

[0074]

[0075] The probability distribution function is f(x|d, a). lo a hi (This provides the condition for given parameters d and a) lo and a hi The probability of finding the value x under the given condition, and is actually the theoretically noise-free pdf ( Figure 5A (to the right) and Gaussian kernel f g Convolution of (x).

[0076] In contrast to the prototype example for MLE estimation, it is impossible to derive an analytical equation in practice because MLE estimation produces a product of summation terms. However, an improved estimate can be obtained using a least-squares approximation. Assume that a sequence of observations for a specific pixel value is received across multiple candidate images. The "max-min" estimate can be viewed as sorting the sequence of observations, weighting the maximum value with +1, the minimum value with -1, and all observations in between with 0. Following this line of thought, the estimation of an ambient light corrected image can be viewed as differently weighting the sorted sequence of observations to obtain a more accurate estimate:

[0077]

[0078] Where w is a pre-calculated set of weights, s sort It is a sorted sequence of pixel values ​​(from low to high).

[0079] Assuming the statistics of the observed sequence, including the estimation of the signal-to-noise ratio (SNR) and the sequence length, i.e., the number of candidate images, can be pre-calculated or predetermined through simulation for estimating a. hi -a lo The optimal weighting. More specifically, in some embodiments, the coefficients applied as weights are different for different SNR values.

[0080] Similarly, in some embodiments, the optimal weighting to be applied can be a function of the SNR. Therefore, typically, a 2D table of coefficients can be pre-computed, where a set of weights can be selected based on an estimate or proxy of the current SNR. Figure 6 The figure illustrates multiple pre-computed weights for least-squares estimation of multiple SNRs. The values ​​shown are the standard deviations of pixel sequences across multiple frames (i.e., candidate images). SNR estimation can be obtained, for example, by computing the standard deviations of the observation sequence (i.e., multiple candidate images), which can be analytically described as:

[0081]

[0082] Where X is a signal composed of the actual signal S and noise N, μ is the average value of signal X, and μ s It is the average value of the signal S (equal to μ), since noise is assumed to be without any bias. The equation shows that the standard deviation of the sequence of measured images increases monotonically with increasing signal level.

[0083] Therefore, for a given or known noise level (variance) The standard deviation may be a good representative of the SNR. For comparative purposes, Table 1 below shows the differences in calculations for both the maximum-minimum estimate and the estimation technique as described in this disclosure, as shown in pseudocode:

[0084] Table 1 – Comparison of estimators using pseudocode

[0085]

[0086]

[0087] exist Figure 7 The results of a series of simulation measurements using the proposed maximum-minimum estimation and estimation techniques are provided, showing that, on average, the proposed estimation techniques are more accurate. More detailed, Figure 7 It includes a comparison of probability distribution functions for estimates of maximum-minimum and least-squares estimates for many different SNRs, where the noise level remains constant in these simulations. Figure 7 The a above each subgraph hi -a lo The value indicates the correct value. By comparing the results associated with the maximum-minimum estimate and the estimation technique described herein, it is clear that the estimation technique described herein achieves less bias and a smaller standard deviation. Therefore, an improved system and method for performing ambient light image correction is provided, which overcomes the existing problems.

[0088] A computer program product comprising a computer-readable medium having computer-readable code contained therein, the computer-readable code being configured to, when executed by a suitable computer or processor, cause the computer or processor to perform at least a portion of the methods described herein. Therefore, it should be understood that this disclosure is also applicable to computer programs, particularly to computer programs on or in a carrier in which embodiments are practiced. The program may be source code, object code, intermediate source code, and object code such as in partially compiled form, or any other form suitable for use in an implementation of the methods according to the embodiments described herein.

[0089] It should also be understood that such a program can have many different architectural designs. For example, the program code implementing the functionality of a method or system can be subdivided into one or more subroutines. Many different ways of distributing functionality among these subroutines will be apparent to those skilled in the art. Subroutines can be stored together in an executable file to form a self-contained program. Such an executable file can include computer-executable instructions, such as processor instructions and / or interpreter instructions (e.g., Java interpreter instructions). Alternatively, one or more subroutines can be stored in at least one external library file and linked to the main program statically or dynamically, for example, at runtime. The main program contains at least one call to at least one of the subroutines. Subroutines can also include function calls to each other.

[0090] Embodiments relating to computer program products include computer-executable instructions corresponding to each processing stage of at least one of the methods set forth herein. These instructions may be subdivided into subroutines and / or stored in one or more files that can be statically or dynamically linked. Another embodiment relating to computer program products includes computer-executable instructions corresponding to each device of at least one of the systems and / or products set forth herein. These instructions may be subdivided into subroutines and / or stored in one or more files that can be statically or dynamically linked.

[0091] The carrier of a computer program can be any entity or device capable of carrying the program. For example, the carrier can include data storage such as ROM, like a CD-ROM or semiconductor ROM, or magnetic recording media such as a hard disk. Furthermore, the carrier can be a transmissible medium such as electrical or optical signals, which can be transmitted via cables or optical fibers or by radio or other means. When a program is contained within such a signal, the carrier can be constituted by such cables or other devices or apparatuses. Alternatively, the carrier can be an integrated circuit in which the program is embedded, the integrated circuit being adapted to perform the relevant method or to be used in the performance of the relevant method.

[0092] By studying the accompanying drawings, the disclosure, and the appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments in practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. A single processor or other unit can implement the functions of several items as described in the claims. The fact that certain measures are recited in mutually different dependent claims does not imply that combinations of these measures cannot be advantageously used. Computer programs can be stored / distributed on suitable media, such as optical storage media or solid-state media provided with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. Any reference numerals in the claims should not be construed as limiting the scope.

Claims

1. A system (100) for performing ambient light image correction, the system comprising: A light source (110) configured to provide pulsed illumination to an object; An imaging unit (120) is configured to capture multiple images of the object while the object is illuminated by pulsed illumination from the light source, wherein each of the multiple images is captured with an exposure time shorter than the wavelength of the pulsed illumination, wherein the pulse frequency of the illumination provided by the light source is not an integer multiple of the frame rate, the multiple images are captured by the imaging unit at an integer multiple of the frame rate, and wherein the total time during which the multiple images are captured is at least half of the wavelength of the pulsed illumination; as well as Control unit (130), the control unit being configured to: Received the pre-ordered number n The captured images, candidate images, wherein the candidate images are n A series of captured images; A sorted list of pixels is generated for each pixel position in each of the plurality of candidate images by sorting the corresponding pixels corresponding to the corresponding pixel positions in the plurality of candidate images, wherein the sorting is based on the pixel value of the corresponding pixel. For each pixel location in the plurality of candidate images, a set of weights is applied to the corresponding sorted list of pixels, wherein the set of weights is associated with a least-squares approximation of the probability density function of the pixel value over time; as well as An estimated ambient light corrected image is generated by summing the multiple weighted and sorted lists of pixels.

2. The system (100) of claim 1, wherein n is predetermined based at least on the pulse frequency of the pulsed illumination and the frame rate at which the plurality of images are captured.

3. The system (100) according to claim 1 or 2, wherein the corresponding pixels, each corresponding to the corresponding pixel position in the plurality of candidate images, are sorted in ascending order.

4. The system (100) according to any one of claims 1 and 2, wherein the control unit (130) is configured to determine the set of weights to be applied to the sorted list of pixels based on the signal-to-noise ratio of the plurality of candidate images.

5. The system (100) according to any one of claims 1 and 2, wherein the control unit (130) is configured to determine the set of weights to be applied to the sorted list of pixels based on at least one of the following: the exposure time of the plurality of images being captured, the sensitivity level of the imaging unit, and the detection light level of the plurality of captured images.

6. The system (100) of claim 4, wherein determining the set of weights to be applied to the sorted list of pixels includes selecting the set of weights to be applied from a plurality of predetermined weight reassemblies.

7. The system (100) according to any one of claims 1 and 2, wherein the set of weights to be applied to the sorted list of pixels is selected from a predetermined coefficient table.

8. The system (100) according to any one of claims 1 and 2, wherein the imaging unit is configured to employ a rolling shutter such that each of the captured plurality of images includes a plurality of bright bands and dark bands, wherein the bright bands correspond to the high state of the pulsed illumination and the dark bands correspond to the low state of the pulsed illumination.

9. A method for performing ambient light image correction, comprising: Provide (202) pulsed illumination to the object; While the object is illuminated by the pulsed illumination, multiple images of the object are captured (204), wherein each of the multiple images is captured with an exposure time shorter than the wave period of the pulsed illumination, wherein the pulse frequency of the illumination provided is not an integer multiple of the frame rate, the multiple images are captured at an integer multiple of the frame rate, and wherein the total time during which the multiple images are captured is at least half of the wave period of the pulsed illumination. Received (206) pre-ordered quantity n The captured images, candidate images, wherein the candidate images are n A series of captured images; (208) A sorted list of pixels for each pixel position in each of the plurality of candidate images is generated by sorting the corresponding pixels corresponding to the corresponding pixel positions in the plurality of candidate images, wherein the sorting is based on the pixel value of the corresponding pixel. For each pixel location in each of the plurality of candidate images, a set of weights is applied (210) to the corresponding sorted list of pixels, wherein the set of weights is associated with a least-squares approximation of the probability density function of the pixel value over time; as well as Based on multiple weighted and sorted lists of pixels, an estimated ambient light corrected image (212) is generated by summing the multiple weighted and sorted lists of pixels.

10. The method of claim 9, wherein n is predetermined based at least on the pulse frequency of the pulsed illumination and the frame rate at which the plurality of images are captured.

11. The method of claim 9 or 10, wherein the respective pixels corresponding to the respective pixel positions in the plurality of candidate images are sorted in ascending order.

12. The method of any one of claims 9 and 10, further comprising determining the set of weights to be applied to the sorted list of pixels based on the signal-to-noise ratio of the plurality of candidate images.

13. The method of any one of claims 9 and 10, wherein the method further comprises determining the set of weights to be applied to the sorted list of pixels based on at least one of the following: the exposure time at which the plurality of images are captured, the sensitivity level of the imaging unit used to capture the plurality of images, and the detection light level of the captured plurality of images.

14. The method of claim 12, wherein determining the set of weights to be applied to the sorted list of pixels comprises selecting the set of weights to be applied from a plurality of predetermined weight reassemblies.

15. A computer program product comprising a computer-readable medium having computer-readable code implemented therein, the computer-readable code being configured to, when executed by a suitable computer or processor, cause the computer or processor to perform the method according to any one of claims 9 to 14.

Images