Flicker compensation

Abstract

Features described herein pertain to flicker compensation. Flicker frequencies of light sources in an environment can be detected and the greatest common factor of the flicker frequencies can be calculated. A first exposure time for capturing an image can be set based on the greatest common factor. A determination can be made that the first exposure time is greater than a threshold exposure time, and, in response, a second exposure time for capturing another image can be set based on the threshold exposure time. Flicker can be compensated by fusing the images captured with the first and second exposure times.

Description

FLICKER COMPENSATIONBACKGROUND

[0001] Electrical distribution systems throughout the World provide electrical power via power grids operating at different frequencies. For example, in some regions of the World, the operating frequency of the regional power grid is 60 Hertz, and, in some other regions of the World, the operating frequency of the regional power grid is 50 Hertz. Light sources powered by electrical power supplied by these power grids tend to flicker at frequencies that correspond to the operating frequencies of the respective power grids. When image capture is performed in such an environment using a rolling shutter technique, the resulting images can include undesirable banding artifacts.SUMMARY

[0002] Embodiments described herein pertain to systems and methods for flicker compensation.

[0003] In various embodiments, a method includes determining a first exposure time based on the greatest common factor of flicker frequencies of a plurality of light sources; determining that the first exposure time is equal to or greater than a threshold exposure time; in response to determining that the first exposure time is equal to or greater than the threshold exposure time, using the threshold exposure time as a second exposure time; capturing a first image with the first exposure time; capturing a second image with the second exposure time; and calculating a third image, wherein calculating the third image comprises combining a difference image with the second image, wherein the difference image is derived from the first image and the second image.

[0004] In some embodiments, the method further includes detecting the flicker frequencies of the plurality of light sources; and calculating the greatest common factor of the flicker frequencies of the plurality of light sources.

[0005] In some embodiments, calculating the third image further comprises performing a luminance normalization operation on the first image.

[0006] In some embodiments, calculating the third image further comprises performing an image processing operation on the first image.

[0007] In some embodiments, calculating the third image further comprises performing an image processing operation on the second image.

[0008] In some embodiments, the difference image is derived from the first image and the second image by calculating a difference between pixel values of the first image and pixel values of the second image.

[0009] In some embodiments, combining the difference image with the second image comprises subtracting the difference image from the second image.

[0010] Some embodiments include a system that includes a processing system and at least one computer-readable medium storing instructions which, when executed by the processing system, cause the system to perform part or all of the operations and / or methods disclosed herein.

[0011] Some embodiments include one or more non-transitory computer-readable media storing instructions which, when executed by at least one processing system, cause a system to perform part or all of the operations and / or methods disclosed herein.

[0012] The techniques described above and below may be implemented in a number of ways and in a number of contexts. Several example implementations and contexts are provided with reference to the following figures, as described below in more detail. However, the following implementations and contexts are but a few of many.BRIEF DESCRIPTION OF THE DRAWINGS

[0013] A further understanding of the nature and advantages of various embodiments may be realized by reference to the following figures. In the appended figures, similar components or features may have the same reference label. Further, various components of the same type may be distinguished by following the reference label by a dash and a second label that distinguishes among the similar components. If only the first reference label is used in the specification, the description is applicable to any one of the similar components having the same first reference label irrespective of the second reference label.

[0014] FIG. 1 illustrates an example of image capturing environment that includes light sources according to some implementations of the present disclosure.

[0015] FIG. 2 illustrates a block diagram of an embodiment of a flicker compensation system according to some implementations of the present disclosure.

[0016] FIG. 3 illustrates an embodiment of a flow for flicker compensation according to some implementations of the present disclosure.

[0017] FIGS. 4A and 4B illustrate an embodiment of a flicker compensation method according to some implementations of the present disclosure.DETAILED DESCRIPTION

[0018] Electronic devices that include image sensors for capturing images of scenes have become prevalent. To facilitate image capture, an image sensor can typically utilize a globalshutter technique and / or a rolling shutter technique for exposing pixels of the image sensor to light. A global shutter technique typically involves exposing all or nearly all of the pixels of the image sensor to light at the same time and for the same exposure time. On the other hand, a rolling shutter technique typically involves exposing rows of pixels of the image sensor to light at different times and for the same exposure time per row. The starts of the exposure times between rows of pixels of the image sensor can be staggered such that a start time for exposing pixels in one row of pixels to light can be offset from a start time for exposing pixels in another row of pixels to light. In the case of a global shutter technique, signals can be read out from the pixels of the image sensor at the end of the exposure time, and, in the case of a rolling shutter technique, signals can be read out from a row of pixels of the image sensor at the end of the exposure time for the row.

[0019] Often a scene will include multiple light sources such as lights for illuminating a room, display screens, appliance lighting, and the like. These lights are often powered by electrical power supplied by a power grid of a local electrical utility. Interconnected electrical systems used to distribute power through the power grid often operate at one or more frequencies such as 50 Hertz (Hz) or 60 Hz. Light sources that are powered by such power grids tend to emit light that flickers at a frequency that is equal to or a multiple of the operating frequency of the power grid in which it is connected. In the case that an image sensor that employs a rolling shutter technique is used to capture images of a scene that includes these light sources, banding artifacts can appear in the captured images due the difference between flicker periods of the light sources and the exposure times of the rows of pixels of the image sensor.

[0020] Conventional flicker compensation and / or anti-banding techniques include synchronizing the exposure time or exposure times of the rows of pixels of the image sensor with the flicker frequencies of the light sources. Depending on the flicker frequencies of the light sources, a long exposure time may be set which in turn can lead to the captured images being over-exposed (z.e., saturated) and distorted with various distortions such as motion blur. To avoid the potential for over-exposure and distortions, the exposure time can be capped such that no exposure time can be set that exceeds the exposure time cap. However, if the exposure time needed to cancel the effects of the flickering of the light sources is greater than the exposure time cap, banding artifacts can still appear in the captured images.

[0021] The features and techniques described herein overcome these challenges and / or others by providing a system and method for flicker compensation. In some implementations, flicker frequencies of light sources in an environment can be detected and the greatest common factor ofthe flicker frequencies can be calculated. A first exposure time for capturing a first image can be determined based on the greatest common factor. A determination can be made that the first exposure time is equal to or greater than a threshold exposure time, and, in response, the threshold exposure time can be used as a second exposure time for capturing a second image. Flicker can be compensated by fusing images captured with the first and second exposure times.

[0022] FIG. 1 illustrates an image capturing environment that includes light sources. As shown in FIG. 1, the image capturing environment 100 can include an electronic device 102 and light sources 106 (which can include light sources 106-1, 106-2, 106-3, 106-4, etc.). In some implementations, the electronic device 102 can be an assistant device such as a Google® Nest® Hub and Google® Nest® Hub Max. In other implementations, the electronic device 102 can be a mobile phone and / or portable computing device such as a Google® Pixel® Phone and Google® Pixel® Tablet. The camera 108 can be configured to capture images of a scene within a field of view 104. The light sources 106 can be electrical light sources that are configured to emit light when powered by an electrical power source such as power supplied by a utility via an electric power grid. Examples of light sources 106 include light-emitting diode lamps, fluorescent lamps, incandescent lamps, halogen lamps, arc lamps, discharge lamps, and the like. Each light source may emit light that flickers at a flicker frequency. In some implementations, the intensity of the light emitted from the light source may fluctuate at the flicker frequency and / or the light source may repeatedly turn off and turn on at the flicker frequency. The flicker frequency of each light source can be a function of an operating frequency of the electric power grid that serves as the power supply for the respective light source. In some implementations, a light source can flicker at the same frequency as the operating frequency of the electric power grid. For example, a power grid having an operating frequency of 50 Hz can cause a light source powered by that power grid to flicker at 50 Hz. In other implementations, a light source can flicker at a frequency that is a multiple of the operating frequency of the electric power grid. For example, a power grid having an operating frequency of 60 Hz can cause a light source powered by that power grid to flicker at 120 Hz (z.e., double the operating frequency of the electric power grid). Examples of frequencies at which the light sources 106 can flicker include, but are not limited to, 60 Hz, 90 Hz, 100 Hz, and 120 Hz.

[0023] The camera 108 can include an image sensor (not shown) and a light sensor (not shown). In some implementations, the light sensor can be configured to measure characteristics of ambient light in the scene and the flicker frequencies of the light sources 106 illuminating the scene and output signals indicative of the measured characteristics and flicker frequencies. The characteristics of the ambient light and the flicker frequencies can be used to set exposure times forcapturing images with the image sensor and the image sensor can capture images of the scene based on the set exposure times. In some implementations, the image sensor can be configured to capture an image using a global shutter technique in which all or nearly all of the pixels (e.g., 70%, 80%, 90%) of the image sensor are exposed to light at the same time and for a set exposure time. Additionally, or alternatively, the image sensor can be configured to capture an image using a rolling shutter technique in which the rows of pixels of the image sensor are exposed to light at different times but with the same set exposure time per row. Using a rolling shutter technique, the starts of the exposure times between adjacent rows of pixels of the image sensor can be staggered such that a start time for exposing pixels in one row of pixels to light can be offset from a start time for exposing pixels in an adjacent row of pixels to light.

[0024] As described herein, capturing images of a scene illuminated with light that flickers at different frequencies using a rolling shutter technique can cause banding artifacts in the captured images due to the difference between the flicker periods (e.g., periods between peak intensities of the emitted light) of the light sources and the exposure times of the rows of pixels.

[0025] To compensate for flicker, the electronic device 102 can be configured to perform flicker compensation. The electronic device 102 can be configured to perform flicker compensation by detecting the flicker frequencies of the light sources illuminating the scene using the light sensor and calculating the greatest common factor of the flicker frequencies. A first exposure time for capturing a first image can be determined based on the greatest common factor of the flicker frequencies. A determination can be made as to whether the first exposure time is equal to or greater than a threshold exposure time (e.g., an exposure time cap determined by an application programming interface (API) of the camera 108 and / or the electronic device 102). In response to determining that the first exposure time is equal to or greater than the threshold exposure time, the threshold exposure time can be used as a second exposure time for capturing a second image. The electronic device 102 can be configured to capture first and second images with the first and second exposure times and fuse the captured images to form an image in which the effects of flicker have been compensated.

[0026] In some implementations, the electronic device can be configured to fuse the captured images by performing a luminance normalization operation on the first image and / or second image and performing an image processing operation on the first and second images and / or the first and second luminance-normalized images. A difference can be calculated between the first processed image and the second processed image to form a difference image. The difference image can be subtracted from the second image to form the flicker-compensated image.

[0027] FIG. 2 illustrates a block diagram of an embodiment of a flicker compensation system. Flicker compensation system 200 (“system 200”) can include electronic device 202, network 228, and cloud server system 230. The electronic device 202 can include a sensing subsystem 204, a processing subsystem 210, a display subsystem 222, an audio subsystem 224, and a communication subsystem 226.

[0028] The sensing subsystem 204 can include an image sensor 206 and a light sensor 208. The image sensor 206 can be configured to capture images of a scene and output signals indicative of light reflected from the scene and collected by the image sensor 206. The image sensor 206 can be a complementary metal-oxide semiconductor (CMOS) image sensor, which can include an array of pixels (e.g., pixels arranged in rows and columns). Each pixel of the image sensor 206 can include one or more photodiodes, one or more sensing circuits, and one or more select circuits. The image sensor 206 can be a digital sensor (e.g., a digital pixel sensor) in which signals generated by the photodiodes of the image sensor 206 can be converted into digital signals. In some implementations, analog-to-digital conversion (ADC) circuitry can be included in each pixel and / or included in the image sensor 206 for one or more groups of pixels. The ADC circuitry can be controlled to adjust a bit depth and / or the dynamic range of the signals output from the image sensor 206. The image sensor 206 can be configured to be sensitive to visible and / or infrared light. In some other implementations, each pixel of the image sensor 206 can include a tunable filter that can be configured to pass and / or block light having a particular wavelength and / or range of wavelengths. Each photodiode of the image sensor 206 can output a signal indicative of the light collected by the respective photodiode. In some implementations, the signals output by the photodiodes can be indicative of the intensity and angle of the light reflected from objects in the scene.

[0029] In order to capture images, each pixel of the array of pixels can be configured to be exposed to light (e.g., natural ambient light, artificial ambient light, light from reflected from the scene, etc.) for a period of time (e.g., an exposure time), convert the received light into an electric charge, and store a voltage corresponding to the electric charge on a charge storage device for the respective pixel. The voltage stored on the charge storage device for a respective pixel can be provided to ADC circuitry for conversion into a pixel value for the respective pixel which can be read out from the array of pixels. Images can be formed by selectively reading out the pixel values from the array of pixels. Although not shown, the sensing subsystem 204 can include other circuitry such as ramp generators, bias generators, amplifiers, power supplies, and the like for facilitating image capture. The pixel values can be read out from the array of pixels and output (e.g., by the sensing subsystem 204) to the processing subsystem 210 for conversion into imagessuch as color images, grayscale images, video frames, depth images, three-dimensional (3D) images, and the like. In some implementations, the processing subsystem 210 can convert the signals into the images at a particular frame rate (e.g., between 30-60 frames per second).

[0030] In some implementations, the image sensor 206 can be configured to capture an image of the scene using a global shutter in which all or nearly all the pixels (e.g., 70%, 80%, 90%) of the image sensor 206 are exposed to light at the same time and for an exposure time. Additionally, or alternatively, the image sensor 206 can be configured to capture an image of the scene using a rolling shutter in which the rows of pixels of the image sensor 206 are exposed to light at different times and for the same exposure time per row. Using a rolling shutter, the starts of the exposure times between rows of pixels of the image sensor 206 can be staggered such that a start time for exposing pixels in one row of pixels to light can be offset from a start time for exposing pixels in another row of pixels to light. In some implementations, the image sensor 206 can be sequentially scanned from a first row of pixels (e.g., a top row of pixels) to a last row of pixels (e.g., a bottom row of pixels) such that an exposure time for one row of pixels can start before an exposure time for an adjacent subsequent row of pixels starts. The exposure times and offsets between exposure start times for adjacent rows of pixels of the image sensor 206 can be determined based on a shutter speed and / or exposure time for the image sensor 206. In some implementations, the exposure time or shutter speed for the image sensor 206 can be divided evenly among the rows of pixels of the image sensor 206 and the offset can be determined based on the number of rows of pixels of the image sensor 206.

[0031] The light sensor 208 can be configured to measure characteristics of ambient light in the scene and flicker frequencies of light sources illuminating the scene and output signals indicative of the measured characteristics and flicker frequencies. Examples of characteristics of ambient light include, but are not limited to, brightness, intensity, color temperature, illuminance, and the like. The light sensor 208 can include one or more components that enable the light sensor 208 to measure the characteristics and the flicker frequencies. For example, the light sensor 208 can include an ambient light sensor for measuring the characteristics and a flicker sensor for measuring the flicker frequencies. In some implementations, the light sensor 208 can be included in the image sensor 206. In other implementations, the image sensor 206 can be configured to function as a light sensor and measure the characteristics and the flicker frequencies. The signals generated by the light sensor 208 can be output (e.g., by the sensing subsystem 204) to the processing subsystem 210 for determining exposure times for capturing images using the image sensor 206.

[0032] The processing subsystem 210 includes one or more memories 212, one or more processors 214, and RAM 216. The one or more processors 214 can read one or more programs from the one or more memories 212 and execute them using RAM 216. Each of the one or more processors 214 can be of any type including but not limited to a microprocessor, a microcontroller, a graphical processing unit, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or any combination thereof. In some embodiments, the one or more processors 214 can include a plurality of cores, a plurality of arrays, one or more coprocessors, and / or one or more layers of local cache memory. The one or more processors 214 can execute one or more programs stored in the one or more memories 212 to perform the operations and / or methods, including parts thereof, described herein.

[0033] Each of the one or more memories 212 can be non-volatile and can include any type of memory device that retains stored information when powered off. Non-limiting examples of memory include electrically erasable and programmable read-only memory (EEPROM), flash memory, or any other type of non-volatile memory. At least one memory of the one or more memories 212 can include a non-transitory computer-readable storage medium from which the one or more processors 214 can read instructions. A computer-readable storage medium can include electronic, optical, magnetic, or other storage devices capable of providing the one or more processors 214 with computer-readable instructions or other program code. Non-limiting examples of a computer-readable storage medium include magnetic disks, memory chips, readonly memory (ROM), RAM, an ASIC, a configured processor, optical storage, or any other medium from which a computer processor can read the instructions.

[0034] The one or memories 212 can include a capture module 218 and a fusion module 220. These modules, which are presented being included in the one or more memories 212, can be implemented as individual hardware and / or software components or may be implemented together, such as in the form of software that can be executed by the one or more processors 214.

[0035] The capture module 218 can be configured to receive the signals generated by the light sensor 208 and determine exposure times for capturing images using the image sensor 206. In some implementations, an exposure time for capturing an image can be determined based on the flicker frequencies of the light sources illuminating the scene. In some implementations, based on the signals generated by the light sensor 208 and received by the capture module 218, the capture module 218 can be configured to calculate the greatest common factor of the flicker frequencies of the light sources illuminating the scene and determine an exposure time for capturing an image based on the greatest common factor. In some implementations, the exposure time can bedetermined based on the greatest common factor by converting the greatest common factor into a fraction of a second and converting the fraction of the second into milliseconds representing the exposure time. For example, for a scene illuminated by a light source that flickers at a frequency of 100 Hz and a light source that flickers at a frequency of 120 Hz, the capture module 218 can calculate the greatest common factor of these frequencies (e.g., 20), convert the greatest common factor into a fraction of a second (e.g., 1 / 20^ second), and convert the fraction of the second into milliseconds (e.g., 50 milliseconds) representing the exposure time.

[0036] In other implementations, based on the signals generated by the light sensor 208 and received by the capture module 218, the capture module 218 can be configured to calculate suitable exposure times for each flicker frequency measured and calculate the least common multiple between the suitable exposure times. A suitable exposure time as used herein refers to an exposure time for capturing a banding-free image. In other words, with a suitable exposure time, any differences between flicker periods of a light source and the suitable exposure time will be minimized thereby reducing the likelihood of banding-type artifacts in the captured image. In some implementations, a suitable exposure time for a particular flicker frequency can be calculated by converting the particular flicker frequency (e.g, 100 Hz) into a fraction of a second (e.g, l / lOO*11) and converting the fraction of the second for the flicker frequency into milliseconds (e.g., 10 milliseconds) representing a base suitable exposure time. Other suitable exposure times can include multiples of the base suitable exposure time for the particular flicker frequency (e.g., 20, 30, 40, 50 milliseconds). In the case a scene includes light sources having different flickering frequencies, the least common multiple between the suitable exposure times for the flicker frequencies of the light sources can be identified as suitable exposure time for the light sources in the scene.

[0037] In some implementations, as shown in FIG. 3, which shows a flow 300 for flicker compensation, the capture module 218 can be configured to determine exposure times for capturing multiple images (e.g., a first image and a second image). As shown in FIG. 3, based on the signals generated by the light sensor 208 and received by the capture module 218, the capture module 218 can determine a first exposure time for capturing a first image 302 based on the greatest common factor and / or least common multiple of the flicker frequencies of the light sources. The capture module 218 can then determine whether the first exposure time for capturing the first image 302 is equal to or greater than a threshold exposure time (e.g., 33.33 milliseconds). In some implementations, the threshold exposure time can be set to reduce the likelihood that a captured image will be over-exposed (z.e., saturated) and / or distorted with various distortions such as motion blur. In other implementations, the threshold exposure time can be set based on anexposure time cap of an API of the sensing subsystem 204 and / or the electronic device 202. In some implementations, in response to determining that the first exposure time is equal to or greater than the threshold exposure time, the capture module 218 can be configured to use the threshold exposure time as a second exposure time for capturing a second image 304. In this way, a first image can be captured that minimizes the potential for banding-type artifacts due to flickering light sources in a scene and a second image can be captured that minimizes the potential for over- exposure and various other distortions caused by longer exposure times.

[0038] In some implementations, the capture module 218 can determine the exposure time for capturing an image and / or exposure times for capturing multiple images in response to receiving an image capture command. In some implementations, the image capture command can be provided to the capture module 218 in response to the depression of a shutter button, a touch input received on a touchscreen display of the electronic device 202, an instruction generated by the processing subsystem 210, and the like. In some implementations, the image capture command can be provided to the capture module 218 in response to the electronic device 202 entering into a flicker compensation mode. For example, based on the signals generated by the light sensor 208 and received by the capture module 218, the capture module 218 can generate an instruction that causes the electronic device 202 to enter into the flicker compensation mode and, in response to entering into the flicker compensation mode, an image capture command can be provided to the capture module 218. In the flicker compensation mode, exposure times for capturing a first image and a second image can be determined as described herein.

[0039] Once the capture module 218 determines an exposure time for capturing an image and / or exposure times for capturing multiple images, the capture module 218 can generate instructions that cause the image sensor 206 to capture image(s) with the determined exposure time(s). For example, continuing to refer to FIG. 3, the capture module 218 can generate instructions for instructing the image sensor 206 to capture a first image 302 with a first exposure time and a second image 304 with a second exposure time. In the case of capturing a first image and a second image in the flicker compensation mode, the sensing subsystem 204 can be configured to provide the captured first and second images to the fusion module 220 where a flicker-compensated image 306 can be calculated. In other cases, the sensing subsystem 204 can be configured to output the captured image(s) to a storage medium (not shown) where they can be stored for later processing and / or display.

[0040] The fusion module 220 can be configured to receive the first image and the second image from the sensing subsystem 204 and calculate a flicker-compensated image 306 from the capturedimages. In some implementations, the fusion module 220 can be configured to calculate a flicker- compensated image 306 by performing an image fusion process on the captured images. For example, the fusion module 220 can be configured to calculate a flicker-compensated image by performing an image fusion process on the first and second images captured with the exposure times determined as described herein. In some implementations, to perform the image fusion process, upon receiving the first and the second images, the fusion module 220 can perform a luminance normalization operation on the first and / or second image to form first and / or second luminance-normalized images and perform an image processing operation on the first and second images and / or on the first and / or second luminance-normalized images to form first and second processed images. Examples of image processing operations include, but are not limited to, denoising, deblurring, smoothing, contrast enhancement, and the like.

[0041] The fusion module 220 can then calculate a difference image by calculating a difference between the first processed image and the second processed image. The difference image can be calculated using any suitable image difference calculation technique. For example, the difference image can be calculated by calculating an elementwise (pixel-by-pixel) absolute difference between pixel values of the first processed image and pixel values of the second processed image. In some implementations, the difference image can be indicative of banding and other artifacts appearing in the second image. The fusion module 220 can then subtract the difference image from the second image to form the flicker-compensated image 306. Any suitable image subtraction technique can be used to subtract the difference image from the second image. For example, the fusion module 220 can invert the second image and add the inverted second image to the difference image. The fusion module 220 can be configured to output the flicker-compensated image 306 to a storage medium (not shown) where it can be stored for later processing and / or display.

[0042] The electronic device 202 also includes a display subsystem 222 that can be configured to display color images, grayscale images, video frames, depth images, three-dimensional (3D) images, and other content on a display screen of the electronic device 202 and receive input from a user of the electronic device 202. Examples of the display screen included in the display subsystem 222 include a liquid crystal display, a light emitting diode display, an organic light emitting diode display, a projector display, a touchscreen display, and the like.

[0043] The electronic device 202 also includes an audio subsystem 224 that can be configured to record sounds from the environment surrounding the electronic device 202 and output sounds to the environment surrounding the electronic device 202. Examples of audio devices included inaudio subsystem 224 include microphones, speakers, and other audio / sound transducers for receiving and outputting audio signals and other sounds.

[0044] The electronic device 202 also includes a communications subsystem 226 that can be configured to enable the electronic device 202 to communicate with various wired or wireless networks and other systems and devices. For example, communications subsystem 226 can enable the electronic device 202 to communicate with network 228. Network 228 can include one or more private and / or public networks, such as the Internet. Network 228 can be used such that electronic device 202 can communicate with the cloud server system 230. Cloud server system 230 can, in some implementations, perform some of the processing functions performed by processing subsystem 210. Additionally, or alternatively, cloud server system 230 can be used to relay notifications and / or store data generated by the electronic device 202. For instance, in association with the user account, alerts and / or notifications generated by electronic device 202 can be logged by cloud server system 230. Examples of communications devices included in communications subsystem 226 include wireless communication modules and chips, wired communication modules and chips, chips for communicating over local area networks, wide area networks, cellular networks, satellite networks, fiber optic networks, and the like, systems on chips, and other circuitry that enables the electronic device 202 to send and receive data.

[0045] Although not shown, the electronic device 202 can also include other components that provide the electronic device 202 with various functionality. Other components can include storage devices, power generating / storing devices, input / output (I / O) components, and the like. The foregoing configurations of the electronic device 202 are not intended to be limiting and the electronic device 202 can include other subsystems, devices, and components.

[0046] FIGS. 4A and 4B illustrate an embodiment of a method 400 for flicker compensation. The processing depicted in FIGS. 4 A and 4B may be implemented in software (e.g., code, instructions, program) executed by a processing system such as the processing subsystem 210 of the electronic device 202. The software may be stored on a non-transitory computer-readable storage medium (e.g., a memory device). The method 400 is intended to be illustrative and nonlimiting. For example, although FIGS. 4 A and 4B depicts the various processing steps occurring in a particular sequence or order, in other embodiments, the steps may be performed in some different order or some steps may also be performed in parallel.

[0047] The method 400 begins at block 402. At block 404, a determination is made as whether an image capture command has been received. The image capture command can be generated in response to the depression of a shutter button, a touch input received on a touchscreen display, aninstruction generated by a processor, and the like. In some implementations, the image capture command can be generated in response to an event occurring in an application or operating system executing in an electronic device such as electronic device 202. In some implementations, the image capture command can be generated in response to an electronic device such as electronic device 202 entering into a flicker compensation mode. In the event an image capture command is received, the method 400 continues to block 406. On the other hand, in the event that an image capture command is not received, the method 400 returns to block 402.

[0048] At block 406, flicker frequencies of light sources are detected. The light sources can be electrical light sources that are configured to emit light when powered by an electrical power source such as power supplied by a utility via an electric power grid. Examples of light sources include light-emitting diode lamps, fluorescent lamps, incandescent lamps, halogen lamps, arc lamps, discharge lamps, and the like. Each light source may emit light that flickers at a flicker frequency. In some implementations, the intensity of the light emitted from the light source may fluctuate at the flicker frequency and / or the light source may repeatedly turn off and turn on at the flicker frequency. The flicker frequency of each light source can be a function of an operating frequency of the electric power grid that serves as the power supply for the respective light source. In some implementations, a light source can flicker at the same frequency as the operating frequency of the electric power grid. For example, a power grid having an operating frequency of 50 Hz can cause a light source powered by that power grid to flicker at 50 Hz. In other implementations, a light source can flicker at a frequency that is a multiple of the operating frequency of the electric power grid. For example, a power grid having an operating frequency of 60 Hz can cause a light source powered by that power grid to flicker at 120 Hz (z.e., double the operating frequency of the electric power grid). Examples of frequencies at which the light sources 106 can flicker include, but are not limited to, 60 Hz, 90 Hz, 100 Hz, and 120 Hz.

[0049] In some implementations, the flicker frequencies of light sources can be detected by a light sensor such as the light sensor 208 of the electronic device 202. The light sensor can be configured to measure characteristics of ambient light in the scene and flicker frequencies of light sources illuminating the scene and output signals indicative of the measured characteristics and flicker frequencies. Examples of characteristics of ambient light include, but are not limited to, brightness, intensity, color temperature, illuminance, and the like. The light sensor can include one or more components that enable the light sensor to measure the characteristics and the flicker frequencies. For example, the light sensor can include an ambient light sensor for measuring the characteristics and a flicker sensor for measuring the flicker frequencies. In some implementations, the light sensor can be included in an image sensor used for capturing images ofa scene such as the image sensor 206 of the electronic device 202. In other implementations, the image sensor can be configured to function as a light sensor and measure the characteristics and the flicker frequencies.

[0050] At block 408, the greatest common factor of the flicker frequencies of the light sources is calculated. For example, for a scene illuminated by a light source that flickers at a frequency of 100 Hz and a light source that flickers at a frequency of 120 Hz, the greatest common factor of these frequencies is 20.

[0051] At block 410, a first exposure time is determined based on the greatest common factor of the flicker frequencies of the light sources. In some implementations, the first exposure time can be used for capturing a first image. The first exposure time can be determined based on the greatest common factor by converting the greatest common factor into a fraction of a second and converting the fraction of the second into milliseconds representing the exposure time. For example, for a scene illuminated by a light source that flickers at a frequency of 100 Hz and a light source that flickers at a frequency of 120 Hz, the greatest common factor of these frequencies (e.g., 20) can be converted into a fraction of a second (e.g., 1 / 20^ second), which can be converted into milliseconds (e.g., 50 milliseconds) representing an exposure time. In some implementations, the first exposure time can be determined based on a least common multiple between suitable exposure times determined for each flicker frequency detected. A suitable exposure time for a particular flicker frequency can be calculated by converting the particular flicker frequency (e.g, 100 Hz) into a fraction of a second (e.g, 1 / 100th) and converting the fraction of the second for the flicker frequency into milliseconds (e.g., 10 milliseconds) representing a base suitable exposure time. Other suitable exposure times can include multiples of the base suitable exposure time for the particular flicker frequency (e.g., 20, 30, 40, 50 milliseconds). The least common multiple between the suitable exposure times for the flicker frequencies of the light sources can be identified as a suitable exposure time for the light sources in the scene.

[0052] At block 412, a determination is made as to whether the first exposure time is equal to or greater than a threshold exposure time. In some implementations, the threshold exposure time can be set to reduce the likelihood that a captured image will be over-exposed ( / . e. , saturated) and / or distorted with various distortions such as motion blur. In other implementations, the threshold exposure time can be set based on an exposure time cap of an API of the image sensor or a sensing subsystem or electronic device incorporating the image sensor such as sensing subsystem 204 and / or the electronic device 202 (e.g., 33.33 milliseconds). In the event that the first exposure time is equal to or greater than the threshold exposure time, the method 400 continues to block 418. Onthe other hand, in the event that the first exposure time is less than the threshold exposure time, the method 400 continues to block 414.

[0053] At block 414, an image is captured with the first exposure time. In some implementations, the image can be stored in a storage device such as a storage device (not shown) of electronic device 202 for later processing and / or display. Subsequently, the method returns to block 402.

[0054] At block 418, the threshold exposure time is used as a second exposure time. In some implementations, the second exposure time can be used for capturing a second image.

[0055] At block 420, a first image is captured with the first exposure time and a second image is captured with the second exposure time. In some implementations, to capture the first image and the second image, instructions can be generated that cause an image sensor such as image sensor 206 of electronic device 202 to capture the first and second images with the first and second exposure times.

[0056] At block 422, a third image is calculated. In some implementations calculating the third image includes combining a difference image derived from the first image and the second image with the second image. In some implementations, the third image can be a flicker-compensated image. The third image can be calculated by performing an image fusion process on the first and second images. To perform the image fusion process, a luminance normalization operation can be performed on the first and / or second image to form first and / or second luminance-normalized images and an image processing operation can be performed on the first and second images and / or on the first and / or second luminance-normalized images to form first and second processed images. Examples of image processing operations include, but are not limited to, denoising, deblurring, smoothing, contrast enhancement, and the like.

[0057] A difference image can then be calculated by calculating a difference between the between the first processed image and the second processed image. The difference image can be calculated using any suitable image difference calculation technique. For example, the difference image can be calculated by calculating an elementwise (pixel-by-pixel) absolute difference between pixel values of the first processed image and pixel values of the second processed image. In some implementations, the difference image can be indicative of banding and other artifacts appearing in the second image. The difference image can then be subtracted from the second image to form the third image (z.e., a flicker-compensated image). Any suitable image subtraction technique can be used to subtract the difference image from the second image. For example, the second image can be inverted and the inverted second image can be added to the difference image.The third image can be stored in a storage device such as a storage device (not shown) of electronic device 202 for later processing and / or display.

[0058] The systems and methods of the present disclosure may be implemented using hardware, software, firmware, or a combination thereof and may be implemented in one or more computer systems or other processing systems. Some embodiments of the present disclosure include a system including a processing system that includes one or more processors. In some embodiments, the system includes a non-transitory computer readable storage medium containing instructions which, when executed on the one or more processors, cause the system and / or the one or more processors to perform part or all of one or more methods and / or part or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computerprogram product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause the system and / or the one or more processors to perform part or all of one or more methods and / or part or all of one or more processes disclosed herein.

[0059] The terms and expressions which have been employed are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention claimed. Thus, it should be understood that although the present invention as claimed has been specifically disclosed by embodiments and optional features, modification, and variation of the concepts herein disclosed may be resorted to by those skilled in the art, and that such modifications and variations are considered to be within the scope of this invention as defined by the appended claims.

[0060] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

[0061] The above description of certain examples, including illustrated examples, has been presented only for the purpose of illustration and description and is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Modifications, adaptations, and uses thereof will be apparent to those skilled in the art without departing from the scope of thedisclosure. For instance, any examples described herein can be combined with any other examples.

Claims

WHAT IS CLAIMED IS:

1. A method comprising: determining a first exposure time based on the greatest common factor of flicker frequencies of a plurality of light sources; determining that the first exposure time is equal to or greater than a threshold exposure time; in response to determining that the first exposure time is equal to or greater than the threshold exposure time, using the threshold exposure time as a second exposure time; capturing a first image with the first exposure time; capturing a second image with the second exposure time; and calculating a third image, wherein calculating the third image comprises combining a difference image with the second image, wherein the difference image is derived from the first image and the second image.

2. The method of claim 1, further comprising: detecting the flicker frequencies of the plurality of light sources; and calculating the greatest common factor of the flicker frequencies of the plurality of light sources.

3. The method of claim 1, wherein calculating the third image further comprises performing a luminance normalization operation on the first image.

4. The method of claim 1, wherein calculating the third image further comprises performing an image processing operation on the first image.

5. The method of claim 1, wherein calculating the third image further comprises performing an image processing operation on the second image.

6. The method of claim 1, wherein the difference image is derived from the first image and the second image by calculating a difference between pixel values of the first image and pixel values of the second image.

7. The method of claim 1, wherein combining the difference image with the second image comprises subtracting the difference image from the second image.

8. A system comprising: a processing system; andat least one computer-readable medium storing instructions which, when executed by the processing system, cause the system to perform operations comprising: determining a first exposure time based on the greatest common factor of flicker frequencies of a plurality of light sources; determining that the first exposure time is equal to or greater than a threshold exposure time; in response to determining that the first exposure time is equal to or greater than the threshold exposure time, using the threshold exposure time as a second exposure time; capturing a first image with the first exposure time; capturing a second image with the second exposure time; and calculating a third image, wherein calculating the third image comprises combining a difference image with the second image, wherein the difference image is derived from the first image and the second image.

9. The system of claim 8, the operations further comprising: detecting the flicker frequencies of the plurality of light sources; and calculating the greatest common factor of the flicker frequencies of the plurality of light sources.

10. The system of claim 8, wherein calculating the third image further comprises performing a luminance normalization operation on the first image.

11. The system of claim 8, wherein calculating the third image further comprises performing an image processing operation on the first image.

12. The system of claim 8, wherein calculating the third image further comprises performing an image processing operation on the second image.

13. The system of claim 8, wherein the difference image is derived from the first image and the second image by calculating a difference between pixel values of the first image and pixel values of the second image.

14. The system of claim 8, wherein combining the difference image with the second image comprises subtracting the difference image from the second image.

15. One or more non-transitory computer-readable media storing instructions which, when executed by at least one processing system, cause a system to perform operations comprising:determining a first exposure time based on the greatest common factor of flicker frequencies of a plurality of light sources; determining that the first exposure time is equal to or greater than a threshold exposure time; in response to determining that the first exposure time is equal to or greater than the threshold exposure time, using the threshold exposure time as a second exposure time; capturing a first image with the first exposure time; capturing a second image with the second exposure time; and calculating a third image, wherein calculating the third image comprises combining a difference image with the second image, wherein the difference image is derived from the first image and the second image.

16. The one or more non-transitory computer-readable media of claim 15, the operations further comprising: detecting the flicker frequencies of the plurality of light sources; and calculating the greatest common factor of the flicker frequencies of the plurality of light sources.

17. The one or more non-transitory computer-readable media of claim 15, wherein calculating the third image further comprises performing a luminance normalization operation on the first image.

18. The one or more non-transitory computer-readable media of claim 15, wherein calculating the third image further comprises performing an image processing operation on the first image and performing an image processing operation on the second image.

19. The one or more non-transitory computer-readable media of claim 15, wherein the difference image is derived from the first image and the second image by calculating a difference between pixel values of the first image and pixel values of the second image.

20. The one or more non-transitory computer-readable media of claim 15, wherein combining the difference image with the second image comprises subtracting the difference image from the second image.

Images