Detection of light source distortion in imaging systems

Abstract

This invention discloses an imaging system comprising a plurality of optical sensors arranged in an array having multiple rows and columns on an integrated circuit. The system includes an interface for communicating with the plurality of optical sensors, a memory for storing operation instructions, and a processing circuit system configured to sample an image using the plurality of optical sensors in a first mode and to sequentially sample at least a portion of the image on a row-by-row basis at a predetermined sampling rate in a second mode to generate a row-by-row sample output. The processing circuit system is further configured to initiate sampling of at least some rows of the plurality of rows of the optical sensors using different timestamps.

Description

Background of the Invention

[0002] Invention Technology

[0003] This invention relates generally to digital imaging, and more specifically to compensating for light source distortion using spectral sensors and interference-based filters.

[0004] Digital imaging has always had a profound impact on the quality and usability of camera technology. At the same time, camera consumers' expectations are constantly rising, especially for cameras embedded in modern smartphones. For example, automatic white balance improves the quality of camera images by compensating for the distorting effects of various light sources on camera output.

[0005] In some digital imaging applications, light source distortion originates from artificial illumination sources with substantially definable temporal light modulation or periodic intensity variations (such as flicker). The detection and classification of light distortion can be used to improve the performance of digital imaging systems.

[0006] Brief description of several views in the diagram

[0007] Figure 1 An exemplary image sensor array having pixels arranged in rows and columns according to the present invention is shown;

[0008] Figure 2 An exemplary dual-modal image sensor array having pixels configured in rows and columns according to the present invention is shown;

[0009] Figure 3A and Figure 3B The sequence and timing of an exemplary flicker detection operation in an image sensor according to the present invention are shown.

[0010] Figure 4A An overview of an exemplary flicker detection method in an image sensor according to the present invention is shown;

[0011] Figure 4B This illustrates the phase discontinuity caused by flicker detection in an image sensor according to the present invention;

[0012] Figure 4C An overview of an exemplary flicker detection method in an image sensor according to the present invention is shown;

[0013] Figure 4D The flicker detection output from an image sensor according to the present invention is shown;

[0014] Figure 4E The flicker detection output of the image sensor according to the present invention using the 2D Fast Fourier Transform method is shown;

[0015] Figure 4FAnother flicker detection output of the image sensor according to the present invention using a 2D Fast Fourier Transform method is shown;

[0016] Figure 5A An exemplary spectrum of a scintillation source according to the present invention is provided;

[0017] Figure 5B An exemplary extracted spectrum from an image is provided for a scintillation source according to the present invention; and

[0018] Figure 6 An exemplary image sensor array according to the present invention is shown, having two types of pixels (photosensitive elements) arranged in rows and columns. Detailed Implementation

[0019] In various embodiments, small-scale digital imaging systems can be adapted for use in applications where correction for lighting anomalies can be beneficial. Examples of such applications include, but are not limited to, smartphones, high-resolution cameras, camcorders, security cameras, calibration systems, inspection systems, and certain industrial applications.

[0020] Incandescent lamps, fluorescent lamps, and light-emitting diodes (LEDs) each distort the light captured by a digital imaging system. Other light sources (such as sodium streetlights) distort the output of the image sensor enough to make most colors indistinguishable. Additionally, artificial lighting sources used for LiDAR detection and digital displays (such as incandescent lamps, fluorescent lamps, LEDs, pulsed VCSELS, and lasers) can have definable temporal light modulation or periodic intensity variations, such as flicker. In examples, flicker can originate from, for example, the AC component of the power supply or from the inherent refresh rate of the digital imager or display. In various embodiments, flicker is used to identify and / or classify the distinctive characteristics of lighting sources in digital imaging applications. In some embodiments, flicker-related information can be used to provide corrections to camera images, such as color and / or white balance corrections.

[0021] Figure 1 An exemplary image sensor array 10 with pixels (photosensitive elements) configured in rows and columns is shown. In this example, the image sensor array 10 includes pixels in rows 1 to N and columns 1 to M to provide a total of N×M pixels. In the example, the imaging sensor array 10 samples signals from the N×M photosensitive elements to capture light from a spatial region of the scene. The exemplary frame rate of the imaging sensor is in the range of 30-100Hz, which may be insufficient to detect flicker from artificial lighting sources. In the example, light source distortion can include artifacts such as banding, image quality issues, and distortion problems.

[0022] Refer again Figure 1The photosensitive elements of the imaging sensor form rows (as part of a pixel array of rows and columns). In an example, the imaging sensor can be configured to operate in a flicker detection mode. In a specific example, the flicker detection mode can be the primary operation of the image sensor. In a specific example of implementation and operation, each row of optical sensors can be configured to sequentially sample the scene to achieve a row sampling rate sufficient to detect flicker (e.g., in the range of kilohertz (kHz) and megahertz (MHz)). In an example, row-by-row sampling can be referred to as using a "rolling shutter". In a specific example, the imaging system can be configured to capture an image using a rolling shutter while simultaneously detecting flicker sources associated with that image in parallel. In a related example, an imaging system operating with a rolling shutter can be configured to detect flicker sources within a predetermined time period before and / or after capturing the image.

[0023] In the example, sampling of different rows (such as different rows R1-RN) of the image sensor array 10 can be configured to overlap, but at least some rows have start and / or stop times at different timestamps. In the example, readout of image sensor samples in flicker detection mode can be performed as a sequence of row images, where a complete or partial image frame is constructed from consecutive row images.

[0024] In the example, the image sensor can be implemented using a focusing lens. In a specific example, the lens is initially positioned out of focus so that the spatial resolution of the scene is blurred to aid flicker detection. In this example, the lens is then positioned in focus to sample the image. In another example, flicker detection may be negatively affected when the flicker frequency is a multiple of the frame rate. In specific examples of implementation and operation, the image sensor system can be adapted such that the frame rate can vary across a series of images, thereby allowing flicker detection to be decoupled from a given frame rate.

[0025] In the example, flicker can be detected by evaluating the output of the rows from the image sensor as a function of time. Therefore, different rows of the image sensor may be affected by varying intensities from the flicker source when elements in the scene, such as light sources or reflections of light, cause flicker. In the example, this could be due to sampling each row at progressively different time points. In the example, each row can be used to "sample" the flicker source during a time interval. In a specific example of the implementation, flicker from one or more illumination (light) sources can be detected, where the frequency of the light source is calculated by analyzing the row output across time using signal processing methods such as Fast Fourier Transform (FFT), Discrete Fourier Transform (DFT), or other frequency-time algorithms. In the example, the given analysis can be simplified by organizing the data from the row-by-row sampling in two dimensions and applying a two-dimensional Fast Fourier Transform (2D-FFT).

[0026] Figure 2 An exemplary dual-modal image sensor array with pixels (photosensitive elements) arranged in rows and columns is shown. In this example, the dual-modal image sensor array 100 is configured with optical sensor rows 102 and optical sensor columns 104, such that incident light 106 from the sampled scene can be sampled by the dual-modal image sensor array 100. In the exemplary configuration, when the dual-modal image sensor array 100 operates in flicker detection mode, optical sensor rows 102 1-4 can be sampled sequentially. In the example, when the dual-modal image sensor array 100 operates in normal or image capture mode, the dual-modal image sensor array 100 can be configured to sample incident light 106 from the scene in a “global shutter” mode, where the entire frame is captured substantially simultaneously. In a specific example, the dual-modal image sensor array 100 is coupled to a control mechanism, such as a processor, a state machine, or another mechanism that can alternate between operating modes of the dual-modal image sensor array 100. In a specific example, the dual-modal image sensor array 100 can be configured to operate in a flash detection mode in a substantially continuous manner until it is ready to capture an image, at which point the dual-modal image sensor array 100 can switch to a global shutter mode during image capture.

[0027] Figure 3A and Figure 3B The sequence and timing of an exemplary flicker detection operation for image sensing are illustrated. In the example, the read / reset window comprises a row of spectral sensors (in the "x" direction), wherein the read / reset window transitions from row to row (in the "y" direction) over a period of time. In the example, the read / reset window begins with a read line (a dotted dashed line extending along the x-direction) and resets with a reset line (a short dashed line extending along the x-direction). Figure 3B It shows the use of, such as Figure 3A The read / reset window and other reset sequences (identified as reset sequence 130, which includes all sensors between the gray and dark sensors) are sampled line by line over time from multiple rows of sensors. In a specific example, the frame time begins at the end of the reset sequence in row 1 and ends at the beginning of the reset sequence in row n. In another example, the read / reset window includes a row of spectral sensors (in the "x" direction).

[0028] In a specific example of the implementation, a Discrete Fourier Transform (DFT) can be performed on the time row values ​​of the sensor, and the frequency of the DFT can be used to represent the scintillation source (F). flicker In the example, the sampling frequency can be the frame rate (F). frame ), flicker rate (F) flicker ) or (m*F frame +n*Fflicker The multiples and multiples of the frame rate. In another example, when the frame rate is known, the flicker frequency can be determined. In a specific example, the resulting signal can be configured to consider a sequence of single frames or multiple frames.

[0029] Figure 4A An overview of an exemplary flicker detection method in an image sensor is shown. In this example, the amplitude of the flicker source as a function of time is shown at the top, while an exemplary image sensor line-by-line integration of several sampled frames is shown below. In the example, the Discrete Fourier Transform (DFT) can be configured to provide the obtained frequency-based peak values ​​for a given flicker source.

[0030] Figure 4B An overview of an exemplary flicker detection method adapted to compensate for time-based discontinuities is shown. In the example, the row values ​​of different frames from the sampling sensor are organized time-wise for the flicker determination signal. When a frequency-time algorithm is used to determine the flicker frequency using the signal, discontinuities in the signal can manifest as artifacts. Figure 4B In the example, when assembling frames 1-n, the sample intensity is discontinuous at frame boundaries 150-1 to 150-n.

[0031] In specific examples of implementation and operation, discontinuity artifacts can be eliminated or mitigated by separating the row values ​​of different frames into a two-dimensional (2D) matrix, and then performing a 2D Fast Fourier Transform (FFT) or Discrete Fourier Transform (DFT) function on the separated row values. In the example, the effect of flicker can be separated from the artifacts introduced by frame change discontinuities.

[0032] Figure 4C and Figure 4D It shows spanning multiple frames ( Figure 4C Line-by-line flicker detection and Figure 4D The correlation between the obtained images. In this example, image flicker behaves as a periodic function in each frame. In the example, the flicker is out of sync with the frame rate, so the flicker shifts phase between frames. In the example, the flicker offset is reflected as a periodic function under the tilt angle.

[0033] Figure 4E The example illustrates flicker in a two-dimensional (2D) Fast Fourier Transform (FFT). In this example, the effect of flicker is represented as a peak in the 2D FFT at non-zero frequencies. In this example, the flicker frequency can be determined once the DC component is eliminated. The 2D Fourier transform in this example comprises multiple one-dimensional Fourier transforms implemented by first transforming each row (i.e., replacing each row with its 1D Fourier transform). In this example, the first step produces an intermediate "picture" where the horizontal axis is the frequency f and the vertical axis is the frame, such as... Figure 3BThe frame shown. The second step is to apply a 1D Fourier transform to the vertical lines of the intermediate image individually. The new image will then be a 2D Fourier transform of the initial image.

[0034] Figure 4F The diagram illustrates a flicker frequency determined using a 2D FFT. In this example, the peak frequency generated by the 2D FFT can then be used to reduce or eliminate flicker in the image captured by the image sensor. In another example, the peak frequency generated by the 2D FFT can be used to eliminate or reduce flicker in the image sensor itself.

[0035] Figure 5A An exemplary sampling spectrum of a scintillation source is provided. In this example, the sampling spectrum has peaks at various frequencies where the amplitude is measured, generated by harmonics. In this example, F frame =Frame rate=31Hz, and F f1icker =Flicker rate=45Hz, where the expected peak is located at m*F frame +n*F flicker Above. In the example, the 31Hz signal is higher than other signals because F frame Dominant.

[0036] Figure 5B Provided flickering sources (such as Figure 5A An exemplary extracted spectrum (from the scintillation source provided in the image) is shown. In this example, the extracted spectrum from the image is in m*F... frame +n*F flicker The frequency peak is represented at [value], where m and n are integer values. In the example, the peak is located at + / -76Hz, + / - [value].

[0037] In a specific example of implementation and operation, the red, green, and blue (RGB) pixels of an RGB sensor can be separated for flicker detection. In this example, samples from each red, green, and blue pixel can be analyzed individually to provide additional information for detection and classification of a given light source. Reference Figure 2 In this example, the image sensor (such as the dual-modal image sensor array 100) can be configured as a spectral sensor to provide additional spectral information about the light source, thereby allowing more information to be used for detection and classification. In a specific example, the dual-modal image sensor array 100 can be a hyperspectral image sensor.

[0038] Refer again Figure 1 and Figure 2 In a specific example of the implementation, all pixels in a given row can be summed to provide information for flicker analysis of a substantially complete image. In another alternative example, pixels in a region of interest in the image can be used to provide information for flicker analysis and / or for determining the spatial location of one or more flicker sources in the image.

[0039] In specific examples of implementation and operation, unprocessed or “raw” data from a photosensitive element can be used for flicker analysis. In alternative examples, the raw data can be preprocessed to allow for temporal variations (such as movement during image capture) and / or frequency-based spatial anomalies (such as stripes and line patterns) can be eliminated before the preprocessed image or region of interest in the image. In a related example, two instances of a row can be processed during subtraction to eliminate unwanted spatial information. In another related example, two consecutive time-series images can undergo a subtraction process to eliminate spatial anomalies.

[0040] In another example, the image sensor can be configured to include a readout mode operable to eliminate irrelevant and / or unwanted frequencies during the scrambling process. In a correlation example, the image sensor is configured to first read line by line from the top of the sensor, then line by line from the bottom of the sensor, to remove irrelevant and / or unwanted frequencies. In the correlation example, filtering operations can be used to extract relevant / desired frequencies from the readout.

[0041] In a specific example of operation, the image sensor is configured to operate at 100 frames per second (fps) and detect flicker occurring at 49 Hz using a line rate of 10 kHz. In a related example, the detected flicker frequency (e.g., 49 Hz) can be used to correct the image output from the image sensor. In another example, the detected flicker frequency can be used to adjust an image sensor adapted to provide a variable frame rate, thereby allowing the image sensor to reduce or eliminate the effect of the flicker source. In yet another example, the detected flicker can be used to adjust the amplitude of the image or the affected portion of the image. In yet another example, the detected flicker can be used to adjust a color balance algorithm in an image processor. In yet another example, the image can be adjusted for spatial effects based on the determined spatial location of the flicker source. In yet another example, adjustments can be made based on the wavelength range of the flicker source so that the adjustment is limited to the relevant colors affected by the flicker source. In an example of implementation, a processor is included in the imaging system to calculate the flicker effect and then correct the image and / or pixel output.

[0042] In another specific example of implementation and operation, the imaging system may be configured with two imaging sensors operable to provide parallel imaging and scintillation correction. In this example, a first imaging sensor is configured to operate in a scintillation detection mode, while a second imaging sensor is configured to operate in an imaging mode. In this example, scintillation information generated by the first imaging sensor can be used to correct image acquisition by the second imaging sensor. In a related example, the scintillation information generated by the first imaging sensor can be used to adjust the settings of the second imaging sensor to reduce or eliminate the undesirable effects of scintillation in the resulting captured image. In yet another example, multiple imaging sensors are used in either scintillation or imaging mode to further improve scintillation correction. In a specific example, a first imaging sensor among the multiple imaging sensors may be configured to have a higher spatial resolution than one or more other imaging sensors in an imaging system comprising multiple imaging sensors.

[0043] In this example, the image sensing system includes a focusing lens. In this example, the focusing lens is used in defocus mode so that the spatial resolution of the scene is blurred to aid in flicker detection. In a related example, the focusing lens is used in focus mode to capture an image.

[0044] In a specific example of the implementation, the imaging system is configured to perform flicker detection using a portion of the rows in the imaging sensor. In an alternative example, the imaging system is configured to perform flicker detection using all rows in the imaging sensor. In this example, the imaging sensor is configured to implement a flicker detection mode that first uses all rows and then uses a subset of rows read sequentially, thereby allowing sampling rates in the kHz-MHz range while preserving some spatial information. In this example, the flicker mode allows for the determination of both the frequency and location of the light source presenting the flicker. In a related example, different flicker sources can be found by employing digital signal processing techniques. In another example of the implementation, the imaging system is configured to use multiple row rates to optimize the measurement time required to provide the desired flicker frequency detection resolution.

[0045] Flicker detection can be adversely affected when the flicker frequency is a multiple of the selected frame rate. In an example of the implementation, the imaging system is configured to include a variable frame rate for a series of captured images. In a related example, flicker detection may be disadvantageous when a given flicker frequency is a multiple of the frame rate. In another related example, multiple frame rates can be used to prevent spatial frequencies associated with the scene or image from being mistakenly detected as flicker frequencies of a light source.

[0046] In a specific example of the implementation, the imaging system includes a plurality of optical sensors on an integrated circuit, wherein the plurality of optical sensors are arranged in an array, and the array includes a plurality of rows and a plurality of columns. In this example, a processing module is operatively coupled to each of the plurality of optical sensors, and when operable within the imaging system, the processing module is configured to sample an image in a first mode and sequentially sample at least a portion of the image on a row-by-row basis at a predetermined sampling rate in a second mode to produce a row-by-row sample output, wherein the processing module is further configured to initiate sampling of at least some of the plurality of rows using different timestamps.

[0047] In the example, the imaging system includes an interface, a memory storing operation instructions, and a second processing module operatively coupled to the interface and the memory. When operable within the imaging system based on the operation instructions, the second processing module is configured to determine periodic intensity variations of a light source associated with an image. These periodic intensity variations are determined by analyzing multiple line-by-line sample outputs based on different timestamps, and the analysis is performed using a frequency-time algorithm. In the example, the frequency-time algorithm is based on at least one of Fast Fourier Transform (FFT) and Discrete Fourier Transform (DFT). In a specific example of operation, the results of the frequency-time algorithm can be compared with known scintillation sources for classification purposes and / or scintillation source attenuation.

[0048] When illuminated by a light source, a photosensitive element (e.g., a photodiode) can necessarily generate photoelectrons (photonic electrons) and photoholes (photonic holes) in the depletion region of the photosensitive element. Exemplary image sensors are conventionally configured to measure light by collecting the photonic electrons generated in the photosensitive element. While the photonic holes generated in the photosensitive element can be evacuated to the ground terminal of the circuit, the associated hole current also represents the collected photonic electrons. In the example, the image sensor is configured to collect substantially all the hole currents of the photosensitive element in the image sensor or a portion of the image sensor and measure them continuously over time. In this example, the resulting time measurement may include full image illumination as well as a flicker component. In the example, the imaging system may be configured to perform frequency analysis of the collected hole currents to provide a flicker frequency. In a related example, the performance of the image sensor's normal function is unaffected because the photosensitive element is still able to detect an image based on the individually collected photocurrents. In a specific example of an implementation, each photodiode in the image sensor may include a photonic hole-side terminal connected to a readout circuit for measuring the hole currents of a set of photodiodes. In a relevant example, the photonic electronic side of each photodiode can be connected to a readout circuit to provide a signal for each individual photodiode.

[0049] In a specific example of implementation and operation, a method performed by one or more processing modules of an imaging system includes: sampling an image from a plurality of optical sensors arranged in an array; and sampling the photocurrents (photonic electrons or photonic holes) of at least some of the optical sensors within a predetermined time period to generate a photocurrent determined for at least some of the optical sensors, wherein the sampling includes sampling the photohole current of each of the at least some optical sensors. The method continues to determine a periodic intensity variation of a light source associated with the image based on the determined photocurrents of the at least some of the optical sensors, wherein the periodic intensity variation is determined by time-frequency analysis of the photocurrents of the at least some of the optical sensors within the predetermined time period.

[0050] Figure 6 An exemplary image sensor array with two types of pixels (photosensitive elements) arranged in rows and columns is shown. In this example, the staggered image sensor array 200 is configured with optical sensor rows 102 and optical sensor columns 104, such that incident light 106 from the sampled scene can be sampled by the staggered image sensor array 200. In this example, the staggered image sensor array 200 includes at least two types of sensors: a first sensor type configured for image acquisition and a second sensor type (flicker sensor 216) configured as a flicker sensor.

[0051] In this example, the scintillation sensor 216 is interleaved among conventional sensors in the interleaved image sensor array 200. In this example, the scintillation sensor 216 may be configured with specific readout operations for the interleaved image sensor array 200. In an alternative example, the scintillation sensor 216 may be configured to perform readout during normal sampling of the interleaved imaging sensor array 200 with image sensors. In this example, the scintillation sensor 216 is configured to provide optimized performance, such as, for example, to accommodate increased dynamic range.

[0052] In the example, scintillation sensor 216 can be used for scintillation readout, and a line-by-line Fast Fourier Transform (FFT) can be used to extract the scintillation component. In this example, scintillation sensor 216 is configured without optical filters (i.e., they are “transparent” sensors). In another example, scintillation sensor 216 uses one or more optical filters specific to a target wavelength range of the intended scintillation source.

[0053] In the example, the image sensor includes an array of standard or ordinary sensors and multiple scintillator sensors optimized for detecting flicker components in the image. In the example, the scintillator sensors are interspersed among the standard sensors. In a specific example, the scintillator sensors are adapted to operate at a readout rate (F) higher than that of the normal sensors. pixel Higher frequency readout rate (F read_flicker The scintillation sensor can be adapted to have higher sensitivity to accommodate higher frequency readout rates (F). read_flicker In a representative example, (F) pixel ) can be 100Hz, while (F read_flicker It is 500Hz.

[0054] In an example of implementation and operation, the imaging system includes: a plurality of first optical sensors on an integrated circuit, wherein the plurality of first optical sensors are arranged in an array; and a plurality of second optical sensors on the integrated circuit, wherein the plurality of second optical sensors are distributed among the plurality of first optical sensors. In the example, a processing module is operatively coupled to each of the plurality of first optical sensors and the plurality of second optical sensors, and when operable within the imaging system, the processing module is configured to sample a scene using the plurality of first optical sensors at a first frequency to generate a first image and to sample the scene using the plurality of second optical sensors at a second frequency to generate a plurality of time-based samples of the plurality of second optical sensors, wherein the second frequency is higher than the first frequency. In an example of operation, the plurality of time-based samples of the plurality of second optical sensors can be used to reduce or eliminate light source distortion in the first image.

[0055] In another example of implementation and operation, the imaging system includes: a plurality of first optical sensors on an integrated circuit, wherein the plurality of first optical sensors are arranged in an array; and a plurality of second optical sensors on the integrated circuit, wherein the plurality of second optical sensors are distributed among the plurality of first optical sensors and arranged in a plurality of rows and columns. In the example, a processing module is operatively coupled to each of the plurality of first optical sensors and the plurality of second optical sensors, and when operable within the imaging system, the processing module is configured to sample a scene at a first frequency using the plurality of first optical sensors to generate a first image and to sequentially sample the scene on a row-by-row basis at a predetermined sampling rate using the plurality of second optical sensors to generate a row-by-row sample output, wherein the processing module is further configured to initiate sampling for at least some of the rows using different timestamps.

[0056] It should be noted that terms such as bit stream, stream, signal sequence, etc. (or their equivalents) used herein have been used interchangeably to describe digital information whose content corresponds to any of a number of desired types (e.g., data, video, speech, text, graphics, audio, etc., any of which may be collectively referred to as 'data').

[0057] As may be used herein, the terms “substantially” and “approximately” provide an industry-acceptable tolerance for the correlation between their respective terms and / or items. For some industries, the industry-acceptable tolerance is less than one percent, while for others it is 10 percent or higher. Other examples of industry-acceptable tolerance ranges are in the range of less than one percent to fifty percent. Industry-acceptable tolerances correspond to, but are not limited to, component values, variations in integrated circuit process technology, temperature variations, rise and fall times, thermal noise, dimensions, transmission errors, dropped data packets, temperature, pressure, material composition, and / or performance metrics. Within an industry, tolerance variations in acceptable tolerances can be greater than or less than a certain percentage level (e.g., a size tolerance of less than + / - 1%). A certain relativity between items can range from a difference of less than one percentage level to several percentages. Other relativity between items can range from a difference of several percentages to a large difference.

[0058] As may also be used herein, the terms “configured to,” “operably connected to,” “connected to,” and / or “connected” include direct connections between items and / or indirect connections between items via intermediate items (e.g., items that include, but are not limited to, components, elements, circuits, and / or modules), wherein, for instances of indirect connections, the intermediate items do not modify information about the signal but may adjust its current level, voltage level, and / or power level. As may be further used herein, inferred connection (i.e., one element is inferredly connected to another element) includes both direct and indirect connections between two items in the same manner as “connected to.”

[0059] As may be used further herein, the terms “configured to,” “operable to,” “connected to,” or “operably connected to” indicate that an item includes one or more of power connections, inputs, outputs, etc., for performing one or more of their respective functions when activated, and may further include inferred connections to one or more other items. As may also be used further herein, the term “associated with” includes direct and / or indirect connections to individual items, and / or one item embedded in another.

[0060] As may be used herein, the term "favorable comparison" indicates that a comparison between two or more items, signals, etc., provides the desired relationship. For example, when the desired relationship is that signal 1 has a larger value than signal 2, a favorable comparison is achieved when the value of signal 1 is greater than the value of signal 2 or when the value of signal 2 is less than the value of signal 1. As may be used herein, the term "unfavorable comparison" indicates that a comparison between two or more items, signals, etc., fails to provide the desired relationship.

[0061] As may be used herein, one or more claims may contain the general phrase “at least one of a, b, and c” or a specific form of the general phrase “at least one of a, b, or c”, having more or fewer elements than “a,” “b,” and “c.” In either wording, the phrase is interpreted the same way. Specifically, “at least one of a, b, and c” is equivalent to “at least one of a, b, or c”, and both mean a, b, and / or c. As examples, it means: “only a,” “only b,” “only c,” “a” and “b,” “a” and “c,” “b” and “c,” and / or “a,” “b,” and “c.”

[0062] As may also be used herein, the terms “processing module,” “processing circuit,” “processor,” “processing circuit system,” and / or “processing unit” can refer to a single processing device or multiple processing devices. This processing device can be a microprocessor, microcontroller, digital signal processor, microcomputer, central processing unit, field-programmable gate array, programmable logic device, state machine, logic circuit system, analog circuit system, digital circuit system, and / or any device that manipulates signals (analog and / or digital) based on the hard decoding of circuit systems and / or operating instructions. A processing module, module, processing circuit, processing circuit system, and / or processing unit can be or further include memory and / or integrated memory elements, which can be a single memory device, multiple memory devices, and / or an embedded circuit system of another processing module, module, processing circuit, processing circuit system, and / or processing unit. This memory device can be read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, and / or any device storing digital information. It should be noted that if a processing module, module, processing circuit, processing circuit system, and / or processing unit comprises more than one processing device, the processing devices can be centrally located (e.g., directly connected together via wired and / or wireless bus structures) or distributed (e.g., cloud computing indirectly connected via a local area network and / or wide area network). It should also be noted that if a processing module, module, processing circuit, processing circuit system, and / or processing unit implements one or more of its functions via a state machine, analog circuit system, digital circuit system, and / or logic circuit system, then a memory and / or memory element storing the corresponding operation instructions can be embedded within or outside the circuit system including the state machine, analog circuit system, digital circuit system, and / or logic circuit system. It should also be noted that the memory element can store, and the processing module, module, processing circuit, processing circuit system, and / or processing unit executes, hard-decoded and / or operation instructions corresponding to at least some of the steps and / or functions shown in one or more figures. This memory device or memory element can be included in the article of manufacture.

[0063] The foregoing has described one or more embodiments by way of illustrating the execution of specified functions and their relationships. For ease of description, the boundaries and order of these functional building blocks and method steps are arbitrarily defined herein. Alternative boundaries and orders may be defined, provided that the specified functions and relationships are properly performed. Therefore, any such alternative boundaries or orders are within the scope and spirit of the claims. Furthermore, for ease of description, the boundaries of these functional building blocks are arbitrarily defined. Alternative boundaries may be defined, provided that certain essential functions are properly performed. Similarly, flowchart blocks may also be arbitrarily defined herein to illustrate certain essential functionalities.

[0064] Within the scope of use, flowchart block boundaries and order can be defined in other ways while still performing certain important functionalities. Therefore, this alternative definition of functional building blocks and flowchart blocks and their order is within the scope and spirit of the claims. Those skilled in the art will also recognize that the functional building blocks and other illustrative blocks, modules, and components described herein can be implemented as shown, or by discrete components, application-specific integrated circuits, processors executing appropriate software, or any combination thereof.

[0065] Additionally, the flowchart may include "Start" and / or "Continue" instructions. The "Start" and "Continue" instructions reflect that the presented steps may optionally be incorporated into or otherwise combined with one or more other routines. Furthermore, the flowchart may include "End" and / or "Continue" instructions. The "End" and / or "Continue" instructions reflect that the presented steps may end as described and shown, or may optionally be incorporated into or otherwise combined with one or more other routines. In this context, "Start" indicates the beginning of the first presented step, which may be preceded by other activities not specifically shown. Furthermore, the "Continue" instruction reflects that the presented steps may be performed multiple times, and / or may be followed by other activities not specifically shown. Moreover, although the flowchart indicates a specific order of steps, other orders are equally possible, as long as the principle of cause and effect is maintained.

[0066] This document uses one or more embodiments to illustrate one or more aspects, features, concepts, and / or instances. Physical embodiments of devices, articles of manufacture, machines, and / or processes may include one or more of the aspects, features, concepts, instances, etc., described with reference to one or more embodiments discussed herein. Furthermore, throughout the figures, embodiments may incorporate functions, steps, modules, etc., with the same or similar names that may use the same or different reference numerals, and thus, said functions, steps, modules, etc., may be the same or similar functions, steps, modules, etc., or may not be the same.

[0067] Unless specifically stated otherwise, signals arriving at or from any of the elements in any of the figures presented herein and / or between elements may be analog or digital, continuous-time or discrete-time, and single-ended or differential. For example, if a signal path is shown as a single-ended path, it also represents a differential signal path. Similarly, if a signal path is shown as a differential path, it also represents a single-ended signal path. Although one or more specific architectures are described herein, other architectures may also be implemented using one or more data buses not explicitly shown, direct connections between elements, and / or indirect connections between other elements as known to those skilled in the art.

[0068] The term "module" is used in the description of one or more embodiments. A module implements one or more functions via means such as a processor or other processing device or other hardware, which may include or operate in association with a memory storing operation instructions. A module may operate independently and / or in combination with software and / or firmware. Also as used herein, a module may contain one or more submodules, and each submodule may be one or more modules.

[0069] As may be further used herein, a computer-readable storage device includes one or more memory elements. A memory element may be a single memory device, multiple memory devices, or a set of memory locations within a memory device. This memory device may be read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, and / or any means of storing digital information. The memory device may be in the form of solid-state memory, hard disk drive memory, cloud storage, USB flash drive, server memory, computing device memory, and / or other physical media for storing digital information.

[0070] While specific combinations of various functions and features of the one or more embodiments described herein have been expressly described, other combinations of these features and functions are equally possible. This disclosure is not limited to the specific instances disclosed herein, and such other combinations are expressly incorporated.

Claims

1. An imaging system comprising: Multiple optical sensors on an integrated circuit, wherein the multiple optical sensors are arranged in an array, wherein the array includes multiple rows and multiple columns; Multiple optical filters are disposed in a layer and have corresponding top and bottom surfaces, wherein the bottom surfaces of the multiple optical filters are located near the proximal ends of the plurality of optical sensors, and each optical filter in one set of optical filters is configured to allow light in a different wavelength range to pass through. An interface configured to interact and communicate with the plurality of optical sensors; The memory stores operation instructions; A processing circuitry system operatively coupled to the interface and the memory, wherein the processing circuitry system is configured to execute the operation instructions to: The image is sampled in the first mode; as well as In the second mode, at least a portion of the image is sampled sequentially on a line-by-line basis at a predetermined sampling rate to produce a line-by-line sample output, wherein the processing circuitry is further configured to initiate sampling of the plurality of lines of the optical sensor sequentially using time-shifted timestamps.

2. The imaging system of claim 1, wherein the processing circuitry is further configured to execute the operation instructions to determine intensity variations that repeat at substantially regular intervals associated with the image based on the line-by-line sample output.

3. The imaging system of claim 2, wherein the processing circuitry is further configured to execute the operation instructions to determine the intensity change using a frequency-time algorithm.

4. The imaging system according to claim 3, wherein the frequency-time algorithm is selected from the group consisting of: Fast Fourier Transform (FFT); Discrete Fourier Transform (DFT); and 2D Fast Fourier Transform (2D-FFT).

5. The imaging system of claim 1, wherein a set of optical filters comprises a plurality of optical filters arranged in a spatial pattern, wherein the spatial pattern includes regions without optical filters.

6. The imaging system of claim 1, wherein the line-by-line sample output indicates intensity variations that repeat at substantially regular intervals.

7. An imaging system comprising: A plurality of optical sensors, wherein the plurality of optical sensors are arranged in an array; A second plurality of optical sensors, wherein the second plurality of optical sensors are distributed among the first plurality of optical sensors; Processing module, wherein the processing module is configured to receive outputs from each of the first plurality of optical sensors and the second plurality of optical sensors, wherein the processing module is further configured to: The scene is sampled at a first frequency using the first plurality of optical sensors to generate a first image; The scene is sampled at a second frequency within a time period T using the second plurality of optical sensors to generate the intensity within T, wherein the second frequency is higher than the first frequency; as well as The optical distortion associated with the scene is determined based on the intensity within T.

8. The imaging system of claim 7, wherein the first plurality of optical sensors and the second plurality of optical sensors are located on the same integrated circuit.

9. The imaging system of claim 7, wherein the first plurality of optical sensors and the second plurality of optical sensors have corresponding photosensitive sensitivities, wherein the photosensitive sensitivities of the second plurality of optical sensors are higher than the photosensitive sensitivities of the first plurality of optical sensors.

10. The imaging system of claim 7, wherein the first frequency is between 80 Hz and 120 Hz, and the second frequency is between 250 Hz and 650 Hz.

11. The imaging system of claim 7, further comprising: Multiple optical filters are disposed in a layer and have corresponding top and bottom surfaces, wherein the bottom surfaces of the multiple optical filters are located near the proximal ends of the plurality of optical sensors, wherein each optical filter in one set of optical filters is configured to allow light in a different wavelength range to pass through, wherein one or more filters in one set of optical filters are adapted to allow light in a wavelength range specific to one or more of the plurality of optical sensors to pass through.

12. The imaging system of claim 11, wherein the processing module is further configured to sequentially sample the scene on a line-by-line basis to produce a line-by-line sample output.

13. The imaging system of claim 11, wherein the processing module is further configured to sample the scene at a predetermined sampling rate.

14. The imaging system of claim 11, wherein the array comprises a plurality of rows and a plurality of columns of optical sensors, and wherein the processing module is further configured to initiate sampling of at least some of the plurality of rows using a plurality of different timestamps.

15. The imaging system of claim 7, wherein each of the second plurality of optical sensors is interleaved among the plurality of optical sensors of the first plurality of optical sensors.

16. The imaging system of claim 15, wherein the processing module is further configured to sample the first plurality of optical sensors sequentially using a first sequence and to sample the second plurality of optical sensors sequentially using a second sequence.

17. The imaging system of claim 11, wherein one or more of the second plurality of optical sensors are configured to have a greater dynamic range than any of the first plurality of optical sensors.

18. A method for imaging a scene, comprising: The scene is sampled using a digital imager in a first mode, wherein the digital imager includes a plurality of optical sensors arranged in an array, wherein the array includes a plurality of rows and a plurality of columns of optical sensors; as well as In the second mode, at least a portion of the scene is sampled sequentially on a line-by-line basis at a predetermined sampling rate to produce a line-by-line sample output, wherein at least some of the multiple lines of the optical sensor are sampled using different timestamps; as well as The frequency-time algorithm is used to determine the intensity changes of the row-by-row sample output.

19. The method of claim 18, wherein determining the intensity change comprises: Determine whether the intensity changes occur at regular intervals.

20. The method of claim 19, wherein the digital imager is adapted to include a plurality of additional optical sensors, wherein the sequential sampling of at least a portion of the scene on a line-by-line basis is performed using the additional optical sensors.

21. The method of claim 18, wherein the second mode includes sampling the photohole current of at least some of the plurality of optical sensors.

22. The method of claim 21, further comprising determining the intensity change based on the photohole current.

23. The method of claim 21, wherein the intensity change is further based on a frequency-time algorithm.

24. The method of claim 18, further comprising: At least a portion of the scene is sampled sequentially on a line-by-line basis at another predetermined sampling rate.

25. The method of claim 18, wherein the digital imager comprises multiple sets of optical filters overlapping the plurality of optical sensors.

26. The method of claim 18, wherein each of the optical filters in the set of optical filters is configured to allow light in a different wavelength range to pass through.

Images