Asynchronous integration in rolling shutter image sensors

Abstract

The invention relates to asynchronous integration in a rolling shutter image sensor. According to an embodiment, a method of capturing an image frame using a pixel array of an image sensor is provided. The method includes sequentially resetting each pixel row of the pixel array; sequentially reading out each pixel row of the first subset of the pixel rows during the readout phase such that the integration time of each pixel of the first subset of the pixel rows is equal to the fixed integration time; and globally resetting a pixel of the pixel array in response to detecting an ambient light level exceeding a threshold level, the global resetting of the pixel occurring prior to reading out a second subset of pixel rows, each pixel row of the second subset of pixel rows having a variable integration time less than the fixed integration time, and the variable integration time begins from a global reset of the pixel.

Description

Technical Field

[0001] The present disclosure relates generally to image sensors and, in particular embodiments, to high dynamic range rolling shutter image sensors using asynchronous integration. Background Art

[0002] Typically, still and video cameras (e.g., digital cameras equipped with complementary metal oxide semiconductor (CMOS) sensors) employ a mechanism known as a rolling shutter to capture images. As opposed to capturing an entire scene at a uniform moment in time, a rolling shutter acquires images sequentially, scanning the image line by line. This procedural scanning is not limited to still photography, but is inherent in cameras and affects the individual frames that make up a video storyboard.

[0003] Under unique conditions, typically during the readout phase of the sensor, there may be significant and sudden fluctuations in illumination. During the readout phase in a frame, such a sudden burst of intense light may overload a row of pixels, resulting in saturation of the pixel output. The resulting image typically exhibits overexposure or saturation in the areas affected by the light burst. This overexposure typically manifests as "blown out" areas within the image, where detail is lost due to the overwhelming light intensity, and the affected pixels or rows of pixels are captured as white or very bright areas. In contrast, the portion of the image read out before the light burst will typically be correctly exposed and appear normal.

[0004] For example, in a scene where an image of a scene is being captured and a flash suddenly occurs. If the camera utilizes a rolling shutter, the resulting image may show the object normally in one part, and excessively bright or distorted in another part, creating a distortion commonly referred to as the "rolling shutter effect." This can be particularly problematic in scientific or professional photography where precision and detail are extremely important, but it can also affect general photography, affecting the aesthetic quality and authenticity of the captured image.

[0005] Likewise, when a camera, such as a security camera, utilizes a rolling shutter, the impact on the captured frames can compromise the quality and reliability of the recorded footage. This can potentially hinder the primary purpose of a security camera, which is to monitor and record clear and accurate visual information about the environment. For example, areas affected by sudden bursts of light in the frame will likely be overexposed, resulting in loss of detail in those areas. Important information or objects in those areas may become unidentifiable. Appropriate solutions to these challenges are desired. Summary of the invention

[0006] Technical advantages are generally achieved by embodiments of the present disclosure, which describe a high dynamic range rolling shutter image sensor using asynchronous integration.

[0007] A first aspect relates to a method for capturing an image frame using a pixel array of an image sensor. The method includes sequentially resetting each pixel row of the pixel array from a first pixel row to a last pixel row; sequentially reading out each pixel row of a first subset of pixel rows during a readout phase, such that an integration time of each pixel of the first subset of pixel rows is equal to a fixed integration time; and globally resetting pixels of the pixel array in response to detecting that an ambient light level exceeds a threshold level, the global resetting of the pixels occurring before reading out a second subset of pixel rows, each pixel row in the second subset of pixel rows having a variable integration time less than the fixed integration time, and the variable integration time starting from the global resetting of the pixels.

[0008] A second aspect relates to an imaging system. The imaging system includes a pixel array; a non-transitory memory storage device including instructions; and a processor in communication with the non-transitory memory storage device and the pixel array. The processor is configured to execute the instructions to sequentially reset each pixel row of the pixel array from a first pixel row to a last pixel row; sequentially read out each pixel row of a first subset of pixel rows during a readout phase, so that the integration time of each pixel of the first subset of pixel rows is equal to a fixed integration time; and globally reset the pixels of the pixel array in response to detecting that the ambient light level exceeds a threshold level, the global reset of the pixels occurring before the second subset of pixel rows is read out, each pixel row in the second subset of pixel rows having a variable integration time less than the fixed integration time, and the variable integration time starting from the global reset of the pixels.

[0009] A third aspect relates to a computer-implemented method for capturing an image frame using a pixel array of an image sensor. The computer-implemented method includes sequentially resetting each pixel row of the pixel array from a first pixel row to a last pixel row; sequentially reading out each pixel row of a first subset of pixel rows during a readout phase such that an integration time of each pixel of the first subset of pixel rows is equal to a fixed integration time; and globally resetting pixels of the pixel array in response to detecting that an ambient light level exceeds a threshold level, the global resetting of the pixels occurring before reading out a second subset of pixel rows, each pixel row in the second subset of pixel rows having a variable integration time less than the fixed integration time, and a variable integration time starting from the global resetting of the pixels.

[0010] Embodiments may be implemented in hardware, software or any combination thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, in which:

[0012] Figure 1 is a block diagram of an embodiment imaging system;

[0013] Figure 2 is a timing diagram of the operation of an imaging system including a rolling shutter;

[0014] Figure 3 is a timing diagram of the operation of a conventional imaging system incorporating a rolling shutter that responds to sudden surges in light levels during readout of a row of pixels;

[0015] Figure 4 is a timing diagram of the operation of the imaging system in response to a sudden surge in light level;

[0016] Figure 5 is a timing diagram of the operation of the imaging system in response to a sudden surge in light level;

[0017] Figure 6 is a flow chart of an embodiment method for capturing frames using an imaging system in response to a sudden surge in light levels; and

[0018] Figure 7 is a flow chart of an embodiment method for capturing frames using an imaging system in response to a sudden surge in light levels. DETAILED DESCRIPTION

[0019] The present disclosure provides many applicable inventive concepts that can be implemented in a variety of specific contexts. The specific embodiments are merely illustrations of specific configurations and do not limit the scope of the claimed embodiments. Unless otherwise stated, features from different embodiments may be combined to form additional embodiments. Various embodiments are shown in the accompanying drawings, in which the same parts and elements are represented by the same reference numerals, and repeated descriptions are omitted for brevity.

[0020] The changes or modifications described in one embodiment may also be applied to other embodiments. In addition, various changes, substitutions and alterations may be made without departing from the spirit and scope of the present disclosure as defined by the appended claims.

[0021] Although aspects of the present invention are primarily described in the context of video cameras, it should also be understood that aspects of the invention may also be applied to still cameras.

[0022] In an embodiment, an imaging system and corresponding method for capturing an image frame using a pixel array of an image sensor are provided. The method includes sequentially resetting each pixel row of the pixel array from a first pixel row to a last pixel row; sequentially reading out each pixel row of a first subset of pixel rows during a readout phase so that an integration time of each pixel of the first subset of pixel rows is equal to a fixed integration time; and globally resetting the pixels of the pixel array in response to detecting that an ambient light level exceeds a threshold level. The global reset of the pixels occurs before reading out a second subset of pixel rows, and each pixel row in the second subset of pixel rows has a variable integration time that is less than the fixed integration time, and the variable integration time starts from the global reset of the pixels.

[0023] Figure 1 A block diagram of an embodiment imaging system 100 is shown. The imaging system 100 can be used to capture, process, and represent visual information in the form of images or videos. The imaging system 100 can be used in various applications, such as photography, videography, security applications, etc. The imaging system 100 can be included in a host device, such as a digital camera, a digital video camera, a smart phone, a tablet computer, a laptop computer, a 3D printer, etc.

[0024] As shown, imaging system 100 includes an ambient light sensor (ALS) 102, an image sensor 104, an image processor 106, and a memory 108, which may (or may not) be arranged as shown. In an embodiment, imaging system 100 is configured to efficiently capture and process images, thereby adapting to various light conditions to optimize captured image data.

[0025] In an embodiment, the ambient light sensor 102 is configured to measure the intensity of the surrounding light. This information helps adjust the "white balance" of the image, ensuring that the display is fine-tuned to present the best looking picture. By using a common centroid arrangement for the various colors, greater accuracy can be achieved, thereby reducing any spatial sensitivity.

[0026] In an embodiment, ambient light sensor 102 includes a charge integrator 110 and a photodetector 111, such as a photodiode, phototransistor or other light-sensitive element. Photodetector 111 converts light energy into an electrical signal, which is fed to charge integrator 110. Charge integrator 110 (which includes, for example, an operational amplifier and a feedback capacitor) integrates the charge over time (e.g., fixed) to produce a voltage. In an embodiment, a transimpedance amplifier (not shown) including, for example, an operational amplifier and a feedback resistor provides a light-sensitive readout.

[0027] When light strikes the sensor, the photosensitive element of the charge integrator 110 generates a current or voltage proportional to the intensity of the incident light. In an embodiment, the charge integrator 110 includes an optical filter 112 (optical filter) to narrow the sensitivity to the visible light spectrum. The optical filter 112 may include an infrared filter to minimize interference from infrared light, thereby focusing primarily on the visible light spectrum relevant to imaging applications.

[0028] In an embodiment, ambient light sensor 102 includes one or more analog-to-digital converters (ADCs), such as a sample-and-hold (S / H) ADC 114 and a differential threshold ADC 116. In an embodiment, S / H ADC 114 converts an electrical signal representing a measured light level from charge integrator 110 into a digital output in a digital format that indicates the ambient light level. For example, S / H ADC 114 may capture or "sample" the voltage of the analog signal from charge integrator 110 at a particular moment in time and then "hold" that voltage constant during the conversion process, which allows the analog signal to remain stable and not change during the time it takes for S / H ADC 114 to complete the conversion, thereby allowing for a more accurate digital representation of the sampled signal.

[0029] In an embodiment, the differential threshold ADC 116 is configured to identify sudden changes or surges in photon levels and provide a specific value when such a surge is detected. The differential threshold ADC 116 receives the electrical signals from the charge integrator 110 and converts these analog signals into their corresponding digital values. The differential threshold ADC 116 takes the electrical signal and converts it into a digital format. Once the analog signal is converted into a digital value, the value is compared with a predetermined threshold level. The threshold level is set to distinguish normal signal levels from those signal levels indicating abnormal conditions, such as a surge of photons, which generally indicates a sudden increase in light intensity. If the digital representation of the received electrical signal exceeds the threshold level, it is interpreted as a photon surge being detected.

[0030] In an embodiment, once the ambient light level is quantified, this information is relayed to the image processor 106. In response, the imaging system 100 dynamically adjusts various parameters such as exposure time, brightness, contrast, and screen illumination to optimize image capture and display under prevailing light conditions. For example, in a camera, the readings of the ambient light sensor may affect the shutter speed, aperture size, and ISO setting so that a well-exposed image can be captured. In mobile devices and displays, the sensor helps adjust the screen brightness to ensure optimal visibility and reduce power consumption under different ambient lighting conditions.

[0031] In an embodiment, image sensor 104 converts an optical image into an electronic signal. It acts as an electronic eye, capturing light and converting it into an electrical charge, which is then converted into an image file by, for example, image processor 106. In an embodiment, image sensor 104 is a complementary metal oxide semiconductor (CMOS) sensor.

[0032] In an embodiment, the image sensor 104 includes a plurality of pixels 121, each of which is capable of detecting and measuring photons (light particles). In an embodiment, the image sensor 104 includes a pixel array 120 consisting of a grid of individual pixels 121. In an embodiment, each pixel 121 of the pixel array 120 is a photodetector that is sensitive to light. In an embodiment, the pixels 121 are arranged in rows and columns to cover the field of view of the image sensor 104. Each pixel 121 corresponds to a specific point in the final image and captures light information. When the light integrated by the photons reaches the pixel array 120, each pixel 121 absorbs the photons and converts them into electric charge. The amount of charge generated in each pixel 121 is proportional to the intensity of the incident light, allowing different brightness levels to be represented.

[0033] In an embodiment, the image sensor 104 includes a color filter array (CFA) 122, such as a Bayer filter, to capture color information by filtering light into red, green, and blue wavelengths before the light reaches the pixel 121. In an embodiment, a microlens array 124 is typically placed above the pixel 121 to focus and direct more light to each pixel 121, thereby improving the efficiency and sensitivity of the sensor. In an embodiment, the image sensor 104 includes an ADC 126 to convert the charge from each pixel 121 from an analog signal to a digital value.

[0034] Traditional cameras like digital single-lens reflex (DSLR) and mirrorless interchangeable lens cameras (MILC) incorporate a mechanical shutter. In contrast, most contemporary cameras have departed from this feature. These modern cameras typically employ active pixel sensors, typically CMOS image sensors, which hold the pixels in a reset state until the appropriate moment for them to begin integration. While in this reset state, the pixels still generate charge due to exposure; however, this charge is typically directed to a power source or, occasionally, to ground.

[0035] When the camera's shutter is open, the pixel array 120 of the image sensor 104 is exposed, and each pixel 121 of the pixel array 120 absorbs photons, generating an electrical charge proportional to the intensity of the light. After exposure, the readout mechanism of the image sensor 104 transfers the charge from each pixel 121 to the ADC 126, converting the analog signal into digital values. These values ​​represent the brightness level of each pixel 121 in the final image.

[0036] In an embodiment, the image sensor 104 includes a high dynamic range (HDR) surge suppression circuit 128 configured to receive a signal indicating a photon surge detection from the differential threshold ADC 116. The HDR surge suppression circuit 128 can manage the high dynamic range and suppress any aberrations caused by a sudden light surge, thereby ensuring the integrity of the captured image data.

[0037] The final step in the operation of the image sensor 104 is the processing of the digital values ​​by the image processor 106, which applies correction, enhancement, and compression to the raw data to produce a final image. The processed image is stored in the memory 108. The image processor 106 can be any component or collection of components suitable for performing calculations or other processing related tasks. In an embodiment, the image processor 106 is an application processor or microcontroller of the host device of the imaging system 100. In an embodiment, the image processor 106 is a signal processing unit. In an embodiment, the image processor 106 is configured to execute instructions to operate the image sensor 104 based on the disclosure herein.

[0038] The memory 108 may be any component or collection of components suitable for storing programs or instructions executed by the image processor 106. In one embodiment, the memory 108 comprises a non-transitory computer-readable medium. In an embodiment, the image processor 106 and the memory 108 are configured to process and store digital information. In an embodiment, the image processor 106 manages the operations and calculations required to capture, process, and potentially enhance image data. In an embodiment, the memory 108 is used as a repository to temporarily or permanently store this processed image data, thereby facilitating retrieval and further use or modification of the image.

[0039] After the readout phase, the digital values ​​from the pixel array 120 are used to reconstruct the captured image. The digital representations of the light intensity and color of each pixel are organized into a two-dimensional array corresponding to the layout of the pixel array 120 on the image sensor 104. The values ​​of the pixel array 120 form a digital image that can be processed, displayed, stored, or transmitted by the imaging system 100.

[0040] The imaging system 100 may include additional components not shown. For example, the imaging system 100 may include a user interface, allowing the user to interact with and manage the settings of the system, view the captured images, and possibly adjust the parameters of the camera. An external flash unit may be incorporated to provide additional lighting when the ambient light is insufficient. In addition, the implementation of advanced software algorithms may enable features such as image stabilization, facial identification, and autofocus, thereby significantly improving the versatility and efficiency of the imaging system. A wireless communication module may also be added to enable sharing of captured images or to connect the camera system to other devices or networks for additional functionality. A power source, typically a rechargeable battery, solar cell, etc., may power the system. External ports for charging, data transfer, and connection to other devices, an image stabilization system, a thermal management system for regulating heat generation, a focusing mechanism, a filter wheel or filter array for color images or specific spectral bands, and a housing for protecting internal components may all be incorporated to enhance the imaging system 100 capabilities and performance.

[0041] Typically, to capture an image, the pixel 121 undergoes three different phases of operation. The first phase involves resetting the pixel 121 and preparing to receive new light information. After resetting, the pixel 121 enters the second phase, in which the pixel 121 is exposed, absorbs photons and converts them into an electric charge proportional to the light intensity. The third and final phase is the readout phase. In this phase, the voltage corresponding to the accumulated charge from the absorbed photons is measured, allowing the received light information to be converted into a digital value that can be further processed to form an image. Typically, each pixel 121 is converted to the digital domain before being transmitted off the chip. In the camera, these phases are repeated continuously.

[0042] An image sensor utilizing the rolling shutter principle (e.g., image sensor 104) operates by sequentially capturing an image row by row, rather than acquiring an entire frame at once, which is a characteristic of a global shutter sensor. In this approach, each pixel 121 in pixel array 120 corresponds to a different light point in the image, and each row of pixels is sequentially exposed to light one after the other.

[0043] Figure 2 A timing diagram 200 of the operation of the imaging system 100 including a rolling shutter is shown. The pixel array 120 may be arranged as an M×N pixel array 121, where M and N are integers greater than zero, M represents the number of rows and N represents the number of columns. The vertical axis represents different rows of the pixel array 120 from 1 to M. As shown, Figure 2 The progression of three different phases of operation for each pixel 121 in a particular column of each of the M rows (ie, N pixels 121 ) is illustrated.

[0044] Initially, the operation begins with the reset phase of frame 1 at time t0, where each of the N first pixel rows is reset to initiate the exposure or light collection process for that particular row, while the other pixel rows (i.e., M-1 pixel rows) remain inactive. During the exposure phase (between time t0 and t1), each pixel 121 in the activated row (i.e., N pixels 121) collects photons and converts them into a charge corresponding to the intensity of the received light. After a set duration (T1), the exposure phase for the row is ended, and the accumulated charge is retained, ready for readout at time t1. After exposure, the readout phase begins at time t1, where the charge stored in the pixels 121 of the complete row is measured and converted into digital values, depicting the varying light intensity. These digital values ​​are then further processed to reconstruct and form an image.

[0045] The same process is then applied to the next row, which continues sequentially until each row in pixel array 120 has been exposed and read, completing the capture of an entire image frame. The time that elapses between the reset of a row and its readout is called the integration time (T1), and can be adjusted by varying the time that elapses between the reset of a row and the subsequent readout of that row.

[0046] When video is captured using an image sensor 104 with a rolling shutter principle, the mechanism involves systematically and consistently capturing individual frames to present a moving image or video sequence. Here, each frame acts as a still image, and when these frames are played back in rapid succession, they produce the motion or video we perceive.

[0047] The frame rate of a video, quantified as frames per second (fps), represents the number of individual frames captured per second. This rate can be standard, such as 24, 30, or 60 fps, or even higher for specific requirements such as slow motion capture. The rolling shutter operates continuously to capture individual frames at this predetermined rate, sequentially processing each pixel row of each frame.

[0048] Within each frame (t0 to t1, t1 to t2, etc.), the rolling shutter performs processing on each row of pixels sequentially. This means that each row within the frame is exposed and read out at a slightly different time, representing a different moment in the overall duration of the frame. The operations of resetting, exposing, and reading the pixels are continuous operations that occur for each frame in the sequence and are synchronized with the recording speed to achieve smooth and coherent playback of the video. For example, in a video with a frame rate of 50fps, this sequence of operations is repeated 50 times per second, ensuring that enough frames are captured to create the illusion of seamless motion in the resulting video.

[0049] Once a frame is captured, the resulting digital values ​​are processed and compressed by the camera's internal mechanisms and stored or transmitted as cohesive video data. Additionally, sophisticated processing techniques can be applied during real-time capture or in post-processing to reduce distortion, enhance the overall quality and color balance of the image, and stabilize the video, among other enhancements.

[0050] In a rolling shutter system, adjusting the integration time in response to changing light levels allows for optimal image exposure. The integration time is the duration that pixels 121 on the image sensor 104 are exposed to light, accumulating photons to be converted into electrical charge, which is then converted into a digital value representing light intensity.

[0051] Typically, the ambient light sensor 102 continuously measures the ambient light conditions, interpreting whether the scene is well-lit, under-lit, or over-lit. Accurately adjusting the integration time helps the image sensor 104 adapt to different lighting conditions. When the ambient light is low, the image sensor 104 requires a longer integration time, allowing the pixels 121 more time to collect photons, thereby increasing the sensor's sensitivity to light. Conversely, under high ambient light conditions, the integration time is shortened by reducing the amount of light accumulated by each pixel 121 to avoid overexposure.

[0052] Adjustment of the integration time is also inherently dependent on the frame rate of the video being captured. The maximum integration time per frame is set to For example, if the frame rate is 50fps, the maximum integration time is Therefore, in this example, the integration time is set to no more than 20ms.The integration time for each row of pixels must therefore be balanced within this frame duration to provide sufficient time for the subsequent readout and reset phases for each row.

[0053] Traditionally, given the inherent nature of rolling shutter, where each pixel row is exposed for a slightly different time within the frame duration, integration time is applied sequentially and evenly to each pixel row to ensure the same exposure over the entire frame. Because light conditions can change dynamically, especially during video capture, camera systems continuously recalibrate integration time to ensure consistent exposure throughout the recording period.

[0054] In a typical scenario, the ambient light level exhibits a gradual shift across multiple imaging frames. This speed allows the sensor's automatic exposure control (AEC) to modulate the integration time over several consecutive frames. Typically, image sensors ensure uniform integration time across all rows to prevent the introduction of undesirable visual distortions, such as alternating light and dark bands in the image. However, under unique but critical conditions, typically during the sensor's readout phase, there may be significant and sudden surges in illumination.

[0055] Figure 3FIG. 300 is a timing diagram showing the operation of a conventional imaging system that includes a rolling shutter that responds to a sudden surge in light level during the readout of a pixel row. If a sudden change in light level such as a lightning flash or an explosion occurs during the readout of a subset of rows of a sensor operating on the rolling shutter principle, it can result in differences in exposure within the same frame.

[0056] When each pixel row is exposed and read out sequentially at different times, if a strong light event occurs during this process, the rows that have been read out before the event (<t1) (i.e., rows 1 to K) will maintain their pre-event exposure levels (i.e., normal exposure) and will appear to be properly exposed. These rows effectively capture the scene before the sudden change in light and are not affected by it because they have already completed their exposure and readout phases.

[0057] Conversely, at the time (t1) of the sudden light change, the rows that are in exposure or yet to be exposed (i.e., rows K+1 to M) are affected differently. These rows are subjected to a sudden and significant increase in light level, and as a result, the pixels in these rows can become saturated due to overexposure. This saturation leads to a loss of detail in the affected region of the frame because the strong light overwhelms the sensor's ability to accurately represent the scene.

[0058] Traditionally, addressing sudden offsets in light level, such as a transition from low to high light level, involves starting approximately at the maximum integration time, typically about 1 divided by the frame rate, and gradually reducing the integration time over multiple frames. The adjustment of the integration time typically occurs at the start of the frame sequence to avoid the generation of image artifacts.

[0059] The human eye can naturally adapt to changing light conditions, but pixels with a relatively limited dynamic range compared to the human retina require manual adjustment of the light level. Cameras, especially those in phones and video recorders, typically gradually adjust the brightness level. This is because significant changes in system gain only in the case of the smallest changes in light can cause the image to darken, which is generally undesirable. Many phones use pixel information to determine whether the image is overexposed or underexposed and adjust the exposure based on an analysis of image statistics until a suitable level is reached.

[0060] When the pixels approach saturation, the system typically starts to reduce the gain by modifying the exposure, and this is supplemented by the analog gain present in the readout. However, when the light level suddenly increases, this balance adjustment can be interrupted, leaving the camera configured for low light to struggle with the higher actual light level and resulting in image saturation.

[0061] In the case of saturation, the system cannot determine how much the light level should be reduced, and incrementally reduces it until the saturation is resolved. However, this approach is often too slow, especially in safety applications where rapid adjustment is critical. Embodiments of the present disclosure are intended to prevent the undesirable effects of sudden light changes, thereby maintaining the integrity of the captured image under changing light conditions.

[0062] Figure 4 A timing diagram 400 illustrates the operation of the imaging system 100 in response to a sudden surge in brightness level. Figure 4 , a sudden surge in light level occurs at time t1 before readout of the first pixel row begins at time t2. The imaging system 100 includes a rolling shutter process.

[0063] In an embodiment, to capture the first frame (i.e., between time t0 and time t4), each pixel row of pixel array 120 from the first pixel row to the last row (i.e., the Mth row) of pixels is sequentially reset from time t0 to time t3. In an embodiment, the first pixel row is reset at time t0, and the last pixel row (i.e., the Mth row) is reset at time t3.

[0064] Once the corresponding row of pixels has been reset, the system waits until the integration time of reading out the pixels of the corresponding row is completed. In an embodiment, in the absence of a sudden surge in light level during the capture of a frame, each row of pixels of the pixel array from the first row of pixels to the last row is sequentially read out during the readout phase so that the integration time of each pixel is equal.

[0065] With the reset, integration and readout phases of the pixel 121 , the light level of the surrounding environment is measured continuously and asynchronously using, for example, the ambient light sensor 102 .

[0066] At time t1, in response to detecting a sudden surge in light levels, a global reset of all pixels in pixel array 120 occurs. In an embodiment, in response to determining that the ambient light level exceeds a threshold level, a global reset signal is generated. The global reset signal is simultaneously communicated to each pixel 121 of pixel array 120, which results in an immediate reset of each pixel 121 of pixel array 120.

[0067] In an embodiment, the threshold level is a programmable threshold. In an embodiment, one or more threshold levels are provided to a user as selectable values. In an embodiment, the threshold level is stored in memory 108. In an embodiment, the threshold level corresponds to the saturation of the pixels of image sensor 104. In an embodiment, one or more threshold levels correspond to various saturation levels of pixel 121.

[0068] In an embodiment, the image processor 106 receives the ambient light level from the S / H ADC 114 and compares the light level to a threshold level. In an embodiment, the differential threshold ADC 116 generates a signal indicating that a photon surge has been detected. In an embodiment, a global reset signal is generated in response to the signal generated by the differential threshold ADC 116.

[0069] like Figure 4 As shown, the global reset at time t1 occurs before reading the first pixel row. Since the global reset occurs before time t2, no pixels are read out. Assuming that the global reset occurs during the reset of the P-th pixel row, where P is less than M, the first subset of pixel rows includes the first P pixel rows, and the second subset of pixel rows includes the P+1 pixel rows until the last pixel row.

[0070] Thus, the first subset of rows has an integration time that is shorter than the initial time interval. For example, the integration time for the first pixel row is from time t1 to time t2, where the first pixel row is read out. Each subsequent pixel row in the first subset of rows has an integration time that is longer than the first pixel row starting at time t1. Thus, while the pixel rows in the first subset of rows have a shorter integration time and are partially exposed (e.g., underexposed), the second subset of pixel rows has an integration time that is equal to the initial time interval corresponding to a normal exposure time. In an embodiment, underexposure of the pixel rows in the first subset of rows may be digitally corrected based on the exposure time (e.g., multiplied by (P-y) / P) using the image processor 106 as a function of the output signal from each pixel (e.g., x, y coordinates) in the first subset of rows.

[0071] Figure 5 A timing diagram 500 illustrates the operation of the imaging system 100 in response to a sudden surge in brightness level. Figure 5 In FIG. 1 , a sudden surge in light level occurs at time t2 after the readout of the first row of pixels begins at time t1 , but before the readout of the last row at time t5 . The imaging system 100 includes a rolling shutter process.

[0072] In an embodiment, to capture the first frame (i.e., between time t0 and time t4), each pixel row of pixel array 120 from the first pixel row to the last row (i.e., the Mth row) of pixels is sequentially reset from time t0 to time t3. In an embodiment, the first pixel row is reset at time t0, and the last pixel row (i.e., the Mth row) is reset at time t3.

[0073] Once the corresponding row of pixels has been reset, the system waits until the integration time of reading out the pixels of the corresponding row is completed. In an embodiment, in the absence of a sudden surge in light level during the capture of a frame, each row of pixels of the pixel array from the first row of pixels to the last row is sequentially read out during the readout phase so that the integration time of each pixel is equal.

[0074] With the reset, integration and readout phases of the pixel 121 , the light level of the surrounding environment is measured continuously and asynchronously using, for example, the ambient light sensor 102 .

[0075] At time t2, in response to detecting the sudden surge in light levels, a global reset of all pixels in pixel array 120 occurs. In an embodiment, in response to determining that the ambient light level exceeds a threshold level, a global reset signal is generated. The global reset signal is simultaneously communicated to each pixel 121 of pixel array 120, which results in an immediate reset of each pixel 121 of pixel array 120.

[0076] In an embodiment, the threshold level is a programmable threshold. In an embodiment, one or more threshold levels are provided to a user as selectable values. In an embodiment, the threshold level is stored in memory 108. In an embodiment, the threshold level corresponds to the saturation of the pixels of image sensor 104. In an embodiment, one or more threshold levels correspond to various saturation levels of pixel 121.

[0077] In an embodiment, the photodetector 111 converts light energy into an electrical signal that is fed to the charge integrator 110. The charge integrator 110 integrates a charge (e.g., fixed) over time to generate a voltage. The image processor 106 receives the ambient light level from the S / H ADC 114 and compares the light level to a threshold level. In an embodiment, the differential threshold ADC 116 generates a signal indicating that a photon surge has been detected. In an embodiment, a global reset signal is generated in response to the signal generated by the differential threshold ADC 116.

[0078] like Figure 5 As shown in , the global reset at time t2 occurs after the first pixel row has been read but before the last pixel row has been read. Therefore, at time t2, the first subset of pixel rows is read out before the global reset. In contrast, at time t2, the second subset of pixel rows has not yet been read out. Assuming that the global reset occurs during the readout of the K-th pixel row, where K is less than M, the first subset of pixel rows includes the first K pixel rows, and the second subset of pixel rows includes the K+1-th pixel row through the last pixel row.

[0079] Each row of the first subset of pixel rows that has been read out prior to the global reset will maintain its pre-event exposure level corresponding to the second time interval and appear to be properly exposed. However, each row of the second subset of pixel rows will have an integration time that is less than the original integration time of the first subset of pixel rows. The first pixel row in the second subset of pixel rows (i.e., row K+1) will have an integration time of zero seconds. The next pixel row in the second subset of pixel rows will have a slightly larger integration time, but much less than the original integration time of the first subset of pixel rows. The last pixel row in the second subset of pixel rows (i.e., row M) will have an integration time equal to the third time interval, which is equal to time t5-time t2.

[0080] In an embodiment, time t2 is stored in metadata of the image frame indicating a discontinuity in processing of rows of the first frame. In an embodiment, time t2 stored in the metadata is used to post-process rows of a second subset of pixel rows having a shorter integration time than the first subset of pixel rows.

[0081] In an embodiment, to capture the second frame, the integration time is reduced from the original integration time in the first frame in response to the sudden surge in light levels in the previous frame. In an embodiment, the integration time of the second frame (and optionally any subsequent frames) is set to the integration time of the last pixel row (i.e., the Mth row) in the second subset of pixel rows, such that the integration time is equal to the third time interval (i.e., time t5-time t2). In an embodiment, the integration time of the second frame (and optionally subsequent frames) is set based on the light level detected from the S / H ADC 114 using the ambient light sensor 102.

[0082] Advantageously, embodiments of the present invention increase the dynamic range of image sensor 104 under asynchronous pulses at high light levels.

[0083] Figure 6 A flow chart of an embodiment method 600 for capturing frames using imaging system 100 in response to a sudden surge in brightness level is shown. Figure 6 In FIG. 1 , there is a sudden surge in light level before readout of the first row of pixels begins.

[0084] At step 602, each pixel row of the pixel array 120 is sequentially placed from a first pixel row to a last pixel row.

[0085] At step 604, in the absence of a sudden surge in light level during the capture of a frame, each pixel row of pixel array 120 is read out during the readout phase such that the integration time of each pixel is equal to the fixed time interval. In an embodiment, the integration time is set based on the ambient light level detected by ambient light sensor 102. In a video camera system, the maximum integration time for each frame is set to This prevents the pixels in the next frame from being reset before the pixels in the current frame are read out.

[0086] At step 606, in response to a sudden surge in light levels before a frame is captured, all pixels 121 of pixel array 120 are simultaneously and globally reset. In an embodiment, the global reset occurs in response to determining that the ambient light level detected by ambient light sensor 102 exceeds a threshold level. In an embodiment, the threshold level is a programmable threshold. In an embodiment, the threshold level can be selected by a user from a set of values. In an embodiment, the threshold level is stored in memory 108. In an embodiment, the threshold level corresponds to the saturation of the pixels of image sensor 104. In an embodiment, one or more threshold levels correspond to various saturation levels of pixels 121.

[0087] In step 606, the set of pixel rows that were pre-reset before the global reset is included in the first subset of pixel rows. The set of pixel rows that have not been reset at the time of the global reset is included in the second subset of pixel rows. Each pixel row in the second subset of pixel rows from the alternate pixel row to the last pixel row has a consistent and fixed integration time. Each pixel row in the first subset of pixel rows from the first pixel row to the alternate pixel row has an integration time that is less than the integration time of the pixels in the second pixel row and has a variable value. The integration time of each pixel in the first subset of pixel rows corresponds to the integration time starting from the global reset.

[0088] At step 608, an integration time for a subsequent frame is determined. In an embodiment, the integration time is reduced from the original integration time in the first frame in response to a sudden surge in light levels in a previous frame. In an embodiment, the integration time for the subsequent frame is set to the integration time of the last row of pixels in the second subset of rows of pixels of the first frame (i.e., the Mth row). In an embodiment, the integration time for the subsequent frame or frames is set based on the light level detected from the S / H ADC 114 using the ambient light sensor 102.

[0089] Figure 7 A flow chart of an embodiment method 700 for using imaging system 100 to capture frames in response to a sudden surge in light levels is shown. Figure 7 In FIG. 5 , a sudden surge in light level occurs after readout of the first row of pixels has begun but before readout of the last row.

[0090] At step 702 , each pixel row of the pixel array 120 from the first pixel row to the last pixel row is sequentially reset.

[0091] At step 704, in the absence of a sudden surge in light level during the capture of a frame, each pixel row of pixel array 120 is read out during the readout phase such that the integration time of each pixel is equal to the fixed time interval. In an embodiment, the integration time is set based on the ambient light level detected by ambient light sensor 102. In a video camera system, the maximum integration time for each frame is set to This prevents the pixels in the next frame from being reset before the pixels in the current frame are read out.

[0092] At step 706, in response to a sudden surge in light levels during the capture of a frame, all pixels 121 of pixel array 120 are simultaneously and globally reset. In an embodiment, the global reset occurs in response to determining that the ambient light level detected by ambient light sensor 102 exceeds a threshold level. In an embodiment, the threshold level is a programmable threshold. In an embodiment, the threshold level can be selected by a user from a set of values. In an embodiment, the threshold level is stored in memory 108. In an embodiment, the threshold level corresponds to the saturation of the pixels of image sensor 104. In an embodiment, one or more threshold levels correspond to various saturation levels of pixels 121.

[0093] In step 706, the set of pixel rows that have been read before the global reset is included in the first subset of pixel rows. The set of pixel rows that have not been read at the time of the global reset is included in the second subset of pixel rows. Each pixel row in the first subset of pixel rows from the first pixel row to the alternate pixel row has a consistent and fixed integration time. Each pixel row in the second subset of pixel rows from the alternate pixel row to the last pixel row has an integration time that is less than the integration time of the pixels in the first pixel row, and the value is variable. The integration time of each pixel in the second subset of pixel rows corresponds to the integration time starting from the global reset.

[0094] At step 708, the integration time for the subsequent frame is determined. In an embodiment, the integration time is reduced from the original integration time in the first frame in response to the sudden surge in light level in the previous frame. In an embodiment, the integration time for the subsequent frame is set to the integration time of the last pixel row in the second subset of pixel rows of the first frame (i.e., the Mth row). In an embodiment, the integration time for the subsequent frame or frames is set based on the light level detected from the S / H ADC 114 using the ambient light sensor 102.

[0095] Note that all steps outlined in the flowchart of the method are not necessarily required and may be optional. In addition, changes in the arrangement of steps, removal of one or more steps and path connections, and addition of steps and path connections are similarly contemplated.

[0096] A first aspect relates to a method for capturing an image frame using a pixel array of an image sensor. The method includes sequentially resetting each pixel row of the pixel array from a first pixel row to a last pixel row; sequentially reading out each pixel row of a first subset of pixel rows during a readout phase such that an integration time of each pixel of the first subset of pixel rows is equal to a fixed integration time; and globally resetting pixels of the pixel array in response to detecting that an ambient light level exceeds a threshold level, the global resetting of the pixels occurring before reading out a second subset of pixel rows, each pixel row in the second subset of pixel rows having a variable integration time less than the fixed integration time and starting from the global resetting of the pixels.

[0097] In such a first implementation of the method according to the first aspect, the method further comprises determining, by the ambient light sensor, that the ambient light level exceeds a threshold level and, in response, generating a signal indicative of a photon surge event.

[0098] In a second implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, the threshold level is a programmable threshold.

[0099] In a third implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, determining that the ambient light level exceeds the threshold level corresponds to a saturation of a pixel of the image sensor.

[0100] In a fourth implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, the method further comprises storing the discontinuity time in metadata of the image frame corresponding to the detection of the ambient light level.

[0101] In a fifth implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, the method further comprises post-processing the second subset of pixels based on the discontinuity times stored in the metadata.

[0102] In a sixth implementation form of the method according to the first aspect itself or any of the foregoing implementation forms of the first aspect, the image frame is a first image frame, and the method further includes sequentially resetting each pixel row of the pixel array from the first pixel row to the last pixel row; and sequentially reading out each pixel row from the first pixel row to the last pixel row during the second readout phase, so that the integration time of each pixel is equal to the second fixed integration time.

[0103] In a seventh implementation form of the method according to the first aspect itself or any preceding implementation form of the first aspect, the second fixed integration time is equal to the integration time of the last pixel row of the first image frame or is based on the ambient light level of the first image frame.

[0104] In an eighth implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, the second fixed integration time is shorter in duration than the fixed integration time of the first image frame.

[0105] In a ninth implementation form of the method according to the first aspect as such or any of the preceding implementation forms of the first aspect, the global reset pixel is asynchronous in operation with respect to the sequential reading of the pixels.

[0106] A second aspect relates to an imaging system. The imaging system includes a pixel array; a non-transitory memory storage device including instructions; and a processor in communication with the non-transitory memory storage device and the pixel array. The processor is configured to execute the instructions to sequentially reset each pixel row of the pixel array from a first pixel row to a last pixel row; sequentially read out each pixel row of a first subset of pixel rows during a readout phase, so that the integration time of each pixel of the first subset of pixel rows is equal to a fixed integration time; and globally reset the pixels of the pixel array in response to detecting that the ambient light level exceeds a threshold level, the global reset of the pixels occurring before reading out a second subset of pixel rows, each pixel row in the second subset of pixel rows having a variable integration time less than the fixed integration time, and starting from the global reset of the pixels.

[0107] In a first implementation of the imaging system according to the second aspect, the imaging system further comprises an ambient light sensor configured to determine that an ambient light level exceeds a threshold level and, in response, generate a signal indicative of a photon surge event.

[0108] In a second implementation form of the imaging system according to the second aspect as such or any preceding implementation form of the second aspect, the threshold level is a programmable threshold.

[0109] In a third implementation form of the imaging system according to the second aspect as such or any preceding implementation form of the second aspect, the processor is configured to execute instructions to store the discontinuity time in metadata corresponding to the detection of the ambient light level.

[0110] In a fourth implementation form of the imaging system according to the second aspect as such or any preceding implementation form of the second aspect, the integration time for capturing subsequent frames is set equal to the integration time of the last pixel row or based on an ambient light level determined during the readout phase.

[0111] A third aspect relates to a computer-implemented method for capturing an image frame using a pixel array of an image sensor. The computer-implemented method includes sequentially resetting each pixel row of the pixel array from a first pixel row to a last pixel row; sequentially reading out each pixel row of a first subset of pixel rows during a readout phase such that an integration time of each pixel of the first subset of pixel rows is equal to a fixed integration time; and globally resetting pixels of the pixel array in response to detecting that an ambient light level exceeds a threshold level, the global resetting of the pixels occurring before reading out a second subset of pixel rows, each pixel row in the second subset of pixel rows having a variable integration time less than the fixed integration time and starting from the global resetting of the pixels.

[0112] In a first implementation form of such a computer-implemented method according to the third aspect, the method further comprises determining, by the ambient light sensor, that the ambient light level exceeds a threshold level and, in response, generating a signal indicative of a photon surge event.

[0113] In a second implementation form of the computer-implemented method according to the third aspect above or any preceding implementation form of the third aspect, the threshold level is a programmable threshold.

[0114] In a third implementation form of the computer-implemented method according to the third aspect or any preceding implementation form of the third aspect, the computer-implemented method further comprises storing the discontinuity time in metadata of the image frame corresponding to the detection of the ambient light level.

[0115] In a fourth implementation of the computer-implemented method according to the third aspect or any preceding implementation of the third aspect, the image frame is a first image frame and the integration time of subsequent frames is set equal to the integration time of the last pixel row of the first image frame or based on the ambient light level of the first image frame.

[0116] Although the present specification has been described in detail, it should be understood that various changes, substitutions and modifications may be made without departing from the spirit and scope of the present disclosure as defined by the appended claims. In the various drawings, the same elements are represented by the same reference numerals. In addition, the scope of the present disclosure is not intended to be limited to the specific embodiments described herein, because it will be readily understood by those of ordinary skill in the art from the present disclosure that the processes, machines, manufacturing methods, material compositions, means, methods or steps currently existing or to be developed later may perform functions substantially the same as the corresponding embodiments described herein or achieve results substantially the same as the corresponding embodiments described herein. Therefore, the appended claims are intended to include such processes, machines, manufacturing, material compositions, means, methods or steps within their scope.

[0117] Accordingly, the specification and drawings are to be considered merely illustrative of the present disclosure, which is defined by the appended claims, and are to cover any and all modifications, variations, combinations or equivalents falling within the scope of the present disclosure.

Claims

1. A method for capturing an image frame using a pixel array of an image sensor, the method comprising: sequentially resetting each pixel row of the pixel array from a first pixel row to a last pixel row; sequentially reading out each pixel row of a first subset of pixel rows during a readout phase such that an integration time of each pixel of the first subset of pixel rows is equal to a fixed integration time; as well as In response to detecting that the ambient light level exceeds a threshold level, the pixels of the pixel array are globally reset, the global reset of the pixels occurring before a second subset of pixel rows are read out, each pixel row in the second subset of pixel rows having a variable integration time that is less than the fixed integration time, and the variable integration time of each pixel row in the second subset of pixel rows begins at the global reset of the pixels. 2 . The method of claim 1 , further comprising determining, by an ambient light sensor, that the ambient light level exceeds the threshold level and, in response, generating a signal indicative of a photon surge event. The method of claim 1 , wherein the threshold level is a programmable threshold. The method of claim 1 , wherein determining that the ambient light level exceeds the threshold level corresponds to saturation of pixels of the image sensor. 5 . The method of claim 1 , further comprising storing a discontinuity time in metadata of the image frame corresponding to the detection of the ambient light level. 6 . The method of claim 5 , further comprising post-processing the second subset of pixels based on the discontinuity times stored in the metadata.

7. The method according to claim 1, wherein the image frame is a first image frame, the method further comprising: sequentially resetting each pixel row of the pixel array from a first pixel row to a last pixel row; as well as Each pixel row is read out sequentially from the first pixel row to the last pixel row during a second readout phase such that an integration time of each pixel is equal to a second fixed integration time. 8 . The method of claim 7 , wherein the second fixed integration time is equal to an integration time of the last pixel row of the first image frame or is based on the ambient light level of the first image frame.

9. The method of claim 7, wherein the second fixed integration time is shorter than the fixed integration time of the first image frame.

10. The method of claim 1, wherein globally resetting the pixels is asynchronous in operation relative to the sequential reading of the pixels.

11. An imaging system comprising: Pixel array; a non-transitory memory storage device including instructions; and a processor in communication with the non-transitory memory storage device and the pixel array, the processor being configured to execute the instructions to: sequentially resetting each pixel row of the pixel array from a first pixel row to a last pixel row; sequentially reading out each pixel row of a first subset of pixel rows during a readout phase such that an integration time of each pixel of the first subset of pixel rows is equal to a fixed integration time; as well as In response to detecting that the ambient light level exceeds a threshold level, the pixels of the pixel array are globally reset, the global reset of the pixels occurring before a second subset of pixel rows are read out, each pixel row in the second subset of pixel rows having a variable integration time that is less than the fixed integration time, and the variable integration time for each pixel row in the second subset of pixel rows begins at the global reset of the pixels.

12. The imaging system of claim 11, further comprising an ambient light sensor configured to determine that the ambient light level exceeds the threshold level and, in response, generate a signal indicative of a photon surge event.

13. The imaging system of claim 11, wherein the threshold level is a programmable threshold.

14. The imaging system of claim 11, wherein the processor is configured to execute the instructions to store a discontinuity time in metadata corresponding to the detection of the ambient light level.

15. The imaging system of claim 11, wherein an integration time for capturing subsequent frames is set equal to an integration time of the last pixel row or based on the ambient light level determined during the readout phase.

16. A computer-implemented method for capturing an image frame using a pixel array of an image sensor, the computer-implemented method comprising: sequentially resetting each pixel row of the pixel array from a first pixel row to a last pixel row; sequentially reading out each pixel row of a first subset of pixel rows during a readout phase such that an integration time of each pixel of the first subset of pixel rows is equal to a fixed integration time; as well as In response to detecting that the ambient light level exceeds a threshold level, the pixels of the pixel array are globally reset, the global reset of the pixels occurring before a second subset of pixel rows are read out, each pixel row in the second subset of pixel rows having a variable integration time that is less than the fixed integration time, and the variable integration time for each pixel row in the second subset of pixel rows begins at the global reset of the pixels.

17. The computer-implemented method of claim 16, further comprising determining, by an ambient light sensor, that the ambient light level exceeds the threshold level and, in response, generating a signal indicative of a photon surge event.

18. The computer implemented method of claim 16, wherein the threshold level is a programmable threshold.

19. The computer-implemented method of claim 16, further comprising storing a discontinuity time in metadata of the image frame corresponding to the detection of the ambient light level.

20. The computer-implemented method of claim 16, wherein the image frame is a first image frame, and wherein an integration time for subsequent frames is set equal to an integration time of the last pixel row of the first image frame or based on the ambient light level of the first image frame.

Images