Floating artifact cancellation in high dynamic range imaging

Abstract

The invention relates to scatter artifact cancellation in high dynamic range imaging. Image sensors, imaging systems, and methods for eliminating scatter artifacts in high dynamic range imaging. The image sensor includes a pixel array and a controller. The controller is configured to detect that the first pixel signal is at or below a barrier value. The first pixel signal is generated during a first exposure. The controller is also configured to compare an adjacent pixel signal to the barrier value. The controller is further configured to determine a first image value of the center pixel based on a second pixel signal when at least one of the adjacent pixel signals is above the barrier value. The second pixel signal is generated during a second exposure period that is shorter than the first exposure. The controller is further configured to set the first image value to the first pixel signal when each of the adjacent pixel signals is at or below the barrier value.

Description

Background Art

[0001] Image sensors are used in electronic devices such as cell phones, cameras, and computers to capture images. Specifically, the electronic device is provided with an array of image sensor pixels arranged in a grid pattern. Each image sensor pixel receives incident photons such as light and converts these photons into electrical signals. A column circuit is coupled to each column for reading out the sensor signal from each image sensor pixel. Summary of the invention

[0002] A high dynamic range (HDR) image may be created by combining pixel signals generated by image sensor pixels during two readouts with different exposure durations. In some cases, when the image sensor pixel with the highest responsiveness to the spectrum of incident light becomes saturated, there is charge overflow to adjacent image sensor pixels. The overflow charge (sometimes referred to as blooming or blooming charge) introduces nonlinearity to the signal of the affected pixel and may cause undesirable image artifacts (sometimes referred to as blooming artifacts) to be included in the HDR image. Accordingly, the present disclosure provides image sensors, imaging systems, and methods that, among other things, eliminate or mitigate blooming artifacts in HDR imaging by detecting image sensor pixels that may be affected by blooming and replacing their long exposure signals with different signals determined based on their short exposure signals.

[0003] The present disclosure provides an image sensor, which includes a pixel array and a controller in a specific implementation. The pixel array includes a plurality of image sensor pixels. The controller is configured to detect that a first pixel signal is at or below a barrier value. The first pixel signal is generated by a central pixel of the plurality of image sensor pixels during a first exposure duration. The controller is also configured to compare a plurality of adjacent pixel signals with the barrier value. The plurality of adjacent pixel signals are generated during the first exposure duration. The controller is also configured to determine a first image value of the central pixel based on a second pixel signal when at least one of the plurality of adjacent pixel signals is higher than the barrier value. The second pixel signal is generated by the central pixel during a second exposure duration shorter than the first exposure duration. The controller is also configured to set the first image value to the first pixel signal when each of the plurality of adjacent pixel signals is at or below the barrier value.

[0004] The present disclosure also provides an imaging system, which in a specific implementation includes a lens system, an image sensor and an imaging controller. The image sensor is in an operative relationship with the lens system. The image sensor includes a pixel array, which includes a plurality of image sensor pixels. The imaging controller is configured to detect that a first pixel signal is at or below a barrier value. The first pixel signal is generated by a central pixel of the plurality of image sensor pixels during a first exposure duration. The imaging controller is also configured to compare a plurality of adjacent pixel signals with the barrier value. The plurality of adjacent pixel signals are generated during the first exposure duration. The imaging controller is also configured to determine a first image value of the central pixel based on a second pixel signal when at least one of the plurality of adjacent pixel signals is above the barrier value. The second pixel signal is generated by the central pixel during a second exposure duration shorter than the first exposure duration. The imaging controller is also configured to set the first image value to the first pixel signal when each of the plurality of adjacent pixel signals is at or below the barrier value.

[0005] The present disclosure also provides a method for eliminating floating artifacts in high dynamic range imaging. The method includes detecting that a first pixel signal is at or below a barrier value. The first pixel signal is generated by a central pixel of a plurality of image sensor pixels included in a pixel array during a first exposure duration. The method also includes comparing a plurality of adjacent pixel signals with the barrier value. The plurality of adjacent pixel signals are generated during the first exposure duration. The method also includes determining a first image value of the central pixel based on a second pixel signal when at least one of the plurality of adjacent pixel signals is above the barrier value. The second pixel signal is generated by the central pixel during a second exposure duration that is shorter than the first exposure duration. The method also includes setting the first image value to the first pixel signal when each of the plurality of adjacent pixel signals is at or below the barrier value. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] To describe in detail exemplary implementations, reference will now be made to the accompanying drawings, in which:

[0007] Figure 1A is a block diagram of an example of an imaging system according to some specific implementations;

[0008] Figure 1B is an illustration of an example of an imaging system incorporated into a vehicle according to some implementations;

[0009] Figure 2 is a partial schematic diagram and a partial block diagram of an example of an image sensor according to some specific implementations;

[0010] Figure 3is a schematic diagram of an example of a circuit in an image sensor pixel according to some specific implementations;

[0011] Figure 4A is a top view of an example of a pixel array having a Bayer color filter array under green light according to some specific implementations;

[0012] Figure 4B is a top view of an example of a pixel array having a Bayer color filter array under red light according to some specific implementations;

[0013] Figure 5A is an illustration of an example of a long exposure image generated by a portion of a pixel array according to some implementations;

[0014] Figure 5B is an illustration of an example of a short exposure image generated by a portion of a pixel array according to some implementations;

[0015] Figure 6 is a flow chart of an example of a method for determining a long exposure correction signal value for an image sensor pixel according to some implementations;

[0016] 7A to 7E is an illustration of an example of long exposure image inspection with different combinations of adjacent imaging sensor pixels to determine whether a center pixel may be affected by blooming according to some implementations;

[0017] Figure 8 is a flow chart of an example of a method for determining a digital lateral overflow value for an image sensor pixel according to some specific implementations;

[0018] Fig. 9 is a flow chart of an example of a method for determining a long exposure correction signal value for an image sensor pixel with noise filtering according to some specific implementations;

[0019] Fig.10 is a graph of examples of long exposure and short exposure signal values ​​generated by red and green image sensor pixels under green light with different amounts of photon flux according to some specific implementations; and

[0020] Fig.11 is a flow chart of an example of a method for removing blooming artifacts in high dynamic range imaging according to some implementations.

[0021] definition

[0022] Various terms are used to refer to specific system components. Different companies may refer to a component by different names - this document is not intended to distinguish between components that differ in name but have the same function. In the following discussion and in the claims, the terms "include" and "comprising" are used in an open-ended manner, and therefore, these terms should be interpreted to mean "including but not limited to...". In addition, the terms "couple" or "couples" are intended to mean either an indirect connection or a direct connection. Thus, if a first device is coupled to a second device, that connection may be made through a direct connection or through an indirect connection via other devices and connections.

[0023] Terms defining altitudes, such as "above", "below", "upper", and "lower", should be positional terms with respect to the direction of light incident on the pixel array and / or image pixels. Light entering should be considered to interact with or pass through objects and / or structures "above" and "upper" before interacting with or passing through objects and / or structures "below" or "lower". Therefore, positional terms may not have any relationship to the direction of gravity.

[0024] In reference to electrical devices, whether stand-alone or as part of an integrated circuit, the terms "input" and "output" refer to electrical connections to the electrical device and should not be construed as verbs requiring operation. For example, a differential amplifier such as an operational amplifier may have a first differential input and a second differential input, and these "inputs" define electrical connections to the operational amplifier and should not be construed as requiring signal inputs to the operational amplifier.

[0025] "Asserting" shall mean changing the state of a Boolean signal. A Boolean signal may be asserted high or have a higher voltage, and a Boolean signal may be asserted low or have a lower voltage, at the discretion of the circuit designer. Similarly, "deasserting" shall mean changing the state of a Boolean signal to a voltage level opposite to the asserted state.

[0026] "Light" or "color" shall refer to visible light in the range of about 380 nanometers and 700 nanometers (nm). "Light" or "color" shall also refer to invisible light, such as infrared light in the range of about 800 nm and 1 mm. "Light" or "color" shall also refer to invisible light, such as ultraviolet light in the range of about 100 nm to 400 nm.

[0027] “Controller” shall mean, alone or in combination, an individual circuit component, an application specific integrated circuit (ASIC), one or more microcontrollers with control software, a reduced instruction set computer (RISC) with control software, a digital signal processor (DSP), one or more processors with control software, a programmable logic device (PLD), a field programmable gate array (FPGA), or a programmable system on a chip (PSOC) configured to read inputs and drive outputs in response to those inputs. DETAILED DESCRIPTION

[0028] The following discussion relates to various specific implementations of the present invention. Although one or more of these specific implementations may be preferred, the disclosed specific implementations should not be interpreted or otherwise used to limit the scope of the present disclosure, including the claims. In addition, it should be understood by those skilled in the art that the following description has broad application, and the discussion of any specific implementation is only intended to be an example of that specific implementation, and is not intended to indicate that the scope of the present disclosure, including the claims, is limited to that specific implementation.

[0029] Various examples relate to systems and methods for eliminating blooming artifacts in high dynamic range (HDR) imaging. More specifically, various examples relate to image sensors and related imaging systems that detect when an image sensor pixel may be affected by blooming based on pixel signals of adjacent image sensor pixels. Even more specifically, various examples relate to image sensors and related imaging systems that replace a long exposure signal generated by an image sensor pixel affected by blooming with a different signal determined based on a short exposure signal generated by the image sensor pixel. The specification now turns to exemplary systems for the guidance of the reader.

[0030] Figure 1A An example of an imaging system 100 is shown. Specifically, the imaging system 100 may be a portable electronic device with imaging capabilities, such as a camera, a cellular phone, a tablet computer, a webcam, a video camera, a video surveillance system, or a video game system. In other cases, the imaging system 100 may be an automotive imaging system. Figure 1A The imaging system 100 shown in includes a camera module 102 that can be used to convert incoming light into digital image data. The camera module 102 may include one or more lenses 104 and one or more corresponding image sensors 106. The lenses 104 may include fixed and / or adjustable lenses. During an image capture operation, light from a scene may be focused onto the image sensor 106 through the lenses 104. The image sensor 106 may include circuitry for converting analog pixel data into corresponding digital image data, which is provided to an imaging controller 108. If desired, the camera module 102 may be provided with an array of lenses 104 and an array of corresponding image sensors 106.

[0031] The imaging controller 108 may include one or more integrated circuits. The imaging circuitry may include image processing circuitry, a microprocessor, and storage devices, such as random access memory and non-volatile memory. The imaging controller 108 may be implemented using components separate from the camera module 102 and / or components forming part of the camera module 102 (e.g., circuitry forming part of the image sensor 106). The imaging controller 108 may be used to process and store digital image data captured by the camera module 102. The processed image data may be provided to external equipment such as a computer, an external display, or other device using a wired communication path and / or a wireless communication path coupled to the imaging controller 108, as desired.

[0032] Figure 1B Another example of an imaging system 100 is shown. Figure 1B The imaging system 100 shown in FIG. 1 includes an automobile or vehicle 110. The vehicle 110 is illustratively shown as a passenger vehicle, but the imaging system 100 may be other types of vehicles, including commercial vehicles, on-road vehicles, and off-road vehicles. Commercial vehicles may include buses and tractor-trailer vehicles. Off-road vehicles may include tractors and crop harvesting equipment. Figure 1B In the example of , the vehicle 110 includes a front-view camera module 102 arranged to capture an image of a scene in front of the vehicle 110. Such a front-view camera module 102 can be used for any suitable purpose, such as lane keeping assistance, collision warning systems, cruise control systems, autonomous driving systems, and proximity detection. The exemplary vehicle 110 also includes a rear-view camera module 102 arranged to capture an image of a scene behind the vehicle 110. Such a rear-view camera module 102 can be used for any suitable purpose, such as a collision warning system, reverse video, autonomous driving systems, proximity detection, monitoring the position of overtaking vehicles, and backing up. The vehicle 110 also includes a side-view camera module 102 arranged to capture an image of a scene beside the vehicle 110. Such a side-view camera module 102 can be used for any suitable purpose, such as blind spot monitoring, collision warning systems, autonomous driving systems, monitoring the position of overtaking vehicles, lane change detection, and proximity detection. In the case where the imaging system 100 is a vehicle, the imaging controller 108 can be a controller of the vehicle 110. Next, turn to the image sensor 106 of the camera module 102 for a more detailed discussion.

[0033] Figure 2 An example of an image sensor 106 is shown. Specifically, Figure 2The image sensor 106 is shown to include a substrate 200 of a semiconductor material (e.g., silicon) encapsulated within a package to produce a packaged semiconductor device or a packaged semiconductor product. Bond pads or other connection points of the substrate 200 are coupled to terminals of the image sensor 106, such as a serial communication channel 202 coupled to a first terminal 204 and a capture input 206 coupled to a second terminal 208. There will be additional terminals, such as a ground terminal, a common terminal, or a power supply terminal, but these additional terminals are omitted to avoid over-complication of the drawing. Although a single example of the substrate 200 is shown, in other specific implementations, multiple substrates may be combined to form the image sensor 106 in a multi-chip module.

[0034] Figure 2 The image sensor 106 shown includes a pixel array 210 having a plurality of image sensor pixels 212 arranged in rows and columns. The pixel array 210, which is one example of a “pixel array,” may include, for example, hundreds or thousands of rows and hundreds or thousands of columns of image sensor pixels 212. Control and readout of the pixel array 210 may be implemented by an image sensor controller 214 coupled to a row controller 216 and a column controller 218. The row controller 216 may receive a row address from the image sensor controller 214 and provide corresponding row control signals to the image sensor pixels 212 such as reset, row select, charge transfer, dual conversion gain, and readout control signals. The row control signals may be communicated via one or more conductors, such as a row control path 220.

[0035] Column controller 218 may be coupled to pixel array 210 via one or more conductors, such as column lines 222. Column controllers may sometimes be referred to as column control circuits, readout circuits, or column decoders. Column lines 222 may be used to read out pixel signals from image sensor pixels 212 and to provide bias currents and / or bias voltages to image sensor pixels 212. If desired, during a pixel readout operation, a row of pixels in pixel array 210 may be selected using row controller 216, and pixel signals generated by image sensor pixels 212 in the pixel row may be read out along column lines 222. Column controller 218 may include sample-and-hold circuits for sampling and temporarily storing pixel signals read out from pixel array 210, amplifier circuits, analog-to-digital conversion (ADC) circuits, bias circuits, column memories, latch circuits for selectively enabling or disabling column circuits, or other circuits coupled to one or more columns of image sensor pixels 212 in pixel array 210 for operating image sensor pixels 212 and for reading out pixel signals from image sensor pixels 212. ADC circuitry in column controller 218 may convert analog pixel values ​​received from pixel array 210 into corresponding digital image data. Column controller 218 may provide the digital image data to image sensor controller 214 and / or imaging controller 108 (eg, via serial communication channel 202). Figure 1A ).

[0036] Figure 3 An example of circuitry in one of image sensor pixels 212 is shown. Image sensor pixel 212 may have Figure 3 The image sensor pixels shown may have fewer, additional, or different components in different configurations. Specifically, Figure 3 It is shown that each of the image sensor pixels 212 may include a photodetector 302 (e.g., a photodiode). A pixel positive supply voltage such as a supply voltage VAAPIX may be provided at a positive supply terminal 304. A ground supply voltage such as a reference voltage Vss may be provided at a ground terminal 306. In some cases, after the light passes through a color filter structure (not shown), the incoming light is collected by the photodetector 302. The photodiode 302 converts this light into an electrical charge.

[0037] Prior to acquiring an image, a reset control signal RST may be asserted. The reset control signal RST turns on the reset transistor 308 and resets the charge storage (CS) node 310 to a voltage equal to or close to the supply voltage VAAPIX. The reset control signal RST may then be deasserted, turning off the reset transistor 308. After the reset process is complete, the transfer gate control signal TX may be asserted to turn on the transfer transistor 312. When the transfer transistor 312 is turned on, the charge generated by the photodetector 302 in response to the incoming light is transferred to the charge storage node 310. The charge storage node 310 exhibits a capacitance that can be used to store the charge that has been transferred from the photodetector 302. The signal associated with the charge stored in the charge storage node 310 is buffered by the source follower transistor 314. The row select transistor 316 connects the source follower transistor 314 to one of the column lines 222.

[0038] When it is desired to read out the value of the charge stored in charge storage node 310, control signal RS is asserted. The readout value may be, for example, the value of charge storage node 310 represented by a signal at source terminal S of source follower transistor 314. When control signal RS is asserted, row select transistor 316 turns on and produces an output signal Vout on output path 318 that represents the magnitude of the charge stored in charge storage node 310. Output signal Vout is an example of a "pixel signal." When control signal RS is asserted, one of column lines 222 may be used to route output signal Vout from image sensor pixel 212 to a readout circuit, such as Figure 2 Column controller 218 in.

[0039] When photodetector 302 of one of image sensor pixels 212 is exposed to incident light, charge begins to accumulate in the photodiode well of photodetector 302. In some cases, more charge may be generated than photodetector 302 can hold, such as when one of image sensor pixels 212 is exposed to extremely bright light. In other words, one of image sensor pixels 212 becomes saturated when the charge exceeds the full well capacity of its photodetector 302. When one of image sensor pixels 212 is saturated, there may be charge overflow to neighboring pixels. For example, Figure 4A 4 shows a top view of a first array 402 of pixels having a Bayer (red-green-green-blue) color filter array under green light. When a green pixel of the first array 402 becomes saturated, charge overflows to the adjacent red and blue pixels of the first array 402, as shown in FIG. Figure 4A As another example, Figure 4BFIG. 4 shows a top view of a second array 404 of pixels having a Bayer color filter array under red light. When a red pixel of the second array 404 becomes saturated, charge overflows to the adjacent green and blue pixels of the second array 404, as shown in FIG. Figure 4B The overflow charge (sometimes referred to as blooming or blooming charge) is added to the light signal, so the pixel signal is no longer proportional to the illumination level. As described in more detail below, the nonlinearity introduced by the blooming charge can cause undesirable image artifacts (sometimes referred to as blooming artifacts) to be included in the HDR image.

[0040] An HDR image may be created by combining pixel signals from two reads of the pixel array 210 at different exposure durations. For example, in some implementations, the image sensor 106 may include a digital lateral overflow (DLO) pipeline that determines a combined linearized HDR response based on pixel signals from a long duration read and a short duration read. The pixel signal generated by the image sensor pixel 212 that has become saturated is not proportional to the illumination. Therefore, the DLO pipeline clips the pixel signals above the predetermined barrier from the long duration read and the short duration read, and assumes that the pixel signals below the predetermined barrier are proportional to the illumination. When affected by blooming, the image sensor pixel 212 may generate a pixel signal that is below the predetermined barrier but is higher than its light signal due to, for example, overflowing charge from one or more adjacent pixels that have become saturated. Even though they are not proportional to the illumination, these pixel signals are not clipped because their values ​​are below the predetermined barrier. Therefore, a combined HDR image created using these nonlinear pixel signals may exhibit color artifacts, especially after color processing. In other words, blooming may cause color artifacts in the combined HDR image. Blooming can also result in fixed pattern noise (sometimes called "structure noise") when pixels become saturated at different light levels, such as when there is variation in full well capacity.

[0041] As described above, the DLO pipeline determines a combined linearized HDR response based on the long-read and short-read pixel signals. Figure 5A is a schematic diagram of a first time image 502 (sometimes referred to as a “T1 image”) generated by a portion of pixel array 210 . Figure 5Bis a schematic diagram of a second time image 504 (sometimes referred to as a “T2 image”) generated by a portion of the pixel array 210. The exposure of the second time image 504 (an example of a “second exposure”) is lower than the exposure of the first time image 502 (an example of a “first exposure”). Exposure is the product of the integration time and the light intensity. Therefore, in some specific implementations, the exposure duration of the second time image 504 (an example of a “second exposure duration”) is shorter than the exposure duration of the first time image 502 (an example of a “first exposure duration”). Alternatively or in addition, the image sensor 106 may include one or more attenuation filters for reducing the intensity of the incident light of the second time image 504. Alternatively or in addition, the image sensor 106 may use a detector with a smaller area to reduce the intensity of the incident light of the second time image 504. As described in more detail below, the image sensor 106 is configured to determine when a first pixel signal T1(c) generated by a center pixel in the first temporal image 502 may be affected by blooming based on two or more of a plurality of adjacent pixel signals T1(1), T1(2), T1(3), T1(4), T1(5), T1(6), T1(7), and T1(8) generated by a plurality of adjacent pixels in the first temporal image 502. The center pixel may be any pixel in the pixel array.

[0042] Figure 6 6 is a flow chart of an example of a method 600 for determining a first image value T1*(c) of a center pixel in a first time image 502 according to some specific implementations. For a given pixel array, the method 600 may be performed for one or more pixels in the given pixel array. To simplify the explanation, the method 600 is described in detail in FIG. Figure 6 The operations are depicted and described as a series of operations. However, these operations can be performed in various orders and / or simultaneously, and / or with other actions not presented and described herein.

[0043] At block 602, a first pixel signal T1(c) is compared to a barrier value B1. The first pixel signal T1(c) is generated by a center pixel in the first temporal image 502. The barrier value B1 may include a predetermined threshold above which the center pixel may be considered saturated or nearly saturated. For example, for an image in a 12-bit data format with a maximum signal level of 4095 digital numbers (DN), the center pixel may become saturated or nearly saturated when the first pixel signal T1(c) is above 3600 DN, and therefore, B1 may be set to 3600 DN. When the first pixel signal T1(c) is above the barrier value B1, at block 604, the first image value T1*(c) is set to the barrier value B1. Alternatively, when the first pixel signal T1(c) is equal to or lower than the barrier value B1, two or more of the adjacent pixel signals T1(1), T1(2), T1(3), T1(4), T1(5), T1(6), T1(7) and T1(8) are compared with the barrier value B1. The adjacent pixel signals T1(1), T1(2), T1(3), T1(4), T1(5), T1(6), T1(7) and T1(8) are generated by adjacent pixels in the first time image 502.

[0044] In some implementations, the neighboring pixel signals T1(1), T1(2), T1(3), T1(4), T1(5), T1(6), T1(7), and T1(8) generated by all eight neighboring pixels are compared with the barrier value B1. Fig. 7A As shown by the shading in , each of the adjacent pixel signals T1(1), T1(2), T1(3), T1(4), T1(5), T1(6), T1(7) and T1(8) generated by all eight adjacent pixels can be compared with the barrier value B1. However, in some cases, it is not necessary to check the adjacent pixel signals generated by some adjacent pixels. For example, the pixel signal generated by a green pixel may not be affected by blooming when one or more of its adjacent green pixels becomes saturated. Therefore, in some specific implementations, less than all eight adjacent pixel signals T1(1), T1(2), T1(3), T1(4), T1(5), T1(6), T1(7) and T1(8) are checked. For example, as Figure 7B As shown by the shading in , the adjacent pixel signals T1(4) and T1(5) generated by two adjacent pixels located next to the central pixel on the same row as the central pixel can be compared with the barrier value B1. As another example, Figure 7C As shown by the shading in , the adjacent pixel signals T1(2) and T1(7) generated by two adjacent pixels located immediately above and below the central pixel on the rows above and below the central pixel can each be compared with the barrier value B1. As another example, Fig.7DAs shown by the shading in , the adjacent pixel signals T1(2), T1(4), T1(5) and T1(7) generated by four adjacent pixels located non-diagonally adjacent to the center pixel can be compared with the barrier value B1. As another example, Fig. 7E As shown by the shading in , the neighboring pixel signals T1(1), T1(3), T1(6) and T1(8) generated by four neighboring pixels located diagonally adjacent to the center pixel can each be compared with the barrier value B1.

[0045] Back to Figure 6 At block 606 in FIG. 1 , each of the neighboring pixel signals T1(2), T1(4), T1(5), and T1(7) of four neighboring pixels located non-diagonally adjacent to the center pixel is compared with the barrier value B1. When all of the neighboring pixel signals T1(2), T1(4), T1(5), and T1(7) are at or below the barrier value B1, at block 608, the first image value T1*(c) is set to the value of the first pixel signal T1(c).

[0046] Alternatively, when any of the adjacent pixel signals T1(2), T1(4), T1(5) and T1(7) is higher than the barrier value B1, the first pixel signal T1(c) may be affected by blooming. As described above, a pixel affected by blooming may generate a pixel signal that is not proportional to the illumination. However, because the exposure duration of the second time image 504 is less than the exposure duration of the first time image 502, the second pixel signal T2(c) generated by the center pixel may not be affected by blooming. Therefore, when any of the adjacent pixel signals T1(2), T1(4), T1(5) and T1(7) is higher than the barrier value B1, the first image value T1*(1) is set to the product of the second pixel signal T2(c) and the exposure ratio G1 at box 610. The exposure ratio G1 can be determined based on the exposure duration and / or gain value of the first time image 502 and the second time image 504. When the gain values ​​of the first time image 502 and the second time image 504 are the same, the exposure ratio G1 may be a ratio of the exposure duration of the first time image 502 to the exposure duration of the second time image 504. For example, when the exposure duration of the first time image 502 is twice the exposure duration of the second time image 504 and the gain value is the same for both images, the exposure ratio G1 may be set to 2.

[0047] Figure 8 is a method for determining the DLO image value T of the center pixel in the first time image 502 according to some specific implementations. DLO (c) is a flowchart of an example of method 800. For a given pixel array, method 800 may be performed for one or more pixels in the given pixel array. To simplify the explanation, method 800 is shown in FIG. Figure 8The operations are depicted and described as a series of operations. However, these operations can be performed in various orders and / or simultaneously, and / or with other actions not presented and described herein.

[0048] At block 802, a first image value T1*(c) is determined. The first image value T1*(c) may be determined using any of the methods described herein (including, for example, method 600 described above). Then, at block 804, a second image value T2*(c) is determined as the smaller of the second pixel signal T2(c) and the barrier value B1. At block 806, a sum image value T2*(c) of a combined HDR image is determined by adding the first image value T1*(c) and the second image value T2*(c). SUM (c) At block 808, by adding the sum image value T SUM (c) Multiply by the gain factor G2 to determine the DLO image value T DLO (c) The gain factor G2 may be determined based on the integration time and / or the gain value. In some implementations, the HDR image is constructed from more than two exposures. When there are N exposures, T1 to T N-1 The image can be processed in a similar manner as image T1 in the double exposure example described herein. N-1 Image, the barrier level can be different, for example B 1_1 To B 1_N-1 , and T N The image can be processed in a similar manner as T2 in the two exposure example described herein.

[0049] In some implementations, a noise threshold may be implemented to avoid reducing the quality of the resulting HDR image. For example, if the second pixel signal T2(c) is low, the noise level of the second pixel signal T2(c) may be high because the noise floor is dominant. Using a low second pixel signal to determine the first image value of one of the image sensor pixels 212 affected by blooming may reduce the signal-to-noise ratio (SNR) of the combined HDR image. Therefore, a noise threshold may be implemented to avoid using a low second pixel signal. In some implementations, the first pixel signal T1(c) is compared to a noise threshold B2 to determine when the second pixel signal T2(c) is low. When the first pixel signal T1(c) is below the noise threshold B2, the second pixel signal T2(c) may not be used to determine the first image value T1*(c) even when one or more of the adjacent pixel signals T1(1), T1(2), T1(3), T1(4), T1(5), T1(6), T1(7), and T1(8) are above the barrier value B1. Conversely, when the first pixel signal T1(c) is lower than the noise threshold B2 and one or more of the adjacent pixel signals T1(1), T1(2), T1(3), T1(4), T1(5), T1(6), T1(7) and T1(8) are higher than the barrier value B1, the first image value T1*(c) may be set to the value of the first pixel signal T1(c). The noise threshold B2 is less than the barrier value B1.

[0050] Fig. 9 is a flow chart of an example of a method 900 for determining a first image value T1*(c) of a center pixel in a first temporal image 502 with noise filtering according to some specific implementations. For a given pixel array, the method 900 may be performed for one or more pixels in the given pixel array. To simplify the explanation, the method 900 is described in detail in FIG. Fig. 9 The operations are depicted and described as a series of operations. However, these operations can be performed in various orders and / or simultaneously, and / or with other actions not presented and described herein.

[0051] At box 902, the first pixel signal T1(c) is compared to a barrier value B1. When the first pixel signal T1(c) is above the barrier value B1, the first image value T1*(c) is set to the barrier value at box 904. Alternatively, when the first pixel signal T1(c) is at or below the barrier value B1, the first pixel signal T1(c) is compared to a noise threshold value B2 at box 906. When the first pixel signal T1(c) is below the noise threshold value B2, the first image value T1*(c) is set to the value of the first pixel signal T1(c) at box 908. Alternatively, when the first pixel signal T1(c) is equal to or above the noise threshold value B2, one or more of the adjacent pixel signals T1(1), T1(2), T1(3), T1(4), T1(5), T1(6), T1(7), and T1(8) are compared to the barrier value B1. For example, at box 910, the neighboring pixel signals T1(1), T1(3), T1(6), and T1(8) of four neighboring pixels located diagonally adjacent to the center pixel are each compared with the barrier value B1. When any of the neighboring pixel signals T1(1), T1(3), T1(6), and T1(8) is higher than the barrier value B1, the first image value T1*(1) is set to the product of the second pixel signal T2(c) and the exposure ratio G1 at box 912. Alternatively, when each of the neighboring pixel signals T1(1), T1(3), T1(6), and T1(8) is at or below the barrier value B1, the first image value T1*(c) is set to the first pixel signal T1(c) at box 908.

[0052] Fig.10 Graphs of T1 and T2 signals generated by red and green pixels of pixel array 210 under green light at different amounts of photon flux are shown. Photon flux is the number of photons striking pixel array 210 per unit time. Fig.10 In the embodiment, the T1 signal generated by the green pixel (referred to herein as the "T1 green pixel signal") increases linearly with the photon flux until the T1 green pixel signal reaches the barrier value B1 and is clipped (e.g., as described above with respect to Figure 6 In addition, Fig.10 In FIG. 1 , when the T1 green pixel signal is clipped to the barrier value B1 and the T1 red pixel signal is lower than the barrier value B1, the T1 signal generated by the red pixel (referred to herein as “T1 red pixel signal”) increases nonlinearly with the photon flux. Fig.10 As shown in FIG. 1 , the T2 signal generated by the red pixel (referred to herein as the "T2 red pixel signal") is linear over different amounts of photon flux for which the T1 red pixel signal is nonlinear. Using the systems and methods described herein, a portion of the T2 red pixel signal that is proportional to the illumination is used to replace the nonlinear portion of the T1 red pixel signal. Fig.10As shown, the result is that the first image value of the red pixel increases linearly with the photon flux until T1 the red pixel signal reaches the barrier value B1 and is clipped.

[0053] Fig.11 is a flow chart of an example of a method 1100 for removing blooming artifacts in high dynamic range imaging according to some specific implementations. To simplify the explanation, the method 1100 is Fig.11 1100 is described in and described as a series of operations. However, these operations can be performed in various orders and / or simultaneously, and / or with other actions not presented and described herein. At box 1102, a first pixel signal is detected to be at or below a barrier value B1. The first pixel signal is generated by a center pixel during a first exposure duration. At box 1104, a plurality of adjacent pixel signals are compared with the barrier value. The plurality of adjacent pixel signals are generated during the first exposure duration. At box 1106, a first image value of the center pixel is determined based on a second pixel signal when at least one of the plurality of adjacent pixel signals is higher than the barrier value. The second pixel signal is generated by the center pixel during a second exposure duration shorter than the first exposure duration. At box 1108, when each of the plurality of adjacent pixel signals is at or below the barrier value, the first image value is set to the first pixel signal. Method 1100 can be performed for one or more additional pixels in a pixel array as a center pixel.

[0054] Many electrical connections in the drawings are shown as directly coupled without intervening devices, but are not explicitly described as such in the above description. However, for electrical connections shown in the drawings without intervening devices, this paragraph should serve as an antecedent basis for the claims for referring to any electrical connection as "directly coupled."

[0055] The above discussion is intended to illustrate the principles and various specific implementations of the present invention. Once the above disclosure is fully understood, many variations and modifications will become apparent to those skilled in the art. The following claims are interpreted as being intended to include all such variations and modifications.

Claims

1. An image sensor, comprising: a pixel array comprising a plurality of image sensor pixels; and A controller, the controller being configured to: For one or more of the plurality of image sensor pixels: detecting a first pixel signal at or below a barrier value, the first pixel signal generated by a pixel of the plurality of image sensor pixels during a first exposure duration, the pixel being a center pixel, comparing a plurality of adjacent pixel signals to the barrier value, the plurality of adjacent pixel signals being generated during the first exposure duration, and When at least one of the plurality of adjacent pixel signals is above the barrier value, a first image value of the center pixel is determined based on a second pixel signal generated by the center pixel during a second exposure duration shorter than the first exposure duration.

2. The image sensor of claim 1 , wherein to determine the first image value based on the second pixel signal, the controller is further configured to set the first image value to a product of the second pixel signal and an exposure ratio between the second exposure duration and the first exposure duration.

3. The image sensor according to claim 1, wherein the controller is further configured to: comparing the first pixel signal to a noise threshold value less than the barrier value, and When the first pixel signal is lower than the noise threshold, the first image value is set as the first pixel signal. 4 . The image sensor of claim 1 , wherein the controller is further configured to set the first image value to the first pixel signal when each of the plurality of adjacent pixel signals is at or below the barrier level. 5 . The image sensor of claim 1 , wherein the plurality of adjacent pixel signals are generated by image sensor pixels located adjacent to the center pixel on a row or column of the pixel array. 6 . The image sensor of claim 1 , wherein the plurality of adjacent pixel signals are generated by image sensor pixels positioned diagonally adjacent to the center pixel.

7. The image sensor of claim 1, wherein the controller is further configured to generate a high dynamic range image based on the first image value for each of the one or more of the plurality of image sensor pixels. 8 . The image sensor according to claim 1 , wherein the controller is further configured to read out the first pixel signal, the second pixel signal, and the plurality of adjacent pixel signals.

9. An imaging system, comprising: Lens system; an image sensor in operative relationship with the lens system and comprising a pixel array including a plurality of image sensor pixels; and An imaging controller, the imaging controller being configured to: For one or more of the plurality of image sensor pixels: detecting a first pixel signal at or below a barrier value, the first pixel signal being generated by a center pixel during a first exposure period, comparing a plurality of adjacent pixel signals to the barrier value, the plurality of adjacent pixel signals being generated during the first exposure period, When at least one of the plurality of adjacent pixel signals is above the barrier value, determining a first image value of the central pixel based on a second pixel signal generated by the central pixel during a second exposure less than the first exposure, and When each of the plurality of adjacent pixel signals is at or below the barrier value, the first image value is set as the first pixel signal.

10. The imaging system of claim 9, wherein to determine the first image value based on the second pixel signal, the imaging controller is further configured to set the first image value to a product of the second pixel signal and an exposure ratio between the second exposure and the first exposure.

11. The imaging system of claim 9, wherein the imaging controller is further configured to: comparing the first pixel signal to a noise threshold value less than the barrier value, and When the first pixel signal is lower than the noise threshold, the first image value is set as the first pixel signal.

12. The imaging system of claim 9, wherein the plurality of adjacent pixel signals are generated by two of the plurality of image sensor pixels that are located adjacent to the center pixel on a row or column of the pixel array.

13. The imaging system of claim 9, wherein the plurality of adjacent pixel signals are generated by four of the plurality of image sensor pixels that are positioned diagonally adjacent to the center pixel.

14. The imaging system of claim 9, wherein the plurality of adjacent pixel signals are generated by eight of the plurality of image sensor pixels that are located around the center pixel.

15. The imaging system of claim 9, wherein the imaging controller is further configured to generate a high dynamic range image based on the first image value for each of the one or more of the plurality of image sensor pixels. 16 . The imaging system of claim 9 , wherein the imaging controller is further configured to receive the first pixel signal, the second pixel signal, and the plurality of adjacent pixel signals from the image sensor.

17. A method for eliminating blooming artifacts in high dynamic range imaging, the method comprising: For one or more of a plurality of image sensor pixels included in the pixel array: detecting a first pixel signal at or below a barrier value, the first pixel signal being generated by a center pixel during a first exposure duration; comparing a plurality of adjacent pixel signals to the barrier value, the plurality of adjacent pixel signals generated during the first exposure duration; determining a first image value of the center pixel based on a second pixel signal when at least one of the plurality of adjacent pixel signals is above the barrier value, the second pixel signal being generated by the center pixel during a second exposure duration that is shorter than the first exposure duration; as well as When each of the plurality of adjacent pixel signals is at or below the barrier value, the first image value is set as the first pixel signal.

18. The method of claim 17, wherein determining the first image value based on the second pixel signal further comprises: The first image value is set to the product of the second pixel signal and an exposure ratio between the second exposure duration and the first exposure duration.

19. The method according to claim 17, further comprising: comparing the first pixel signal to a noise threshold value less than the barrier value; as well as When the first pixel signal is lower than the noise threshold, the first image value is set as the first pixel signal.

20. The method according to claim 17, wherein the method further comprises: A high dynamic range image is generated for each of the one or more of the plurality of image sensor pixels based on the first image values.

Images