Parallax-preserving merging for phase-detection autofocus in digital imaging systems

By adopting the technology of retained parallax pixel merging in the digital imaging system, the cost and efficiency problems caused by the increase of PDAF pixels in the PDAF method are solved, fast and accurate automatic focus is achieved, and imaging performance under low-light conditions is improved.

CN114827500BActive Publication Date: 2024-11-19SHENZHEN GOODIX TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210225877.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-09-27
Filing Date
2022-03-09
Publication Date
2024-11-19
Estimated Expiration
2042-03-09

AI Technical Summary

Technical Problem

In existing digital imaging systems, the phase detection automatic focus (PDAF) method faces increasing the number of PDAF pixels, resulting in increased hardware costs, extended processing time, reduced imaging resolution and reduced low-light focusing effect when improving the auto focus speed and accuracy. question.

Method used

By adopting a system and method of retained parallax pixel merging in the image sensor array, using consistent application control of pixel actuators, the combined reading of PDAF pixels and imaging pixels is realized, horizontal PDAF parallax information is retained, and the parallax induces structure configuration is configured, Position the merged parallax PDAF pixels to improve the efficiency and accuracy of autofocus.

Benefits of technology

It effectively reduces the number of PDAF pixels, improves the speed and accuracy of automatic focus, and reduces hardware cost and processing time, and improves the imaging quality under low-light conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114827500B_ABST
    Figure CN114827500B_ABST
Patent Text Reader

Abstract

Techniques are described for pixel merging that preserves parallax during coherently merged parallel readout of an imaging sensor array having phase detection autofocus (PDAF) pixels and imaging pixels. Each group of PDAF pixels and each group of imaging pixels are coupled to a pixel actuator according to a particular arrangement such that coherently applied control of the pixel actuator results in a desired merging of the PDAF pixels with the imaging pixels. According to some embodiments, while such control of the pixel actuator is coherently applied to pixels of the array, parallel readout of the sensor array produces diagonally merged imaging pixels, but vertically merged PDAF pixels to preserve horizontal PDAF parallax information. Additionally or alternatively, a parallax inducing structure is configured to position PDAF pixels of the same parallax such that coherently applied control of the pixel actuator preserves parallax information during merging.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to digital imaging systems, and more particularly, to parallax-preserving merging for digital imaging systems, such as for phase-detection autofocus in smartphone cameras and / or other digital cameras. Background Art

[0002] Many electronic devices include digital imaging systems. For example, most modern smartphones include one or more digital cameras. Almost all modern digital cameras (and many other digital imaging systems) include autofocus functions implemented according to various autofocus methods. Two common methods are called contrast detection autofocus (CDAF) and phase detection autofocus (PDAF). CDAF is based on the principle that higher contrast is generally associated with better focus. Therefore, the CDAF method generally involves an iterative process of measuring contrast, adjusting the focus setting, and measuring contrast again until the best focus setting is found (e.g., the highest contrast image is formed at the best focus setting). CDAF tends to produce very sharp focus, but the iterative method may tend to be too slow to work well in applications such as tracking moving objects, focusing during video capture, etc.

[0003] PDAF tends to produce significantly faster autofocus solutions than CDAF. PDAF is based on the principle that even slight shifts in the viewpoint from which the image is captured will tend to shift the relative position of each point by an amount and distance corresponding to that point's distance from the focal length of the imaging system (i.e., the disparity). For example, a point that is "in focus" may not have any disparaty, closer points may exhibit positive (or negative) disparaty, and farther points may exhibit negative (or positive) disparaty. Therefore, PDAF methods typically use a PDAF sensor to capture the same image simultaneously from slightly different viewpoints, and the data from the different viewpoints is used to calculate disparity measurements. The imaging system can quickly adjust the focus to the optimal focal length setting based on the amount and direction of the disparity measurement.

[0004] Conventionally, the effect of the PDAF method may be related to the number of PDAF sensors used to obtain data from different viewpoints. For example, a greater number of PDAF sensors can provide higher data resolution for different viewpoints, which may tend to provide more information for the PDAF system to perform more accurate autofocus using this information. However, increasing the number of sensors may also result in increased hardware costs, increased processing time, reduced imaging resolution, reduced low-light focusing effects, and / or other undesirable consequences. Summary of the invention

[0005] Embodiments provide systems and methods for performing parallax-preserving pixel merging during consistent merging parallel readout of an imaging sensor array having phase detection autofocus (PDAF) pixels and imaging pixels. Each group of PDAF pixels and each group of imaging pixels are coupled to a pixel actuator according to a particular arrangement such that consistent application of control of the pixel actuator results in a desired merging of the PDAF pixels and the imaging pixels. According to some embodiments, although such control of the pixel actuator is consistently applied to the pixels of the array, parallel readout of the sensor array produces vertically merged PDAF pixels (thereby preserving horizontal PDAF parallax information) and diagonally merged imaging pixels. According to other embodiments, the parallax-inducing structure is configured to position the same parallax PDAF pixels to be merged together (e.g., by using diagonally placed metal shields) according to the consistent application of control of the pixel actuator so that the parallax information is preserved during merging.

[0006] According to one set of embodiments, a system for preserving parallax merging of phase detection autofocus (PDAF) pixels in an image sensor array is provided. The system includes a pixel block, a pixel actuator, and a controller. The pixel block has an array of photodetector elements, the array of photodetector elements including at least one square array of PDAF pixels interspersed with a plurality of imaging pixels, the at least one square array having first parallax PDAF pixels and second parallax PDAF pixels, each first parallax PDAF pixel being configured to generate a corresponding pixel response corresponding to a first parallax orientation, and each second parallax PDAF pixel being configured to generate a corresponding pixel response corresponding to a second parallax orientation complementary to the first parallax orientation. The pixel actuators are each coupled to a corresponding one of the PDAF pixels and a corresponding subset of the imaging pixels. The controller will selectively actuate the pixel actuators to perform a consistent merged parallel readout of the pixel block, such that the consistent merged parallel readout includes: simultaneously actuating a first subset of the pixel actuators at a first time to output a combination of corresponding pixel responses of the first parallax PDAF pixels as a first merged pixel response corresponding to the first parallax orientation; and simultaneously actuating a second subset of the pixel actuators at a second time to output a combination of corresponding pixel responses of the second parallax PDAF pixels as a second merged pixel response corresponding to the second parallax orientation.

[0007] According to another set of embodiments, a method for preserving parallax merging of phase detection autofocus (PDAF) pixels in an image sensor array is provided. The method includes performing a coherently merged parallel readout of a pixel block having an array of photodetector elements having at least one square array of PDAF pixels interspersed with imaging pixels, the at least one square array including first parallax PDAF pixels and second parallax PDAF pixels, each first parallax PDAF pixel being configured to generate a corresponding pixel response corresponding to a first parallax orientation, and each second parallax PDAF pixel being configured to generate a corresponding pixel response corresponding to a second parallax orientation complementary to the first parallax orientation. The execution includes: resetting the readout circuit for a first time, the readout circuit being configured to generate a pixel output signal based on the actuation state of multiple pixel actuators, each pixel actuator being coupled to a corresponding one of the PDAF pixels and a corresponding subset of the imaging pixels; after the first reset, simultaneously actuating the first subset of the pixel actuators for a first time to output a combination of corresponding pixel responses of the first parallax PDAF pixels as a first merged pixel response corresponding to the first parallax orientation, thereby causing the readout circuit to generate a pixel output signal as a first merged pixel output corresponding to the first parallax orientation; resetting the readout circuit for a second time to reset the pixel output signal after the first simultaneous actuation; and after the second reset, simultaneously actuating the second subset of the pixel actuators for a second time to output a combination of corresponding pixel responses of the second parallax PDAF pixels as a second merged pixel response corresponding to the second parallax orientation, thereby causing the readout circuit to generate a pixel output signal as a second merged pixel output corresponding to the second parallax orientation. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The accompanying drawings illustrate embodiments of the present disclosure, and are incorporated herein by reference and constitute a part of this disclosure. The drawings and the description together serve to explain the principles of the present invention.

[0009] Figure 1 Shown is a block diagram of an imaging sensor environment as context for various embodiments described herein.

[0010] Figure 2 A more detailed view of an example implementation of an imaging sensor array configured to support a phase detection autofocus (PDAF) feature is shown.

[0011] Figure 3 A simplified schematic diagram of a portion of an example image sensor including pixels and readout circuitry supporting one or more binning schemes for use with embodiments described herein is shown.

[0012] Figures 4A-4C Timing diagrams for various unmerged and merged readout schemes are shown.

[0013] Figure 5 An example of applying this merging scheme to a 4x4 pixel block configured according to a standard Bayer RGB color filter array (CFA) is shown.

[0014] Figure 6 An example of a particular diagonal binning scheme applied to an 8×8 pixel block of photodetector elements configured according to a particular RGBW CFA is shown.

[0015] Figure 7 A first example of a pixel block having PDAF pixels and imaging pixels configured to preserve parallax combining according to various embodiments described herein is shown.

[0016] Figure 8 Shows Figure 7 Illustrative output of a coherently merged parallel readout of a block of pixels.

[0017] Fig. 9 A second example of a pixel block having PDAF pixels and imaging pixels configured to preserve parallax combining according to various embodiments described herein is shown.

[0018] Fig.10 Shows Fig. 9 Illustrative output of a coherently merged parallel readout of a block of pixels.

[0019] Fig.11 A third example of a pixel block having PDAF pixels and imaging pixels configured to preserve parallax combining according to various embodiments described herein is shown.

[0020] Fig.12 A fourth example of a pixel block having PDAF pixels and imaging pixels configured to preserve parallax combining according to various embodiments described herein is shown.

[0021] Fig.13 A flow chart of an illustrative method for parallax-preserving merging of PDAF pixels in an image sensor array is shown in accordance with various embodiments described herein.

[0022] Fig.14A and 14B A simplified schematic diagram of a portion of an example image sensor including pixels and readout circuitry is shown to illustrate various methods of merging 4×4 pixel blocks.

[0023] In the drawings, similar components and / or features may have the same reference number. In addition, various components of the same type may be distinguished by following the reference number with a second reference number that distinguishes between the similar components. If only the first reference number is used in the specification, the description applies to any of the similar components having the same first reference number, regardless of the second reference number. DETAILED DESCRIPTION

[0024] In the following description, many specific details are provided for a thorough understanding of the present invention. However, it should be understood by those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, features and techniques known in the art will not be described for the sake of brevity.

[0025] Steering Figure 1 , a block diagram of an imaging sensor environment 100 is shown as a context for various embodiments described herein. The imaging sensor environment 100 is shown as including a processor 110 in communication with a processor-controlled system 120 and a complementary metal oxide semiconductor (CMOS) imaging sensor (CIS) system 130. The imaging sensor environment 100 can be used to implement a digital imaging system in any suitable application context. For example, the processor 110, the processor-controlled system 120, and the CIS system 130 can all be implemented in a smartphone, a digital camera, a wearable device, an implantable device, a laptop, a tablet computer, an e-reader, an Internet of Things (IoT) appliance, or any other suitable context.

[0026] The processor-controlled system 120 is generally intended to represent any suitable system or systems to provide any suitable features of the imaging sensor environment 100, other than features of the CIS system 130. For example, in a smart phone, the processor-controlled system 120 may include subsystems for providing phone and communication features, display features, user interaction features, application processing features, etc. Embodiments of the imaging sensor environment 100 may include one or more processors 110. In some embodiments, the one or more processors 110 are shared between the processor-controlled system 120 and the CIS system 130. In other embodiments, the one or more processors 110 are used by the processor-controlled system 120, and the CIS system 130 has its own one or more dedicated processors 110.

[0027] An embodiment of the CIS system 130 includes a sensor array 140 and a sensor control circuit 150. As described below, the sensor array 140 and the sensor control circuit 150 can communicate via an interface channel 145. The basic function of the sensor array 140 is to capture photons using photodetector elements 142 and convert them into electrons. Therefore, the sensor array 140 can be implemented as an array of such photodetector elements 142, which can be implemented by any suitable photosensitive component or group of components (e.g., photodiodes). In some cases, the sensor array 140 is a high pixel count array, such as a 48 million pixel array. In general, the design of modern image sensors with high pixel counts (e.g., 48 million pixel sensors, etc.) attempts to achieve high frame rates, low read noise, high dynamic range, and / or other features (e.g., longer battery life, less heat, etc.) with minimal power consumption. In some embodiments, each photodetector element 142 may include a photodiode and a filter, which is configured to detect light energy in one or more frequency bands. The photodetector elements 142 may implement a color filter array (CFA) such that the sensor array 140 may detect light energy within at least the color range of the visible spectrum and may output corresponding electrical signals.

[0028] As described more fully herein, some of the pixels of the sensor array 140 may be configured to facilitate a phase detection autofocus (PDAF) feature. For example, multiple groups of photodetector elements 142 may be fully or partially covered by a microlens structure (e.g., an on-chip lens or OCL), a metal shield, or other parallax inducing structure. Such a parallax inducing structure causes different photodetector elements 142 in a particular group to receive light from different portions of a main lens that focuses the light onto the sensor array 140. For example, one or more parallax inducing structures may be applied to a group of four photodetector elements 142 (a 2×2 array) such that each photodetector element 142 in the group receives light substantially only from the left or right side of the main lens, from the top or bottom side of the main lens, from a particular quadrant of the main lens, etc.

[0029] Although not explicitly shown, the sensor array 140 also includes a readout circuit system. In particular, the electrons captured by the photodetector elements 142 (by converting photons) are read out by the readout circuit system. As described below, the readout circuit system can generally include a readout line to selectively feed the analog output signals from the photodetector elements 142 to analog-to-digital converters (ADCs), which can convert the analog output signals into digital output signals for output to the sensor control circuit 150 through the interface channel 145. In some cases, it is desirable to perform so-called "pixel merging" to essentially group information from adjacent pixels together for combined readout. For example, the signals from two adjacent "red" pixels can be combined into a single readout (e.g., fed to a single ADC) to obtain a single digital output value of the combined analog pixel information. One feature of such merging is an increase in signal-to-noise ratio. For example, in low light conditions, combining multiple adjacent pixels of the same color can produce more signal information. Another feature of such merging is a reduction in readout resources. For example, combining pixels for readout can significantly reduce the number of readouts and the associated readout time; and readout can be performed with significantly fewer ADCs, which can reduce area and power consumption.

[0030] The sensor control circuit 150 may include any suitable processor and / or circuit for directing the operation of the sensor array 140, processing signals received from the sensor array 140, and interfacing with other systems (e.g., the processor 110). Some embodiments of the sensor control circuit 150 are implemented as or include a companion integrated circuit (IC) having an integrated interface component, a storage component, and a processing component. For example, the processing component of the sensor control circuit 150 may include one or more central processing units (CPUs), application specific integrated circuits (ASICs), application specific instruction set processors (ASIPs), graphics processing units (GPUs), physical processing units (PPUs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, microcontroller units, reduced instruction set computer (RISC) processors, complex instruction set computer (CISC) processors, microprocessors, etc., or any combination thereof.

[0031] As described herein, the CIS system 130 is configured to provide a novel autofocus feature. Autofocus features are ubiquitous in digital imaging systems, and many such autofocus features are implemented according to a contrast detection autofocus (CDAF) method or a PDAF method. CDAF is based on the principle that higher contrast is generally associated with better focus (e.g., blurred edges in an image can appear as lower overall contrast in the image). Therefore, the CDAF method generally involves an iterative process of measuring contrast, adjusting the focus setting, and measuring the contrast again until the best focus setting is found (e.g., the highest contrast image is formed at the best focus setting). For example, imaging data is obtained by the sensor array 140, and the imaging data can be downsampled and high-pass filtered to obtain a contrast image indicating the contrast level on the image. The CDAF method can analyze the contrast image by performing a global and local search of pixels to actually determine how blurry the image looks. In each of a plurality of subsequent iterations, the focus setting can be adjusted, an updated contrast image can be obtained, and the updated contrast image can be reanalyzed to determine whether the image looks more blurry or less blurry.

[0032] Although CDAF can produce very sharp focus, it generally has various limitations. One limitation is that the contrast information at any individual iteration does not indicate how out of focus the image is, or in which direction (i.e., which direction of focus adjustment will improve focus). Therefore, CDAF may initially adjust the focus setting in the wrong direction, making focus worse, before detecting the error and reversing the process; finding a correct focus setting requires multiple iterations, each time with only small setting changes. In addition, the contrast information at any particular iteration may not indicate whether the best focus setting has been reached. Therefore, CDAF generally continues to adjust focus until it reaches and then passes the best focus, at which point CDAF can determine that the focus has deteriorated again and can respond by falling back to the best focus setting. For at least these reasons, CDAF may tend to be too slow to work well in many applications, such as tracking moving objects, focusing during video capture, etc. For example, CDAF may consume about 30 frames to determine the best focus setting; by that time, many moving objects will have changed positions and be out of focus again. Even for still image capture, users are frustrated by the delay in taking photos caused by CDAF.

[0033] PDAF is based on completely different principles than CDAF and tends to reach the optimal focus setting significantly faster than CDAF. In general, PDAF works similar to human depth perception. Because the human eyes are separated horizontally, each eye sees a slightly different image (i.e., each eye sees the same scene from a slightly different viewpoint). In particular, when both eyes are focused on a particular point in space, that point is actually in the same position in the left and right eye images. However, any point that is closer or farther away from the eyes will tend to be spatially offset in one eye's image relative to the other eye's image. The amount and direction of the offset is mathematically (e.g., geometrically) related to the separation between the eyes and the distance of the object from the eyes (e.g., or the distance from the focal length). The amount and direction of the spatial (phase) offset between two viewpoints is called "parallax."

[0034] PDAF exploits this principle by configuring the PDAF sensors to simultaneously capture the same image from slightly different viewpoints, and using the data from the different viewpoints to calculate disparity measurements. For example, masking and / or lenses are used to configure some sensors to detect a first viewpoint, and other sensors to simultaneously detect a second viewpoint. Typically, the different viewpoints are configured to be complementary, such as right and left, up and down, etc. Instead of iteratively trying different settings to find the best amount and direction of focus adjustment as in CDAF, the disparity measurement calculated in the PDAF method directly indicates the amount and direction of focus adjustment to apply for the best focus.

[0035] Figure 2 A more detailed view of an example embodiment of an imaging sensor array 140 configured to support a PDAF feature is shown. As described above, the imaging sensor array 140 includes an array of photodetector elements (e.g., photodiodes) 142. An embodiment of the imaging sensor array 140 arranges the photodetector elements 142 into an array of pixel blocks 210, such that each pixel block 210 is an instance of an N×M photodiode array, where N and M are positive integers. In some embodiments, N can be equal to M, so that each pixel block is a square photodiode array. In some embodiments, each pixel block 210 has the same number of photodetector elements 142. In some embodiments, each pixel block 210 can generally be arranged according to the same color filter array (CFA) pattern (e.g., a Bayer pattern, etc.). The photodetector elements 142 configured as part of a CFA for normal imaging (e.g., as opposed to for supporting PDAF) are referred to herein as "imaging pixels" 218.

[0036] As described above, groups of adjacent photodetector elements 142 of the imaging sensor array 140 are configured as PDAF pixels 220. Each PDAF pixel 220 is in optical communication with one or more parallax-inducing structures, such as by being fully or partially covered by a microlens structure (e.g., OCL) or a metal shield. Such parallax-inducing structures cause different photodetector elements 142 in a particular group to receive light from different parts of the main lens, which focuses the light onto the sensor array 140. In the illustrated embodiment, groups of four PDAF pixels 220 are covered by a 2×2 OCL, which effectively causes each of the individual PDAF pixels 220 in any particular group to receive light from a corresponding quadrant of the main lens. The illustrated embodiment considers both the upper left quadrant and the lower left quadrant to be “left” and both the upper right quadrant and the lower right quadrant to be “right”, such that a group of four PDAF pixels 220 essentially includes two “left” PDAF pixels (PDAF pixels 220a and 220c) and two “right” PDAF pixels (PDAF pixels 220b and 220d). The illustrated configuration is intended to illustrate only one of various configurations of PDAF pixel 220 groups that may be used with the novel features described herein. In an alternative embodiment, PDAF pixels 220a and 220b are covered by a first 2×1 OCL to form a first left-right pair of PDAF pixels 220, and PDAF pixels 220c and 220d are covered by a second 2×1 OCL to form a second left-right pair of PDAF pixels 220. In another alternative embodiment, PDAF pixels 220a and 220c are covered by a first metal shield (or each by its own metal shield), the first metal shield covering the right half of the pixels, and PDAF pixels 220b and 220d are covered by a second metal shield (or each by its own metal shield), the second metal shield covering the left half of the pixels.

[0037] In some embodiments, each pixel block 210 is configured as a PD block 212 including at least one PDAF pixel 220. In other embodiments, only a subset of the pixel blocks 210 are PD blocks 212; and the other pixel blocks in the pixel blocks 210 that are not PD blocks 212 are non-PD blocks 214. In the illustrated configuration, one-sixth of the pixel blocks 210 are PD blocks 212, and the remaining five-sixths of the pixel blocks 210 are non-PD blocks 214. Each non-PD block 214 can be arranged according to any suitable CFA pattern, etc. For example, the illustrated non-PD blocks 214 are each arranged in a 16×16 photodetector array (i.e., 256 imaging pixels 218) in an “RGGBW” CFA pattern (red, green, green, blue, white). Some such RGGBW blocks are configured as so-called “HexD-W” blocks, in which white pixels (also called “brightness pixels”) are evenly interspersed at a density of approximately fifty percent.

[0038] In the illustrated embodiment, each PD block 212 is implemented as a non-PD block 214, except that some of the CFA pixels are replaced by PDAF pixels 220. As described below, this embodiment consistency can simplify the readout structure, merging and / or other features, as described herein. As described above, the PDAF method involves using data from the PDAF pixels 220 to calculate disparity measurements. Calculating disparity measurements involves complex calculations, including accurate pattern recognition and matching between different viewpoints. Conventionally, the effect of the PDAF method may be related to the number of PDAF pixels 220 used to obtain data from different viewpoints. A larger number of PDAF pixels 220 can provide higher data resolution for pattern matching and disparity calculations in different viewpoints, which can tend to provide more information for the PDAF system to perform more accurate autofocus. For example, some conventional PDAF methods configure approximately 5-10% of the pixels of the imaging sensor array 140 as PDAF pixels 220, while other conventional methods configure up to 100% of the pixels of the imaging sensor array 140 as PDAF pixels 220. Although such a large number of PDAF pixels 220 provide certain features, increasing the number of PDAF pixels 220 may also result in increased hardware costs, increased processing time, reduced imaging resolution, reduced low-light focusing effects, and / or other undesirable consequences. For example, using a large number of OCLs to form a large number of PDAF pixels 220 may be relatively expensive and complex to manufacture; alternatively, using a large number of metal shields to form a large number of PDAF pixels 220 may reduce the amount of light reaching the photodetector element 142 and may reduce imaging quality.

[0039] Some embodiments may include additional limitations on the actual number of PDAF pixels 220 that may be included in imaging sensor array 140. Return to Reference Figure 1 , the illustrated imaging sensor environment 100 shows that the sensor control circuit 150 communicates with the imaging sensor array 140 via an interface channel 145, and the maximum number of conventional PDAF pixels 220 that can be implemented in the sensor array 140 may be limited by the speed and / or bandwidth of the interface channel 145. In an example embodiment, the imaging sensor array 140 has tens of millions of photodetector elements 142 (e.g., a 48 million pixel sensor array), and the imaging sensor array 140 communicates via a relatively low bandwidth interface channel 145 (e.g., a mobile industrial processor interface or ) communicates with a companion IC of the sensor control circuit 150. In such an embodiment, each additional percentage of pixels implemented as PDAF pixels 220 may result in hundreds of thousands of additional PDAF pixel 220 outputs being transmitted via the interface channel 145.

[0040] One approach to solving the above and / or other problems is to reduce the number of PDAF pixels 220. For example, Figure 2 In the illustrated embodiment, only one sixth of the pixel blocks 210 of the imaging sensor array 140 are PD blocks 212, and Figure 2 Each PD block 212 of the array only configures 16 of its 256 photodetector elements 142 as PDAF pixels 220. Thus, in the illustrated embodiment, only about one percent of the photodetector elements 142 are PDAF pixels 220. While it may be desirable to implement a relatively small percentage of the photodetector elements 142 as PDAF pixels 220 from the perspectives of cost, manufacturability, data communication, etc., this approach still has certain limitations. One limitation is that reducing the percentage of PDAF pixels 220 tends to reduce the resolution of the data available for performing the PDAF feature. Therefore, any benefit of further reducing the PDAF pixel 220 count tends to be accompanied by a reduction in autofocus quality. Another limitation is that even a relatively small percentage of a very large array is a relatively large number. Thus, as the size (resolution) of the imaging sensor array 140 increases, even configuring a relatively small percentage of the photodetector elements 142 as PDAF pixels 220 will result in a large number of PDAF pixels 220, which will result in a large amount of PDAF pixel 220 data to be read out, a large amount of disparity calculations to be processed, etc. For example, a 48-megapixel imaging sensor array 140 in which only approximately 1% of its pixels are configured as PDAF pixels 220 will still generate PDAF data from nearly 500,000 PDAF pixels 220 for readout and processing.

[0041] As mentioned above Figure 1 As mentioned, information from photodetector elements 142 of imaging sensor array 140 is read out (by readout circuitry) to generate a set of pixel output information corresponding to an image. In some embodiments, it may be desirable to use the same readout circuitry (e.g., or at least the same readout architecture) to read out imaging pixels 218 and PDAF pixels 220. For example, analog information from the pixels is passed to an ADC, which outputs digital pixel output information.

[0042] In many applications, it is desirable to use pixel binning to group information from adjacent pixels together for combined readout, thereby increasing signal-to-noise ratio (e.g., for improved low-light performance), reducing readout time (e.g., for improved frame rate), reducing area and power consumption (e.g., by using fewer ADCs for each readout cycle), etc. As used herein, "binning" refers to so-called charge binning, in which the charges of multiple pixels (e.g., corresponding to the amount of light detected by the pixel, the number of photons, etc.) are summed, averaged, or otherwise combined onto a readout line at one or more levels. For purposes of illustration, Figure 3A simplified schematic diagram 300 of a portion of an example image sensor is shown, including pixels and readout circuitry that supports one or more binning schemes for use with the embodiments described herein. A portion of the image sensor may be Figure 1 1. A partial embodiment of a sensor array 140 of FIG. 1. The illustrated portion of the array has four pixels, but the features illustrated by schematic diagram 300 may be extended to any suitable number of pixels.

[0043] Each pixel is shown to include a photodiode (PD) (e.g., or any other suitable photosensitive element) and a transfer transistor (TX) coupled to the PD. The TXs of the pixels are all coupled to a floating diffusion node (FD) of the readout circuit system. The readout circuit system includes a reset transistor (RST) having a drain node coupled to a reset voltage reference (VDD_RST), a source node coupled to the FD (i.e., to the TX), and a gate node controlled by a reset signal (RST). RST is configured to charge VDD_RST when RST is turned on, thereby resetting the FD. Each PD can be reset together with the FD by turning on its corresponding TX (e.g., by asserting or de-asserting the TXn control signal). The readout circuit system also includes a source follower transistor (SF) having a drain node coupled to a source follower voltage reference (VDD_SF), a gate node coupled to the FD (i.e., to the TX), and a source node coupled to the drain node of the row select transistor (SEL). SEL has a gate node coupled to a row select line and a source node coupled to a voltage readout line that provides an analog output pixel signal (Vout) to the ADC for data conversion. The source node of SEL is also coupled to a current source (IBIAS). In the illustrated embodiment, TX, RST, SF, and SEL are NMOS transistors. Alternative embodiments may be designed using PMOS transistors and / or other suitable components.

[0044] As shown, embodiments may include a controller that generates clocks and control signals. The controller may be implemented using any suitable hardware, firmware, etc. In some embodiments, the controller is integrated with the sensor array as a sensor component (e.g., as part of the sensor array 140). In other embodiments, the controller is implemented as a sensor external component by a separate controller or processor (e.g., implemented by the sensor control circuit 150). In other embodiments, the features of the controller are distributed between one or more sensor components and one or more sensor external components. For example, the sensor control circuit 150 (outside the sensor) may generate a command that instructs the timing and / or generation of a specific control signal by the controller on the sensor. In some embodiments, the controller may include a processing circuit system, a logic state machine, a phase-locked loop, etc., to provide a clock and control signal to the image sensor.

[0045] In addition, embodiments may include a data memory configured to store digital data representing pixel signals after pixel conversion by the ADC. In some embodiments, the data memory includes a buffer and / or register for temporarily storing the readout data before transmitting the data to other processing components (e.g., before transmitting to the sensor control circuit 150 via the interface channel 145). For example, the readout data is buffered in the data memory for fast, serial transmission to the sensor control circuit 150.

[0046] As shown, each pixel (e.g., a pixel on each row or column) has its own TX. The TXn control signal can be generated by the controller with specific timing to support different merging schemes. For illustration purposes, Figures 4A-4C The timing diagrams of various unmerged and merged readout schemes are shown. Figure 4A As shown in the timing diagram of , schematic 300 can be used to implement a non-merged readout by sequentially actuating each TX with a reset between each actuation to clear the readout circuit system. For example, the RST signal is asserted to turn on the RST transistor, thereby resetting the FD to VDD_RST (e.g., clearing the charge on the readout circuit system). Then, a single TX (e.g., TX1) is actuated to turn on a single PD (e.g., PD1), thereby individually transferring charge from the single PD to the FD. The charge on the FD is then transferred to the readout line and ADC via SF and SEL for data conversion into digital data, which can then be stored in a data memory. Therefore, the readout of four pixels consumes four readout cycles, but the individual pixel information from each pixel can be retained in the initial digital conversion.

[0047] Go to Figure 4B, which represents a first binning scheme, wherein schematic 300 is used to implement readout with four-way binning. For example, the RST signal is asserted to turn on the RST transistor, thereby resetting the FD to VDD_RST (e.g., zeroing the charge on the readout circuitry). Subsequently, all four TXs (TX1–TX4) are actuated simultaneously to turn on their corresponding PDs (PD1–PD4), thereby collectively transferring charge from the multiple PDs to the FD. The combined charge now on the FD is then transferred to the readout line and ADC via SF and SEL for data conversion into digital data, which can then be stored in a data memory. Thus, four pixels are read out in a single readout cycle, but the individual pixel information from each pixel is not retained (i.e., it is combined as part of the readout and initial digital conversion).

[0048] Steering Figure 4C , which shows a second merging scheme, wherein schematic 300 is used to implement dual-path merged readout. For example, the RST signal is asserted to turn on the RST transistor, thereby resetting the FD to VDD_RST (e.g., clearing the charge on the readout circuit system). Subsequently, the first pair of TXs (e.g., TX1 and TX2) are simultaneously actuated to turn on their corresponding PDs (e.g., PD1 and PD2), thereby collectively transferring charge from the two PDs to the FD. The combined charge now on the FD is then transferred to the readout line and ADC via SF and SEL for data conversion to digital data, which can then be stored in a data memory. In the next cycle, RST is again asserted to reset the readout circuit system, and the second pair of TXs (e.g., TX3 and TX4) are simultaneously actuated to turn on their corresponding PDs (e.g., PD3 and PD4), thereby collectively transferring charge from the two PDs to the FD. The combined charge now on the FD is then transferred to the readout line and ADC via SF and SEL for data conversion to digital data, which can then be stored in a data memory. Thus, four pixels are read out in two readout cycles and pixel information is retained only at the pixel pair level (ie, information from pixel pairs is combined as part of the readout and initial digital conversion).

[0049] Different merging schemes may be supported based on which pixels share a common FD, how many pixels share a common FD, the capabilities of the controller, etc. In some embodiments, one or more merging schemes may involve multiple levels of merging. For example, a first merging scheme performs charge merging on 32 RGB pixels to generate 16 merged RGB pixel outputs, and a second merging scheme may re-merge the 16 merged RGB pixel outputs to generate 4 re-merged RGB pixel outputs. Although the above description relates to charge merging, the embodiments may be implemented with other suitable types of merging, such as digital merging, merging in a later post-processing stage, etc.

[0050] Many different CIS systems are implemented with many different types of CFAs and many different types of binning schemes. For example, in a conventional Bayer pattern CFA, pixels of the same color may be grouped prior to ADC readout; in a conventional quad-Bayer pattern, 2×2 pixel clusters may be grouped prior to ADC readout; and so on. Some binning schemes use "horizontal binning," in which horizontally adjacent (usually the same color) pixels are binned together. Some binning schemes use "vertical binning," in which vertically adjacent (usually the same color) pixels are binned together. Some binning schemes use "diagonal binning," in which diagonally adjacent (usually the same color) pixels are binned together.

[0051] To make it clearer, Figure 5 An example of applying this merging scheme to a 4×4 pixel block 510 configured according to a standard Bayer RGB CFA is shown. The illustrated pixel block 510 may be a larger array such as Figure 1 For example, the entire CIS may include millions of total pixels implemented as millions of instances of pixel block 510. Application Reference Figure 4B An example application of the described binning scheme is to actuate all of TX1-TX4 simultaneously. This may actually create binning groups, each with four imaging pixels 218, as follows: R1-R4; G1–G4; G5–G8; and B1–B4. Application Reference Figure 4C Another example application of the described binning scheme is to actuate TX1 and TX2 simultaneously, and then subsequently actuate TX3 and TX4 simultaneously. This may actually create horizontal binning groups, each with two imaging pixels 218, as follows: R1 and R2; R3 and R4; G1 and G2; G3 and G4; G5 and G6; G7 and G8; B1 and B2; and B3 and B4.

[0052] Conventionally, pixel merging schemes tend to be used during imaging frames in which imaging pixels 218 are merged. For example, such pixel merging can allow millions of imaging pixels 218 to be read out with low pixel noise, high frame rates, and / or other characteristics. However, during PDAF frames, conventional CIS systems tend to read out PDAF pixels 220 individually, for example to retain as much parallax information as possible. It can be seen that in merged mode, complications arise between imaging pixels 218 and PDAF pixels 220. CIS systems typically use a row addressing scheme with column-parallel readout. That is, as shown, a control signal is generated that extends from right to left on the array and determines which pixels are read out. Typically, the behavior of all pixels in a row is determined by only a few control signals transmitted in the pixel row, and it is not practical to select pixels in the row with different behaviors.

[0053] For example, assume that the B1 and B2 pixels are replaced by a pair of PDAF pixels 220 (e.g., covered by 1×2 OCL), such that the B1 imaging pixel 218 becomes the “left” PDAF pixel 220, and the B2 imaging pixel 218 becomes the “right” PDAF pixel 220. A horizontal diagonal binning scheme applied consistently to all pixels would bin together the two PDAF pixels 220. This would effectively bin the left and right information, and would not preserve any of the disparity information used to perform PDAF.

[0054] Similar problems arise for other types of CFAs in other merging modes. For example, Figure 6 An example of a particular diagonal binning scheme 300 is shown for an 8×8 pixel block 610 of photodetector elements 142 configured according to a particular RGBW CFA. The illustrated RGBW CFA includes approximately fifty percent W pixels and approximately fifty percent RGB pixels; each position in the array alternates between RGB pixels and W pixels, such that the W pixels are evenly interspersed between the RGB pixels. The illustrated pixel block 610 may be a larger array, such as Figure 1 610. For example, the entire CIS may include millions of total pixels implemented as thousands of instances of pixel block 610.

[0055] exist Figure 6, pixel blocks 610 are merged to generate a 4×8 downsampled block 620. Each marked box in the pixel block 610 represents an unmerged pixel 615. Each marked box in the downsampled block 620 represents a merged pixel 625. Each mark indicates that the color is "R", "G", "B" or "W", representing red, green, blue or white, respectively. Each mark also indicates a binning index. For example, the pixel at the lower right position in the illustrated array is shown as "R0", which indicates that the pixel is red and is part of the 0th binning group; and the pixel at the upper left position in the illustrated array is shown as "B24582", which indicates that the pixel is blue and is part of the 24582nd binning group. Figure 6 The particular merge index number shown in may represent the case where the illustrated pixel block 610 is the lower rightmost block of a larger array that includes thousands of such blocks 610 .

[0056] As shown, each unmerged pixel 615 in each row of the pixel block 610 has a different merge index than each other unmerged pixel 615 in the row. For example, the bottom row of the pixel block 610 includes eight unmerged pixels 615 indexed from "0" to "7". However, each unmerged pixel 615 in each second row of the pixel block 610 shares a merge index with a diagonally adjacent unmerged pixel in the unmerged pixel 615 in the row immediately below it. For example, the unmerged pixel 615a in the top row (i.e., the eighth row) of the pixel block 610 is a white pixel in the 24,583rd merge group, and the diagonally adjacent unmerged pixel 615b in the immediately next row (i.e., the seventh row) of the pixel block 610 is also a white pixel in the 24,583rd merge group. This is intended to mean that binning of pixels in pixel block 610 involves combining (e.g., summing, averaging, etc., their charges) those unbinned pixels 615 having the same index (e.g., by turning on those pixels simultaneously to pass the combined charge to readout circuitry, as described with reference to FIG. Figure 3 For example, unbinned pixel 615a is merged with diagonally adjacent unbinned pixel 615b to be read out as a single binned pixel 625a in downsampling block 620.

[0057] Using the method illustrated, it can be seen that binning essentially produces the same number of columns and half the number of rows. Therefore, binning can read out the entire array with half the number of read operations. This binning method generally falls into the category of "diagonal binning" and may be more specifically referred to as a "1Hx2V" binning method, which indicates that the horizontal dimension is divided by '1' (i.e., the array is not downsampled in the horizontal dimension), and the vertical dimension is divided by '2' (i.e., the array is downsampled twice in the vertical dimension).

[0058] like Figure 5As shown, it can be seen that in the illustrated diagonal merging mode, complications arise between imaging pixels 218 and PDAF pixels 220. For example, assume that two R0 pixels and two W1 pixels are replaced by a 2×2 PDAF pixel group 220 (e.g., covered by a 2×2 OCL), so that the pair of R0 and W1 pixels on the left side of the group are actually "left" pixels, while the pair of R0 and W1 pixels on the right side of the group are actually "right" pixels. The diagonal merging scheme applied consistently across pixel block 610 will generate two merged pixels 625, each of which has a combination of left and right information that cannot retain any information in the PDAF disparity information.

[0059] Conventionally, various techniques may be used in CIS systems to retain PDAF information. One such conventional technique is to perform pixel merging only on rows that do not include any PDAF pixels 220 (i.e., pixel merging is not performed on any pixel rows that include PDAF pixels 220). This approach may retain the parallax information of the PDAF pixels 220, but at the expense of any benefits of pixel merging of the imaging pixels 218 in those rows. Another such conventional technique is to order the pixel readout process so that the PDAF pixels 220 are read out individually in a first time frame and the imaging pixels 218 are read out in subsequent time frames with pixel merging. This approach may retain parallax information and may obtain the benefits of pixel merging of all imaging pixels 218, but this approach tends to increase row read times, increase noise, and greatly increase readout complexity. In the above and other conventional methods, there is often a tradeoff between performing pixel merging and retaining PDAF parallax information.

[0060] Embodiments described herein seek to apply consistent pixel binning across the imaging sensor array 140 to the PDAF pixels 220 and imaging pixels 218 in a manner that preserves PDAF disparity information. Embodiments operate in the context of a pixel block comprising an array of photodetector elements 142 forming a portion of the imaging sensor array 140, the pixel block comprising a group of imaging pixels 218 and PDAF pixels 220. Each group of PDAF pixels 220 is arranged such that a consistent binning parallel (column-parallel or row-parallel) readout across the imaging pixels 218 and the PDAF pixels 220 preserves the PDAF disparity in the PDAF pixel group 220. In some embodiments, such an arrangement involves coupling the PDAF pixels 220 to corresponding pixel actuators in an order that preserves PDAF disparity during a consistent binning parallel readout of the imaging sensor array 140. For example, the imaging pixels 218 are coupled to the pixel actuators in an order that produces diagonal binning, while the PDAF pixels 220 are coupled to the same pixel actuators in an order that produces vertical binning. In other embodiments, such an arrangement involves shielding PDAF pixels 220 to generate disparity information such that the disparity information is preserved during a coherent binning parallel readout of imaging sensor array 140. For example, all pixels (imaging pixels 218 and PDAF pixels 220) of imaging sensor array 140 are coupled to pixel actuators in an order to produce diagonal binning, and metal shielding is applied to PDAF pixels 220 in a diagonal manner such that such diagonal binning preserves horizontal PDAF disparity information.

[0061] Figure 7-12 Various examples of pixel blocks 710 having PDAF pixels 220 and imaging pixels 218 configured to preserve parallax merging are shown in accordance with various embodiments described herein. The pixel blocks 710 may be Figure 1 710. For example, the imaging sensor array 140 may be composed of a large number (e.g., thousands) of instances of the pixel block 710. Figure 7-12 As shown, the pixel block 710 includes an array of photodetector elements (e.g., photodetector elements 142) that includes at least one square array 720 of PDAF pixels 220 interspersed with a plurality of imaging pixels 218. For example, the illustrated pixel block 710 includes two such square arrays 720, each having four PDAF pixels 220. In other embodiments, each pixel block 710 may include only a single square array 720, more than two square arrays 720, a larger square array, etc.

[0062] Each square array 720 includes a plurality of (e.g., two) first parallax PDAF pixels 220A and a plurality of (e.g., two) second parallax PDAF pixels 220B. Each of the first parallax PDAF pixels 220A is configured to generate a corresponding pixel response corresponding to a first parallax orientation, and each of the second parallax PDAF pixels 220B is configured to generate a corresponding pixel response corresponding to a second parallax orientation complementary to the first parallax orientation. In the illustrated embodiment, the first parallax PDAF pixel 220A may be a "left" PDAF pixel (corresponding to receiving light typically from the left half of the main imaging lens), and the second parallax PDAF pixel 220B may be a "right" PDAF pixel (corresponding to receiving light typically from the right half of the main imaging lens). Alternatively, the first parallax PDAF pixel 220A can be an "upper" PDAF pixel (corresponding to receiving light generally from the upper half of the main imaging lens), and the second parallax PDAF pixel 220B can be a "lower" PDAF pixel (corresponding to receiving light generally from the lower half of the main imaging lens); or, the first parallax PDAF pixel 220A and the second parallax PDAF pixel 220B can correspond to a pair of any suitable complementary parallax orientations.

[0063] As described above, the parallax of the PDAF pixel 220 may be generated by an optical interaction between the PDAF pixel 220 and one or more parallax inducing structures disposed on top of the PDAF pixel 220. Such parallax inducing structures may typically include an on-chip lens (e.g., also referred to as a microlens) and / or a metal shield. Figure 7-12Each of the square arrays 720 shown in is a 2×2 array. In some embodiments of such a 2×2 array of PDAF pixels 220, a parallax inducing structure is implemented by a 2×2 on-chip lens (OCL) disposed on top of the 2×2 array so that the first PDAF pixel 220Aa and the third PDAF pixel 220Ab (i.e., the two first parallax PDAF pixels 220A) of the 2×2 array jointly receive light generally from the first half of the main imaging lens, and so that the second PDAF pixel 220Ba and the fourth PDAF pixel 220Bb (i.e., the two second parallax PDAF pixels 220B) of the 2×2 array jointly receive light generally from the second half of the main imaging lens. For example, the OCL is configured so that light generally from the left side of the main imaging lens is focused onto the first parallax PDAF pixel 220A, and light generally from the right side of the main imaging lens is focused onto the second parallax PDAF pixel 220B. In another embodiment of such a 2×2 array of PDAF pixels 220, the parallax inducing structure is implemented by two 2×2 OCLs, each OCL being disposed on corresponding two of the 2×2 PDAF pixels 220. For example, the first of the 2×1 OCLs is disposed on top of the first PDAF pixel 220Aa and the second PDAF pixel 220Ba of the 2×2 array to form a first pair of complementary PDAF pixels 220 (e.g., a first pair of "left"-"right" pixels), and the second of the 2×1 OCLs is disposed on top of the third PDAF pixel 220Ab and the fourth PDAF pixel 220Bb of the 2×2 array to form a second pair of complementary PDAF pixels 220 (e.g., a second pair of "left"-"right" pixels). In another embodiment of such a 2×2 array of PDAF pixels 220, the parallax inducing structure is implemented by metal shields, each of which is disposed on top of the corresponding PDAF pixel 220 to cover approximately half of the PDAF pixel 220, so that the PDAF pixel 220 actually receives light only from one side. For example, the first metal shield and the third metal shield cover the right half of each of the first PDAF pixel 220Aa and the third PDAF pixel 220Ab (i.e., the two first parallax PDAF pixels 220A) so that they jointly receive light that is generally from the left half of the main imaging lens, and the second metal shield and the fourth metal shield cover the left half of each of the second PDAF pixel 220Ba and the fourth PDAF pixel 220Bb (i.e., the two second parallax PDAF pixels 220B) so that they jointly receive light that is generally from the right half of the main imaging lens. As used herein, phrases such as "light generally comes from" are not intended to be vague, but rather to convey a practical reality that any type of parallax-inducing structure can generally, but not perfectly, direct light from different parts of the main lens into any particular PDAF pixel 220.

[0064] for Figure 7-12 In each of the pixel blocks 710 shown, it is assumed that each pixel is electrically coupled to a pixel actuator. The pixel actuator may include any circuit and / or structure that selectively actuates the corresponding coupled pixel in response to a control signal. Some embodiments of the pixel actuator are described in detail in detail. Figure 3-5 For example, each pixel actuator includes a transistor, a switch, a gate, and / or other device electrically coupled between the output of one or more corresponding pixels and the input of the readout circuit. In the actuation mode, the pixel actuator actually transmits the pixel response (e.g., an analog voltage response based on the number of photons detected) of its coupled pixel to the readout circuit (e.g., for conversion into a digital output signal). Although the pixel actuator is not in the actuation mode, the pixel actuator is not electrically coupled to the readout circuit. Figure 7-12 , but their coupling is indicated by the "TX" designator. For example, "TX1" marked in a specific pixel indicates that the specific pixel is coupled to the first pixel actuator, so that the first pixel actuator can be selectively used to actuate the specific pixel.

[0065] As described above, embodiments operate in the context of coherent merged parallel readout of imaging sensor array 140. One intention of the term "coherent merged parallel readout" is that Figure 7-12 Readout of the pixel block 710 (e.g., and potentially the entire imaging sensor array 140) is performed in a column-parallel or row-parallel manner. For example, each pixel actuator is coupled to each of the pixels in a row of the pixel block 710 via a pixel actuation bus; and a column-parallel readout is performed to sequentially read out the pixel rows in parallel. Alternatively, each pixel actuator is coupled to each of the pixels in a column of the pixel block 710 via a pixel actuation bus; and a row-parallel readout is performed to sequentially read out the pixel columns in parallel. Another intention of the term "consistently merged parallel readout" is that the readout follows a particular merging scheme, and that the merging scheme is consistently applied to the pixel block 710 (e.g., and potentially to the entire imaging sensor array 140) for both the imaging pixels 218 and the PDAF pixels 220. For example, if part of a parallel readout involves simultaneously actuating a particular set of pixel actuators (e.g., a pair of pixel actuators) to cause charge merging to occur, the same set of pixel actuators will be actuated simultaneously regardless of whether the readout is of a set of imaging pixels 218 or a set of PDAF pixels 220 (although, as described herein, based on the pixel coupling arrangement, such a consistent merging scheme may actually result in diagonal merging of imaging pixels 218 and vertical merging of PDAF pixels 220, etc.).

[0066] Specifically, turning Figure 7, shows a first example of a pixel block 710a having PDAF pixels 220 and imaging pixels 218 configured for parallax-preserving merging according to various embodiments described herein. The illustrated pixel block 710a includes two square arrays 720, each having a 2×2 arrangement of PDAF pixels 220. Specifically, each square array 720 includes two first parallax PDAF pixels 220A and second parallax PDAF pixels 220B, which are respectively shown as "left" and "right" PDAF pixels 220 relative to the orientation illustrated in the pixel block 710a. The imaging pixels 218 of the pixel block 710a substantially follow a HexD-W CFA arrangement for which diagonal merging is ideal. For example, it can be seen that the 2×2 block of 4 imaging pixels 218 adjacent to the first square array 720a includes a first "blue" imaging pixel 218a, a first "white" imaging pixel 218b, a second "blue" imaging pixel 218c, and a second "white" imaging pixel 218d. During merging, it is desirable to diagonally merge "blue" pixel information from diagonally adjacent imaging pixels 218a and 218d together; and diagonally merge "white" pixel information from diagonally adjacent imaging pixels 218b and 218c together. However, it can be seen that for PDAF pixels 220, diagonal merging will result in each merged pair including a first parallax PDAF pixel 220A and a second parallax PDAF pixel 220B, thereby losing left and right PDAF parallax information.

[0067] Embodiments arrange electrical couplings between the individual pixels of each square array 720 such that consistent binning results in diagonal binning of imaging pixels 218 and vertical binning of PDAF pixels 220. As shown, in each 2×2 block of imaging pixels 218, a first (e.g., upper left) imaging pixel 218a is actuated by a first pixel actuator (TX1), a second (e.g., upper right) imaging pixel 218b is actuated by a second pixel actuator (TX2), a third (e.g., lower left) imaging pixel 218c is actuated by a third pixel actuator (TX3), and a fourth (e.g., lower right) imaging pixel 218d is actuated by a fourth pixel actuator (TX4). In each square array 720 of the PDAF pixels 220, the first (e.g., upper left) PDAF pixel 220 (first first parallax PDAF pixel 220Aa) is coupled to the first pixel actuator (TX1), the second (e.g., upper right) PDAF pixel 220 (first second parallax PDAF pixel 220Ba) is coupled to the second pixel actuator (TX2), the third (e.g., lower left) PDAF pixel 220 (second first parallax PDAF pixel 220Ab) is coupled to the fourth pixel actuator (TX4), and the fourth (lower right) PDAF pixel 220 (second second parallax PDAF pixel 220Bb) is coupled to the third pixel actuator (TX3).

[0068] As reference Figure 3-5 As described above, the controller may use control signals to selectively actuate pixel actuators, including actuating multiple pixel actuators simultaneously. Figure 7 In the context of , the controller may selectively actuate the pixel actuators to perform a consistent merged parallel readout of the pixel block 710a. At a first time, this may involve simultaneously actuating the first pixel actuator and the fourth pixel actuator; at a second time, this may involve simultaneously actuating the second pixel actuator and the third pixel actuator. As described above, a reset may occur between each of these times (e.g., before the first time and also before the second time). The simultaneous actuation of the first pixel actuator and the fourth pixel actuator results in vertical merging of the first and second first parallax PDAF pixels 220A in each square array 720 (merging of the "left" parallax information), but results in diagonal merging of the first imaging pixel 218a and the fourth imaging pixel 218d in each block of imaging pixels 218. Similarly, simultaneous actuation of the second pixel actuator and the third pixel actuator results in vertical merging of the first and second second parallax PDAF pixels 220B in each square array 720 (merging of the “right” parallax information), but results in diagonal merging of the second imaging pixel 218b and the third imaging pixel 218c in each block of imaging pixels 218.

[0069] Figure 8 Shows Figure 7 illustrative output of a consistent merged parallel readout of pixel block 710a. Figure 6 According to the convention of FIG. 710 , each pixel in pixel block 710 a includes a flag for the pixel type and the binning group. For example, “W 8197” indicates a white (W) imaging pixel 218 in the 8197th image pixel binning group, and “PDAF 2” indicates a PDAF pixel 220 in the second PDAF pixel binning group. Figure 8 The convention in assumes that the coherently merged parallel readout starts at the lower right corner of pixel block 710a (which is the lower right instance of the thousands of instances of pixel block 710a in large imaging sensor array 140) and proceeds to the left and then upward.

[0070] As reference Figure 7 As described, the pixels are coupled to the pixel actuators such that simultaneous actuation of the pixel actuators results in vertical binning of the PDAF pixels 220, but diagonal binning of the imaging pixels 218. The result of this coincident binning and parallel readout is given by Figure 8For example, applying a specific simultaneous actuation of the pixel actuators causes two diagonally adjacent "G16386" pixels in pixel block 710a to be diagonally merged to produce a single "G 16386" merged pixel output in merged pixel block 810a; and applying the same simultaneous actuation of the pixel actuators causes two vertically adjacent "PDAF 0" pixels in pixel block 710a to be vertically merged to produce a single "PDAF 0" merged pixel output in merged pixel block 810a. Similarly, application of specific simultaneous actuation of the pixel actuators causes two diagonally adjacent “W 16387” pixels in pixel block 710a to be diagonally merged to produce a single “W 16387” merged pixel output in merged pixel block 810a; and application of the same simultaneous actuation of the pixel actuators causes two vertically adjacent “PDAF 1” pixels in pixel block 710a to be vertically merged to produce a single “PDAF 1” merged pixel output in merged pixel block 810a.

[0071] Figure 7 and Figure 8 The illustrated embodiment assumes the use of four pixel actuators to perform coherent binning parallel readout of pixel block 710. Other embodiments may use a different number of pixel actuators. For example, Fig. 9 A second example of a pixel block 710b configured for preserving parallax merging according to various embodiments described herein is shown, the pixel block 710b having PDAF pixels 220 and imaging pixels 218. For greater clarity, Fig. 9 The pixel arrangement of the pixel block 710b is shown as Figure 7 The pixel arrangement of the pixel block 710a is the same, however some electrical couplings between the pixels and the pixel actuators are different. Specifically, Fig. 9 The illustrated embodiment assumes eight pixel actuators for coherent binning parallel readout of pixel block 710 b. However, embodiments arrange electrical coupling between individual pixels in each square array 720 such that coherent binning results in diagonal binning of imaging pixels 218 and vertical binning of PDAF pixels 220.

[0072] As shown, eight pixel actuators (TX1–TX8) are coupled to pairs of 2×2 pixel blocks according to a particular arrangement. Each pair of 2×2 blocks of imaging pixels 218 can be arranged as follows: the first (e.g., the upper left of the first 2×2 block) imaging pixel 218a is actuated by the fifth pixel actuator (TX5), the second (e.g., the upper right of the first 2×2 block) imaging pixel 218b is actuated by the sixth pixel actuator (TX6), the third (e.g., the lower left of the first 2×2 block) imaging pixel 218c is actuated by the seventh pixel actuator (TX7), and the fourth (e.g., the lower right of the first 2×2 block) imaging pixel 218d is actuated by the eighth pixel actuator (TX8). The actuator (TX8) is actuated, the fifth (e.g., the upper left of the second 2×2 block) imaging pixel 218e is actuated by the first pixel actuator (TX1), the sixth (e.g., the upper right of the second 2×2 block) imaging pixel 218f is actuated by the second pixel actuator (TX2), the seventh (e.g., the lower left of the second 2×2 block) imaging pixel 218g is actuated by the third pixel actuator (TX3), and the eighth (e.g., the lower right of the second 2×2 block) imaging pixel 218h is actuated by the fourth pixel actuator (TX4). Each pair of square arrays 720 of the PDAF pixels 220 may be arranged as follows: the first (e.g., upper left of the first square array 720a) PDAF pixel 220 (the first first parallax PDAF pixel 220Aa) is coupled to the first pixel actuator (TX1), the second (e.g., upper right of the first square array 720a) PDAF pixel 220 (the first second parallax PDAF pixel 220Ba) is coupled to the second pixel actuator (TX2), the third (e.g., lower left of the first square array 720a) PDAF pixel 220 (the second first parallax PDAF pixel 220Ab) is coupled to the fourth pixel actuator (TX4), and the fourth (e.g., lower right of the first square array 720a) PDAF pixel 220 (the second second parallax PDAF pixel 220Bb) is coupled to the fourth pixel actuator (TX4). Coupled to the third pixel actuator (TX3), the fifth (for example, the upper left of the second square array 720b) ​​PDAF pixel 220 (the third first parallax PDAF pixel 220Aa) is coupled to the fifth pixel actuator (TX5), the sixth (the upper right of the second square array 720b) ​​PDAF pixel 220 (the third second parallax PDAF pixel 220Ba) is coupled to the sixth pixel actuator (TX6), the seventh (for example, the lower left of the second square array 720b) ​​PDAF pixel 220 (the fourth first parallax PDAF pixel 220Ab) is coupled to the eighth pixel actuator (TX8), and the eighth (for example, the lower right of the second square array 720b) ​​PDAF pixel 220 (the fourth second parallax PDAF pixel 220Bb) is coupled to the seventh pixel actuator (TX7).

[0073] As described above, the controller may selectively actuate the pixel actuators to perform a consistent merged parallel readout of the pixel block 710b. At a first time, this may involve simultaneously actuating the first pixel actuator, the fourth pixel actuator, the fifth pixel actuator, and the eighth pixel actuator; at a second time, this may involve simultaneously actuating the second pixel actuator, the third pixel actuator, the sixth pixel actuator, and the seventh pixel actuator. As described above, a reset may occur between each of these times (e.g., before the first time and also before the second time). The simultaneous actuation of the first pixel actuator, the fourth pixel actuator, the fifth pixel actuator, and the eighth pixel actuator results in a vertical merge of the four first parallax PDAF pixels 220A in the pair of square arrays 720 (merging of the “left” parallax information), but results in a diagonal merge of the imaging pixels 218a, 218d, 218e, and 218h in each pair of 2×2 blocks 218 of imaging pixels 218. Similarly, simultaneous actuation of the second, third, sixth, and seventh pixel actuators results in vertical merging of the four second parallax PDAF pixels 220B in the pair of square arrays 720 (merging of “right” parallax information), but results in diagonal merging of the imaging pixels 218b, 218c, 218f, and 218g in each pair of 2×2 blocks 218 of imaging pixels 218.

[0074] Fig.10 Shows Fig. 9 illustrative output of a consistent merged parallel readout of pixel block 710b. Figure 8 Each pixel in pixel block 710b includes a pixel type and a flag for the binning group. By using eight pixel actuators to bin pairs of 2×2 pixel blocks (rather than using four pixel actuators to bin a single 2×2 pixel block, as shown in FIG. Figure 7 and 8 As shown in , there can be far fewer merge groups, and the resulting merges are more aggressive. Fig. 9 As described above, the pixels are coupled to the pixel actuators such that simultaneous actuation of the pixel actuators results in vertical binning of the PDAF pixels 220, but diagonal binning of the imaging pixels 218. The result of this coincident binning and parallel readout is Fig.10 The merged pixel blocks 810b are shown. The merged pixel blocks 810b are shown as a first merged pixel block 810b-1 having only the merged output of the imaging pixels 218 and a second merged pixel block 810b-2 having only the merged output of the PDAF pixels 220. In practice, these separate blocks may be output as part of a single merged pixel block 810b (e.g., stored as a single output data set), or as separate outputs (e.g., storing the merged imaging data separately from storing the merged PDAF data).

[0075] In the illustrated example, application of a particular simultaneous actuation of the pixel actuators results in the diagonal merging of the four "B8194" pixels in pixel block 710b to produce a single "B 8194" merged pixel output in merged pixel block 810b-1; and application of the same simultaneous actuation of the pixel actuators results in the vertical merging of the four "PDAF 0" pixels in pixel block 710b to produce a single "PDAF 0" merged pixel output in merged pixel block 810b-2. Similarly, application of a particular simultaneous actuation of the pixel actuators results in the diagonal merging of the four "W 8195" pixels in pixel block 710b to produce a single "W8195" merged pixel output in merged pixel block 810b-1; and application of the same simultaneous actuation of the pixel actuators results in the vertical merging of the four "PDAF1" pixels in pixel block 710b to produce a single "PDAF 1" merged pixel output in merged pixel block 810b-2.

[0076] Such as Fig. 9 and 10 The merging of the 4×4 blocks in can be implemented in various ways. One method is to split the 4×4 block into four 2×2 sub-blocks. Each 2×2 sub-block can be basically based on Figure 3 Schematic 300 of is implemented with its respective readout circuitry such that each sub-block has its own source follower output path through a respective select transistor. The source follower output paths can be shorted together to form a common analog-to-digital converter (ADC) block. In one such embodiment, each of the four PDs of two of the sub-blocks is coupled to a respective one of four transfer transistors (e.g., TX1–TX4), and each of the four PDs of the other two of the sub-blocks is coupled to a respective one of different four transfer transistors (TX5–TX8). For example, as Fig. 9 As shown, TX1, TX4, TX5, and TX8 may be actuated simultaneously in a particular readout cycle to produce a combined output from PDAF pixels 220Aa, 220Ab, 220Ac, and 220Ad (ie, the four “left” PDAF pixels 220).

[0077] In another such embodiment, each of the four PDs in all four sub-blocks is coupled to a corresponding one of the four transfer transistors (eg, TX1 - TX4 ). Fig. 9Each pair of TXs shown in can be combined into a single TX, such that TX1 and TX5 become only TX1, TX2 and TX6 become only TX2, TX3 and TX7 become only TX3, and TX4 and TX8 become only TX4 (e.g., the actuators labeled TX1 and TX5 are actuated by a single signal, the actuators are shorted together, etc.). In such an embodiment, when any particular TX is actuated, the coupled PDs from each of the four sub-blocks (i.e., the four PDs) are simultaneously selected for readout. For example, applying this modification to Fig. 9 , it can be seen that simultaneously actuating TX1 and TX4 in a particular readout cycle produces a merged output from PDAF pixels 220Aa, 220Ab, 220Ac, and 220Ad (ie, the four “left” PDAF pixels 220 ).

[0078] Fig.14A and 14B A simplified schematic diagram 1400 of a portion of an example image sensor including pixels and readout circuitry is shown to illustrate merging 4×4 pixel blocks such as Fig. 9 and 10 For greater clarity, schematic 1400 is similar to Figure 3 Shown and referenced Figure 3 The described approach is shown except that modifications are made for the merging of 4x4 pixel blocks. Fig.14A A first embodiment is shown in which a 4×4 pixel block is divided into two 2×4 sub-blocks. Each 2×4 sub-block is coupled to a corresponding readout structure to output a partially merged result to a corresponding voltage readout line via a corresponding source follower path to provide a corresponding analog output pixel signal (Vout). Each of the four pairs of PDs in the first sub-block is coupled to a corresponding one of a first set of four transfer transistors (e.g., TX1–TX4), and each of the four pairs of PDs in the second sub-block is coupled to a corresponding one of a second set of four transfer transistors (TX5–TX8). Both source follower paths can be selected by a common selection signal (i.e., a single signal selects the outputs of both sub-blocks). The corresponding analog output pixel signal (Vout) from the corresponding voltage readout line can be fed into a single analog-to-digital converter (ADC) block.

[0079] Fig. 14B A second embodiment is shown, in which a 4×4 pixel block is divided into two 2×4 sub-blocks. Fig.14AAs shown, each 2×4 sub-block is coupled to a corresponding readout structure to output the partially merged result via a corresponding source follower path. Each of the four pairs of PDs in the first sub-block is coupled to a corresponding one of the first set of four transfer transistors (e.g., TX1–TX4), and each of the four pairs of PDs in the second sub-block is coupled to a corresponding one of the second set of four transfer transistors (TX5–TX8). Fig.14A different, Fig. 14B Each of the two source follower paths in is selected by its respective corresponding selection signal, and the corresponding analog output pixel signals (Vout) from the two sub-blocks are output to a shared voltage readout line coupled to the ADC block.

[0080] As described above, the PDAF pixels 220 may be configured to generate disparity information by using any suitable disparity inducing structure. Figure 7-10 Embodiments of can be produced using OCLs, metal shields, etc. Fig.11 A third example of a pixel block 710c having PDAF pixels 220 and imaging pixels 218 configured to preserve parallax binning is shown in accordance with various embodiments described herein. The pixel block 710c may be an example of a pixel block 710c that explicitly uses a metal shield. Figure 7 In the illustrated example, the right half of each first parallax PDAF pixel 220A is covered by a metal shield, so that the first parallax PDAF pixel 220A generally receives light from the left half of the main imaging lens; and the left half of each second parallax PDAF pixel 220B is covered by a metal shield, so that the second parallax PDAF pixel 220B generally receives light from the right half of the main imaging lens.

[0081] refer to Figure 7-11 The depicted example electrically couples the pixels of pixel block 710 with pixel actuators such that simultaneous actuation of the pixel actuators results in vertical binning of PDAF pixels 220, but diagonal binning of imaging pixels 218. In other embodiments, the parallax inducing structures may be arranged such that diagonal binning of PDAF pixels 220 also preserves parallax. Fig.12A fourth example of a pixel block 710d having PDAF pixels 220 and imaging pixels 218 configured to preserve parallax merging is shown in accordance with various embodiments described herein. The illustrated pixel block 710d includes two square arrays 720, each having a 2×2 arrangement of PDAF pixels 220. In particular, each square array 720 includes two first parallax PDAF pixels 220A and second parallax PDAF pixels 220B, which are shown as “left” and “right” PDAF pixels 220, respectively, relative to the schematic orientation of the pixel block 710d. For greater clarity, the pixel block 710d is shown with Figure 7 The pixel block 710 a is identical except that a metal shield is used to configure the PDAF pixels 220 so that PDAF pixels 220 of the same parallax are diagonally adjacent.

[0082] Embodiments arrange electrical coupling between individual pixels in each square array 720 such that coincident binning results in diagonal binning of imaging pixels 218 and diagonal binning of PDAF pixels 220. As shown, in each 2×2 block of imaging pixels 218, a first (e.g., upper left) imaging pixel 218a is actuated by a first pixel actuator (TX1), a second (e.g., upper right) imaging pixel 218b is actuated by a second pixel actuator (TX2), a third (e.g., lower left) imaging pixel 218c is actuated by a third pixel actuator (TX3), and a fourth (e.g., lower right) imaging pixel 218d is actuated by a fourth pixel actuator (TX4). Similarly, in each square array 720 of PDAF pixels 220, the first (e.g., upper left) PDAF pixel 220 (first first parallax PDAF pixel 220Aa) is coupled to the first pixel actuator (TX1), the second (e.g., upper right) PDAF pixel 220 (first second parallax PDAF pixel 220Ba) is coupled to the second pixel actuator (TX2), the third (e.g., lower left) PDAF pixel 220 (second second parallax PDAF pixel 220Bb) is coupled to the third pixel actuator (TX3), and the fourth (e.g., lower right) PDAF pixel 220 (second first parallax PDAF pixel 220Ab) is coupled to the fourth pixel actuator (TX4).

[0083] As described above, the controller can selectively actuate the pixel actuators to perform a consistent merged parallel readout of the pixel block 710d. At a first time, this may involve actuating the first pixel actuator and the fourth pixel actuator simultaneously; at a second time, this may involve actuating the second pixel actuator and the third pixel actuator simultaneously. As described above, a reset can occur between each of these times (e.g., before the first time and also before the second time). The simultaneous actuation of the first pixel actuator and the fourth pixel actuator results in a diagonal merge of the first and second first parallax PDAF pixels 220A in each square array 720 (a merge of the "left" parallax information), and results in a diagonal merge of the first imaging pixel 218a and the fourth imaging pixel 218d in each block of imaging pixels 218. Similarly, simultaneous actuation of the second pixel actuator and the third pixel actuator results in diagonal merging of the first and second second parallax PDAF pixels 220B in each square array 720 (merging of the “right” parallax information), and results in diagonal merging of the second imaging pixel 218b and the third imaging pixel 218c in each block of imaging pixels 218.

[0084] Fig.13 A flow chart of an illustrative method 1300 for preserving parallax merging of PDAF pixels in an image sensor array according to various embodiments described herein is shown. An embodiment of method 1300 includes: performing a consistent merged parallel readout of a pixel block at stage 1304. Assume that the pixel block has an array of photodetector elements, the array of photodetector elements including at least one square array of PDAF pixels interspersed with a plurality of imaging pixels, the at least one square array including a plurality (e.g., two) first parallax PDAF pixels and a plurality (e.g., two) second parallax PDAF pixels, each first parallax PDAF pixel being configured to generate a corresponding pixel response corresponding to a first parallax orientation, and each second parallax PDAF pixel being configured to generate a corresponding pixel response corresponding to a second parallax orientation that is complementary to the first parallax orientation (e.g., left-right, up-down, etc.). Such execution at stage 1304 may involve iteration through multiple stages, such as stages 1308-1320.

[0085] At stage 1308, an embodiment may reset a readout circuit for a first time, the readout circuit being configured to generate pixel output signals based on an actuation state of a plurality of pixel actuators, each pixel actuator being coupled to a respective one of the PDAF pixels and a respective subset of the imaging pixels. At stage 1312, after the first reset, an embodiment may simultaneously actuate a first subset of the pixel actuators for a first time to output a combination of respective pixel responses of the first parallax PDAF pixels as a first binned pixel response corresponding to a first parallax orientation, thereby causing the readout circuitry to generate pixel output signals as a first binned pixel output corresponding to the first parallax orientation. At stage 1316, an embodiment may reset the readout circuit for a second time to reset the pixel output signals after the first simultaneous actuation of stage 1312. At stage 1320, an embodiment may simultaneously actuate a second subset of the pixel actuators for a second time after the second reset to output a combination of respective pixel responses of the second parallax PDAF pixels as a second binned pixel response corresponding to a second parallax orientation, thereby causing the readout circuitry to generate pixel output signals as a second binned pixel output corresponding to the second parallax orientation.

[0086] In some embodiments, the first parallax orientation and the second parallax orientation correspond to light that typically reaches the pixel block through the left and right sides of the main imaging lens, respectively. In such embodiments, simultaneous actuation at stage 1312 results in vertical charge merging of corresponding pixel responses of the plurality of first parallax PDAF pixels; and simultaneous actuation at stage 1320 results in vertical charge merging of corresponding pixel responses of the plurality of second parallax PDAF pixels. In some such embodiments, simultaneous actuation at stage 1312 also results in diagonal charge merging of corresponding pixel responses of corresponding subsets of imaging pixels coupled to the first subset of pixel actuators; and simultaneous actuation at stage 1320 also results in diagonal charge merging of corresponding pixel responses of corresponding subsets of imaging pixels coupled to the second subset of pixel actuators.

[0087] In some embodiments, a first row of the photodetector element array includes, in sequence: a first PDAF pixel of the first parallax PDAF pixel coupled to the first pixel actuator, a second PDAF pixel of the second parallax PDAF pixel coupled to the second pixel actuator, a first imaging pixel coupled to the first pixel actuator, and a second imaging pixel coupled to the second pixel actuator; and a second row of the photodetector element array includes, in sequence: a third PDAF pixel of the first parallax PDAF pixel coupled to the fourth pixel actuator, a fourth PDAF pixel of the second parallax PDAF pixel coupled to the third pixel actuator, a third imaging pixel coupled to the third pixel actuator, and a fourth imaging pixel coupled to the fourth pixel actuator. In such an embodiment, the simultaneous actuation at stage 1312 includes actuating the first pixel actuator and the fourth pixel actuator to cause vertical charge merging of the corresponding pixel responses of the first PDAF pixel and the third PDAF pixel and diagonal charge merging of the corresponding pixel responses of the first imaging pixel and the fourth imaging pixel during a consistent combined parallel readout of the pixel block; and the simultaneous actuation at stage 1320 includes actuating the second pixel actuator and the third pixel actuator to cause vertical charge merging of the corresponding pixel responses of the second PDAF pixel and the fourth PDAF pixel and diagonal charge merging of the corresponding pixel responses of the second imaging pixel and the third imaging pixel during a consistent combined parallel readout of the pixel block.

[0088] In some embodiments, a first row of the photodetector element array includes, in sequence: a first PDAF pixel of the first parallax PDAF pixel coupled to the first pixel actuator, a second PDAF pixel of the second parallax PDAF pixel coupled to the second pixel actuator, a first imaging pixel coupled to the fifth pixel actuator, and a second imaging pixel coupled to the sixth pixel actuator; a second row includes, in sequence: a third PDAF pixel of the first parallax PDAF pixel coupled to the fourth pixel actuator, a fourth PDAF pixel of the second parallax PDAF pixel coupled to the third pixel actuator, a third imaging pixel coupled to the seventh pixel actuator, and a fourth PDAF pixel of the second parallax PDAF pixel coupled to the eighth pixel actuator. The fourth imaging pixel of the embodiment of the present invention; the third row includes in sequence: a fifth imaging pixel coupled to the first pixel actuator, a sixth imaging pixel coupled to the second pixel actuator, a fifth PDAF pixel of the first parallax PDAF pixel coupled to the fifth pixel actuator, and a sixth PDAF pixel of the second parallax PDAF pixel coupled to the sixth pixel actuator; and the fourth row includes in sequence: a seventh imaging pixel coupled to the third pixel actuator, an eighth imaging pixel coupled to the fourth pixel actuator, the seventh PDAF pixel of the first parallax PDAF pixel coupled to the eighth pixel actuator, and an eighth PDAF pixel of the second parallax PDAF pixel coupled to the seventh pixel actuator. In such an embodiment, the simultaneous actuation at stage 1312 includes actuating the first pixel actuator, the fourth pixel actuator, the fifth pixel actuator, and the eighth pixel actuator to cause vertical charge merging of the corresponding pixel responses of the first PDAF pixel, the third PDAF pixel, the fifth PDAF pixel, and the seventh PDAF pixel and diagonal charge merging of the corresponding pixel responses of the first imaging pixel, the fourth imaging pixel, the fifth imaging pixel, and the eighth imaging pixel during the consistent merging parallel readout of the pixel block; and the simultaneous actuation at stage 1320 includes actuating the second pixel actuator, the third pixel actuator, the sixth pixel actuator, and the seventh pixel actuator to cause vertical charge merging of the corresponding pixel responses of the second PDAF pixel, the fourth PDAF pixel, the sixth PDAF pixel, and the eighth PDAF pixel and diagonal charge merging of the corresponding pixel responses of the second imaging pixel, the third imaging pixel, the sixth imaging pixel, and the seventh imaging pixel during the consistent merging parallel readout of the pixel block.

[0089] Although the present disclosure contains many details, these details should not be interpreted as limitations on the scope of any invention or the content claimed, but rather as descriptions of specific features of specific embodiments that may be directed to a particular invention. Certain features described in the context of a separate embodiment in this patent document may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations, and even initially claimed as such, in some cases, one or more features from the claimed combination may be deleted from the combination, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0090] Similarly, while operations are described in a particular order in the drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, in order to achieve the desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0091] Only a few implementations and examples are described herein, and other implementations, enhancements, and variations may be made based on what is described and illustrated in this patent document.

[0092] The description of "one", "an" or "the" is intended to mean "one or more", unless there is a special statement to the contrary. The range can be expressed here as from "about" a specified value, and / or to "about" another specified value. The term "about" used here means approximately, in the vicinity, roughly or around. When the term "about" is used in conjunction with a numerical range, it modifies the range by extending the boundaries above and below the stated numerical values. In general, the term "about" is used herein to modify the numerical values ​​above and below the stated value 10% variance. When expressing such a range, another embodiment includes from a specific value and / or to another specified value. Similarly, when a value is expressed as an approximate value by using the antecedent "about", it should be understood that the specified value forms another embodiment. It should also be understood that the endpoints of each range in these ranges are included in the range.

[0093] All patents, patent applications, publications, and descriptions mentioned herein are hereby incorporated by reference in their entirety for all purposes. None is admitted to be prior art.

Claims

1. A system for parallax-preserving merging of phase detection autofocus (PDAF) pixels in an image sensor array, the system comprising: a pixel block having an array of photodetector elements, the array of photodetector elements comprising at least one square array of PDAF pixels interspersed with a plurality of imaging pixels, the at least one square array comprising a plurality of first parallax PDAF pixels and a plurality of second parallax PDAF pixels, each first parallax PDAF pixel being configured to generate a respective pixel response corresponding to a first parallax orientation, and each second parallax PDAF pixel being configured to generate a respective pixel response corresponding to a second parallax orientation complementary to the first parallax orientation; a plurality of pixel actuators, each pixel actuator coupled to a respective one of the PDAF pixels and a respective subset of the imaging pixels; a controller that selectively actuates the plurality of pixel actuators to perform a coherent binning parallel readout of the pixel block, such that the coherent binning parallel readout comprises: actuating a first subset of the pixel actuators simultaneously at a first time to output a combination of respective pixel responses of the plurality of first parallax PDAF pixels as a first binned pixel response corresponding to the first parallax orientation; and A second subset of the pixel actuators are actuated simultaneously at a second time to output a combination of respective pixel responses of the plurality of second parallax PDAF pixels as a second binned pixel response corresponding to the second parallax orientation.

2. The system according to claim 1, further comprising: a readout circuit electrically coupled to the plurality of pixel actuators to generate pixel output signals based on an actuation state of the plurality of pixel actuators at each of a plurality of times during the coherent binning parallel readout such that: actuating the first subset of the pixel actuators simultaneously so that the readout circuitry generates the pixel output signals at the first time as first binned pixel outputs corresponding to the first parallax orientation; and The second subset of the pixel actuators is actuated simultaneously so that the readout circuitry generates the pixel output signals at the second time as a second binned pixel output corresponding to the second parallax orientation.

3. The system of claim 2, wherein: each pixel actuator coupled to a respective one of the PDAF pixels and to at least some of a respective subset of the imaging pixels via a respective row-by-row pixel actuation bus; as well as The coherent binning parallel readout is a column-parallel readout of the pixel block.

4. The system of claim 1, wherein: The first parallax orientation and the second parallax orientation correspond to light that normally reaches the pixel block through the left side and the right side of the main imaging lens, respectively; said simultaneous actuation of said first subset of said pixel actuators results in vertical charge combining of respective pixel responses of said plurality of first parallax PDAF pixels; and The simultaneous actuation of the second subset of the pixel actuators results in vertical charge combining of respective pixel responses of the second plurality of parallax PDAF pixels.

5. The system of claim 1, wherein: actuating the first subset of the pixel actuators simultaneously resulting in vertical charge combining of respective pixel responses of the plurality of first parallax PDAF pixels coupled to the first subset of the pixel actuators and diagonal charge combining of respective pixel responses of the respective subset of the imaging pixels; as well as Simultaneously actuating the second subset of the pixel actuators results in vertical charge combining of corresponding pixel responses of the plurality of second parallax PDAF pixels coupled to the second subset of the pixel actuators and diagonal charge combining of corresponding pixel responses of the corresponding subset of the imaging pixels.

6. The system according to claim 1, wherein: The at least one square array of PDAF pixels includes at least one 2×2 array of four PDAF pixels.

7. The system according to claim 1, wherein: The at least one square array of PDAF pixels includes at least two 2×2 arrays, each 2×2 array including four PDAF pixels.

8. The system of claim 1, wherein: A first row of the photodetector element array includes, in order: a first PDAF pixel of a plurality of first parallax PDAF pixels coupled to a first pixel actuator, a second PDAF pixel of a plurality of second parallax PDAF pixels coupled to a second pixel actuator, a first imaging pixel coupled to the first pixel actuator, and a second imaging pixel coupled to the second pixel actuator; The second row of the photodetector element array includes, in order: a third PDAF pixel of the plurality of first parallax PDAF pixels coupled to a fourth pixel actuator, a fourth PDAF pixel of the plurality of second parallax PDAF pixels coupled to a third pixel actuator, a third imaging pixel coupled to the third pixel actuator, and a fourth imaging pixel coupled to the fourth pixel actuator; Simultaneously actuating the first subset of the pixel actuators comprises: actuating the first pixel actuator and the fourth pixel actuator to cause vertical charge combining of the respective pixel responses of the first PDAF pixel and the third PDAF pixel and diagonal charge combining of the respective pixel responses of the first imaging pixel and the fourth imaging pixel during the coincident binning parallel readout of the pixel block; and Simultaneously actuating the second subset of the pixel actuators includes actuating the second pixel actuator and the third pixel actuator during the consistent merging parallel readout of the pixel block to cause vertical charge merging of the corresponding pixel responses of the second PDAF pixel and the fourth PDAF pixel and diagonal charge merging of the corresponding pixel responses of the second imaging pixel and the third imaging pixel.

9. The system of claim 1, wherein: The first row of the photodetector element array includes, in order: a first PDAF pixel of a plurality of first parallax PDAF pixels coupled to a first pixel actuator, a second PDAF pixel of a plurality of second parallax PDAF pixels coupled to a second pixel actuator, a first imaging pixel coupled to a fifth pixel actuator, and a second imaging pixel coupled to a sixth pixel actuator; The second row of the photodetector element array includes, in order: a third PDAF pixel of the plurality of first parallax PDAF pixels coupled to a fourth pixel actuator, a fourth PDAF pixel of the plurality of second parallax PDAF pixels coupled to a third pixel actuator, a third imaging pixel coupled to a seventh pixel actuator, and a fourth imaging pixel coupled to an eighth pixel actuator; A third row of the array of photodetector elements includes, in order: a fifth imaging pixel coupled to the first pixel actuator, a sixth imaging pixel coupled to the second pixel actuator, a fifth PDAF pixel of the plurality of first parallax PDAF pixels coupled to the fifth pixel actuator, and a sixth PDAF pixel of the plurality of second parallax PDAF pixels coupled to the sixth pixel actuator; The fourth row of the photodetector element array includes, in order: a seventh imaging pixel coupled to the third pixel actuator, an eighth imaging pixel coupled to the fourth pixel actuator, a seventh PDAF pixel of the plurality of first parallax PDAF pixels coupled to the eighth pixel actuator, and an eighth PDAF pixel of the plurality of second parallax PDAF pixels coupled to the seventh pixel actuator; Simultaneously actuating the first subset of the pixel actuators comprises: actuating the first, fourth, fifth, and eighth pixel actuators to cause vertical charge combining of respective pixel responses of the first, third, fifth, and seventh PDAF pixels and diagonal charge combining of respective pixel responses of the first, fourth, fifth, and eighth imaging pixels during a coincident binning parallel readout of the pixel block; and Simultaneously actuating the second subset of the pixel actuators includes actuating the second pixel actuator, the third pixel actuator, the sixth pixel actuator, and the seventh pixel actuator during a consistent merged parallel readout of the pixel block to cause vertical charge merging of the corresponding pixel responses of the second PDAF pixel, the fourth PDAF pixel, the sixth PDAF pixel, and the eighth PDAF pixel and diagonal charge merging of the corresponding pixel responses of the second imaging pixel, the third imaging pixel, the sixth imaging pixel, and the seventh imaging pixel.

10. The system according to claim 1, wherein: The pixel block also includes at least one parallax inducing structure disposed on top of the plurality of PDAF pixels to configure the plurality of first parallax PDAF pixels to generate corresponding pixel responses corresponding to the first parallax orientation, and to configure the plurality of second parallax PDAF pixels to generate corresponding pixel responses corresponding to the second parallax orientation.

11. The system according to claim 10, wherein: The at least one square array is a 2×2 array, and the at least one parallax inducing structure is a 2×2 on-chip lens disposed on top of the 2×2 array such that the first PDAF pixel and the third PDAF pixel of the 2×2 array jointly receive light typically from a first half of the main imaging lens, and such that the second PDAF pixel and the fourth PDAF pixel of the 2×2 array jointly receive light typically from a second half of the main imaging lens.

12. The system according to claim 10, wherein: The at least one square array is a 2×2 array, and the at least one parallax inducing structure comprises: a first 1×2 on-chip lens disposed on top of the first PDAF pixel and the second PDAF pixel of the 2×2 array such that the first PDAF pixel receives light generally from a first half of the main imaging lens and the second PDAF pixel receives light generally from a second half of the main imaging lens; and A second 1×2 on-chip lens is disposed on top of the third PDAF pixel and the fourth PDAF pixel of the 2×2 array so that the third PDAF pixel receives light generally from the first half of the main imaging lens and the fourth PDAF pixel receives light generally from the second half of the main imaging lens.

13. The system according to claim 10, wherein: The at least one parallax inducing structure comprises: a plurality of first metal shields, each of the first metal shields being disposed on top of a corresponding one of the plurality of first parallax PDAF pixels so that the corresponding one of the plurality of first parallax PDAF pixels receives light generally from a first half of the main imaging lens; A plurality of second metal shields, each second metal shield is disposed on top of a corresponding one of the plurality of second parallax PDAF pixels so that the corresponding one of the plurality of second parallax PDAF pixels receives light generally from the second half of the main imaging lens.

14. The system of claim 13, wherein: The at least one square array is a 2×2 array having, in order: in a first row, a first PDAF pixel of the plurality of first parallax PDAF pixels and a second PDAF pixel of the plurality of second parallax PDAF pixels; and in a second row, a third PDAF pixel of the plurality of second parallax PDAF pixels and a fourth PDAF pixel of the plurality of first parallax PDAF pixels; actuating the first subset of the pixel actuators simultaneously resulting in diagonal charge combining of respective pixel responses of the plurality of first parallax PDAF pixels; and Simultaneously actuating the second subset of the pixel actuators results in diagonal charge combining of respective pixel responses of the second plurality of parallax PDAF pixels.

15. The system of claim 1, further comprising: An image sensor array includes multiple instances of a pixel block.

16. A method for parallax-preserving merging of phase detection autofocus (PDAF) pixels in an image sensor array, the method comprising: Performing a coherent binned parallel readout of a pixel block having an array of photodetector elements, the array of photodetector elements comprising at least one square array of PDAF pixels interspersed with a plurality of imaging pixels, the at least one square array comprising a plurality of first parallax PDAF pixels and a plurality of second parallax PDAF pixels, each first parallax PDAF pixel being configured to generate a respective pixel response corresponding to a first parallax orientation, each second parallax PDAF pixel being configured to generate a respective pixel response corresponding to a second parallax orientation complementary to the first parallax orientation, the performing comprising: resetting a readout circuit for a first time, the readout circuit being configured to generate a pixel output signal based on an actuation state of a plurality of pixel actuators, each pixel actuator being coupled to a respective one of the PDAF pixels and a respective subset of the imaging pixels; After the first resetting, simultaneously actuating a first subset of the pixel actuators for a first time to output a combination of respective pixel responses of the plurality of first parallax PDAF pixels as a first binned pixel response corresponding to the first parallax orientation, thereby causing the readout circuit to generate the pixel output signal as a first binned pixel output corresponding to the first parallax orientation; resetting the readout circuit a second time to reset the pixel output signal after the first simultaneous actuation; and After the second reset, a second subset of the pixel actuators is simultaneously actuated for a second time to output a combination of corresponding pixel responses of the plurality of second parallax PDAF pixels as a second merged pixel response corresponding to the second parallax orientation, thereby causing the readout circuit to generate the pixel output signal as a second merged pixel output corresponding to the second parallax orientation.

17. The method of claim 16, wherein: The first parallax orientation and the second parallax orientation correspond to light that normally reaches the pixel block through the left side and the right side of the main imaging lens, respectively; The first simultaneous actuation results in vertical charge combining of respective pixel responses of the plurality of first parallax PDAF pixels; as well as The second simultaneous actuation results in vertical charge combining of corresponding pixel responses of the plurality of second parallax PDAF pixels.

18. The method of claim 16, wherein: the first simultaneous actuation causing vertical charge combining of respective pixel responses of the plurality of first parallax PDAF pixels coupled to the first subset of the pixel actuators and diagonal charge combining of respective pixel responses of the respective subset of the imaging pixels; as well as The second simultaneous actuation results in vertical charge combining of respective pixel responses of the plurality of second parallax PDAF pixels coupled to the second subset of the pixel actuators and diagonal charge combining of respective pixel responses of the respective subset of the imaging pixels.

19. The method of claim 16, wherein: A first row of the array of photodetector elements includes, in order: a first PDAF pixel of the plurality of first parallax PDAF pixels coupled to a first pixel actuator, a second PDAF pixel of the plurality of second parallax PDAF pixels coupled to a second pixel actuator, a first imaging pixel coupled to the first pixel actuator, and a second imaging pixel coupled to the second pixel actuator; The second row of the photodetector element array includes, in order: a third PDAF pixel of the plurality of first parallax PDAF pixels coupled to a fourth pixel actuator, a fourth PDAF pixel of the plurality of second parallax PDAF pixels coupled to a third pixel actuator, a third imaging pixel coupled to the third pixel actuator, and a fourth imaging pixel coupled to the fourth pixel actuator; The first simultaneous actuation includes actuating the first pixel actuator and the fourth pixel actuator to cause vertical charge combining of the respective pixel responses of the first PDAF pixel and the third PDAF pixel and diagonal charge combining of the respective pixel responses of the first imaging pixel and the fourth imaging pixel during a coincident binning parallel readout of the pixel block; and The second simultaneous actuation includes actuating the second pixel actuator and the third pixel actuator during a consistent merged parallel readout of the pixel block to cause vertical charge merging of the corresponding pixel responses of the second PDAF pixel and the fourth PDAF pixel and diagonal charge merging of the corresponding pixel responses of the second imaging pixel and the third imaging pixel.

20. The method of claim 16, wherein: The first row of the photodetector element array includes, in order: a first PDAF pixel of the plurality of first parallax PDAF pixels coupled to the first pixel actuator, a second PDAF pixel of the plurality of second parallax PDAF pixels coupled to the second pixel actuator, a first imaging pixel coupled to the fifth pixel actuator, and a second imaging pixel coupled to the sixth pixel actuator; The second row of the photodetector element array includes, in order: a third PDAF pixel of the plurality of the first parallax PDAF pixels coupled to a fourth pixel actuator, a fourth PDAF pixel of the plurality of second parallax PDAF pixels coupled to a third pixel actuator, a third imaging pixel coupled to a seventh pixel actuator, and a fourth imaging pixel coupled to an eighth pixel actuator; A third row of the array of photodetector elements includes, in order: a fifth imaging pixel coupled to the first pixel actuator, a sixth imaging pixel coupled to the second pixel actuator, a fifth PDAF pixel of the plurality of first parallax PDAF pixels coupled to the fifth pixel actuator, and a sixth PDAF pixel of the plurality of second parallax PDAF pixels coupled to the sixth pixel actuator; The fourth row of the photodetector element array includes, in order: a seventh imaging pixel coupled to the third pixel actuator, an eighth imaging pixel coupled to the fourth pixel actuator, a seventh PDAF pixel of the plurality of first parallax PDAF pixels coupled to the eighth pixel actuator, and an eighth PDAF pixel of the plurality of second parallax PDAF pixels coupled to the seventh pixel actuator; The first simultaneous actuation includes actuating the first pixel actuator, the fourth pixel actuator, the fifth pixel actuator, and the eighth pixel actuator to cause vertical charge merging of the respective pixel responses of the first PDAF pixel, the third PDAF pixel, the fifth PDAF pixel, and the seventh PDAF pixel and diagonal charge merging of the respective pixel responses of the first imaging pixel, the fourth imaging pixel, the fifth imaging pixel, and the eighth imaging pixel during a coincident binning parallel readout of the pixel block; and The second simultaneous actuation includes actuating the second pixel actuator, the third pixel actuator, the sixth pixel actuator and the seventh pixel actuator during the consistent merging and parallel readout of the pixel block to cause vertical charge merging of the corresponding pixel responses of the second PDAF pixel, the fourth PDAF pixel, the sixth PDAF pixel and the eighth PDAF pixel and diagonal charge merging of the corresponding pixel responses of the second imaging pixel, the third imaging pixel, the sixth imaging pixel and the seventh imaging pixel.