Method, image processor unit and computer program for processing image data of an image sensor - Patents.com
The method addresses computational complexity and image defects in PDAF by calculating image pixel data at phase detection pixel locations based on symmetry or asymmetry in color channels, enhancing image quality through low-complexity concealment techniques.
Patent Information
- Application Number
- JP2022580837
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2021-12-08
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2041-12-08
AI Technical Summary
Existing phase detection autofocus (PDAF) techniques in imaging systems suffer from computational complexity and require tuning or control parameters, leading to visible defects in images due to phase detection pixels, which affect image quality.
A method for calculating image pixel data at predefined locations of phase detection pixels based on symmetrical or asymmetrical positioning in color channels, using nearest neighbor concealment and directional filtering to hide phase detection pixels without requiring pre-calibration or parameter tuning, thus maintaining low computational complexity.
The method effectively conceals phase detection pixels, improving image quality by reducing computational load and eliminating the need for parameter tuning, making it suitable for real-time image processing.
Smart Images

Figure 2024544800000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a method for processing image data of an image sensor, the image sensor comprising a sensor area of a pixel matrix providing image pixel data, the pixel matrix comprising phase detection pixels at predefined locations, and a set of image pixel data comprising phase detection information of the phase detection pixels.
[0002] The invention further relates to an image processor unit for processing image data of an image sensor and to a computer program comprising instructions which, when executed by the processing unit, cause the processing unit to process image data of the image sensor according to the above-mentioned method.
[0003] The image sensor comprises a sensor area of a pixel matrix providing image pixel data, the pixel matrix comprising phase detection pixels at predefined locations, the image sensor providing the image pixel data at an input of the image processor unit, the set of image pixel data comprising phase detection information of the phase detection pixels. [Background technology]
[0004] Digital imagers are widely used in everyday products such as smartphones, tablets, notebooks, cameras, cars, and wearables. Many of the imaging systems in these products have the function of automatic focusing (AF) to generate clear images / videos. The traditional approach to automatic focusing is based on contrast detection. The lens is moved to the position where the scene contrast is highest. Contrast autofocus is generally slow, so the technique of phase detection autofocus (PDAF) can be combined with contrast autofocus to improve the speed and sometimes the accuracy of focusing.
[0005] PDAF sensors have so-called phase detection (PD) pixels arranged throughout the sensor area. The phase information derived from the phase detection pixels can be used to determine the focal distance, which then drives the lens to move to a position with optimal focus, a process that is typically very fast if the phase information is accurate enough. The phase detection pixels are typically arranged periodically in the horizontal and vertical directions throughout the sensor. After extracting the phase information from those pixels, they must be concealed or corrected. Otherwise, they may appear as a mesh of clustered defects across the image. Therefore, to maintain high image quality, it is essential to have a phase detection pixel concealment (PPC) module in the image signal processor.
[0006] N. El-Yamany: “Robust Defect Pixel Detection and Correction for Bayer Imaging Systems,” in: IS&T International Symposium on Electronic Imaging 2017, p. 46-51, discloses a method for identifying hot pixels, cold pixels, or mixed singlets and couplets of both types. If those pixels are not corrected early in the image processing pipeline, demosaicing and filtering operations would diffuse them and make them appear as coloured clusters, which is detrimental to image quality. The method works on the raw data coming from the Bayer sensor. A defective pixel is identified if two conditions are met. In the first condition, it is checked whether the pixel is significantly different from its same-colour neighbours in an S×S Bayer window centred on the pixel. The second condition tests whether the local luminance difference at a pixel is significantly higher (for a hot pixel) or lower (for a cold pixel) than the minimum local luminance difference per color channel when a 3x3 Bayer window is centered on the pixel.
[0007] Detected defective pixels are replaced with robust, detail-preserving estimates, which are determined through the use of directional filters.
[0008] E. Chang: “Kernel-size selection for defect pixel identification and correction”, in: Proc. SPIE 6502, Digital Photography III, 65020J, 20-02-2007 describes a bounding min-max filter with varying kernel size for defect pixel correction.
[0009] S. Wang, S. Yao, O. Faurie, and Z. Shi: “Adaptive defect correction and noise suppression module in the CIS image processing system”, in: Proc. SPIE Int. Symposium on Photoelectronic Detection and Imaging, vol. 7384, p. 73842V-1-6, describes a spatial adaptive noise suppression algorithm that combines defective pixel correction functions for implementation in a CMOS image sensor chip. A centre weighted median filter is provided to correct the defective pixels. Random noises are processed separately according to their background detail level.
[0010] A. Tanbakuchi, A. van der Sijde, B. Dillen, A. Theuwissen and W. de Haan: “Adaptive pixel defect correction”, in: Proc. SPIE Sensors and Camera Systems for Scientific, Industrial and Digital Photography Applications IV, vol. 5017, pp. 360-370, 2003 presents a defect correction algorithm that uses raw Bayer image data. When a pixel in an image is found to be defective, the neighboring pixels provide the best information to interpolate the defective pixel. Directional derivatives are used to correlate the closest point to the defect color plane.
[0011] M. Schoberi, J. Seiler, B. Kasper, S. Foessel and A. Kaup: “Sparsity-based detect pixel compensation for arbitrary camera raw images”, in: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1257-1260, 2011, discloses an algorithm for detecting defective pixels and interpolating missing values for the defective pixels. Previously interpolated pixels are reused. Summary of the Invention
[0012] The object of the present invention is to provide an improved method and image processor unit, which provides a robust scheme for phase pixel hiding in imaging sensors in a generic framework, keeping the computational complexity low and not requiring any tuning or control parameters.
[0013] This object is achieved by a method comprising the features of claim 1, an image processor unit comprising the features of claim 9 and a computer program for processing comprising the features of claim 11. Preferred embodiments are disclosed in the dependent claims.
[0014] It is proposed to perform a step of calculating image pixel data for predefined locations of the phase detection pixels depending on either a symmetric or asymmetric positioning of the respective phase detection pixels in the colour channels.
[0015] This allows for the correction of defective pixels with low computational complexity and without the need for any tuning or control parameters.
[0016] To provide image data at predefined locations of phase detection pixels to complete the image and avoid artifacts at predefined PD pixel locations, image pixel data are calculated for these locations. The image raw data of the image sensor is processed and the missing image pixel data at the phase detection pixel locations are determined in two different ways depending on the symmetric or asymmetric positioning of each phase detection pixel in each color channel.
[0017] The phase detection pixels can be positioned in one or more color channels in the color filter array of the image sensor, for example, the phase detection pixels can be positioned in the B and W channels of an RGBW sensor or in the B channel in a standard Bayer sensor.
[0018] When the phase detection pixel is positioned in a color channel that exhibits symmetry, such as the W channel in an RGBW color channel arrangement, or the R, G, and B channels in a standard Bayer color channel arrangement, image pixel data for a predefined location of the phase detection pixel is calculated in a first procedure specified for the symmetric color channel arrangement.
[0019] When the arrangement of the color channels is asymmetric, the image pixel data for the predefined positions of the phase detection pixels are calculated according to a second procedure.
[0020] From this it is crucial whether the phase detection pixel is positioned in a color channel exhibiting symmetry or in a color channel exhibiting asymmetricity.
[0021] This allows for robust phase detection pixel concealment without the need to pre-calibrate or know in advance the phase detection pixel behavior. No parameter tuning is required and the computational complexity can be kept very low. Therefore, the method is suitable for real-time resource-constrained image signal processing.
[0022] When the phase detection pixel is asymmetrically positioned in the color channel, i.e., the color channel exhibits asymmetry, the image pixel data can preferably be calculated according to adjacent image pixel values of sensor pixels of the same color according to the color assigned to the position of the phase detection pixel.
[0023] Preferably, the image pixel data is calculated as an average value of the nearest neighbor image pixel values of the same-color pixels located around the selected phase detection pixel location. For example, when the phase detection pixel is located in a color channel that does not exhibit symmetry, such as the R, G, or B channel in an RGBW color filter array pad, or the R and G and B channels in a Quad Bayer color filter array pattern, the phase pixel concealment relies only on one step of nearest-neighbor concealment. The image pixel data for the phase detection pixel location is calculated as a robust estimate determined from the nearest neighbors of the phase detection pixel location. One possibility for such an estimate is, for example, the alpha-trimmed mean of the same-color pixels in the neighborhood of the pixel being concealed. Another possibility is some weighted average of those same-color neighbors.
[0024] In approved methods for computing image pixel data for asymmetrically positioned phase-detected pixels, orientation or content-aware concealment can give better image quality than non-orientation correction. Therefore, when interpolating nearest neighbors, the interpolation strategy preferably tries to identify the orientation as much as possible or takes into account the content in the raw support at the pixel being corrected. An option is to weight the image pixel values of nearest neighbors of the same color pixel.
[0025] To calculate image pixel data for phase detection pixels symmetrically positioned in each color channel, it is preferable to estimate image pixel data for phase detection pixels located in a predetermined area around each selected phase detection pixel selected for calculating associated pixel image data, identify a feature direction at the location of the selected phase detection pixel, and calculate the image pixel data according to interpolated pixel data located adjacent to the selected phase detection pixel in the identified feature direction.
[0026] From this, for color channels that exhibit symmetry, a series of steps are processed to calculate image pixel data for the locations of phase detection pixels. Specifically, when the phase detection pixel is positioned in a color channel that exhibits symmetry, such as the W channel in an RGBW color filter array, or the R, G, and B channels in a standard Bayer color filter array, directional estimation and correction are facilitated due to symmetry. First, a robust estimate is calculated for each of the locations of the phase detection pixel. Then, directional filtering is performed based on the identified feature direction. Allowed determination of robust estimates for phase detection pixels around the locations of the phase detection pixel improves the quality of directional filtering. The image pixel data for the locations of the phase detection pixel are then calculated as directional occlusion or non-directional occlusion.
[0027] Preferably, each image pixel value is calculated in response to pixel data located in a selected area adjacent to and surrounding the selected phase detection pixel when the feature direction cannot be identified.
[0028] Otherwise, the pixel value at the location of the selected phase detection pixel is calculated by using the neighboring pixel values associated with the same color as the associated color channel assigned to the location of the selected phase detection pixel.
[0029] Preferably, in a first step, a symmetric or asymmetric arrangement of phase detection pixels in the colour channels is determined depending on colour channel parameters.
[0030] The status of symmetric or asymmetric arrangement of the phase detection pixels may also be mentioned in advance and therefore known, so that no decision step is required.
[0031] The phase detection pixels may be positioned in the image sensor as clusters of connected pixels. Regardless of the particular arrangement of the color filter array or phase detection pixels, the input raw data may be processed in a first step to decompose the phase detection clusters of pixels into singlets of phase detection pixels per color channel, or into couplets in the case of larger clusters. The phase detection clusters of pixels repeat periodically in the horizontal and vertical directions across the image sensor.
[0032] For example, it is preferable to decompose a cluster of phase detection pixels into singlets and / or couplets, especially when the phase detection pixels are periodically repeated in the horizontal and / or vertical directions of the image sensor area. As an example, it can be performed by decomposing a couplet of displaced detection pixels in a color channel (e.g., B channel) into a quadruplet in the same color channel (e.g., B channel). A quadruplet can be further decomposed into four singlets in the same color channel. Two couplets can be decomposed, for example, into four singlets of the same color channel. Many other decompositions of the cluster into singlets and / or couplets are possible, depending on the decomposition scheme. Each singlet phase detection pixel will have its own neighbors and will be occluded independently of other connected phase detection pixels.
[0033] The invention will now be described, by way of example only, with the aid of the enclosed drawings, in which: [Brief description of the drawings]
[0034] [Figure 1] 1 illustrates exemplary phase detection pixel locations in an RGBW color filter array. [Diagram 2] Example of a CMOS RGBW sensor with phase detection pixels. [Diagram 3] 4 is a flow diagram of a method for processing image data of an image sensor.
[0035] FIG. 1 shows a portion of an exemplary phase detection arrangement on RGBW color filter array data. The red R, green G, blue B, and white W pixels in the 8×8 array and the pixel locations are identified by numbers, and the letters in parentheses identify the respective colors of the pixels. A complete pattern may consist of blue blocks (1(B), 2(W), 3(B), 4(W); 9(W), 10(B), 11(W), 12(B); 17(B), 18(W), 19(B), 20(W); 25(W), 26(B), 27(W), 28(B)) each formed by a matrix of alternating blue and white pixels shown in a first 4×4 matrix on the upper left side of FIG. 1, green blocks each formed by a matrix of alternating green and white pixels shown in a second 4×4 matrix on the upper right side of FIG. 1, and red blocks each formed by a 4×4 matrix of alternating red and white pixels shown in a second 4×4 matrix on the lower right side of FIG. 1. A first line of a complete pattern may be formed by a sequence of alternating blue and green blocks, an adjacent second line may be formed by a sequence of alternating green and red blocks, etc. Such groups of first and second lines are repeated to form all the lines of the pattern, e.g. the height and width of the pattern are determined by the length of the lines.
[0036] The phase detection pixels are identified by the thick frame.
[0037] At the locations of the phase detection pixels, the respective assigned color image pixel values are missing, which has the effect that defects in the image become visible, especially after processing the raw sensor data.
[0038] The phase detection pixels arranged in each color channel form a couplet. For example, the four phase detection pixels in the blue channel 1 (B), 10 (B), 19 (B), and 28 (B) form a quadruplet, while the first pair of phase detection pixels in the white channel 2 (W) and 9 (W) form a first couplet, and the second pair of phase detection pixels in the white channel 20 (W) and 27 (W) form a second couplet in the white channel. The problem then reduces to the concealment of the quadruplet in the blue channel and the two couplets in the white channel. The four phase detection pixels in the blue channel (quadruplet) are corrected independently of the other four phase detection pixels in the white channel (two couplets).
[0039] 2 presents an example of a CMOS sensor with green G, red R, and blue B pixels and a pair of phase detection pixels PA, PB. The image sensor includes color sensors on pixels arranged in horizontal and vertical directions, where all colors are arranged in each line in those directions, as described in US 8,531,563 B2.
[0040] The phase detection pixels are located within the repeating 6×6 grid of green image pixel locations used in this exemplary image sensor.
[0041] In order to conceal image data for the associated color at the location of the phase detection pixel, the method distinguishes between the locations of the phase detection pixel in the color channels that exhibit symmetry and the locations of the phase detection pixel in the color channels that exhibit asymmetry.
[0042] However, the phase detection pixels can be positioned in one or more color channels in the color filter array, for example, in the B and W channels of an RGBW sensor as shown in FIG.
[0043] In addition, the phase detection pixels can also be positioned as clusters of connected pixels as shown in FIGS.
[0044] Regardless of the particular arrangement of the color filter array and phase detection pixels, it is preferable to decompose the clusters of phase detection pixels that repeat periodically in the horizontal and vertical directions across the image sensor into singlets of phase detection pixels per color channel, or couplets in the case of larger clusters.
[0045] For example, the phase detection pixels in the phase detection pixel arrangement shown in FIG. A) can be decomposed into a quadruplet in the B channel, which can be further decomposed into four singlets in the B channel, or B) It can be resolved into two couplets in the W channel, which can be further resolved into four singlets in the W channel.
[0046] Each singlet phase detection pixel has its own neighborhood and will be occluded independently of other connected phase detection pixels.
[0047] FIG. 3 illustrates a method for processing image data of an image sensor, which comprises a sensor area of a pixel matrix providing image pixel data, as illustratively shown in FIGS.
[0048] The method distinguishes between phase detection pixels located in color channels that exhibit symmetry and phase detection pixels located in color channels that do not exhibit symmetry, i.e., asymmetric color channels.
[0049] The image pixel raw data is processed in step A) by the method of closest neighbours concealment when the phase detection pixel is located in a colour channel that does not exhibit symmetry, which is for example the case for the R, G or B channels or the R and G and B channels in an RGBW colour filter array pattern, as well as for a Quad Bayer colour filter array pattern.
[0050] In step A), a robust estimate is used to occlude the phase detection pixel. One possibility for such an estimate is, for example, calculating the alpha trimmed average of the same color pixels in the neighborhood of the pixel being occluded. Another possibility is some weighted average of those image pixels directly neighboring the phase detection pixel that have the same color assigned to its position in the color filter array.
[0051] For example, to conceal a phase detection pixel PA(G) in the green color channel at a location in line 2, column 3 of FIG. 2, an image value for the displayed phase detection pixel is calculated by interpolating the four immediately neighboring image pixel values of the green pixel at locations in line 1, column 4, line 2, column 2, line 3, columns 2 and 3.
[0052] Image quality can be improved by weighting neighboring image pixels of the same color according to their content or orientation. This orientation or content aware concealment attempts to identify the orientation as far as possible or interpolates nearest neighbors using a strategy that takes into account the content in the raw support of the pixel being corrected.
[0053] The method determines depending on the color channel arrangement in the color filter array, the path to be taken is either in step A) or in the sequence of steps B), C), and D) or E).
[0054] When the phase detection pixel is located in a color channel that exhibits symmetry, the upper path is selected, starting with step D) of non-directional intermediate concealment. This is the case, for example, for a phase detection pixel located in the W channel in an RGBW color filter array, or in the R and G and B channels in a standard Bayer color filter array. The symmetry facilitates directional estimation in the correction.
[0055] In step B) a non-directional pre-concealment is performed: a robust estimate (pre-concealment) is calculated for each of the phase detection pixels. All other non-phase detection pixels are kept untouched.
[0056] The purpose of this pre-concealment, i.e. pre-processing in step B), is to provide a clean estimate of the middle of the raw support around the occluded phase-detection pixel, which in turn facilitates robust orientation identification at the location of that phase-detection pixel, without being disturbed by the presence of other phase-detection pixels in the raw support.
[0057] Let P_pd_est(x,y) denote a robust estimate of the PD pixel at location (x,y), where x and y are the pixel coordinates of the sensor. The raw support can be defined as an N×N window centered on pixel P(x,y), e.g., N=3 or 5. P_pd_est(x,y) must be robust to the presence of other same-color phase pixels in the raw window centered on P(x,y). Any robust estimate can be used, such as an alpha-trimmed average (e.g., using a weighting factor alpha=2) of the same-color weighted image pixel values around the PD pixel at location (x,y). The estimate may exclude other phase detection pixels in the raw window, if necessary.
[0058] In the following step c), directional filtering is performed to identify feature directions. In this step, the objective is to identify feature directions at the location of the occluded phase detection pixel P(x,y). Thus, directional filtering is performed on the pre-occlusion result. Specifically, at the location of the phase detection pixel, the pixel together with its same-color neighbors is convolved with a set of directional filters. Based on the maximum absolute response of the filter set, a feature direction is identified at the location of the phase detection pixel.
[0059] The number, size and coefficients of the filters depend on several factors selected from: a) the number of directions that are planned to be identified and supported in the concealment system; b) The size of the raw support centered on the phase detection pixel; c) color filter array channel arrangement, and d) The type of filter (gradient-based or other).
[0060] When the feature direction at the position P(x,y) of each phase detection pixel is identified, the phase detection pixel P(x,y) is concealed in step D) by directional concealment together with interpolating the image pixels assigned to the color along the identified direction. To reduce or eliminate the possibility of correction artifacts, the directional correction is typically restricted to a small neighborhood of the pixel. This may be restricted, for example, to an N×N window centered on the concealed phase detection pixel P(x,y), e.g., N=3 or 5.
[0061] If the feature direction is not identified, e.g. in very smooth regions, a step E) of non-directional concealment is performed instead of step D) by replacing the phase detection pixel P(x,y) by its robust estimate, e.g. P_pd_est(x,y). When concealing the phase detection pixel P(x,y) via direct interpolation in step D), if one or more phase detection pixels lie along the identified direction, its robust estimate P_pd_est(x,y) calculated in the pre-concealment step B) should be used to ensure robust concealment results and achieve better image quality.
[0062] The disclosed method does not need to know the value of the phase detection pixel; only its location is required. Also, no assumptions about the phase detection pixel behavior are required. This makes the method applicable to phase detection pixel hiding regardless of the type of color filter placed in front of the phase detection pixel and regardless of the accuracy in the color channels of the color channel array.
[0063] The corrected image pixel raw data is then further processed.
Claims
1. 1. A method for processing image data of an image sensor, the image sensor comprising a sensor area of a pixel matrix providing image pixel data, the pixel matrix comprising phase detection pixels at predefined locations, a set of image pixel data comprising phase detection information for the phase detection pixels, the method comprising: The method is characterized by a step of calculating image pixel data for the predefined locations of the phase detection pixels according to either symmetric or asymmetric positioning of each of the phase detection pixels in each color channel.
2. 2. The method of claim 1, wherein the image pixel data for asymmetrically positioned phase detection pixels in the color channels is calculated according to adjacent image pixel values of sensor pixels of the same color, according to a color assigned to the position of the phase detection pixel.
3. 3. The method of claim 2, wherein the image pixel data is calculated as an average value of nearest neighboring image pixel values of pixels of the same color located around the position of the selected phase detection pixel.
4. a) calculating the image pixel data for symmetrically positioned phase detection pixels in a color channel, the calculation comprising: b) estimating image pixel data for phase detection pixels located in a predetermined area around each selected phase detection pixel selected in step a) for calculating associated pixel image data; c) identifying feature directions at the selected phase detection pixel locations; d) calculating image pixel values according to interpolated pixel data located adjacent to the selected phase detection pixel in the identified feature direction; The method according to any one of claims 1 to 3, wherein the calculation is performed by:
5. 5. The method of claim 4, wherein each image pixel value is calculated according to the pixel data located in a selected area adjacent to and surrounding the selected phase detection pixel when a feature direction cannot be identified in step c).
6. 6. The method according to claim 1, wherein the pixel value for the location of the selected phase detection pixel is calculated by using neighboring pixel values associated with the same color as the associated color channel assigned to the position of the selected phase detection pixel.
7. Method according to any one of claims 1 to 6, characterized by a first step of determining a symmetric or asymmetric arrangement of the phase detection pixels in a color channel depending on a color channel parameter.
8. The method according to any one of claims 1 to 6, wherein the state of the symmetric or asymmetric arrangement of the phase detection pixels is preset.
9. 1. An image processor unit for processing image data of an image sensor, the image sensor comprising a sensor area of a pixel matrix providing image pixel data, the pixel matrix comprising phase detection pixels at predefined locations, the image sensor providing image pixel data at an input of the image processor unit, the set of image pixel data comprising phase detection information of the phase detection pixels, characterized in that the image processor unit is configured for image pixel data calculated for the predefined locations of the phase detection pixels depending on either symmetric or asymmetric positioning of each of the phase detection pixels in a color channel.
10. Image processor unit according to claim 9, characterized in that the image processor unit is arranged to process image data by performing the steps according to any one of claims 1 to 8.
11. A computer program comprising instructions which, when executed by a processing unit, cause the processing unit to perform the steps of the method of any one of claims 1 to 8.
Citation Information
Patent Citations
Imaging apparatus and imaging method
JP2009094881A
Crosstalk amount detection device and crosstalk amount detection method
JP2018078427A
Method for generating output signals of PDAF pixels
JP2022553726A
Color image sensor, imaging device, and imaging program
WO2012127702A1
Method for generating an output signal of a PDAF pixel
WO2021078473A1
