Image data correction method for image sensor

Abstract

The image data correction method for an image sensor provided in this paper includes: extracting the operating range from the image sensor; obtaining weighting factors for the operating range in each direction based on the readout value of the second photodiode within the operating range; using the weighting factors to obtain the interpolation value of the second photodiode at the position of the first photodiode within the first pixel group of the operating range; and estimating the correction value B of the target first photodiode through Equation I. target :, where G int B is the interpolated value of the second photodiode at the location of the first photodiode of the target. mean It is the average value of the readout values ​​of the first photodiodes in the corresponding first pixel group, including the target first photodiode, G mean It is the average value of the interpolation value of the second photodiode at the position of the first photodiode in the first pixel group.

Description

Technical Field

[0001] This disclosure relates to an image data correction method for an image sensor. Background Technology

[0002] Image sensors comprise pixel arrays implemented using photosensitive elements such as photodiodes and optical structures such as color filters, microlenses, or combinations thereof. In one implementation, multiple photodiodes are covered by a single microlens, enabling phase-detection autofocus (PDAF). However, the individual readouts of photodiodes covered by a common microlens are often inaccurate due to the optical side effects of the shared microlens. Therefore, the image data from image sensors needs to be improved. Summary of the Invention

[0003] According to an embodiment of the present invention, a method for correcting image data of an image sensor is as follows. The image sensor is connected to a processing computer and includes a first pixel group and a second pixel group arranged alternately along a horizontal and vertical direction. Each of the first pixel groups includes four first photodiodes commonly covered by first microlenses, and each of the second pixel groups includes four second photodiodes respectively covered by four microlenses. The method includes: extracting an operating range from the image sensor; obtaining weighting factors for the operating range in each direction based on readout values ​​of the second photodiodes within the operating range; obtaining interpolated values ​​of the second photodiodes at the positions of the first photodiodes in the first pixel group within the operating range using the weighting factors; and estimating a correction value for a target first photodiode using Equation I.

[0004]

[0005] Among them B target It is the target correction value of the first photodiode, G. int B is the interpolation value of the second photodiode at the location of the first photodiode on the target. mean It is the average value of the readout values ​​of the first photodiodes in the corresponding first pixel group, including the target first photodiode, and G mean It is the average value of the interpolated value of the second photodiode at the position of the first photodiode in the first pixel group. Attached Figure Description

[0006] Figure 1 A portion of an image sensor is illustrated according to some embodiments of this disclosure.

[0007] Figure 2 An image data correction method for an image sensor is illustrated schematically according to some embodiments of the present disclosure.

[0008] Figures 3 to 6 The photodiode array 100A and the pixel group array 100A' of the extracted operating range are described respectively to explain the steps for determining the weighting factors in each direction.

[0009] Figures 7 to 10 The steps for extracting the operating range of the photodiode array 100A and displaying the component interpolation value of the second photodiode at the target position are described separately.

[0010] Figure 11 A method for estimating the correction value of a first photodiode used to correct image data from an image sensor is illustrated schematically.

[0011] Corresponding reference characters indicate the corresponding parts in several views of the accompanying drawings. Those skilled in the art will understand that the elements in the drawings are shown for simplicity and clarity and are not necessarily drawn to scale. For example, the dimensions of some elements in the drawings may be exaggerated relative to other elements to aid in understanding the various embodiments of the invention. Detailed Implementation

[0012] References to an implementation, embodiment, example, and / or similar content in this specification mean that a particular feature, structure, characteristic, and / or similar content described in relation to a particular implementation and / or embodiment is included in at least one implementation and / or embodiment of the claimed subject matter. Therefore, such phrases appearing throughout this specification, for example, are not necessarily intended to refer to the same implementation and / or embodiment or any particular implementation and / or embodiment. Furthermore, it should be understood that the particular features, structures, characteristics, and / or similar content described can be combined in various ways in one or more implementations and / or embodiments, and thus within the contemplated scope of the claims. Generally, and of course as has always been the case with patent application specifications, these and other issues are likely to vary in a particular context of use. In other words, throughout the disclosure, the specific context of the description and / or use provides useful guidance regarding reasonable inferences to be drawn; however, similarly, the phrase "in this context" without further elaboration generally refers at least to the context of this patent application.

[0013] Figure 1 A portion of an image sensor is illustrated schematically according to some embodiments of this disclosure. Figure 1In this embodiment, image sensor 100 is connected to processing computer 102, which can receive and process sensed image data from image sensor 100 to output image data of the sensed image. In some embodiments, the sensed image can be displayed by an electronic device. Image sensor 100 includes a first pixel group 110 and a second pixel group 120 arranged alternately along the horizontal direction H and the vertical direction V. In this embodiment, each of the first pixel groups 110 includes four first photodiodes 112 covered by a first microlens 114. Each first pixel group 110 is implemented using a four-quadrant photodiode (QPD) configuration. Each second pixel group 120 includes four second photodiodes 122, each covered by a microlens 124. The second pixel group 120 is implemented using a one-photodiode-to-one-microlens configuration (also referred to as a 4C or 4-unit configuration).

[0014] In some embodiments, the second pixel group 120 is configured to sense green incident light, half of the first pixel group 110 is configured to sense blue incident light, and the other half of the first pixel group 110 is configured to sense red incident light to form a Bayer-like pattern configuration. For example, two first pixel groups 110 and two second pixel groups 120 are arranged in a 2×2 array to form a pixel unit 104. The two first pixel groups 110 can be arranged diagonally along a first diagonal direction D1, and the two second pixel groups 120 can be arranged diagonally along a second diagonal direction D2. The two first pixel groups 110 in the pixel unit 104 can be a blue pixel group and a red pixel group, and the two second pixel groups 120 in the pixel unit 104 can be two green pixel groups.

[0015] Photodiodes 112 in one of the first pixel groups 110 share a first microlens 114, which facilitates phase detection autofocus (PDAF). Therefore, the image sensor 100 enables fast autofocus. Each of the second pixel groups 120 achieves and enables high sensing resolution by covering a single photodiode with a single microlens. Specifically, the image sensor 100 with a half-QPD configuration can leverage the advantages of both QPD and 4C configurations.

[0016] In some embodiments, the individual readout values ​​of the first photodiodes 112 in the first pixel group 110 implemented by the QPD configuration may be inaccurate due to optical side effects and / or phase shift effects in a design where a microlens covers four photodiodes. However, the total energy of the incident light received by each first photodiode 112 in the first pixel group 110 is not affected by optical side effects and / or phase shift effects. Therefore, the average value of the individual readout values ​​of the first photodiodes 112 in a first pixel group 110 is accurate. In this embodiment, the image data of the image sensor 100 can be corrected using the following method.

[0017] Figure 2 A method for correcting image data of an image sensor according to some embodiments of the present disclosure is illustrated schematically. Figure 2 The method can be derived from Figure 1 The processing computer 102 shown in the image is executing the task. Figure 2 In step S01, the process window is extracted from the image sensor. For example... Figure 1 As shown, 10 × 10 photodiodes from the first photodiode 112 and the second photodiode 122 are extracted from the image sensor 100 as the operating range. In some embodiments, Figure 1 The processing computer 102 shown can perform preprocessing on the readout values ​​provided by the first photodiode 112 and the second photodiode 122 before executing step S01.

[0018] Then, step S02, obtaining the weighting factors of the operating range in each direction, is performed. Specifically, the weighting factors can be obtained based on the readout values ​​of the second photodiode 122 within the operating range. In step S02, the weighting factors of the operating range can be determined based on the gradients of the readout values ​​of the second photodiode 122 in each direction. The gradients of the readout values ​​of the second photodiode 122 in each direction can be related to the graphic features of the sensed image. For example, if the gradient of the readout value of the second photodiode 122 is small in one component direction, the image may involve slight variations in the component direction, such that the readout values ​​of adjacent second photodiodes 122 in the component direction can represent similar features with high correlation. On the other hand, if the gradient of the readout value of the second photodiode 122 is large in one component direction, the image may involve significant variations in the component direction, such that the readout values ​​of adjacent second photodiodes 122 in the component direction can represent different features with very low correlation. Therefore, in this embodiment, the weighting factors are determined based on the rule that the larger the gradient, the smaller the weighting factor.

[0019] Step S03 is performed to obtain the interpolation value of the second photodiode 122 at the position of the first photodiode 112 in the first pixel group 110 within the operating range. Specifically, the interpolation value of the second photodiode 122 at the position of the first photodiode 112 can be obtained by a suitable interpolation method. In this embodiment, the interpolation value of the second photodiode 122 at the position of the first photodiode 112 is obtained by using the weighting factor obtained from step S02 and the readout values ​​of the second photodiodes 122 adjacent to the position of the first photodiode 112 in each direction.

[0020] Subsequently, as shown in step S04, a correction value for each first photodiode 112 can be estimated. In this embodiment, the correction value for each first photodiode 112 can be estimated using the interpolation value of the second photodiode 122 at the location of each first photodiode 112. For example, the correction value for each first photodiode is obtained through Equation I: B target It is the correction value of the target first photodiode in the first photodiode 112, G int It is the interpolated value of the second photodiode 122 at the location of the first photodiode 112, B. mean It is the average value of the readout values ​​corresponding to the first photodiode 112 in the first pixel group 110, including the target first photodiode 112, G mean It is the average of the interpolated values ​​of the second photodiodes 122 at the positions corresponding to the first photodiode 112 in the first pixel group 110, which includes the target first photodiode 112. After obtaining the correction value of the first photodiode 112, the correction value of the first photodiode 112 can be used as the image data output of the image sensor 100. In other words, the output image data of the image sensor 100 can be composed of the readout value of the second photodiode 122 and the correction value of the first photodiode 112, which provides an accurate image, even if the first pixel group 110 is implemented through a QPD configuration.

[0021] Figures 3 to 6 The photodiode array 100A and the pixel group array 100A' of the extracted operating range are described separately. The photodiode array 100A of the extracted operating range includes 10×10 photodiodes, and the pixel group array 100A' of the extracted operating range includes 5×5 pixel groups. Each pixel group includes 2×2 photodiodes binned together. Figures 3 to 6 This can be used to explain the steps of obtaining internal and global gradients to determine weight factors in various directions. In some embodiments, from Figures 3 to 6 Each gradient value obtained can be converted into a weighting factor using Equation II: , where Y n It is a weighting factor, X n Here, θ represents the gradient value, n indicates the component direction, and θ is a positive number. In this embodiment, when the component direction is horizontal (H), the letter "n" is "h"; when the component direction is vertical (V), the letter "n" is "v"; when the component direction is the first diagonal direction (D1), the letter "n" is "p"; and when the component direction is the second diagonal direction (D2), the letter "n" is "q". Furthermore, θ can be selected from 0.7 to 3.0, but this disclosure is not limited thereto.

[0022] like Figure 3 As shown, each second photodiode 122 in the photodiode array 100A provides a readout value R122. Each internal gradient H122 is obtained by calculating the absolute value of the difference between the readout values ​​R122 of two adjacent second photodiodes 122 in the horizontal direction H. In the photodiode array 100A, the double arrows of each display internal gradient H122 point to two adjacent second photodiodes 122 in the horizontal direction H. Each second pixel group 120 in the pixel group array 100A' provides a binning value R120', and the binning value R120' of each second pixel group 120 is obtained by binning the four second photodiodes 122 in each second pixel group 120 (e.g., ...). Figure 1 The readout values ​​(shown) are binned to obtain the global gradient H120'. Each global gradient H120' is obtained by calculating the absolute value of the difference between the binning values ​​R120' of two adjacent second pixel groups 120 arranged in the horizontal direction H. In the pixel group array 100A', each double arrow displaying the global gradient H120' points to two adjacent second pixel groups 120 in the horizontal direction H. After obtaining the internal gradient H122 and the global gradient H120', the sum of the internal gradient H122 is multiplied by the sum of the global gradient H120' to obtain the gradient value X in the horizontal direction H. h Then based on the gradient value X h Obtain the weight factor in the horizontal direction H described in step S01, where the gradient value X h The larger the value, the smaller the weighting factor. For example, the weighting factor in the horizontal direction H can be represented as Y. h And from Equation II: , where θ is 0.9 or other positive number, and h indicates the horizontal direction H.

[0023] like Figure 4As shown, each second photodiode 122 in the photodiode array 100A provides a readout value R122. Each internal gradient V122 is obtained by calculating the absolute value of the difference between the readout values ​​R122 of two adjacent second photodiodes 122 in the vertical direction V. In the photodiode array 100A, each double arrow displaying the internal gradient V122 points to two adjacent second photodiodes 122 in the vertical direction V. Each second pixel group 120 in the pixel group array 100A' provides a binning value R120', and the binning value R120' of each second pixel group 120 is obtained by binning the four second photodiodes 122 in each second pixel group 120 (e.g., ...). Figure 1 The readout values ​​(shown) are binned to obtain the global gradient V120'. Each global gradient V120' is obtained by calculating the absolute value of the difference between the binning values ​​R120' of two adjacent second pixel groups 120 arranged in the vertical direction V. In the pixel group array 100A', each double arrow displaying the global gradient V120' points to two adjacent second pixel groups 120 in the vertical direction V. After obtaining the internal gradient V122 and the global gradient V120', the sum of the internal gradients V122 is multiplied by the sum of the global gradients V120' to obtain the gradient value X in the vertical direction V. v Based on gradient value X v Obtain the weight factor in the vertical direction V described in step S02, where the gradient value X v The larger the value, the smaller the weighting factor. For example, the weighting factor in the vertical direction V can be represented as Y. v And from Equation II: , where n indicates the direction of the component is the vertical direction V, and θ is 0.9 or other positive numbers.

[0024] like Figure 5 As shown, each second photodiode 122 in the photodiode array 100A provides a readout value R122. Each internal gradient P122 is obtained by calculating the absolute value of the difference between the readout values ​​R122 of two adjacent second photodiodes 122 in the first diagonal direction D1. In the photodiode array 100A, the double arrows of each display internal gradient P122 point to two adjacent second photodiodes 122 in the first diagonal direction D1. Each second pixel group 120 in the pixel group array 100A' provides a binning value R120', and the binning value R120' of each second pixel group 120 is obtained by binning the four second photodiodes 122 in each second pixel group 120 (e.g., ...). Figure 1The readout values ​​(shown) are binned to obtain the global gradient P120'. Each global gradient P120' is obtained by calculating the absolute value of the difference between the binning values ​​R120' of two adjacent second pixel groups 120 arranged in the first diagonal direction D1. In the pixel group array 100A', each double arrow displaying the global gradient P120' points to two adjacent second pixel groups 120 in the first diagonal direction D1. After obtaining the internal gradient P122 and the global gradient P120', the sum of the internal gradients P122 is multiplied by the sum of the global gradients P120' to obtain the gradient value X in the first diagonal direction D1. p Based on gradient value X p Obtain the weight factor on the first diagonal direction D1 described in step S02, where the gradient value X p The larger the value, the smaller the weighting factor. For example, the weighting factor on the first diagonal direction D1 can be represented as Y. p And from Equation II: , where p indicates the direction of the component as the first diagonal direction D1, and θ is 0.9 or other positive numbers.

[0025] like Figure 6 As shown, each second photodiode 122 in the photodiode array 100A provides a readout value R122. Each internal gradient Q122 is obtained by calculating the absolute value of the difference between the readout values ​​R122 of two adjacent second photodiodes 122 in the second diagonal direction D2. In the photodiode array 100A, the double arrows of each display internal gradient Q122 point to two adjacent second photodiodes 122 in the second diagonal direction D2. Each second pixel group 120 in the pixel group array 100A' provides a binning value R120', and the binning value R120' of each second pixel group 120 is obtained by binning the four second photodiodes 122 in each second pixel group 120 (e.g., ...). Figure 1 The readout values ​​(shown) are binned to obtain the global gradient Q120'. Each global gradient Q120' is obtained by calculating the absolute value of the difference between the binning values ​​R120' of two adjacent second pixel groups 120 arranged in the second diagonal direction D2. In the pixel group array 100A', each double arrow displaying the global gradient P120' points to two adjacent second pixel groups 120 in the second diagonal direction D2. After obtaining the internal gradient Q122 and the global gradient Q120', the sum of the internal gradients Q122 is multiplied by the sum of the global gradients Q120' to obtain the gradient value X in the second diagonal direction D2. q Based on gradient value X q Obtain the weighting factor on the second diagonal direction D2 described in step S02, where the gradient value X q The larger the value, the smaller the weighting factor. For example, the weighting factor on the second diagonal direction D2 can be represented as Y. q And from Equation II: We obtain , where q indicates the direction of the component as the second diagonal direction D2, and θ is 0.9 or another positive number.

[0026] Figures 7 to 10 The photodiode array 100A with its extracted operating range is described and displayed. Figure 1 The step of obtaining component interpolation values ​​of the second photodiode in various directions at the target position of one of the first photodiodes 112 in the first pixel group 110 shown. Here, each direction includes, for example, Figure 1 The four component directions are shown. Specifically, Figure 7 The steps for obtaining component interpolation values ​​in the horizontal direction H are shown. Figure 8 The steps for obtaining component interpolation values ​​in the vertical direction V are shown. Figure 9 This shows the steps for obtaining component interpolation values ​​in the first diagonal direction D1. Figure 10 The steps for obtaining component interpolation values ​​in the second diagonal direction D2 are shown. In some embodiments, one of the component interpolation values ​​in one component direction is obtained by Equation III: V n It is the component interpolation value in the component direction, where n indicates the component direction, Rn i F is the readout value of the i-th photodiode out of m second photodiodes. i It is the component weight value.

[0027] Figure 7 The display operating range includes 10×10 photodiodes to illustrate the operation of... Figure 1 The step of obtaining the component interpolation value of the second photodiode 122 at the target position (4,4) of one of the first photodiodes 112 in the first pixel group 110 (e.g., the target first photodiode 112A). Specifically, this method is used to correct the position of the second photodiode 122. Figure 7 The readout value of the target first photodiode 112A at position (4,4) is shown. To obtain the component interpolation value at position (4,4) in the component direction (e.g., horizontal direction H), the readout values ​​of m second photodiodes 122 are used. The m selected second photodiodes 122 are arranged in the horizontal direction H and adjacent to the target position (4,4) of the target first photodiode 112A. In this embodiment, m is 4 as an example. The four selected second photodiodes 122 are equidistantly located on opposite sides of the target position (4,4) of the target first photodiode 112A in the horizontal direction H. For example, in the horizontal direction H, two of the selected second photodiodes 122 are located to the left of the target position (4,4), and the other two selected second photodiodes 122 are located to the right of the target position (4,4).

[0028] Here, Equation III: It can be used to obtain component interpolation values, where V n It is the component interpolation value in the component direction, where n indicates the component direction, Rn i F is the readout value of the i-th photodiode out of m second photodiodes. i It is the component weight value. Specifically, for Figure 7 In the example shown, second photodiode 122A provides a readout value Rh1, second photodiode 122B provides a readout value Rh2, second photodiode 122C provides a readout value Rh3, and second photodiode 122D provides a readout value Rh4. n is denoted as "h" to indicate the horizontal direction H. Equation III is extended and expressed as... V h These are the component interpolation values ​​in the horizontal direction H, Rh1 ~ Rh4 are the readout values ​​of the four selected second photodiodes 122, and F1 ~ F4 are the component weight values ​​determined by the position of the corresponding second photodiode 122.

[0029] Second photodiodes 122A and 122B are located on the first side of the target position (4,4), with second photodiode 122A being closer to the target position (4,4) than second photodiode 122B. The component weight value F1 of the readout value Rh1 is greater than the component weight value F2 of the readout value Rh2. For example, component weight value F1 can be 4, while component weight value F2 can be 2. Meanwhile, second photodiodes 122C and 122D are located on the second side of the target position (4,4), with second photodiode 122C being closer to the target position (4,4) than second photodiode 122D. The component weight value F3 of the readout value Rh3 can be 2, while the component weight value F4 of the readout value Rh4 can be 1. Here, the second photodiode 122A is closer to the target position (4,4) than the second photodiode 122C, and the component weight value F1 of the readout value Rh1 is greater than the component weight value F3 of the readout value Rh3, but the disclosure is not limited thereto. In some embodiments, the values ​​of F1 to F4 can be combinations of other values ​​that satisfy F1 > F2 and F3 > F4.

[0030] like Figure 8As shown, to obtain the component interpolation value at position (4,4) in the vertical direction V from the photodiode array 100A, the readout values ​​of m second photodiodes 122 are used. The selected m second photodiodes 122 are arranged in the vertical direction V, adjacent to the target position (4,4) of the first photodiode 112. In this embodiment, m is 4 as an example. The four selected second photodiodes 122 are equally distributed on opposite sides of the target position (4,4) of the target first photodiode 112A in the component direction, i.e., the vertical direction V. For example, two of the selected second photodiodes 122 are located above the target position (4,4), and the other two are located below the target position (4,4) in the vertical direction V.

[0031] Here, Equation III: It can be used to obtain component interpolation values, where V n It is the component interpolation value in the component direction, where n indicates the component direction, Rn i F is the readout value of the i-th photodiode out of m second photodiodes. i These are component weight values. In this embodiment, the second photodiode 122E provides the readout value Rv1, the second photodiode 122F provides the readout value Rv2, the second photodiode 122G provides the readout value Rv3, and the second photodiode 122H provides the readout value Rv4. Specifically, for the component interpolation value in the vertical direction V, denoted as "v", Equation III is extended and expressed as... ,in These are the component interpolation values ​​in the vertical direction V. Rv1~Rv4 are the readout values ​​of the four selected second photodiodes 122, respectively. F1~F4 are the component weight values ​​determined by the position of the corresponding second photodiode 122.

[0032] The second photodiode 122E and the second photodiode 122F are located on the first side of the target position (4,4), with the second photodiode 122E being closer to the target position (4,4) than the second photodiode 122F. The component weight value F1 of the readout value Rv1 is greater than the component weight value F2 of the readout value Rv2. For example, the component weight value F1 can be 3, while the component weight value F2 can be 1. Meanwhile, the second photodiode 122G and the second photodiode 122H are located on the second side of the target position (4,4), with the second photodiode 122G being closer to the target position (4,4) than the second photodiode 122H. The component weight value F3 of the readout value Rv3 can be 3, while the component weight value F4 of the readout value Rv4 can be 1. In some embodiments, the values ​​of F1 to F4 can be combinations of other values ​​satisfying F1 > F2 and F3 > F4.

[0033] like Figure 9As shown, to obtain the component interpolation value at position (4,4) in the first diagonal direction D1, the readout values ​​of m second photodiodes 122 are used. The selected m second photodiodes 122 are arranged in the first diagonal direction D1 and adjacent to the target position (4,4) of the target first photodiode 112A. In this embodiment, m is 4 as an example. The four selected second photodiodes 122 are equally distributed on opposite sides of the target position (4,4) of the target first photodiode 112A in the component direction, i.e., the first diagonal direction D1. For example, two of the selected second photodiodes 122 are located on the upper right side of the target position (4,4) in the first diagonal direction D1, and the other two of the selected second photodiodes 122 are located on the lower left side of the target position (4,4) in the first diagonal direction D1.

[0034] Here, Equation III: It can be used to obtain component interpolation values, where V n It is the component interpolation value in the component direction, where n indicates the component direction, Rn i F is the readout value of the i-th photodiode out of m second photodiodes. i These are component weight values. In this embodiment, the second photodiode 122I provides the readout value Rp1, the second photodiode 122J provides the readout value Rp2, the second photodiode 122K provides the readout value Rp3, and the second photodiode 122L provides the readout value Rp4. Specifically, for the component interpolation value in the first diagonal direction D1, note that n is "p", Equation III is extended and expressed as... V p These are the component interpolation values ​​in the first diagonal direction D1, Rp1~Rp4 are the readout values ​​of the four selected second photodiodes 122, and F1~F4 are the component weight values ​​determined by the position of the corresponding second photodiode 122.

[0035] The second photodiode 122I and the second photodiode 122J are located on the first side of the target position (4,4), with the second photodiode 122I being closer to the target position (4,4) than the second photodiode 122J. The component weight value F1 of the readout value Rv1 is greater than the component weight value F2 of the readout value Rv2. For example, the component weight value F1 can be 4, while the component weight value F2 can be 2. Meanwhile, the second photodiode 122K and the second photodiode 122L are located on the second side of the target position (4,4), with the second photodiode 122K being closer to the target position (4,4) than the second photodiode 122L. The component weight value F3 of the readout value Rv3 can be 2, while the component weight value F4 of the readout value Rv4 can be 1. In some embodiments, the values ​​of F1 to F4 can be combinations of other values ​​that satisfy F1 > F2 and F3 > F4.

[0036] like Figure 10 As shown, to obtain the component interpolation value at position (4,4) in the component direction, second diagonal direction D2, the readout values ​​of m second photodiodes 122 are used. The selected m second photodiodes 122 are arranged in the second diagonal direction D2 and adjacent to the target position (4,4) of the target first photodiode 112A. In this embodiment, m is 8 as an example. The selected eight second photodiodes 122 are equally distributed on opposite sides of the target position (4,4) of the first photodiode 112 in the component direction, second diagonal direction D2. For example, four of the selected second photodiodes 122 are located on the upper left side of the target position (4,4), and the other four of the selected second photodiodes 122 are located on the lower right side of the target position (4,4) in the second diagonal direction D2.

[0037] In the photodiode array 100A, a first photodiode 112 is positioned along the same linear path as the target position (4,4) along the second diagonal direction D2. Therefore, the selected second photodiode 122 is not arranged along the same linear path as the target position (4,4) along the second diagonal direction D2. Here, second photodiodes 122M, 122N, 122O, and 122P are arranged along the same linear path along the second diagonal direction D2, while second photodiodes 122Q, 122R, 122S, and 122T are arranged along the same linear path along the second diagonal direction D2. Along the second diagonal direction D2, the second photodiode 122M and the second photodiode 122Q can maintain the same distance from the first photodiode 112A at the target position (4,4), the second photodiode 122N and the second photodiode 122R can maintain the same distance from the first photodiode 112A at the target position (4,4), the second photodiode 122O and the second photodiode 122S can maintain the same distance from the first photodiode 112A at the target position (4,4), and the second photodiode 122P and the second photodiode 122T can maintain the same distance from the first photodiode 112A at the target position (4,4).

[0038] Here, Equation III can be used: To obtain the component interpolation values, where V n It is the component interpolation value in the component direction, where n indicates the component direction, Rn i F is the readout value of the i-th photodiode out of m second photodiodes. iThese are component weight values. Specifically, in the embodiment, the second photodiode 122M provides a readout value Rq1, the second photodiode 122N provides a readout value Rq2, the second photodiode 122O provides a readout value Rq3, and the second photodiode 122P provides a readout value Rq4. Furthermore, the second photodiode 122Q provides a readout value Rq5, the second photodiode 122R provides a readout value Rq6, the second photodiode 122S provides a readout value Rq7, and the second photodiode 122T provides a readout value Rq8. For the component interpolation value in the second diagonal direction D2, note that n is "q", Equation III is extended and expressed as... V q These are the component interpolation values ​​in the second diagonal direction D2, Rq1~Rq8 are the readout values ​​of the eight selected second photodiodes 122 respectively, and F1~F8 are the component weight values ​​determined by the position of the corresponding second photodiode 122.

[0039] The second photodiode 122M and the second photodiode 122N are located on the first side of the target position (4,4), with the second photodiode 122M being closer to the target position (4,4) than the second photodiode 122N. The component weight value F1 of the readout value Rq1 is greater than the component weight value F2 of the readout value Rq2. For example, the component weight value F1 can be 3, while the component weight value F2 can be 1. The second photodiode 122Q and the second photodiode 122R are located on the first side of the target position (4,4), with the second photodiode 122Q being closer to the target position (4,4) than the second photodiode 122R. The component weight value F5 of the readout value Rq5 is greater than the component weight value F6 of the readout value Rq6. For example, the component weight value F5 can be 3, while the component weight value F6 can be 1.

[0040] Simultaneously, the second photodiode 122O and the second photodiode 122P are located on the second side of the target position (4,4), with the second photodiode 122O being closer to the target position (4,4) than the second photodiode 122P. The component weight value F3 of the readout value Rq3 can be 3, while the component weight value F4 of the readout value Rq4 can be 1. The second photodiode 122S and the second photodiode 122T are located on the second side of the target position (4,4), with the second photodiode 122S being closer to the target position (4,4) than the second photodiode 122T. The component weight value F7 of the readout value Rq7 can be 3, while the component weight value F8 of the readout value Rq8 can be 1. In some embodiments, the values ​​of F1 to F8 can be combinations of other values ​​satisfying F1>F2, F3>F4, F5>F6, and F7>F8.

[0041] The interpolated component values ​​V of the target position (4,4) in four directions are obtained: horizontal direction H, vertical direction V, first diagonal direction D1, and second diagonal direction D2. h V v V p and V q Subsequently, the interpolated value of the second photodiode 122 at the target position (4,4) is obtained through equation IV: G int It is the interpolated value, Y n V is a weighting factor for the operational range of the component directions in each direction. n Yn represents the component interpolation value in each direction, where n indicates the component direction. In this embodiment, Yn can be obtained by executing... Figures 3 to 6 The steps described in the text are used to obtain V. n By executing Figures 7 to 10 The steps described herein are used to obtain the value. Specifically, the interpolation value G for the second photodiode 122 at the target position (4,4) is obtained. int Equation IV can be extended and expressed as , where h represents the horizontal direction H as the component direction, v represents the vertical direction V as the component direction, p represents the first diagonal direction D1 as the component direction, and q represents the second diagonal direction D2 as the component direction.

[0042] Figure 11 The diagram schematically illustrates a method for estimating the correction value of a first photodiode for calibrating image data from an image sensor, wherein the photodiode array 100A... Figure 1 The image sensor 100 depicted is selected as the operating range. Figure 11 In this method, the readout value RT of the first photodiode 112A located at the target position (4,4) is displayed. A Corrected as an example, Figure 1 Each first photodiode 112 in the image sensor 100 shown can be calibrated using the same method. In this embodiment, the target first photodiode 112A belongs to the corresponding first pixel array 110A surrounded by a dashed square, and the corresponding first pixel array 110A includes four photodiodes, first photodiodes 112A to 122D.

[0043] like Figure 1As described, the first photodiodes 112A-122D share a single microlens 114, and due to certain optical side effects of the shared microlens, may provide inaccurate readout values ​​compared to the second photodiode 122. However, the total energy received by the first photodiodes 112A-122D in a first pixel array 110 is least likely to be affected by optical effects and is substantially equivalent to that of the second photodiode 122 in a second pixel array 120. Therefore, the average readout value provided by the first photodiodes 112A-122D is likely to be accurate and reliable. Furthermore, the readout value of the second photodiode 122 is accurate and reliable because the second photodiode 122 is implemented in a 4C (4-unit) configuration. The correction method in this embodiment can be based on the assumption that the readout value RT of the target first photodiode 112A is... A The first ratio of the average value of the readout values ​​provided by the first photodiodes 112A to 122D is the same as the second ratio of the average value of the interpolation value of the second photodiode 122 at the position (4,4) of the target first photodiode 112A to the average value of the interpolation values ​​of the second photodiode 122 at the positions of the first photodiodes 112A to 122D.

[0044] The target first photodiode 112A is configured to provide a readout value RT A The first photodiode 112B is configured to provide a readout value RT B The first photodiode 112C is configured to provide a readout value RT C The first photodiode 112D is configured to provide a readout value RT. D In this embodiment, the interpolation value of the second photodiode 122 at the positions of the first photodiodes 112A-122D can be obtained by performing an operation on each position of the first photodiodes 112A-122D. Figures 7 to 10 The method described in the text is used to obtain the corresponding interpolation value GA. int GB int GC int and GD int And calculate the corresponding interpolation value GA int GB int GC int and GD int average G mean .

[0045] Specifically, in this embodiment, the correction value of each of the first photodiodes 112A~112D can be obtained through Equation I: B target It is the correction value for each of the first photodiodes (taking the target first photodiode 112A as an example), G intIt is the interpolation value of the second photodiode 112 at one of the positions of the first photodiodes 112A~112D, B mean It is the average value of the readout values ​​of the first photodiodes 112A~112D in the first pixel group 110A, including the target first photodiode 112A, G mean The interpolation value GA of the second photodiode 122 at the position corresponding to the first photodiodes 112A~112D in the first pixel group 110A is... int GB int GC int and GD int The average value. Then, the correction value B. target The output is used as the sensed image data of the target first photodiode 112A in the image sensor.

[0046] In view of the above, the image data correction method of the image sensor according to embodiments of the present disclosure utilizes a pixel array implemented through a semi-QPD configuration. The readout values ​​of photodiodes under the QPD design are corrected using interpolated values ​​of photodiodes under a 4C (4-cell) design. The corrected values ​​are more accurate than the individual readout values ​​of photodiodes under the QPD design because the individual readout values ​​of photodiodes under the QPD design introduce inaccuracies. The image sensor outputs image data using the corrected values ​​of photodiodes under the QPD design instead of the individual readout values ​​of photodiodes under the QPD design, thus resulting in output image data with good image quality.

[0047] In some embodiments of this disclosure, an image data correction method for an image sensor is provided. The image sensor is connected to a processing computer and includes a first pixel group and a second pixel group arranged alternately along horizontal and vertical directions. Each pixel group contains four first photodiodes, collectively covered by a first microlens. Each pixel group contains four second photodiodes, each covered by a microlens. The method includes extracting an operating range from the image sensor; obtaining weighting factors for the operating range in various directions based on readout values ​​of the second photodiodes within the operating range; obtaining interpolation values ​​of the second photodiodes at their positions within the first pixel group within the operating range using the weighting factors; and estimating a correction value for a target first photodiode using Equation I. B target It is the correction value of the target first photodiode in the first photodiode, G. int B is the interpolation value of the second photodiode at the location of the first photodiode on the target. mean It is the average value of the readout values ​​of the first photodiodes in the corresponding first pixel group, including the target first photodiode, G meanIt is the average value of the interpolation value of the second photodiode at the position of the first photodiode in the corresponding first pixel group.

[0048] In some embodiments of this disclosure, the operating range includes 10 × 10 photodiodes in the first and second photodiodes. Each weight factor in each component direction is obtained by: obtaining the internal gradient of the second photodiode readout value in the component direction by calculating the absolute value of the difference between the readout values ​​of two adjacent second photodiodes in the component direction; obtaining the global gradient of the second pixel group binning value in the component direction by calculating the absolute value of the difference between the binning values ​​of two adjacent second pixel groups, wherein the binning value of each second pixel group is obtained by binning the readout values ​​of the four second photodiodes within each second pixel group; multiplying the sum of the internal gradients in the component direction by the sum of the global gradients to obtain a gradient value; and determining a weight factor based on the gradient value in the component direction, wherein a larger gradient value results in a smaller weight factor.

[0049] In some embodiments of this disclosure, the gradient values ​​in the component directions are converted into weighting factors in the component directions using Equation II: , where Y n It is a weighting factor, X n This is the gradient value, where n indicates the component direction, and θ is a positive number. θ is chosen to be between 0.7 and 3.0.

[0050] In some embodiments of this disclosure, each direction includes a horizontal direction, a vertical direction, a first diagonal direction, and a second diagonal direction.

[0051] In some embodiments of this disclosure, the interpolation value of the second photodiode at the location of the target first photodiode is obtained based on component interpolation values ​​in various directions.

[0052] In some embodiments of this disclosure, one of the component interpolation values ​​in each direction is obtained from the readout values ​​of m second photodiodes, where m is a multiple of 2. The m second photodiodes are arranged in the component direction and adjacent to the target first photodiode, and are equidistantly located on opposite sides of the target first photodiode in the component direction. One of the component interpolation values ​​in the component direction is obtained by Equation III: V n It is the component interpolation value in the component direction, where n indicates the component direction, Rn i F is the readout value of the i-th photodiode out of m second photodiodes. i This is the component weight value. In some embodiments of this disclosure, for one of the m second photodiodes closer to the target first photodiode location, F... iIt is larger than another of the m second photodiodes located relatively far from the target first photodiode. In some embodiments of this disclosure, for one of the m second photodiodes more adjacent to the target first pixel group including the target first photodiode, F i It is larger than another of the m second photodiodes that are relatively far from the target first pixel group.

[0053] In some embodiments of this disclosure, each of the interpolated values ​​is obtained via Equation IV: G int It is the interpolated value, Y n It is the weighting factor of the operating range in each direction of the component directions, V n It is the component interpolation value in the component direction, where n indicates the component direction.

[0054] It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed embodiments without departing from the scope or spirit of this disclosure. In view of the foregoing, this disclosure is intended to cover modifications and variations falling within the scope of the following claims and their equivalents.

Claims

1. A method of correcting image data of an image sensor, wherein, The image sensor is connected to a processing computer and includes a first pixel group and a second pixel group arranged alternately along a horizontal and vertical direction. Each of the first pixel groups includes four first photodiodes commonly covered by first microlenses, and each of the second pixel groups includes four second photodiodes respectively covered by four microlenses. The method includes: Extract the operating range from the image sensor; The weighting factor of the operating range in each direction is obtained based on the readout value of the second photodiode within the operating range; The interpolation value of the second photodiode at the position of the first photodiode in the first pixel group within the operating range is obtained by using the weighting factor. as well as The target correction value of the first photodiode is estimated using Equation I: Among them B target G is the correction value of the target first photodiode of the first photodiode. int B is the interpolation value of the second photodiode at the location of the first photodiode in the target. mean It is the average value of the readout values ​​of the first photodiodes in the corresponding first pixel group, including the target first photodiode, and G mean It is the average value of the interpolation value of the second photodiode at the position of the first photodiode in the corresponding first pixel group.

2. The method according to claim 1, wherein the operating range includes 10 × 10 photodiodes of the first photodiode and the second photodiode.

3. The method of claim 1, wherein each of the weighting factors in the component directions of each direction is obtained by: The internal gradient of the readout value of the second photodiode in the component direction is obtained by calculating the absolute value of the difference between the readout values ​​of two adjacent second photodiodes in the component direction; The global gradient of the binning value of the second pixel group in the component direction is obtained by calculating the absolute value of the difference between the binning values ​​of two adjacent second pixel groups, wherein the binning value of each of the second pixel groups is obtained by binning the readout values ​​of the four second photodiodes in each of the second pixel groups; The gradient value is obtained by multiplying the sum of the internal gradients in the component direction by the sum of the global gradients. as well as The weighting factor is determined based on the gradient value in the component direction, wherein the larger the gradient value, the smaller the weighting factor.

4. The method of claim 3, wherein the gradient value in the component direction is converted into the weighting factor in the component direction by Equation II: , where Y n X is the weighting factor. n θ is the gradient value, n indicates the direction of the component, and θ is a positive number.

5. The method according to claim 4, wherein θ is selected from 0.7 to 3.

0.

6. The method according to claim 1, wherein each direction includes a horizontal direction, a vertical direction, a first diagonal direction, and a second diagonal direction.

7. The method of claim 1, wherein the interpolation value of the second photodiode at the target first photodiode location is obtained based on the component interpolation values ​​in each direction.

8. The method of claim 7, wherein one of the component interpolation values ​​in the component directions of each direction is obtained from the readout values ​​of m second photodiodes, m being a multiple of 2, and the m second photodiodes are arranged in the component directions and adjacent to the position of the target first photodiode, and are equidistantly located on opposite sides of the position of the target first photodiode in the component directions.

9. The method of claim 8, wherein one of the component interpolation values ​​in the component direction is obtained by Equation III: V n It is the component interpolation value in the component direction, where n indicates the component direction. It is the readout value of the i-th of the m second photodiodes, and F i It is the component weight value.

10. The method of claim 9, wherein for one of the m second photodiodes located closer to the target first photodiode, F i It is larger than the other of the m second photodiodes located relatively far from the target first photodiode.

11. The method of claim 9, wherein for one of the m second photodiodes that is closer to a target first pixel group including the target first photodiode, F i It is larger than the other of the m second photodiodes that is relatively far from the target first pixel group.

12. The method of claim 7, wherein each interpolated value is obtained via equation IV: G int The interpolated value, Y n V is the weighting factor for the operating range in the component directions of each direction. n is the component interpolation value in the component direction, and n indicates the component direction.

Images