Image sensor, calibration method of image sensor, and electronic device

Abstract

Disclosed is a method for calibrating an image sensor, the image sensor including a color filter array having a first type of array pattern. The method includes capturing a target using the image sensor to generate first image data, performing a white balance operation on the first image data to generate second image data, performing an inverse white balance operation on the second image data based on a second type of array pattern to generate third image data, the second type of array pattern being different from the first type of array pattern, and calibrating the image sensor based on the third image data.

Description

[0001] Cross - reference to related applications

[0002] This application claims priority to Korean Patent Application No. 10 - 2020 - 0019240, filed on February 17, 2020, with the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical field

[0003] Some example embodiments described herein relate to semiconductor devices, and more particularly, to an image sensor capable of reducing calibration time, a calibration method of the image sensor, and / or an electronic device for calibrating the image sensor. Background art

[0004] An image sensor can generate image data of a target or a scene by capturing the target or the scene. With the improvement of the performance of mobile devices such as smart phones and smart tablets, image sensors are adopted in mobile devices. The image sensors adopted in mobile devices generate image data, and thus, the image sensors can be used to create image - based content.

[0005] After the image sensor is manufactured and before the image sensor is coupled to a mobile device, the image sensor can be calibrated. Calibrating the image sensor can include adjusting the resolution by adjusting the distance between the lens and the pixels of the image sensor, or adjusting the tilt of the lens by horizontally positioning the lens and the pixels of the image sensor. When the image sensor is fully calibrated, the image sensor can have performance that meets the design specifications of the image sensor.

[0006] As the number of electronic devices (or, mobile devices) adopting image sensors increases and as the number of image sensors adopted in each electronic device (or, mobile device) increases, the demand for image sensors may increase rapidly. Image sensor calibration delays may increase the manufacturing time of image sensors, thereby increasing the manufacturing cost of image sensors. Summary of the invention

[0007] Some example embodiments provide an image sensor with reduced calibration time, a calibration method of the image sensor, and / or an electronic device for calibrating the image sensor.

[0008] According to some example embodiments, a method of calibrating an image sensor, the image sensor including a color filter array having a first type of array pattern, the method comprising: using the image sensor to capture a target to generate first image data, performing a white balance operation on the first image data to generate second image data, performing an inverse white balance operation on the second image data based on a second type of array pattern to generate third image data, the second type of array pattern being different from the first type of array pattern, and calibrating the image sensor based on the third image data.

[0009] According to some example embodiments, an image sensor includes: a lens; a color filter array below the lens, the color filter array including a plurality of color filters having a first array pattern; a pixel array below the color filter array, the pixel array including a plurality of pixels respectively corresponding to the plurality of color filters, and the pixel array being configured to sense light incident through the lens and the color filter array; and a processing circuit configured to digitize the light sensed by the pixel array to generate first image data, perform a white balance operation on the first image data to generate second image data, and perform an inverse white balance operation on the second image data based on a second array pattern, the second array pattern being different from the first array pattern.

[0010] According to some example embodiments, an electronic device includes a processing circuit configured to: perform a white balance operation on first image data received from an image sensor to generate second image data, the image sensor including a color filter array having a first type of array pattern; perform an inverse white balance operation on the second image data based on a second type of array pattern to generate third image data, the second type of array pattern being different from the first type of array pattern; and generate a signal for calibrating the image sensor based on the third image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects and features of some example embodiments will become apparent by describing its examples in detail with reference to the drawings.

[0012] Figure 1 An image sensor calibration system according to some example embodiments is shown.

[0013] Figure 2 An example of color filters arranged depending on a first type of array pattern of a first color filter array is shown.

[0014] Figure 3 An image sensor calibration system according to some example embodiments is shown.

[0015] Figure 4 An example of color filters arranged depending on a second type of array pattern of a second color filter array is shown.

[0016] Figure 5 An example of an image sensor calibration method according to some example embodiments is shown.

[0017] Figure 6 The intensities of the G, R, and B hues of the first image data according to the intensity of the incident light are shown.

[0018] Figure 7 The intensities of the G, R, and B hues of the second image data according to the intensity of the incident light are shown.

[0019] Figure 8 An example of performing an inverse white balance operation on a second color filter array is shown.

[0020] Figure 9 An example of an electronic device configured to calibrate an image sensor is shown.

[0021] Figure 10 An image sensor calibration system according to some example embodiments is shown.

[0022] Figure 11 Shows Figure 10 An example of an operation method of an image sensor.

[0023] Figure 12 Components of an image sensor according to some example embodiments are shown.

[0024] Figure 13 An example of calibrating an image sensor by using a part of image data is shown. Detailed Description

[0025] Hereinafter, some example embodiments may be described in detail and clearly to such an extent that an ordinary person skilled in the art can easily implement them.

[0026] Figure 1 An image sensor calibration system 100 according to some example embodiments is shown. Refer to Figure 1 , the image sensor calibration system 100 may include an image sensor 110 and / or an electronic device 120 configured to calibrate the image sensor 110.

[0027] The image sensor 110 may be based on a complementary metal-oxide semiconductor (CMOS) image sensor (CIS) or a charge-coupled device (CCD). The image sensor 110 may include a lens 111, a support 112, a first color filter array 113 (CFA1), a pixel array 114 (PA), an analog-to-digital block 115 (A-D), and / or a first physical block 116 (PHY1). According to some example embodiments, operations described herein as being performed by the image sensor calibration system 100, the image sensor 110, the electronic device 120, the first color filter array 113 (CFA1), the pixel array 114 (PA), the analog-to-digital block 115 (A-D), and / or the first physical block 116 (PHY1) may be performed by a processing circuit. As used in this disclosure, the term "processing circuit" may refer, for example, to: hardware including logic circuits; a hardware / software combination such as a processor executing software; or a combination thereof. For example, the processing circuit may more specifically include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field-programmable gate array (FPGA), a system-on-chip (SoC), a programmable logic unit, a microprocessor, an application-specific integrated circuit (ASIC), etc.

[0028] The lens 111 may transmit light incident from the outside to the first color filter array 113. The support 112 may be a structure for supporting the lens 111. The support 112 may be configured to adjust the distance between the lens 111 and the first color filter array 113 or the distance between the lens 111 and the pixel array 114. The support 112 may be configured to adjust the inclination between the lens 111 and the first color filter array 113 and / or the inclination between the lens 111 and the pixel array 114.

[0029] The first color filter array 113 may include color filters applied to the light incident through the lens 111. The first color filter array 113 may have a first type of array pattern. The color filters of the first color filter array 113 may be arranged depending on the first type of array pattern.

[0030] Figure 2 An example of color filters arranged depending on the first type of array pattern of the first color filter array 113 is shown. Referring Figure 1 and Figure 2 , the first color filter array 113 may include color filters for three colors: red R, green G, and blue B. The color filters of the first color filter array 113 may be arranged in rows and columns in units of a first basic unit BU1 depending on the first type of array pattern.

[0031] The first basic unit BU1 may include four color filters. The first basic unit BU1 may include G, R, G, and B color filters arranged in sequence in a clockwise direction starting from the leftmost end. In some example embodiments, the first type array pattern of the first color filter array 113 may be a Bayer pattern.

[0032] The pixel array 114 may include pixels arranged in rows and columns. The pixels of the pixel array 114 may respectively correspond to the color filters of the first color filter array 113. According to some example embodiments, each pixel of the pixel array 114 may correspond to a different color filter of the first color filter array 113. The pixels corresponding to the G color filter may output information corresponding to the amount of green light in the form of current or voltage.

[0033] The pixels corresponding to the R color filter may output information corresponding to the amount of red light in the form of current or voltage. The pixels corresponding to the B color filter may output information corresponding to the amount of blue light in the form of current or voltage.

[0034] The analog-to-digital block 115 may receive the result of sensing the amount of light at the pixels of the pixel array 114 (e.g., the current and / or voltage values output by the pixels of the pixel array 114). The received result may be in analog form. The analog-to-digital block 115 may digitize the received result to generate image data. The image data may include pixel data corresponding to the pixels of the pixel array 114 (or may be composed of the pixel data), or may be generated from the pixels.

[0035] The first physical block 116 may receive the image data from the analog-to-digital block 115. The first physical block 116 may output the image data based on the corresponding communication protocol among various communication protocols. For example, the communication protocol may include D-PHY and / or C-PHY.

[0036] Since the first type array pattern of the first color filter array 113 is a Bayer pattern, the signal output from the first physical block 116 may be a Bayer signal BS corresponding to the Bayer pattern.

[0037] The electronic device 120 may include a processing circuit (e.g., a computer) configured (or organized) to calibrate the image sensor 110 or a calibration device designed or manufactured to calibrate the image sensor 110. According to some example embodiments, the calibration device may be implemented using the processing circuit. The electronic device 120 may receive the Bayer signal BS from the image sensor 110.

[0038] The electronic device 120 may include a second physical block 121 (PHY2) and / or an image signal processor 122 (ISP). The second physical block 121 may have the same structure or a similar structure as the first physical block 116 of the image sensor 110, and may be based on the same communication protocol or a similar communication protocol as the first physical block 116. According to some example embodiments, the operations described herein as being performed by the second physical block 121 (PHY2) and / or the image signal processor 122 (ISP) may be performed by a processing circuit.

[0039] The image signal processor 122 may be installed in the electronic device 120 and may be configured to execute a library encoded to calibrate the image sensor 110. Alternatively, the image signal processor 122 may include a processing circuit (e.g., an integrated circuit) designed and manufactured to calibrate the image sensor 110 and installed in the electronic device 120.

[0040] The image signal processor 122 may be implemented based on the Bayer signal BS (BS-based) and may process the Bayer signal BS based on a first type of array pattern (i.e., the Bayer pattern). The image signal processor 122 may detect the resolution of the image sensor 110 from the Bayer signal BS and may transmit a feedback signal FBS for calibrating the resolution to the support 112 of the image sensor 110.

[0041] For example, the resolution may be determined by the distance between the lens 111 and the pixel array 114. The feedback signal FBS may include calibration information related to the distance between the lens 111 and the pixel array 114. The support 112 may adjust the distance between the lens 111 and the pixel array 114 in response to the feedback signal FBS.

[0042] The image signal processor 122 may detect the tilt of the lens 111 of the image sensor 110 based on the Bayer signal BS and may transmit a feedback signal FBS for calibrating the tilt to the support 112 of the image sensor 110.

[0043] For example, in a case where the lens 111 and the first color filter array 113 or the pixel array 114 are not parallel to each other, phenomena such as diffusion may occur in a partial region of the image data. The feedback signal FBS may include calibration information related to the tilt of the lens 111. The support 112 may adjust the tilt of the lens 111 in response to the feedback signal FBS.

[0044] In Figure 1An example is shown in which a feedback signal FBS is transmitted to the support member 112, and the resolution and tilt are adjusted by the support member 112. The image sensor calibration system 100 may further include an adjustment device for adjusting the resolution and / or tilt of the image sensor 110. The feedback signal FBS may be transmitted to the adjustment device, and the adjustment device may control the support member 112 to adjust the resolution and / or tilt of the image sensor 110. In some example embodiments, the adjustment device may be included in the electronic device 120. According to some example embodiments, the adjustment device may be implemented using a processing circuit.

[0045] Figure 3 An image sensor calibration system 200 according to some example embodiments is shown. Referring Figure 3 to, the image sensor calibration system 200 may include an image sensor 210 and / or an electronic device 220 configured to calibrate the image sensor 210.

[0046] The image sensor 210 may be based on a complementary metal-oxide-semiconductor image sensor (CIS) or a charge-coupled device (CCD). The image sensor 210 may include a lens 211, a support member 212, a second color filter array 213 (CFA2), a pixel array 214 (PA), an analog-to-digital block 215 (A-D), and / or a first physical block 216 (PHY1). According to some example embodiments, the operations described herein as being performed by the image sensor calibration system 200, the image sensor 210, the electronic device 220, the second color filter array 213 (CFA2), the pixel array 214 (PA), the analog-to-digital block 215 (A-D), and / or the first physical block 216 (PHY1) may be performed by a processing circuit.

[0047] Except that the image sensor 210 includes a second color filter array 213 instead of the first color filter array 113, the operation and / or configuration of the image sensor 210 may be the same as or similar to Figure 1 that of the image sensor 110. According to some example embodiments, the lens 211, the support member 212, the pixel array 214 (PA), the analog-to-digital block 215 (A-D), and / or the first physical block 216 (PHY1) may be the same as and similar to the lens 111, the support member 112, the pixel array 114 (PA), the analog-to-digital block 115 (A-D), and / or the first physical block 116 (PHY1), respectively. Therefore, additional descriptions will be omitted to avoid redundancy. Regarding Figure 1 the configuration, operation, and / or features described for the image sensor 110 may be applied to the image sensor 210 in the same or similar manner.

[0048] The second color filter array 213 may include color filters applied to light incident through the lens 211. The second color filter array 213 may have a second type of array pattern. The color filters of the second color filter array 213 may be arranged depending on the second type of array pattern. The second type of array pattern may be different from the Bayer pattern of the first type of array pattern.

[0049] Figure 4 An example of color filters arranged depending on the second type of array pattern of the second color filter array 213 is shown. Refer to Figure 3 and Figure 4 , the second color filter array 213 may include color filters for three colors: red R, green G, and blue B. The color filters of the second color filter array 213 may be arranged in rows and columns in units of the second basic unit BU2 depending on the second type of array pattern.

[0050] The second basic unit BU2 may include 16 color filters. In the case where the second basic unit BU2 is equally divided into four regions based on the horizontal axis and the vertical axis, the second basic unit BU2 may include four G color filters in the upper left quadrant, four R color filters in the upper right quadrant, four B color filters in the lower left quadrant, and four G color filters in the lower right quadrant.

[0051] In some example embodiments, the second basic unit BU2 may include three or more color filters that are adjacent to each other and correspond to the same color or similar colors. The second type of array pattern of the second color filter array 213 may not be the Bayer pattern.

[0052] Since the second type of array pattern of the second color filter array 213 may not be the Bayer pattern, the signal output from the first physical block 216 may be a non - Bayer signal NBS that does not correspond to the Bayer pattern.

[0053] The electronic device 220 may include a computer configured (or, organized) to calibrate the image sensor 210 or a calibration device designed or manufactured to calibrate the image sensor 210. The electronic device 220 may receive the non - Bayer signal NBS from the image sensor 210.

[0054] The electronic device 220 may include a second physical block 221 (PHY2) and / or an image signal processor 222 (ISP). The second physical block 221 may have the same structure or a similar structure as the first physical block 216 of the image sensor 210, and may be based on the same communication protocol or a similar communication protocol as the first physical block 216.

[0055] The image signal processor 222 can be installed in the electronic device 220 and can be configured to execute a library encoded for calibrating the image sensor 210. Alternatively, the image signal processor 222 can include processing circuitry (e.g., an integrated circuit) designed and manufactured to calibrate the image sensor 210 and installed in the electronic device 220.

[0056] The image signal processor 222 can be implemented based on the non-Bayer signal NBS. For the image signal processor 222 to process the non-Bayer signal NBS, the image signal processor 222 can convert the non-Bayer signal NBS into a Bayer signal BS.

[0057] Conventional devices and methods for performing image sensor calibration (including various operations for maintaining image data quality) convert the non-Bayer signal generated by the image signal processor into a Bayer signal. However, this conversion of the non-Bayer signal to the Bayer signal results in excessive latency (e.g., latency greater than a reference time). This excessive conversion latency leads to excessive sensor calibration latency. When calibrating a large number of image sensors, such as during initial calibration performed by an image sensor manufacturer, the negative impact of this calibration latency is amplified, further resulting in excessive manufacturing costs and latency.

[0058] However, the image signal processor 222 according to some example embodiments can provide white balance operations and inverse white balance operations, converting non-Bayer image data into Bayer image data faster than conventional devices and methods convert non-Bayer signals into Bayer signals. Thus, compared to conventional devices and methods, the image signal processor 222 provides reduced conversion latency (e.g., latency less than a reference time), and thus, provides reduced calibration latency as well as reduced manufacturing costs and latency.

[0059] In addition to the fast converter 223 being included in the image signal processor 222, the configuration and / or operation of the image signal processor 222 of the electronic device 220 can be the same as or similar to Figure 1 the configuration and / or operation of the image signal processor 122. Thus, additional descriptions will be omitted to avoid redundancy. The configurations, operations, and / or features described with respect to Figure 1 the electronic device 120 can be applied to the electronic device 220 in the same or similar manner. According to some example embodiments, the operations described herein as being performed by the image signal processor 222, the fast converter 223 (FC), and / or the second physical block 221 (PHY2) can be performed by processing circuitry.

[0060] Figure 5 An example of an image sensor calibration method according to some example embodiments is shown. Referring to Figure 3 and Figure 5, in operation S110, an operation of the image sensor 110 capturing a non - color target to generate first image data may be performed.

[0061] Color can be represented by hue, chroma, and / or value. The chroma of the non - color target can be "0". The G, R, and B hue values of the non - color target can be the same or similar. The non - color target can have various values.

[0062] The non - color target can include various patterns with various values, enabling information for calibrating the resolution and tilt of the image sensor 210 to be obtained. For example, the ISO12233 resolution chart can be used as the non - color target.

[0063] Operation S120 can be performed after operation S110. An operation of performing a white - balance operation on the first image data to generate second image data can be performed in operation S120. The white - balance operation can adjust the gains of the G, R, and B hues of the light reflected from the target captured by the image sensor 210.

[0064] In some example embodiments, the first image data can include information on the light reflected from the non - color target. The light transmittance (or, sensitivity) of the G filter, R filter, and / or B filter of the second color filter array 213 can be different from each other. Therefore, the intensities of the G, R, and / or B hues of the first image data of the non - color target captured by the pixel array 214 of the image sensor 210 can be different from each other.

[0065] Figure 6 The intensities of the G, R, and B hues of the first image data according to the intensity of the incident light are shown. In Figure 6 , the horizontal axis represents the intensity of the incident light (e.g., received intensity), and the vertical axis represents the captured intensity.

[0066] Referring to Figure 3 and Figure 6 , the first line L1 shows the intensity captured by the pixels corresponding to the G filter. The second line L2 shows the intensity captured by the pixels corresponding to the R filter. The third line L3 shows the intensity captured by the pixels corresponding to the B filter.

[0067] As Figure 6 shown, when light of the same intensity or similar intensity reflected from a non - color target that does not include chroma is incident, the intensity captured through the G filter can be greater than the intensity captured through the R filter. Similarly, the intensity captured through the R filter can be greater than the intensity captured through the B filter.

[0068] That is to say, the first image data generated according to the achromatic target can be color image data with chromaticity. The white balance operation can adjust the gain of the intensities of the G, R, and B hues of the first image data to generate second image data (e.g., achromatic image data). According to some example embodiments, the intensity gain corresponding to each pixel of the first image data can be adjusted according to whether each pixel among the respective pixels is associated with the G, R, or B hue (associated with the corresponding G, R, or B color filter).

[0069] Figure 7 Shows the intensities of the G, R, and B hues of the second image data according to the intensity of the incident light. In Figure 7 the horizontal axis represents the intensity of the incident light (e.g., received intensity), and the vertical axis represents the captured intensity.

[0070] Refer to Figure 7 , the fourth line L4 shows the intensities corresponding to the G, R, and B color filters according to the intensity of the incident light. In the second image data, the intensities of the G, R, and B hues of the light with the same intensity or similar intensity reflected from the achromatic target can be the same or similar.

[0071] The second image data can include information of an image captured from the achromatic target, and in the information of the second image data, the intensities of the G, R, and B hues can be calibrated to the original G, R, and B hues of the achromatic target.

[0072] That is to say, the white balance operation can be an operation of adjusting the gain of the intensities of the G, R, and B hues of the first image data to be adjusted to an achromatic color. The second image data generated by the white balance operation can include achromatic information.

[0073] In some example embodiments, Figure 6 the difference between the captured intensities shown in can be based on the characteristics of the second color filter array 213. The white balance operation can adjust the gains of the G, R, and B hues based on the characteristics of the second color filter array 213.

[0074] Since the white balance operation is performed on the first image data generated through the second color filter array 213, the white balance operation can be based on the second type of array pattern of the second color filter array 213.

[0075] Even if the intensities of the G, R, and B hues are calibrated during the process of converting the first image data into the second image data, the shape and different values of the achromatic pattern captured from the achromatic target can be maintained identically or similarly. The shape and different values of the achromatic pattern can be used to calibrate the image sensor 210.

[0076] Return Figure 3 and Figure 5, operation S130 may be performed after operation S120. In operation S130, an inverse white balance operation may be performed on the second image data, and third image data may be generated. The inverse white balance operation may be based on the first type of array pattern of the first color filter array 113, i.e., the Bayer pattern.

[0077] As described in Figure 2 and Figure 4 , the positions of the color filters according to the first type of array pattern of the first color filter array 113 are different from the positions of the color filters according to the second type of array pattern of the second color filter array 213.

[0078] As described in Figure 5 , to calibrate the image sensor 210, first image data may be generated based on the result of capturing a non-color target. Similarly, second image data may be generated by performing a white balance operation on the first image data.

[0079] The intensities of the G, R, and B chromaticities of the non-color target may be the same or similar, the chromaticity may be "0", and the values may be different. Similarly, the intensities of the G, R, and B hues of the second image data may be the same or similar, the chromaticity may be "0", and the values may be different. That is, the second image data may include all information related to the color of the non-color target.

[0080] Third image data corresponding to the first type of array pattern may be generated by adjusting the gains of the G, R, and B hues of the second image data (e.g., non-color image data), which is opposite to the white balance operation.

[0081] Figure 8 shows an example of performing an inverse white balance operation on the second color filter array 213. Refer to Figure 2 , Figure 4 and Figure 8 , the second color filter array 213 may be divided into sub-units SU1 to SU4 having the same or similar size as the first basic unit BU1. Such an example is shown in Figure 8 , where the size (or granularity) of the second basic unit BU2 is an integer multiple (e.g., four times) of the size (or granularity) of the sub-unit, but some example embodiments are not limited thereto.

[0082] In each sub-unit, the color filters may be converted to correspond to the first type of array pattern. For example, in the first sub-unit SU1, one of the G color filters may be converted to an R color filter (G→R), and the other may be converted to a B color filter (G→B). In the second sub-unit SU2, two of the R color filters may be converted to G color filters (R→G), and the other may be converted to a B color filter (R→B).

[0083] In the third sub-unit SU3, two of the B color filters can be converted to G color filters (B→G), and the other can be converted to an R color filter (B→R). In the fourth sub-unit SU4, one of the G color filters can be converted to an R color filter (G→R), and the other can be converted to a B color filter (G→B). According to some example embodiments, converting the hue of a corresponding color filter to a different hue may include applying a defined value to second image data corresponding to pixels associated with the corresponding color filter (e.g., adding, subtracting, multiplying, etc.) to obtain a converted array pattern.

[0084] When converting the color filters as shown in Figure 8 , the converted color filter array may have the same array pattern or a similar array pattern as the first type of array pattern. The inverse white balance operation may adjust the gain of the G, R, and / or B hues depending on the converted array pattern. According to some example embodiments, the gain applied to each pixel in the first image data used to generate the second image data during the white balance operation is stored, and the gain opposite to the stored gain is applied to each pixel in the second image data corresponding to the converted array pattern to generate third image data during the inverse white balance operation.

[0085] The inverse white balance operation may adjust the intensity of the G, R, and / or B hues of the second image data as shown in Figure 7 to the intensity of the G, R, and / or B hues as shown in Figure 6 . The third image data generated by the inverse white balance operation may be the same as or similar to the image data captured from a non-color target using the first color filter array 113 having the first type of array pattern.

[0086] In some example embodiments, Figure 6 the difference between the capture intensities as shown in

[0087] may be based on the characteristics of the second color filter array 213. The inverse white balance operation may adjust the gain of the G, R, and / or B hues based on the characteristics of the second color filter array 213. Figure 3 and Figure 5 , operation S140 may be performed after operation S130. In operation S140, the image sensor 210 may be calibrated by using the third image data.

[0088] As described above, the calibration method according to some example embodiments includes: generating first image data according to a non-color target; performing a white balance operation on the first image data to generate second image data; and performing an inverse white balance operation on the second image data to generate third image data corresponding to the Bayer pattern.

[0089] The third image data may be a Bayer signal BS based on a Bayer pattern. The white balance operation and the inverse white balance operation may be performed and completed within a time less than a reference time. For example, the above operations of the calibration method may be performed in real time when calibrating the image sensor 210. Accordingly, the calibration time of the image sensor 210 may be reduced.

[0090] In some example embodiments, when calibrating an image sensor, at least a part of a target captured by the image sensor may be non-color, and the remaining part may include color. In this case, the calibration method may be performed by using a part of the non-color part of the target. Figure 5 of the calibration method.

[0091] In some example embodiments, generating the first image data by using the image sensor 210 may be performed in a calibration environment with a fixed color temperature. The white balance operation and the inverse white balance operation may be performed based on information of the fixed color temperature.

[0092] In some example embodiments, information on the color temperature in a calibration environment may be obtained before generating the first image data by using the image sensor 210. The white balance operation and the inverse white balance operation may be performed based on information of the fixed color temperature.

[0093] Figure 9 An example of an electronic device 300 configured to calibrate an image sensor is shown. The electronic device 300 may correspond to Figure 3 the electronic device 220. Referring to Figure 3 and Figure 9 , the electronic device 300 may include a bus 301, a processing circuit (e.g., a processor) 310, a main memory 320, a storage device 330, a second physical block 340, a display device 350, and / or a feedback signal generator 360. According to some example embodiments, operations described herein as being performed by the electronic device 300, the second physical block 340, and / or the feedback signal generator 360 may be performed by the processing circuit (e.g., the processing circuit 310).

[0094] The bus 301 may provide a channel (e.g., a communication channel and / or a power channel) between components of the electronic device 300. The processing circuit 310 may control the operation of the electronic device 300 and may execute an operating system and applications. The processing circuit 310 may be a general-purpose processor or a dedicated processor designed and manufactured to calibrate an image sensor.

[0095] The main memory 320 may be accessed by the processing circuit 310. The main memory 320 may temporarily store code and / or data that the processing circuit 310 may execute. The main memory 320 may include DRAM, SRAM, and / or storage class RAM (SCRAM).

[0096] The storage device 330 can be a non-volatile memory that can store code and / or data that can be used in the electronic device 300. The storage device 330 can include various storage devices, such as a hard disk drive (HDD), a solid state drive (SSD), and / or an optical disc drive (ODD).

[0097] The storage device 330 can include image signal processing code 351 (ISP code). The image signal processing code 351 can include various code and / or data for calibrating the image sensor 210 by using the BS-based image data.

[0098] The image signal processing code 351 and / or the storage device 330 can include a fast converter 361 (FC). As described with reference to Figure 5 the fast converter 361 can include code and / or data for performing a white balance operation and / or an inverse white balance operation on the non-Bayer signal to convert the non-Bayer signal NBS into the Bayer signal BS. According to some example embodiments, the operations described herein as being performed by the fast converter 361 (FC) can be performed by the processing circuit.

[0099] The processing circuit 310 can load the image signal processing code 351 including the fast converter 361 or a part of the image signal processing code 351 onto the main memory 320. The processing circuit 310 can access the specific code or data loaded onto the main memory 320 from the image signal processing code 351 (depicted as image signal processing code 352 when loaded onto the main memory 320) including the fast converter 361 (depicted as fast converter 362 when loaded onto the main memory 320), or a part of the image signal processing code 351.

[0100] The processing circuit 310 can receive first image data from the image sensor 210 through the second physical block 340 (PHY2).

[0101] The processing circuit 310 can execute the accessed image signal processing code 351 including the fast converter 361 (depicted as fast converter 363 when loaded on the processing circuit 310) (depicted as image signal processing code 353 when executed on the processing circuit 310), and can perform operations for calibrating the image sensor 210 based on the first image data. For example, the processing circuit 310 can execute the fast converter 361 to generate third image data (e.g., the Bayer signal BS). Similarly, the processing circuit 310 can execute the image signal processing code 351, and can calculate information for adjusting the resolution and / or tilt of the image sensor 210 according to the third image data.

[0102] The processing circuit 310 may display the calculation result through the display device 350. Similarly, based on the calculation result, the processing circuit 310 may allow the feedback signal generator 360 to generate a feedback signal FBS including information for adjusting the resolution and / or tilt of the image sensor 210.

[0103] Figure 10 An image sensor calibration system 400 according to some example embodiments is shown. Referring Figure 10 , the image sensor calibration system 400 may include an image sensor 410 and / or an electronic device 420 configured to calibrate the image sensor 410.

[0104] The image sensor 410 may be based on a complementary metal-oxide-semiconductor (CMOS) image sensor (CIS) or a charge-coupled device (CCD). The image sensor 410 may include a lens 411, a support 412, a second color filter array 413 (CFA2), a pixel array 414 (PA), an analog-to-digital block 415 (A-D), and / or a first physical block 416 (PHY1). According to some example embodiments, operations described herein as being performed by the image sensor calibration system 400, the image sensor 410, the electronic device 420, the second color filter array 413 (CFA2), the pixel array 414 (PA), the analog-to-digital block 415 (A-D), and / or the first physical block 416 (PHY1) may be performed by a processing circuit.

[0105] Except that a fast converter 417 is included in the analog-to-digital block 415, the configuration and / or operation of the image sensor 410 may be the same as or similar to Figure 3 that of the image sensor 210. According to some example embodiments, the lens 411, the support 412, the second color filter array 413 (CFA2), the pixel array 414 (PA), and / or the first physical block 416 (PHY1) may be the same as or similar to the lens 211, the support 212, the second color filter array 213 (CFA2), the pixel array 214 (PA), the analog-to-digital block 215 (A-D), and / or the first physical block 216 (PHY1), respectively. Thus, additional descriptions will be omitted to avoid redundancy. The configurations, operations, and / or features described with respect to Figure 3 the image sensor 210 may be applied to the image sensor 410 in the same or similar manner.

[0106] The analog-to-digital block 415 may include a fast converter 417. In a first mode, the analog-to-digital block 415 may activate the fast converter 417. In a second mode, the analog-to-digital block 415 may deactivate the fast converter 417. The so-activated fast converter 417 may perform reference Figure 5The described white balance operation and / or inverse white balance operation. That is, when the fast converter 417 is activated, the image sensor 410 can output the Bayer signal BS. When the fast converter 417 is deactivated, the image sensor 410 can output the non-Bayer signal NBS. According to some example embodiments, the operations described herein as being performed by the fast converter 417 (FC) can be performed by a processing circuit.

[0107] The electronic device 420 can include a processing circuit (e.g., a computer) configured (or, organized) to calibrate the image sensor 410 or a calibration device designed or manufactured for calibrating the image sensor 410. The electronic device 420 can receive the Bayer signal BS from the image sensor 410. According to some example embodiments, the operations described herein as being performed by the calibration device can be performed by a processing circuit.

[0108] The electronic device 420 can include a second physical block 421 (PHY2) and / or an image signal processor 422 (ISP). The second physical block 421 can have the same structure or a similar structure as the first physical block 416 of the image sensor 410 and can be based on the same communication protocol or a similar communication protocol as the first physical block 416. According to some example embodiments, the operations described herein as being performed by the second physical block 421 (PHY2) and / or the image signal processor 422 (ISP) can be performed by a processing circuit.

[0109] The image signal processor 422 can be installed in the electronic device 420 and can be configured to execute a library encoded to calibrate the image sensor 410. Alternatively, the image signal processor 422 can include a processing circuit (e.g., an integrated circuit) designed and manufactured to calibrate the image sensor 410 and installed in the electronic device 420. The image signal processor 422 can be implemented based on the Bayer signal BS.

[0110] Figure 11 An example of the operation method of the Figure 10 image sensor 410 is shown. Referring to Figure 10 and Figure 11 , the image sensor 410 can operate in one of a first mode and a second mode. The first mode can be a calibration mode for calibrating the image sensor 410. The second mode can be a normal operation mode in which the image sensor 410 is installed in a mobile device to create image content.

[0111] The first mode can include operation S210 to operation S240. In the first mode, the fast converter 417 can be activated. Operations S210 to S230 of the first mode can be performed as being related to Figure 5Operations S110 to S130 are the same or similar. The image sensor 410 can generate first image data based on a non-color target, perform a white balance operation on the first image data to generate second image data, and perform an inverse white balance operation on the second image data to generate third image data.

[0112] In operation S240 of the first mode, the image sensor 410 can output the third image data as a Bayer signal BS. The third image data can be output to a calibration device (e.g., the electronic device 420) configured to calibrate the image sensor 410.

[0113] The second mode can include operation S250 and operation S260. In the second mode, the fast converter 417 can be deactivated. In operation S250 of the second mode, the image sensor 410 can capture a target and / or a scene to generate fourth image data. In operation S260, the image sensor 410 can output the fourth image data as a non-Bayer signal NBS.

[0114] The fourth image data can be output to a processing circuit (e.g., a processor) of a device (e.g., a mobile device) in which the image sensor 410 is installed after the image sensor 410 is fully calibrated.

[0115] Figure 12 Components of an image sensor 500 according to some example embodiments are shown. The image sensor 500 can correspond to Figure 10 the image sensor 410. Referring to Figure 10 and Figure 12 , the image sensor 500 can include a pixel array 510, a timing generator 520, a row scan circuit 530, a first sample-and-hold circuit 54-1 to an nth sample-and-hold circuit 54-n (SH), a ramp circuit 550, a first analog-to-digital converter 56-1 to an nth analog-to-digital converter 56-n (ADC), a latch circuit 570, and / or a digital processor 580. According to some example embodiments, what is described herein as being performed by the image sensor 500, the pixel array 510, the timing generator 520, the row scan circuit 530, the first sample-and-hold circuit 54-1 to the nth sample-and-hold circuit 54-n (SH), the ramp circuit 550, the first analog-to-digital converter 56-1 to the nth analog-to-digital converter 56-n (ADC), the latch circuit 570, and / or the digital processor 580 can be performed by a processing circuit.

[0116] The pixel array 510 can include pixels (e.g., image sensor pixels) arranged in rows and columns. The pixel array 510 can correspond to Figure 10The pixel array 414. The pixel array 510 may generate image data in a scanning manner to sequentially select rows of pixels. The pixels in the selected row may convert and output the incident light amount in the form of current and / or voltage. In some example embodiments, the second color filter array 413 and the lens 411 on the pixel array 510 may be omitted in Figure 12 In.

[0117] The timing generator 520 may provide a first signal S1 to the row scanning circuit 530, may provide a second signal S2 to the first sample and hold circuit 54-1 to the nth sample and hold circuit 54-n, may provide a third signal S3 to the ramp circuit 550, and / or may provide a fourth signal S4 to the latch circuit 570. The timing generator 520 may control the first signal S1, the second signal S2, the third signal S3, and / or the fourth signal S4 such that the row scanning circuit 530, the first sample and hold circuit 54-1 to the nth sample and hold circuit 54-n, the ramp circuit 550, and / or the latch circuit 570 operate at appropriate timings.

[0118] The row scanning circuit 530 is connected to the rows of pixels in the pixel array 510 through the first word line WL1 to the mth word line WLm. The row scanning circuit 530 may sequentially select the first word line WL1 to the mth word line WLm in response to the first signal S1. The row scanning circuit 530 may apply an effective voltage (e.g., a positive voltage) to the selected word line and may apply an ineffective voltage (e.g., a ground voltage) to the unselected word line.

[0119] The first sample and hold circuit 54-1 to the nth sample and hold circuit 54-n may be connected to the pixel columns of the pixel array 510 through the first bit line BL1 to the nth bit line BLn. The first sample and hold circuit 54-1 to the nth sample and hold circuit 54-n may sense the voltage and / or current of the first bit line BL1 to the nth bit line BLn in response to the second signal S2 and may hold the sensed voltage and / or current.

[0120] The ramp circuit 550 may generate a ramp signal RS having a gradually increasing (or decreasing) level in response to the third signal S3. The ramp signal RS generated by the ramp circuit 550 may be provided to the first analog-to-digital converter 56-1 to the nth analog-to-digital converter 56-n.

[0121] The first analog-to-digital converter 56-1 to the nth analog-to-digital converter 56-n can receive a ramp signal RS from the ramp circuit 550 and can receive voltages and / or currents held by the first sample-and-hold circuit 54-1 to the nth sample-and-hold circuit 54-n. The first analog-to-digital converter 56-1 to the nth analog-to-digital converter 56-n can compare the received voltage (and / or current) with the ramp signal RS and can generate a digital value based on the comparison result.

[0122] The latch circuit 570 can sequentially store the digital values received from the first analog-to-digital converter 56-1 to the nth analog-to-digital converter 56-n in response to the fourth signal S4. The latch circuit 570 can transmit the stored digital values to the digital processor 580.

[0123] The digital processor 580 can perform subsequent processing on the digital values. For example, the digital processor 580 can include a fast converter 590. The fast converter 590 can perform a white balance operation and / or an inverse white balance operation. The digital processor 580 can be configured to activate or deactivate the fast converter 590 depending on the operation mode of the image sensor 500. According to some example embodiments, the operations described herein as being performed by the fast converter 590 can be performed by a processing circuit.

[0124] The digital processor 580 can transmit the image data as a Bayer signal BS or as non-Bayer signal NBS to the first physical block 416. The timing generator 520, the line scan circuit 530, the first sample-and-hold circuit 54-1 to the nth sample-and-hold circuit 54-n, the ramp circuit 550, the first analog-to-digital converter 56-1 to the nth analog-to-digital converter 56-n, the latch circuit 570, and / or the digital processor 580 can be included in the analog-to-digital block 415.

[0125] Figure 13 An example of calibrating an image sensor by using a part of the image data IMG is shown. Refer to Figure 3 、 Figure 10 and Figure 13 , the image data IMG can be, for example, analog image data captured by the pixel array 214 or 414. The analog-to-digital block 215 or 415 can perform analog-to-digital conversion on a part of the analog image data.

[0126] The electronic device 220 can perform a calibration operation based on the data after analog-to-digital conversion. Alternatively, Figure 5 the operation of the first mode of Figure 11 can be performed by the analog-to-digital block 415, and the calibration can be performed by the electronic device 420.

[0127] For another example, the image data IMG can be the first image data generated by the analog-to-digital block 215. The analog-to-digital block 215 can output all or a part of the first image data to the electronic device 220. The electronic device 220 can perform Figure 5 a calibration operation based on a part of the first image data. Alternatively, the analog-to-digital block 415 can perform Figure 11 an operation of the first mode based on a part of the first image data.

[0128] In some example embodiments, different first and second calibration operations can be performed based on a first part of the image data IMG and a second part of the image data.

[0129] In some example embodiments, the above components are described using terms such as "first", "second", "third", etc. However, the terms "first", "second", "third", etc. can be used to distinguish components from each other and do not limit some example embodiments. For example, the terms "first", "second", "third", etc. do not involve any form of order or numerical meaning.

[0130] In some example embodiments, the above components are described using blocks. These blocks can be implemented using various hardware devices, such as integrated circuits, application-specific ICs (ASICs), field-programmable gate arrays (FPGAs), and complex programmable logic devices (CPLDs), firmware driven in the hardware device, software such as an application, or a combination of hardware device and software. Similarly, these blocks can include circuits implemented using semiconductor elements in an integrated circuit or circuits registered as intellectual property (IP).

[0131] The various operations of the above method can be performed by any suitable device capable of performing the operations, such as the above processing circuit. For example, as described above, the operations of the above method can be performed by various hardware and / or software implemented in some form of hardware (e.g., a processor, an ASIC, etc.).

[0132] Software can include an ordered list of executable instructions for implementing logical functions and can be embodied in any "processor-readable medium" for use by or in connection with an instruction execution system, apparatus, or device (e.g., a single-core or multi-core processor or a system including a processor).

[0133] The methods or algorithms and the functions of the blocks or operations described in connection with some example embodiments disclosed herein may be implemented directly in hardware, in software modules executed by a processor, or in a combination of both. If implemented in software, these functions may be stored on or transmitted through a tangible non-transitory computer-readable medium as one or more instructions or code. The software modules may reside in random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0134] The acts and symbolic representations of operations (e.g., in the form of a flowchart, a flow diagram, a data flow diagram, a structure diagram, a block diagram, etc.) that can be implemented with reference to the units and / or devices that will be discussed in more detail below can be used to describe some example embodiments. Although discussed in a specific manner, the functions or operations specified in a particular block can be performed in a manner different from the flow specified in the flowchart, the flow diagram, etc. For example, functions or operations that are considered to be serially executed in two consecutive blocks can actually be executed in parallel, simultaneously, or in some cases, in the reverse order.

[0135] Spatial relative terms such as "beneath", "below", "lower", "above", "upper", etc. may be used herein for ease of description to describe the relationship of one element or feature to another element or feature as shown in the figures. For example, as used herein, the terms "above", "higher than", "on", and / or "at the top of" may refer to an element or feature being further in a first direction relative to another element or feature, while the terms "lower", "below", and / or "beneath" may refer to an element or feature being further in a direction opposite to the first direction relative to another element or feature. It will be understood that, in addition to the orientations depicted in the figures, the spatial relative terms are also intended to cover different orientations of the device during use or operation. For example, if the device in the figures is flipped, an element described as "below" or "beneath" other elements or features will be oriented "above" the other elements or features. Thus, the term "below" can cover both an upper and a lower orientation. The device may be otherwise oriented (rotated 90 degrees or in other orientations), and the spatial relative descriptors used herein can be interpreted accordingly.

[0136] According to some example embodiments, image data obtained by an image sensor having a first type of color filter array is converted into image data corresponding to a second type of color filter array based on white balance and inverse white balance. The image sensor can be calibrated based on the converted image data. Since the operation of converting the image data corresponding to the first type of color filter array into the image data corresponding to the second type of color filter array is performed based on white balance and inverse white balance, the time taken to calibrate the image sensor is reduced.

[0137] Although some example embodiments have been described with reference to their examples, it will be apparent to those of ordinary skill in the art that various changes and modifications can be made thereto without departing from the spirit and scope of some example embodiments as set forth in the appended claims.

Claims

1. A method for calibrating an image sensor, the image sensor including a color filter array having a first type of array pattern, the method comprising: Using the image sensor to capture a target to generate first image data; Performing a white balance operation on the first image data to generate second image data; Performing an inverse white balance operation on the second image data based on a second type of array pattern to generate third image data, the second type of array pattern being a Bayer pattern and the first type of array pattern being a non - Bayer pattern; And Calibrating the image sensor based on the third image data, wherein each of the first image data, the second image data, and the third image data includes pixel data corresponding to a plurality of pixels of the image sensor, and Performing the inverse white balance operation includes: converting pixel data corresponding to at least one of three or more pixels among the plurality of pixels to a color different from the original color.

2. The method according to claim 1, wherein, The first type of array pattern includes three or more color filters adjacent to each other, and each of the three or more color filters corresponds to the same specific color.

3. The method according to claim 2, wherein, The three or more pixels correspond to the three or more color filters.

4. The method according to claim 1, wherein The target includes non - color colors.

5. The method according to claim 4, wherein Calibrating the image sensor includes: calibrating the image sensor based on a part of the third image data corresponding to the non - color colors.

6. The method according to claim 1, wherein The image sensor further includes: A plurality of pixels below the color filter array, and Lenses on the color filter array; and Calibrating the image sensor includes: adjusting at least one of the tilt of the lens or the resolution of the image sensor based on the third image data.

7. The method according to claim 1, wherein The capturing, the performing of the white balance operation, and the performing of the inverse white balance operation are performed by the image sensor.

8. The method according to claim 1, wherein, The capturing is performed by the image sensor, and the performing of the white balance operation and the performing of the inverse white balance operation are performed by a processing circuit for calibrating the image sensor.

9. The method according to claim 1, wherein The capturing includes: Digitizing a part of the information captured by the image sensor to generate the first image data.

10. The method according to claim 1, wherein The capturing includes: Digitizing a part of the information captured by the image sensor to generate fourth image data including the first image data.

11. An image sensor, comprising: Lenses; A color filter array below the lenses, the color filter array including a plurality of color filters having a first array pattern; A pixel array below the color filter array, the pixel array including a plurality of pixels respectively corresponding to the plurality of color filters, and the pixel array being configured to sense light incident through the lenses and the color filter array; And A processing circuit configured to: Digitize the light sensed by the pixel array to generate first image data, Perform a white balance operation on the first image data to generate second image data, and Performing an inverse white balance operation on the second image data based on a second array pattern, where the second array pattern is a Bayer pattern and the first array pattern is a non-Bayer pattern, wherein each of the first image data, the second image data, and the third image data includes pixel data corresponding to the plurality of pixels, and the processing circuit is configured to perform the inverse white balance operation, and the inverse white balance operation includes converting the pixel data corresponding to at least one of three or more pixels among the plurality of pixels into a color different from the original color.

12. The image sensor according to claim 11, wherein, the processing circuit is configured to perform the white balance operation and the inverse white balance operation in a first mode; and the processing circuit is configured to output the first image data in a second mode.

13. The image sensor according to claim 12, wherein, The image sensor is configured to: output the third image data to an external device configured to calibrate the image sensor; and output the first image data to a device in which the image sensor is installed.

14. The image sensor according to claim 11, wherein, The processing circuit is configured to digitize a part of the light sensed by the pixel array to generate the first image data.

15. The image sensor according to claim 11, wherein, The processing circuit is configured to digitize a part of the light sensed by the pixel array to generate fourth image data, and the first image data is a part of the fourth image data.

16. An electronic device, comprising: a processing circuit configured to: perform a white balance operation on first image data received from an image sensor to generate second image data, the image sensor including a color filter array of a first type of array pattern, perform an inverse white balance operation on the second image data based on a second type of array pattern to generate third image data, the second type of array pattern is a Bayer pattern, and the first type of array pattern is a non-Bayer pattern, and generate a signal for calibrating the image sensor based on the third image data, wherein each of the first image data, the second image data, and the third image data includes pixel data corresponding to a plurality of pixels of the image sensor; and the processing circuit is configured to perform the inverse white balance operation, and the inverse white balance operation includes converting the pixel data corresponding to at least one of three or more pixels among the plurality of pixels into a color different from the original color.

17. The electronic device according to claim 16, wherein, The signal includes information related to the resolution of the image sensor and the tilt of the lens of the image sensor.

18. The electronic device according to claim 16, wherein, The first type of array pattern includes three or more color filters adjacent to each other, and the three or more color filters correspond to the same specific color.

19. The electronic device according to claim 18, wherein, The three or more pixels correspond to the three or more color filters.

Images