Method for calibrating image sensor and device for calibrating image sensor

Abstract

A method for calibrating an image sensor includes: obtaining multiple multi-light source images generated from multiple image sensor modules; obtaining multiple crosstalk levels and multiple color-specific correction coefficients based on the multiple multi-light source images; generating modeling data based on the relationship between the crosstalk level of a first color and the multiple color-specific correction coefficients; and obtaining color-specific correction coefficients of target pixels of an image sensor based on a single light source image captured by a first image sensor module and the modeling data, wherein the image sensor is arranged in the first image sensor module, and the color-specific correction coefficients can be used to correct the crosstalk of the target pixels.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims the benefit of priority from Korean Patent Application No. 10-2020-0021764 filed on February 21, 2020, in the Korean Intellectual Property Office, and Korean Patent Application No. 10-2020-0075937 filed on June 22, 2020, in the Korean Intellectual Property Office, the disclosures of which are incorporated herein in their entirety by reference. Technical Field

[0003] Example embodiments of the present disclosure relate to calibration of an image sensor, and more particularly, to a method and apparatus for calibrating an image sensor by obtaining a correction coefficient for correcting crosstalk in the image sensor. Background Art

[0004] The image sensor may include a pixel array that senses the received light signal. In order to obtain high-quality images, the pixels of the image sensor are highly integrated. In addition, since the image sensor has an autofocus function, the pixel array may include phase detection pixels, and the phase detection pixels may be arranged discontinuously and regularly within the pixel array. As the pixel size in the image sensor decreases and the pattern of the pixel array becomes uneven, the problem of increased crosstalk between pixels arises. Crosstalk refers to the change in the spectral characteristics of the signal generated from any pixel due to the influence of the signal generated from the adjacent pixel on the arbitrary pixel, resulting in reduced color reproducibility. Therefore, the image quality of the image sensor may deteriorate. Therefore, crosstalk correction is required to adjust the pixel value in order to eliminate or minimize the impact of crosstalk. Summary of the invention

[0005] Example embodiments of the present disclosure provide a method and apparatus for calibrating an image sensor by efficiently obtaining a correction coefficient for correcting crosstalk in the image sensor.

[0006] According to one aspect of an example embodiment, a method for calibrating an image sensor is provided, comprising: obtaining a plurality of multi-light source images generated from a plurality of image sensor modules; obtaining a plurality of crosstalk levels and a plurality of color-specific correction coefficients based on the plurality of multi-light source images; generating modeling data based on a relationship between a crosstalk level of a first color and the plurality of color-specific correction coefficients; and obtaining a color-specific correction coefficient for a target pixel of an image sensor based on a single-light source image captured by a first image sensor module and the modeling data, the image sensor being disposed in the first image sensor module, and the color-specific correction coefficient being capable of being used to correct the crosstalk of the target pixel.

[0007] According to one aspect of an example embodiment, a device for calibrating an image sensor is provided, the device comprising: a memory configured to store modeling data, the modeling data representing a color-specific correction coefficient function of a target pixel; and a processor configured to obtain a crosstalk level of the target pixel from a single-light-source image obtained by the image sensor, and based on the modeling data and the crosstalk level of the target pixel, obtain a color-specific correction coefficient of the target pixel, wherein the color-specific correction coefficient can be used to correct the crosstalk of the target pixel.

[0008] According to one aspect of an example embodiment, a method for calibrating an image sensor is provided, the method comprising: obtaining a crosstalk level of a target pixel from a single light source image obtained by the image sensor; obtaining a color-specific correction coefficient of the target pixel by applying the crosstalk level to a pre-obtained color-specific correction coefficient function of the target pixel, the color-specific correction coefficient being capable of being used to correct the crosstalk of the target pixel. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and / or other aspects will become more apparent by describing certain example embodiments with reference to the accompanying drawings, in which:

[0010] Figure 1 is a block diagram illustrating an image sensor and an electronic device including the image sensor according to example embodiments;

[0011] Figure 2A and Figure 2B An example of a pixel array is shown;

[0012] Figure 3A A vertical cross-sectional view of a normal pixel is schematically shown, and Figure 3B A vertical cross-sectional view schematically shows a phase difference detection pixel;

[0013] Figure 4 is a flow chart illustrating a method of calibrating an image sensor according to example embodiments;

[0014] Figure 5 An example of dividing a pixel array into a plurality of grids is shown;

[0015] Fig. 6A , Figure 6B and Figure 6C is a graph showing the relationship between the crosstalk level and the correction coefficient;

[0016] Figure 7 An example of an apparatus for calibrating an image sensor according to an example embodiment is shown;

[0017] Figure 8shows modeling data according to an example embodiment;

[0018] Fig. 9 An example of an apparatus for calibrating an image sensor according to an example embodiment is shown; and

[0019] Fig.10 is a block diagram schematically illustrating an electronic system according to example embodiments. DETAILED DESCRIPTION

[0020] Hereinafter, various exemplary embodiments will be described with reference to the accompanying drawings.

[0021] Figure 1 is a block diagram illustrating an image sensor and an electronic device including the image sensor according to example embodiments.

[0022] Reference Figure 1 , an electronic device 1000 according to an example embodiment may include an image sensor 100 and a processor 200. The image sensor 100 may convert an optical signal of an object incident through an optical lens LS into image data. The image sensor 100 may be mounted on an electronic device having an imaging function and / or a light sensing function. For example, the image sensor 100 may be mounted on an electronic device 1000 such as a digital camera, a digital video camera, a smart phone, a wearable device, an Internet of Things (IoT) device, a tablet personal computer (PC), a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, etc. In addition, the image sensor 100 may be mounted on an electronic device 1000 that is set as a component of a vehicle, furniture, a manufacturing facility, a door, various measuring devices, etc. For example, the electronic device 1000 may be implemented as a mobile device such as a smart phone or a wearable device, and the processor 200 may be an application processor (AP).

[0023] Reference Figure 1 , the image sensor 100 may include a pixel array 110, a readout circuit 120, and an image signal processor 130. In an embodiment, the pixel array 110, the readout circuit 120, and the image signal processor 130 may be implemented as a single semiconductor chip or a single semiconductor module. In an embodiment, the pixel array 110 and the readout circuit 120 may be implemented as one semiconductor chip or one semiconductor module, and the image signal processor 130 may be implemented as another semiconductor chip or another semiconductor module.

[0024] The pixel array 110 can be implemented, for example, using a photoelectric conversion element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and can be implemented using various types of photoelectric conversion devices. The pixel array 110 includes a plurality of pixels that convert a received optical signal (light) into an electrical signal, and the plurality of pixels can be arranged in a matrix.

[0025] Each of the pixels includes a light detection element (or a photoelectric conversion device). For example, the light detection element may include a photodiode, a phototransistor, a photogate, a pinned photodiode, a perovskite photodiode, an organic photodiode, an organic photoconductive film, etc. Alternatively or alternatively, various light detection elements may be used.

[0026] Multiple pixels can detect light using a light detection element, and can convert the detected light into an electrical signal. Multiple pixels can detect light in a specific spectral region. For example, multiple pixels may include: pixels that convert light in a red spectral region into an electrical signal (hereinafter referred to as red pixels), pixels that convert light in a green spectral region into an electrical signal (hereinafter referred to as green pixels), and pixels that convert light in a blue spectral region into an electrical signal (hereinafter referred to as blue pixels). However, one or more embodiments are not limited thereto, and multiple pixels may also include, for example, white pixels. As another example, multiple pixels may include pixels combined in different color configurations, for example, yellow pixels, cyan pixels, and green pixels.

[0027] A color filter array that transmits light in a specific spectral region may be provided on a plurality of pixels, and a color detectable by a corresponding pixel may be determined based on the color filter provided on each of the plurality of pixels. However, one or more embodiments are not limited thereto, and in an embodiment, light in a specific wavelength band may be converted into an electrical signal based on the level of an electrical signal of a light detection element applied to each of the plurality of pixels.

[0028] Figure 2A and Figure 2B An example of a pixel array is shown.

[0029] Reference Figure 2A , the pixel array 110a may include a color filter having nine pixels PX of the same color arranged in a 3×3 matrix. The pixel array 110a may have an arrangement structure in which a pixel group PG is repeatedly arranged, and the pixel group PG includes 9 red pixels R, 9 green pixels Gr, 9 blue pixels B, and 9 green pixels Gb (hereinafter, the green pixel Gr will be referred to as the first green pixel, and the green pixel Gb will be referred to as the second green pixel) arranged in a 3×3 matrix. The pixel array 110a having such a structure may be referred to as a Nona color filter array (CFA).

[0030] Even between pixels of the same color, a signal difference may occur between signals generated from the pixel PX due to the influence of pixels arranged around the pixel PX. Figure 2A , the first pixel PX1 and the second pixel PX2 are both red pixels R, and can be arranged at similar positions in the pixel array 110a. However, the first green pixel Gr and the red pixel R are arranged near the first pixel PX1, and the first pixel PX1 is affected by the surrounding first green pixel Gr and the surrounding red pixel R. Near the second pixel PX2, in addition to the first green pixel Gr and the red pixel R, the second green pixel Gb and the blue pixel B are also arranged. Therefore, the second pixel PX2 is affected by the surrounding first green pixel Gr, the red pixel R, the second green pixel Gb, and the blue pixel B.

[0031] In addition, in addition to Figure 2A In addition to the Nona CFA shown, the pixel array 110 a may also have a CFA structure in which pixels of the same color are arranged in a k×k matrix (k is an integer greater than or equal to 2).

[0032] Reference Figure 2B In the pixel array 110b, some pixel groups PG arranged repeatedly may include phase difference detection pixels PX PD In the embodiment, a plurality of phase difference detection pixels PX PD Can be arranged continuously in rows or columns.

[0033] The pixel array 110b may include red pixels, green pixels, and blue pixels, and four pixels PX arranged in a 2×2 matrix may include color filters having the same color. A pixel group PG including 4 red pixels R, 4 green pixels Gr, 4 blue pixels B, and 4 green pixels Gb, all arranged in a 2×2 matrix, is repeatedly arranged. The pixel array 110b having such a structure may be referred to as a four-Bayer color filter array (CFA). However, one or more embodiments are not limited thereto, and the pixel array 110b may have a Bayer CFA structure in which a pixel group PG including one red pixel, two green pixels, and one blue pixel arranged in a 2×2 matrix is ​​repeatedly arranged, or the Bayer CFA structure has a CFA structure in which pixels of the same color are arranged in a k×k matrix (k is an integer of 2 or greater).

[0034] Will refer to Figure 3A and Figure 3B Describe the phase difference detection pixel PX PD .

[0035] Figure 3A A vertical cross-sectional view of a normal pixel is schematically shown, and Figure 3BA vertical cross-sectional view of a phase difference detection pixel is schematically shown.

[0036] Reference Figure 3A , each of the plurality of pixels PX includes a light detecting element such as a photodiode PD and a color filter CF, and a microlens ML may be disposed on the color filter CF. The microlens ML may have a circular shape.

[0037] Reference Figure 3B , phase detection pixel PX PD Including with Figure 3A A plurality of sub-pixels SPX of the same size as the pixel PX may be provided with a microlens ML on the plurality of sub-pixels SPX. PD When two sub-pixels SPX are included, the microlens ML may have an elliptical shape. PD The structure of can be different from ordinary pixels (for example, Figure 3A The structure of the pixel PX).

[0038] Then, refer to Figure 2A and Figure 2B , in a pixel array with a regular pattern, the crosstalk between pixels PX is similar, so the visibility of the crosstalk may be weak. Figure 2A and Figure 2B As shown, when the patterns of the pixel arrays 110a and 110b are irregular, in other words, when the surrounding pixels of the same color pixel PX are arranged differently, the visibility of the crosstalk may be strong due to the difference in the crosstalk between the pixels PX. PD Different crosstalk may occur between pixels PX of the same color due to the shape of the microlens, etc. Since different crosstalk causes signal differences between pixels, the image quality will be degraded. Therefore, crosstalk correction is required to adjust the pixel value to eliminate or minimize the impact of crosstalk.

[0039] Then, refer to Figure 1 The readout circuit 120 receives an electrical signal from the pixel array 110 and converts the electrical signal into digital data, thereby generating image data (or an image) including a pixel value corresponding to each of a plurality of pixels. The image data is raw data.

[0040] The image signal processor 130 may perform image processing on the first image data IDT1 output from the readout circuit 120. For example, the image signal processor 130 may perform image processing such as bad pixel correction, crosstalk correction, and noise removal on the image data (eg, the first image data IDT1).

[0041] The image signal processor 130 may include a color corrector 131 and a memory 132, and the color corrector 131 may perform correction (hereinafter referred to as crosstalk correction) to obtain a color signal from each pixel (eg, pixels PX1, PX2 (see Figure 2A ))’s pixel values ​​are used to remove crosstalk.

[0042] Correction coefficients Cr, Cg, and Cb (e.g., red correction coefficient Cr, green correction coefficient Cg, and blue correction coefficient Cb) set for each color for each pixel PX requiring crosstalk correction may be stored in the memory 132, and the color corrector 131 may perform crosstalk correction based on the correction coefficients Cr, Cg, and Cb.

[0043] In the calibration process of the image sensor 100, the correction coefficients Cr, Cg, and Cb may be calculated based on the crosstalk level of the single light source image data generated by the image sensor 100 that captures a single light source (e.g., white light). More specifically, based on a plurality of multi-light source image data captured by a plurality of image sensors (e.g., various primary color images generated by capturing various primary color light sources), a function of each correction coefficient may be generated, and by applying the crosstalk level of the single light source image data generated in the image sensor 100 to the function, the correction coefficients Cr, Cg, and Cb for each color of the image sensor 100 may be calculated from the function. This will be referred to later. Figure 4 A method of calibrating the image sensor 100 according to example embodiments is described in detail.

[0044] exist Figure 1 , the memory 132 is shown as being disposed inside the image signal processor 130 , but is not limited thereto, and the memory 132 may be disposed in the image signal processor 130 or may be implemented as a separate semiconductor chip.

[0045] In the embodiments, as referenced Figure 2A As described, when the crosstalk between pixels of the same color within a pixel group is different, the color corrector 131 may perform crosstalk correction on the pixels.

[0046] In the embodiments, as referenced Figure 2B As described, based on the phase detection pixel PX PD The correction coefficients Cr, Cg, and Cb are set for each pixel PX arranged, and the color corrector 131 can detect the phase of the pixel PX based on the correction coefficients Cr, Cg, and Cb. PD Perform crosstalk correction.

[0047] In addition to the color corrector 131, the image signal processor 130 may also include multiple image processing intellectual property (IPs) for performing the above-mentioned image processing, and may provide the image data after image processing (e.g., the second image data IDT2) to the processor 200 (e.g., an application processor, a main processor of the electronic device 1000, a graphics processor, etc.).

[0048] Figure 4 is a flowchart illustrating a method of calibrating an image sensor according to example embodiments.

[0049] By using Figure 4 By using a calibration method, a correction coefficient for each color used for crosstalk correction of an image sensor can be obtained. As an example, a calibration method for extracting (or calculating) a correction coefficient for each color used to correct crosstalk with respect to a phase detection pixel will be described.

[0050] The calibration method according to example embodiments may be performed by a calibration device (eg, a setting facility (or setting device)) on a device having the image sensor 100 (see Figure 1 ) is performed in a calibration operation of an image sensor module (or camera module). In this case, the image sensor module may include an image sensor 100 (see Figure 1 ), optical lens LS (see Figure 1 ) and a housing forming the exterior of the image sensor module, and may also include: a filter (e.g., an infrared cut filter), an actuator, and a driving circuit for driving the actuator.

[0051] The plurality of image sensor modules and the specific image sensor modules mentioned in the example embodiments may have the same configuration and physical characteristics. For example, the pixel array structure, pixel arrangement, and structure and number of phase detection pixels of the image sensor 100 provided in the image sensor module may be the same between the image sensor modules, and the viewing angle of the optical lens LS may be the same between the image sensor modules.

[0052] Reference Figure 4 , the calibration method according to example embodiments may include a multi-light source calibration operation S10 and a single-light source calibration operation S20. The multi-light source calibration operation S10 may be performed using a plurality of image sensor modules. In an embodiment, the multi-light source calibration operation S10 is performed before a mass production operation of the image sensor modules, and in the mass production operation of the image sensor modules, the single-light source calibration operation S20 may be performed in various parameter setting operations of a specific image sensor module (e.g., in a setting process of a calibration process).

[0053] A plurality of multi-light source images may be obtained from each of a plurality of image sensor modules (S11). Here, a plurality of multi-light source images refers to a plurality of primary color images generated by using an image sensor module to capture each of a plurality of primary color light sources of different colors. In order to calculate the correction coefficient of each color (e.g., the correction coefficients Cr, Cg, and Cb of red, green, and blue, respectively), at least three or more primary color images may be used. Each of a plurality of image sensor modules captures those light sources corresponding to a plurality of colors, such as a red light source, a green light source, and a blue light source, thereby generating a red image, a green image, and a blue image, each of which is a multi-light source image. In an embodiment, a plurality of image sensor modules may also generate a white image by capturing a white light source.

[0054] A plurality of crosstalk levels and a plurality of color-specific correction coefficients (S12) may be calculated based on the plurality of multi-light source images. A crosstalk level and a color-specific correction coefficient (e.g., Cr, Cg, and Cb) of each target pixel (a target pixel for crosstalk correction, e.g., a pixel arranged around a phase detection pixel) may be calculated based on the plurality of multi-light source images obtained in operation S11.

[0055] The crosstalk XT can be expressed by Equation 1.

[0056] [Formula 1]

[0057] XT=Cr×Dr+Cg×Dg+Cb×Db

[0058] Here, Cr, Cg, and Cb represent red correction coefficients, green correction coefficients, and blue correction coefficients, respectively, and Dr, Dg, and Db are pixel values ​​of target pixels in the captured image for red, green, and blue, respectively. The values ​​of Dr, Dg, and Db can be calculated based on the pixel values ​​of pixels around the target pixel for which crosstalk is calculated.

[0059] The crosstalk XT1, XT2, and XT3 of the target pixel can be calculated based on each of the multi-light source images (e.g., red image, green image, and blue image) generated by photographing multiple light sources using one image sensor module. For example, in each multi-light source image, the crosstalk XT1, XT2, and XT3 of the target pixel can be calculated by comparing pixel values. The crosstalk XT1, XT2, and XT3 can be expressed by Equation 2.

[0060] [Formula 2]

[0061] XT1=Cr×Dr1+Cg×Dg1+Cb×Db1

[0062] XT2=Cr×Dr2+Cg×Dg2+Cb×Db2

[0063] XT3=Cr×Dr3+Cg×Dg3+Cb×Db3

[0064] Here, Dr1, Dg1, and Db1 are the pixel values ​​of the target pixels in the first multi-light source image for red, green, and blue, respectively; Dr2, Dg2, and Db2 are the pixel values ​​of the target pixels in the second multi-light source image for red, green, and blue, respectively; and Dr3, Dg3, and Db3 are the pixel values ​​of the target pixels in the third multi-light source image for red, green, and blue, respectively.

[0065] Equation 2 can be converted into Equation 3.

[0066] [Formula 3]

[0067]

[0068] The color-specific correction coefficients Cr, Cg, and Cb may be calculated according to Equation 3. The crosstalk level and the color-specific correction coefficient may be calculated for each multi-light source image, and thus, multiple crosstalk levels and multiple color-specific correction coefficients may be calculated based on multiple multi-light source images obtained from multiple image sensors.

[0069] As shown in Equations 2 and 3, crosstalks XT1, XT2, and XT3 of pixels in an image of at least three colors are required in order to calculate correction coefficients of the three colors.

[0070] In an embodiment, in operation S12, the crosstalk level of the white image may be calculated. For example, a white image may be generated based on a red image, a green image, and a blue image, and the crosstalk level in the white image may be calculated. However, one or more embodiments are not limited thereto, and a white image may be obtained by photographing a white light source using an image sensor, and the crosstalk level of pixels in the white image may be calculated.

[0071] In an embodiment, pixel array 110 (see Figure 1 ) can be divided into multiple grids, and for each of the multiple grids, the crosstalk level and color-specific correction coefficient of each target pixel can be calculated.

[0072] Figure 5 An example of dividing a pixel array into a plurality of grids is shown.

[0073] Reference Figure 5 , the pixel array 110 may include nine grids GRD, and each of the plurality of grids GRD may include a plurality of pixel groups PG and have the same pattern (eg, the same pixel arrangement). Figure 2A and Figure 2BAs described above, each of the plurality of pixel groups PG may include a plurality of color pixels, such as red pixels, green pixels, and blue pixels. In an embodiment, at least one pixel group among the plurality of pixel groups PG included in one grid GRD may include a phase detection pixel PX PD (See Figure 3B ). In each of the plurality of grids GRD, the phase detection pixel PX PD (See Figure 3B ) can be located in the same location.

[0074] In this way, the plurality of grids GRD may have the same structure. However, the crosstalk between pixels located at the same position in each grid of the plurality of grids GRD may be different. For example, even if the correction target pixels of the same color (referred to as the first pixel and the second pixel, respectively) are located at the same point P1 in the first grid GRD1 and the second grid GRD2, and the type and color of the pixels located around the first pixel and the second pixel are the same, the crosstalk of the first pixel of the first grid GRD1 and the second pixel of the second grid GRD2 may also be different. For example, according to the dispersion caused by the manufacturing process of the image sensor 100 or the position of the first grid GRD1 and the second grid GRD2, the amount of light received by each of the first grid GRD1 and the second grid GRD2 may be different. Therefore, the characteristics of the first grid GRD1 and the second grid GRD2 may be different, and therefore, the crosstalk of the first pixel and the crosstalk of the second pixel may be different. Therefore, when correcting the crosstalk of the first pixel and the second pixel, different color-specific correction coefficients may be applied. Therefore, in operation S12, the color-specific correction coefficient of the target pixel may be calculated for each grid in the plurality of grids GRD.

[0075] In addition, although it has been described that the correction coefficients Cr, Cg, and Cb for red, green, and blue are calculated respectively, one or more embodiments are not limited thereto, and as described above, the pixel array 110 may include pixels corresponding to combinations of different colors, and correction coefficients for different colors may be calculated. As an example, the pixel array includes yellow pixels, cyan pixels, and green pixels, and corresponding correction coefficients for yellow, cyan, and green may be calculated.

[0076] Then, refer to Figure 4, modeling data indicating the relationship between the crosstalk level and the correction coefficient for each color may be generated and stored (S13). Based on the trends of the target pixel-specific crosstalk level and the color-specific correction coefficient calculated in operation S12, the relationship between the color-specific correction coefficient and the crosstalk level may be modeled as a linear function or a multi-order function, and data representing the function (e.g., coefficients of each order of the function) may be generated as the modeling data. The modeling data may include data for each of the multiple target pixels for each of the multiple grids.

[0077] Fig. 6A , Figure 6B and Figure 6C is a graph showing the relationship between the crosstalk level and the correction coefficient.

[0078] exist FIG. 6A to FIG. 6C In the curve diagram, the horizontal axis represents the crosstalk level L XT , and the vertical axis represents the color-specific correction coefficient. Figure 4 The plurality of crosstalk levels and the plurality of color-specific correction coefficients calculated in operation S12 are classified, and the plurality of crosstalk levels and the plurality of color-specific correction coefficients can be classified in the following manner: FIG. 6A to FIG. 6C The graph shown shows the crosstalk level L XT The crosstalk level distribution of the color-specific correction coefficients Cr, Cg and Cb.

[0079] In an embodiment, the crosstalk level L XT It may be a value calculated based on a white image. However, one or more embodiments are not limited thereto. XT The value may be calculated based on another color image of the multi-light source image obtained in operation S11.

[0080] For each color, the crosstalk level L can be calculated by regression analysis. XT The relationship between the correction coefficients is modeled as an optimization function. For example, the red correction coefficient Cr can be modeled as a function F_Cr, the green correction coefficient Cg can be modeled as a function F_Cg, and the blue correction coefficient Cb can be modeled as a function F_Cb. The inputs of the functions F_Cr, F_Cg, and F_Cb are the crosstalk level L XT .

[0081] exist FIG. 6A to FIG. 6C In FIG. 1 , the functions F_Cr, F_Cg, and F_Cb are mainly shown as linear functions, but are not limited thereto, and the crosstalk level L XT The relationship between and the correction coefficient can be modeled as a multi-order function.

[0082] Then, refer to Figure 4 , for each target pixel of each grid GRD, the crosstalk level LXT The relationship between the color-specific correction coefficients is modeled, and the coefficients of each order of the modeling function can be generated as modeling data. The modeling data can be stored in a calibration device of the image sensor, for example, in a setting device of the image sensor module.

[0083] After the modeling data is generated, the single light source calibration operation S20 may be performed. The single light source calibration operation S20 may be performed for each image sensor module based on the modeling data generated in the multi-light source calibration operation S10.

[0084] A single light source image may be obtained from the image sensor module (S14). The image sensor module may generate a single light source image by photographing one light source. In an embodiment, the image sensor module may generate a white image by photographing a white light source.

[0085] A color-specific correction coefficient (S15) may be calculated based on a single light source image and modeling data. A crosstalk level of a target pixel may be calculated from a single light source image such as a white image, and based on the calculated crosstalk level of the target pixel and the modeling data, a color-specific correction coefficient (e.g., Cr, Cg, and Cb) of the target pixel may be calculated. For example, based on the modeling data, a function of a color-specific correction coefficient for a crosstalk level (a function for red, green, and blue) may be restored, and by applying the crosstalk level of the target pixel to the reconstructed function, the color-specific correction coefficient of the target pixel may be calculated. For each target pixel in each of the plurality of grids, a color-specific correction coefficient may be calculated based on a corresponding modeling function and the crosstalk level of the target pixel.

[0086] In this way, in the single-light source calibration operation S20, by using the modeling data generated in the multi-light source calibration operation S10, a color-specific correction coefficient can be calculated based on a single-light source image obtained by photographing a single light source using the image sensor module. The calculated color-specific correction coefficient can be stored in a memory in the image sensor module, for example, in Figure 1 During the image sensing operation of the image sensor 100, the color corrector 131 (see Figure 1 ) can perform crosstalk compensation on target pixels based on color-specific correction coefficients.

[0087] As described above, based on the calibration method of the image sensor according to the example embodiment, in the multi-light source calibration operation S10, the plurality of image sensor modules respectively capture the plurality of light sources to generate the plurality of multi-light source images, and the modeling data representing the relationship between the crosstalk level and the color-specific correction coefficient can be generated based on the plurality of multi-light source images. Thereafter, in order to extract the color-specific correction coefficient of the specific image sensor module, the single-light source calibration operation S20 is performed, and at this time, the specific image sensor module captures the single light source to generate the single-light source image without capturing the plurality of light sources, and according to the crosstalk level of the target pixel calculated based on the single-light source image and the modeling data, the color-specific correction coefficient of each target pixel can be calculated.

[0088] As a comparative example, in a calibration operation of an image sensor of the related art (e.g., a calibration operation in a mass production operation of an image sensor module), when a calibration device obtains a correction coefficient based on a single light source image generated by capturing a single light source by the image sensor, an image generated by capturing an object including various colors may not be properly corrected. On the other hand, in order to calculate a color-specific correction coefficient, a multi-light source image corresponding to the number of colors is required, and therefore, in a calibration operation of an image sensor (or image sensor module) of the related art, if multiple light sources are captured to generate a multi-light source image, such as red, green, and blue images, and the calibration device calculates the color-specific correction coefficient based on these images, the burden of installing multiple light sources and the burden of multi-light source capturing and multi-light source image processing are high (e.g., processing time, memory usage, and cost increase).

[0089] However, based on the reference Figure 4 In the method of calibrating an image sensor according to the described example embodiment, modeling data representing the relationship between the crosstalk level and the color-specific correction coefficients based on the multi-light source image is pre-generated and stored in the calibration device, and in the calibration operation (e.g., Figure 4 In S20), the image sensor captures a single light source to generate a single light source image, and the calibration device can calculate the crosstalk level of the target pixel from the single light source image, and calculate the color-specific correction coefficient of the target pixel based on the calculated crosstalk level of the target pixel and the modeling data.

[0090] Therefore, there is no need to install additional equipment required for multi-light source image acquisition, such as multiple light sources, and the burden of image processing in the calibration operation of the image sensor and the time required for calibration (eg, setup time) can be reduced.

[0091] Figure 7 An example of a calibration apparatus for an image sensor according to example embodiments is illustrated.

[0092] Reference Figure 7, the calibration device 2000 may include a calibration processor 2100 and a memory 2200 .

[0093] The memory 2200 may store modeling data representing a color-specific correction coefficient function for each of the plurality of target pixels. For example, the modeling data may include coefficients for each order of the color-specific correction coefficient function. Figure 4 The modeling data generated in the multi-light source calibration operation S10 may be stored in the memory 2200. For example, the modeling data may be generated based on a plurality of multi-light source images generated by photographing a plurality of light sources in a plurality of image sensor modules, and the modeling data may include coefficients of each order of a color-specific correction coefficient function.

[0094] Figure 8 Modeling data according to an example embodiment is shown.

[0095] Reference Figure 8 , the image sensor of the image sensor module 1100 (reference Figure 7 ) includes a pixel array divided into a plurality of grids, and the modeling data may include modeling data MD_GRD1 to MD_GRDm (m is an integer greater than or equal to 4) of a specific grid for each of the plurality of grids.

[0096] The modeling data of a specific grid (e.g., the modeling data MD_GRD1 of the first grid) may include color-specific modeling data MDr1, MDg1, and MDb1 (n is a positive integer) of each target pixel of the plurality of target pixels PX1 to PXn included in the first grid. The color-specific modeling data MDr1, MDg1, and MDb1 may include coefficients of each order of a color-specific correction coefficient function of the target pixel. For example, the first color modeling data MDr1 may include coefficients of each order of a correction coefficient function of a first color (e.g., red); the second color modeling data MDg1 may include coefficients of each order of a correction coefficient function of a second color (e.g., green); and the third color modeling data MDb1 may include coefficients of each order of a correction coefficient function of a third color (e.g., blue).

[0097] Then, refer to Figure 7 , the calibration device 2000 may receive a single light source image IDT_S from the image sensor module 1100, and based on the single light source image IDT_S and the modeling data, may calculate color-specific correction coefficients Cr, Cg, and Cb for each of a plurality of target pixels of the image sensor module 1100. In this case, the single light source image IDT_S may be generated by photographing a single light source 2300 using the image sensor module 1100. In an embodiment, the single light source 2300 may be a white light source, and the single light source image IDT_S may be a white image.

[0098] The calibration apparatus 2000 calculates a crosstalk level of each of a plurality of target pixels from a single light source image IDT_S, and calculates color-specific correction coefficients Cr, Cg, and Cb by applying the crosstalk level to a color-specific correction coefficient function of each of the plurality of target pixels restored based on modeling data.

[0099] The calibration apparatus 2000 may store color-specific correction coefficients Cr, Cg, and Cb for each of a plurality of target pixels in the image sensor module 1100. When the image sensor module 1100 performs image sensing, a color corrector (eg, Figure 1 131) can perform crosstalk correction based on color-specific correction coefficients Cr, Cg and Cb corresponding to multiple target pixels respectively.

[0100] In an embodiment, the calibration device 2000 may be implemented by a combination of a general processor such as a microcontroller unit (MCU) and a central processing unit (CPU) and firmware (or software module). For example, the firmware or software module may include a program or algorithm having the functions of the above-mentioned calibration device 2000. However, one or more embodiments are not limited thereto, and the calibration device 2000 may be implemented using hardware logic such as a field programmable gate array (FPGA), an application specific IC (ASIC), or a complex programmable logic device (CPLD) that performs the above-mentioned functions.

[0101] As reference Figure 7 As described, the calibration device 2000 according to the example embodiment can pre-store modeling data generated based on multiple multi-light source images, and calculate the color-specific correction coefficients Cr, Cg and Cb of the image sensor module 1100 based on the single light source image IDT_S captured by the image sensor module 1100 and the modeling data.

[0102] Fig. 9 An example of a calibration apparatus for an image sensor according to example embodiments is illustrated.

[0103] Reference Fig. 9 , the calibration device 3000 may include a processor 3100 and a memory 3200. The memory 3200 may store a multi-light source calibration module 3210 and a single-light source calibration module 3220. Here, the multi-light source calibration module 3210 and the single-light source calibration module 3220 may be software modules. The multi-light source calibration module 3210 includes a processor 3100 and a memory 3200 for performing the multi-light source calibration operation S10 (see Figure 4 ), and the single light source calibration module 3220 may include a program (or calibration tool) for performing the single light source calibration operation S20 (see Figure 4) procedure (or calibration tool).

[0104] The processor 3100 may execute the multi-light source calibration module 3210 and the single-light source calibration module 3220 to perform the calibration method of the image sensor according to the above-described exemplary embodiment. Thus, the processor 3100 may calculate color-specific correction coefficients (eg, Cr, Cg, and Cb) for crosstalk correction.

[0105] The processor 3100 may execute the multi-light source calibration module 3210 to receive a plurality of captured multi-light source images IDT_M from a plurality of arbitrary image sensor modules 1100, model the relationship between the color-specific correction coefficient and the crosstalk level as a correction coefficient function based on the plurality of multi-light source images IDT_M, and calculate the modeling data, for example, the coefficient of each order of the correction coefficient function. In an embodiment, the memory 3200 may store the modeling data, and the modeling data may be used in a single light source calibration operation.

[0106] The processor 3100 executes the single light source calibration module 3220 so that the image sensor module 1100 (e.g., a specific image sensor module 1100) can be controlled to capture a single light source and generate a single light source image IDT_S. In an embodiment, the processor 3100 can receive the single light source image IDT_S from the image sensor module 1100, and calculate the color-specific correction coefficients Cr, Cg, and Cb based on the single light source image IDT_S and the modeling data. The calibration device 3000 can store the color-specific correction coefficients Cr, Cg, and Cb in the image sensor module 1100.

[0107] Fig.10 is a block diagram schematically illustrating an electronic system according to example embodiments.

[0108] Reference Fig.10 , the electronic system 4000 may include an image sensor module 4100 and an application processor (AP) 4200. The electronic system 4000 may be installed on a smart phone, for example, and may also include general components such as a communication module, a working memory, an input / output module, and a display module.

[0109] The image sensor module 4100 may include the image sensor 100 (see Figure 1 ), and the image sensor 100 can store the color-specific correction coefficient calculated by the calibration method of the image sensor according to the above-mentioned embodiment for each target pixel, and perform crosstalk correction based on the color-specific correction coefficient during the image sensing operation. Therefore, the image quality of the image generated by the image sensor can be improved.

[0110] The AP 4200 controls the overall operation of the electronic system 4000 and may be implemented as a system on chip (SoC) that drives an application and an operating system. The AP 4200 may control the image sensor module 4100, generate a control signal for controlling the image sensor module 4100, and provide the control signal to the image sensor module 4100. For example, the AP 4200 may receive a phase difference signal from the image sensor module 4100 and control the image sensor module 4100 to perform autofocus based on the phase difference signal. As described above, the phase difference signal may be generated by a phase detection pixel. The AP 4200 may provide a control signal for autofocusing to the image sensor module 4100 based on the phase difference. The image sensor module 4100 may perform autofocusing by moving the lens in a direction that increases or decreases the distance to the object based on the control signal.

[0111] In addition, the AP 4200 may receive an image from the image sensor module 4100, perform additional image processing (e.g., image quality compensation, processing for increasing the dynamic range of an image, gamma compensation, etc.) on the received image, and store the image-processed image in a memory, or provide the image-processed image to a display module (e.g., a display device). In addition, the AP 4200 may extract additional information (e.g., object recognition information of the image and the position of the object in the image) from the image, provide the information to a user, and control the electronic system 4000 based on the information.

[0112] While the inventive concepts have been particularly shown and described with reference to example embodiments thereof, it will be understood that various changes in form and details may be made therein without departing from the spirit and scope of the appended claims.

Claims

1. A method for calibrating an image sensor, the method comprising: obtaining a plurality of multi-light source images generated from a plurality of image sensor modules, wherein each of the plurality of image sensor modules generates at least three multi-light source images; obtaining a plurality of crosstalk levels and a plurality of color-specific correction coefficients based on the plurality of multi-light source images; generating modeling data representing a relationship between a crosstalk level of a first color and the plurality of color-specific correction coefficients; receiving a single light source image captured by a first image sensor module for a light source of the first color; and Based on the received single-light-source image and the modeling data, a color-specific correction coefficient of a target pixel of an image sensor of the first image sensor module is obtained, and the color-specific correction coefficient can be used to correct crosstalk of the target pixel.

2. The method according to claim 1, wherein: The method of obtaining a color-specific correction coefficient for a target pixel includes: Obtaining a crosstalk level of the target pixel from the single light source image; restoring a color-specific correction coefficient function corresponding to the target pixel by using the modeling data; and By applying the crosstalk levels of the target pixels to the color-specific correction coefficient functions, respectively, the color-specific correction coefficients of the target pixels are obtained.

3. The method according to claim 2, wherein: The first color is white, and the single light source image is a white image.

4. The method according to claim 1, wherein: The color-specific correction coefficients include a correction coefficient for each of a plurality of colors, the plurality of colors being different from one another.

5. The method according to claim 1, wherein: The color-specific correction coefficients include a correction coefficient for red, a correction coefficient for green, and a correction coefficient for blue.

6. The method according to claim 1, further comprising: The color-specific correction coefficients for the target pixel are stored in the first image sensor module.

7. The method according to claim 1, wherein: The generative modeling data includes: modeling a correction factor function for each color-specific correction factor in the plurality of color-specific correction factors by performing a regression analysis on the plurality of crosstalk levels and the plurality of color-specific correction factors; and The modeling data representing the correction coefficient function is obtained.

8. The method according to claim 7, wherein: The modeling data includes coefficients for each order of the correction coefficient function.

9. The method according to claim 1, wherein: The target pixel is one of a plurality of pixels disposed around a phase detection pixel located in a pixel array of the image sensor.

10. The method according to claim 1, wherein: The pixel array of the image sensor includes a plurality of pixels of the same color arranged in a k×k matrix, k being an integer greater than or equal to 2, and The target pixel is at least one of the multiple pixels.

11. A device for calibrating an image sensor, the device comprising: a memory configured to store modeling data representing a color-specific correction coefficient function of a target pixel, the color-specific correction coefficient function representing a relationship between a crosstalk level of a first color and a plurality of color-specific correction coefficients, the modeling data being obtained based on a plurality of multi-light source images; and A processor, the processor being configured to: receiving a single light source image captured by a first image sensor for a light source of the first color, obtaining a crosstalk level of a target pixel from the single light source image obtained by the first image sensor, and Based on the modeling data and the crosstalk level of the target pixel, a color-specific correction coefficient of the target pixel is obtained, and the color-specific correction coefficient can be used to correct the crosstalk of the target pixel.

12. The device according to claim 11, wherein The color-specific correction coefficient function is generated based on a plurality of crosstalk levels and the plurality of color-specific correction coefficients obtained from the plurality of multi-light source images.

13. The device according to claim 11, wherein: The color-specific correction coefficient functions include a first correction coefficient function for a first color, a second correction coefficient function for a second color, and a third correction coefficient function for a third color, and The first color, the second color and the third color are different from each other.

14. The apparatus according to claim 11, wherein Each of the color-specific correction coefficient functions represents a relationship between the crosstalk level of the first color and a corresponding color-specific correction coefficient.

15. The apparatus according to claim 11, wherein: The image sensor includes a pixel array divided into a plurality of grids, and The modeling data represents the color-specific correction coefficient function for each target pixel in a plurality of target pixels, and the plurality of target pixels are arranged in the plurality of grids.

16. The apparatus according to claim 11, wherein: The modeling data includes coefficients for each order of the color-specific correction coefficient function.

17. A method for calibrating an image sensor, the method comprising: receiving a single light source image captured by a first image sensor for a light source of a first color; obtaining a crosstalk level of a target pixel of the first image sensor from the single light source image; The color-specific correction coefficient of the target pixel is obtained by applying the crosstalk level to a color-specific correction coefficient function, wherein the color-specific correction coefficient function represents the relationship between the crosstalk level of the first color and multiple color-specific correction coefficients, the color-specific correction coefficient function is obtained based on multiple multi-light source images, and the color-specific correction coefficient can be used to correct the crosstalk of the target pixel.

18. The method according to claim 17, wherein: The color-specific correction coefficient function is pre-obtained based on a plurality of crosstalk levels and the plurality of color-specific correction coefficients, which are obtained from the plurality of multi-light source images.

19. The method according to claim 17, wherein: The single light source image is an image generated by photographing a white light source using the image sensor.

20. The method according to claim 17, wherein: The color-specific correction coefficient functions include a correction coefficient function for each of at least three colors.

Images