Image processing apparatus and image processing method for color correction, and image processing system including the same

Abstract

An image processing device includes a memory and a color correction circuit. The memory stores first correction information for correcting a first pixel value among a plurality of pixel values. The plurality of pixel values are received from an autofocus image sensor including first pixels configured to detect a phase difference and second pixels configured to detect an image. The first pixel value is obtained from the first pixels and corresponds to a first color. The first correction information is for correcting the first pixel value to correspond to a second color different from the first color. The color correction circuit receives first image frame data including the plurality of pixel values from the autofocus image sensor, loads the first correction information from the memory, and generates first corrected image frame data by correcting the first pixel value included in the first image frame data to correspond to the second color based on the first correction information.

Description

[0001] Cross-reference of related applications

[0002] This application claims priority to Korean Patent Application No. 10-2020-0119077, filed on September 16, 2020 with the Korean Intellectual Property Office (KIPO), the entire contents of which are incorporated herein by reference. Technical Field

[0003] The example embodiments generally relate to semiconductor integrated circuits, and more specifically to image processing apparatus and image processing methods for color correction, and image processing systems including said image processing apparatus. Background Technology

[0004] Complementary metal-oxide-semiconductor (CMOS) image sensors are image capture devices manufactured using CMOS processes. CMOS image sensors typically have lower manufacturing costs and smaller pixel sizes than traditional charge-coupled device (CCD) image sensors, which usually use higher voltage analog circuitry. Therefore, CMOS image sensors can have lower power consumption requirements than CCD image sensors. Furthermore, with continuous improvements in the performance of CMOS image sensors, their use is becoming increasingly widespread in mobile electronic devices such as smartphones, tablet PCs, and digital cameras.

[0005] In digital image processing devices such as cameras, detecting the focus control state of a photographic lens to automatically control the lens's focus can be beneficial. To achieve this, conventional digital image processing devices include additional focus detection equipment separate from and / or different from the image sensor. In this case, the cost of the focus detection equipment and / or the additional optical lens may increase, and / or the overall size of the digital image processing device may increase due to the focus detection equipment. To address these issues, autofocus image sensors using methods for detecting phase differences have been developed. Furthermore, various schemes for processing the image signal output from the autofocus image sensor have been investigated to improve or enhance image quality. Summary of the Invention

[0006] At least one example embodiment of this disclosure provides an image processing apparatus capable of correcting pixel values ​​of a specific color output from an autofocus image sensor or an image sensor.

[0007] At least one example embodiment of this disclosure provides an image processing method capable of correcting pixel values ​​of a specific color output from an autofocus image sensor or an image sensor.

[0008] At least one example embodiment of this disclosure provides an image processing system including an image processing device.

[0009] According to an example embodiment, an image processing apparatus includes a memory and a color correction circuit. The memory stores first correction information for correcting a first pixel value among a plurality of pixel values. The plurality of pixel values ​​are received from an autofocus image sensor including a first pixel configured to detect a phase difference and a second pixel configured to detect an image. The first pixel value is obtained from the first pixel and corresponds to a first color. The first correction information is used to correct the first pixel value to correspond to a second color different from the first color. The color correction circuit receives first image frame data including the plurality of pixel values ​​from the autofocus image sensor, loads the first correction information from the memory, and generates first corrected image frame data by correcting the first pixel value included in the first image frame data to correspond to the second color based on the first correction information.

[0010] According to an example embodiment, in an image processing method, first correction information is generated for correcting a first pixel value among a plurality of pixel values. The plurality of pixel values ​​are received from an autofocus image sensor including a first pixel configured to detect a phase difference and a second pixel configured to detect an image. The first pixel value is obtained from the first pixel and corresponds to a first color. The first correction information is used to correct the first pixel value to correspond to a second color different from the first color. The first correction information is stored in a memory. First image frame data including the plurality of pixel values ​​is received from the autofocus image sensor. The first correction information is loaded from the memory. First corrected image frame data is generated by correcting the first pixel value included in the first image frame data to correspond to the second color based on the first correction information.

[0011] According to an example embodiment, an image processing system includes an autofocus image sensor and an image processing device. The autofocus image sensor includes a first pixel configured to detect a phase difference and a second pixel configured to detect an image. The image processing device performs image processing operations on image frame data provided from the autofocus image sensor. The image processing device includes a calibration circuit, a memory, and a color correction circuit. The calibration circuit receives reference image frame data obtained by capturing a reference pattern using the autofocus image sensor, compares original image data corresponding to the reference pattern with the reference image frame data, and calculates first correction information based on the result of comparing the original image data with the reference image frame data. The first correction information is used to correct a first pixel value among a plurality of pixel values ​​received from the autofocus image sensor. The first pixel value is obtained from the first pixel and corresponds to a first color. The first correction information is used to correct the first pixel value to correspond to a second color different from the first color. The memory receives the first correction information from the calibration circuit and stores the first correction information. The color correction circuit receives first image frame data including the plurality of pixel values ​​from the autofocus image sensor, loads first correction information from the memory, and generates first corrected image frame data by correcting the first pixel values ​​included in the first image frame data to correspond to the second color based on the first correction information. The first correction information includes multiple gains, multiple biases, and multiple position data. The multiple gains and the multiple biases are used to convert the first pixel values ​​corresponding to the first color into first corrected pixel values ​​corresponding to the second color. The multiple position data represent the position of the first pixel. The calibration circuit and the memory generate and store the first correction information before the autofocus image sensor operates normally.

[0012] In the image processing apparatus, image processing method, and image processing system according to the example embodiments, the pixel values ​​of the phase detection pixels included in the autofocus image sensor can be corrected to or converted into different colors. The image processing apparatus can obtain correction information for the correction operation in advance, can store the correction information in advance in the memory, can load correction information for each image frame, and can perform the correction operation based on the loaded correction information, rather than generating and calculating correction information for each image frame. Furthermore, pixel values ​​received from an image sensor different from the autofocus image sensor can also be corrected to or converted into different colors based on the correction information obtained and stored in advance. Therefore, the computational load, computation time, power consumption, etc., for the correction operation can be reduced, and the image processing apparatus can have improved or better performance. Attached Figure Description

[0013] The illustrative, non-limiting exemplary embodiments will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings.

[0014] Figure 1 This is a block diagram illustrating an image processing apparatus according to an example embodiment.

[0015] Figure 2 This is a block diagram illustrating an image processing system according to an example embodiment.

[0016] Figure 3 This is a block diagram illustrating an example of an autofocus image sensor included in an image processing system according to an example embodiment.

[0017] Figure 4 It is shown Figure 3 A circuit diagram of an example of pixels included in a pixel array in an autofocus image sensor.

[0018] Figure 5A , Figure 5B and Figure 5C It is shown Figure 3 An example diagram of the pixel array included in an autofocus image sensor.

[0019] Figure 6A , Figure 6B and Figure 6C It is used to describe the use Figure 3 A diagram illustrating an example of an autofocus image sensor performing automatic focusing.

[0020] Figure 7A , Figure 7B , Figure 7C , Figure 8A , Figure 8B and Figure 9This is a diagram used to describe the operation of an image processing device according to an example embodiment.

[0021] Figure 10 This is a block diagram illustrating an image processing apparatus according to an example embodiment.

[0022] Figure 11 It is shown Figure 10 A block diagram of an example calibration circuit included in an image processing device.

[0023] Figure 12 and Figure 13 This is a block diagram illustrating an image processing apparatus according to an example embodiment.

[0024] Figure 14 This is a block diagram illustrating an image processing apparatus according to an example embodiment.

[0025] Figure 15 This is a block diagram illustrating an image processing system according to an example embodiment.

[0026] Figure 16 yes Figure 15 A plan view of an example pixel array included in an image sensor in an image processing system.

[0027] Figure 17A , Figure 17B , Figure 17C , Figure 17D , Figure 18A , Figure 18B , Figure 18C and Figure 18D This is a diagram used to describe the operation of an image processing device according to an example embodiment.

[0028] Figure 19 This is a flowchart illustrating an image processing method according to an example embodiment.

[0029] Figure 20 It is shown Figure 19 The flowchart shows an example of generating the first correction information.

[0030] Figure 21 and Figure 22 This is a flowchart illustrating an image processing method according to an example embodiment.

[0031] Figure 23 This is a flowchart illustrating an image processing method according to an example embodiment.

[0032] Figure 24 This is a block diagram illustrating an electronic system including an image processing apparatus according to an example embodiment. Detailed Implementation

[0033] Various exemplary embodiments will be described more fully with reference to the accompanying drawings, in which embodiments are illustrated. However, this disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In this application, similar reference numerals denote similar elements.

[0034] Figure 1 This is a block diagram illustrating an image processing apparatus according to an example embodiment.

[0035] refer to Figure 1 The image processing device 100 includes a memory 120 and a color correction circuit 140.

[0036] The memory 120 stores and outputs first correction information CINF1. The first correction information CINF1 is used to correct the autofocus image sensor (e.g., located outside the image processing device 100) positioned outside the image processing device 100. Figure 2 The first pixel value among multiple pixel values ​​received by the autofocus image sensor 220 in the image sensor. (Refer to...) Figure 5A , Figure 5B and Figure 5C Described, the autofocus image sensor includes a first pixel for detecting a phase difference and a second pixel for detecting an image. The first pixel value is obtained from the first pixel and corresponds to a first color, and first correction information CINF1 is used to correct the first pixel value to correspond to a second color different from the first color. For example, the first correction information CINF1 may include data for correcting or converting the pixel value of the phase-detecting pixel to a different color. For example, each of the plurality of pixel values ​​may include the grayscale value, illuminance value, and / or luminance value of the corresponding pixel among the plurality of pixels.

[0037] In some example embodiments, memory 120 may include: any non-volatile memory, such as electrically erasable programmable read-only memory (EEPROM), flash memory, phase-change random access memory (PRAM), resistive random access memory (RRAM), nanofloating gate memory (NFGM), polymer random access memory (PoRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), thyristor random access memory (TRAM), etc.; and / or any volatile memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), etc.

[0038] In some example embodiments, the first correction information CINF1 may be obtained in advance (or obtained ahead of time) and may be stored in advance (or stored ahead of time) in memory 120 before the image processing device 100 is in normal operation and / or before the autofocus image sensor is in normal operation.

[0039] In some example embodiments, the first correction information CINF1 may be provided by a calibration circuit located inside or outside the image processing device 100 (e.g., Figure 10 The calibration circuit 110 in the reference is generated. Figure 10 and Figure 11 Describe the operation for generating the first correction information CINF1.

[0040] Color correction circuit 140 receives first image frame data FDAT1, including multiple pixel values, from an autofocus image sensor, loads first correction information CINF1 from memory 120, and generates first corrected image frame data CFDAT1 based on the first correction information CINF1 and the first image frame data FDAT1. For example, in one embodiment, color correction circuit 140 generates first corrected image frame data CFDAT1 by correcting first pixel values ​​included in the first image frame data FDAT1 to correspond to a second color based on the first correction information CINF1. For example, the first image frame data FDAT1 and the first corrected image frame data CFDAT1 may correspond to an image frame provided from an autofocus image sensor. (Refer to...) Figures 7A to 9 The operation of the color correction circuit 140 is described in more detail.

[0041] The image processing apparatus 100 according to the example embodiment can correct or convert the pixel values ​​of the phase detection pixels included in the autofocus image sensor to different colors. The image processing apparatus 100 can obtain correction information CINF1 for the correction operation in advance, can store the correction information CINF1 in advance in the memory 120, can load the correction information CINF1 for each image frame, and can perform the correction operation based on the loaded correction information CINF1, rather than generating and calculating correction information for each image frame. Therefore, the computational load, computation time, power consumption, etc., for the correction operation can be reduced, and the image processing apparatus 100 can have improved or better performance.

[0042] Figure 2 This is a block diagram illustrating an image processing system according to an example embodiment.

[0043] refer to Figure 2 The image processing system 200 includes an autofocus image sensor 220 and an image processing device 240.

[0044] The autofocus image sensor 220 sequentially outputs multiple image frame data FDATs, each including information associated with an image of a target based on incident light. Each of the multiple image frame data FDATs may include multiple pixel values ​​obtained from multiple pixels included in the autofocus image sensor 220. For example, Figure 1The first image frame data FDAT1 can be included in multiple image frame data FDATs. (Refer to...) Figures 3 to 6C The configuration and operation of the autofocus image sensor 220 are described in more detail.

[0045] Image processing apparatus 240 sequentially receives multiple image frame data FDATs and sequentially generates multiple corrected image frame data CFDATs by sequentially correcting the multiple image frame data FDATs. Image processing apparatus 240 can be implemented according to an example embodiment. Image processing apparatus 240 can obtain and store correction information for the correction operation in advance (e.g., ...). Figure 1 The first correction information (CINF1) in the image can be used to perform correction operations for each image frame based on the pre-stored correction information.

[0046] In some example embodiments, the image processing device 240 may be Figure 1 Image processing device 100. In other example embodiments, image processing device 240 may be... Figure 10 Image processing equipment 100a Figure 12 Image processing device 100b and Figure 13 One of the image processing devices 100c. The image processing device 240 may be referred to as an image signal processor (ISP).

[0047] In some example embodiments, the image processing device 240 may also perform at least one image processing operation, such as image interpolation, depigmentation, white balance, gamma correction, color conversion, etc.

[0048] Figure 3 This is a block diagram illustrating an example of an autofocus image sensor included in an image processing system according to an example embodiment.

[0049] refer to Figure 3The autofocus image sensor 500 includes a pixel array 510, a correlated double sampling (CDS) block 530, and an analog-to-digital converter (ADC) block 540. The autofocus image sensor 500 may also include a line driver 520, a digital signal processor 550, a ramp signal generator 560, and a timing controller 580. As is commonly practiced in the disclosed art, features and embodiments are described and illustrated in the accompanying drawings as functional blocks, units, and / or modules, or components described using a certain device. Those skilled in the art will appreciate that these blocks, units, and / or modules are physically implemented by electronic (or optical) circuitry such as logic circuits, discrete components, microprocessors, hardwired circuitry, memory elements, wiring connections, etc., which can be formed using semiconductor-based manufacturing techniques or other manufacturing techniques. Where blocks, units, and / or modules are implemented by microprocessors, etc., they can be programmed using software (e.g., microcode) to perform the various functions discussed herein, and may optionally be driven by firmware and / or software.

[0050] Pixel array 510 includes a plurality of pixels (or unit pixels) PX arranged in a matrix. Each of the plurality of pixels PX can be connected to a corresponding row of a plurality of rows RW1, RW2, ..., RWX and a corresponding column of a plurality of columns CL1, CL2, ..., CLY, where X and Y are both natural numbers greater than or equal to 2. Pixel array 510 generates a plurality of analog pixel signals VP1, VP2, ..., VPY based on incident light. Pixels implemented in hardware (e.g., pixels PX) may be referred to herein as sensor pixels.

[0051] Multiple pixels (PX) can include a first pixel for detecting phase difference and a second pixel for detecting the image. (Refer to...) Figure 4 and Figure 5A , Figure 5B , Figure 5C The configuration of each pixel in the multiple pixel PX and the pixel array 510 is described in detail.

[0052] The row driver 520 can be connected to multiple rows RW1 to RWX of the pixel array 510. The row driver 520 can generate drive signals to drive the multiple rows RW1 to RWX. For example, the row driver 520 can drive multiple pixels PX included in the pixel array 510 row by row.

[0053] The correlated double sampling block 530 may include multiple correlated double sampling circuits (CDS) 530a, 530b, ..., 530c. These multiple correlated double sampling circuits 530a to 530c may be connected to multiple columns CL1 to CLY of the pixel array 510. These multiple correlated double sampling circuits 530a to 530c may perform correlated double sampling operations on multiple analog pixel signals VP1 to VPY output from the pixel array 510.

[0054] Analog-to-digital converter block 540 includes multiple analog-to-digital converters 540a, 540b, ..., 540c. The multiple analog-to-digital converters 540a to 540c are connected to multiple columns CL1 to CLY of pixel array 510 via multiple correlated double-sampling circuits 530a to 530c. The multiple analog-to-digital converters 540a to 540c perform column-level analog-to-digital conversion operations in parallel (e.g., simultaneously or concurrently) converting multiple analog pixel signals VP1 to VPY (e.g., multiple correlated double-sampling analog pixel signals output from the multiple correlated double-sampling circuits 530a to 530c) into multiple digital signals CNT1, CNT2, ..., CNTY.

[0055] Each of the plurality of analog-to-digital converters 540a to 540c may include a corresponding comparator among a plurality of comparators 542a, 542b, ..., 542c and a corresponding counter among a plurality of counters (CNTs) 544a, 544b, ..., 544c. For example, the first analog-to-digital converter 540a may include a first comparator 542a and a first counter 544a. The first comparator 542a may compare a first analog pixel signal VP1 (e.g., a correlated double-sampled first analog pixel signal output from a first correlated double-sampled circuit 530a) with a ramp signal VRAMP to generate a first comparison signal CS1. The first counter 544a may count the level transitions of the first comparison signal CS1 to generate a first digital signal CNT1.

[0056] The operations of the associated double sampling block 530 and the analog-to-digital converter block 540 can be performed row by row on the multiple pixels PX included in the pixel array 510.

[0057] Multiple correlated dual sampling circuits 530a to 530c and multiple analog-to-digital converters 540a to 540c can form multiple column drive circuits. For example, the first correlated dual sampling circuit 530a and the first analog-to-digital converter 540a can form a first column drive circuit.

[0058] The digital signal processor 550 can perform digital signal processing operations based on multiple digital signals CNT1 to CNTY. For example, operations such as obtaining autofocus information using a first pixel and obtaining an image frame using a second pixel can be performed by the digital signal processor 550. The digital signal processor 550 can sequentially output multiple image frame data FDAT generated by the digital signal processing operations.

[0059] In some example embodiments, the digital signal processor 550 may be omitted, and the aforementioned digital signal processing operations may be performed by a signal processing device located outside the autofocus image sensor 500 (e.g., Figure 2 The image processing device 240 in the middle is used for execution.

[0060] The ramp signal generator 560 can generate a ramp signal VRAMP. The timing controller 580 can control the overall operating timing of the autofocus image sensor 500 and can generate control signals including a count enable signal CNT_EN and a clock signal (not shown).

[0061] Figure 4 It is shown Figure 3 A circuit diagram of an example of pixels included in a pixel array in an autofocus image sensor.

[0062] refer to Figure 4 The pixel (or unit pixel) 600 may include a photoelectric conversion unit 610 and a signal generation unit 612. Figure 3 The first pixel AFP for detecting phase difference and the second pixel for detecting image among the multiple pixels PX included in the pixel array 510 can have the same pixel structure and circuit structure, but the configurations of the microlenses and color filters are different from each other, and will be referenced. Figure 5A , Figure 5B and Figure 5C Describe it.

[0063] The photoelectric conversion unit 610 can perform photoelectric conversion operations. For example, the photoelectric conversion unit 610 can convert incident light into photocharge during integration mode. If the image sensor including pixel 600 is a complementary metal-oxide-semiconductor (CMOS) image sensor, image information related to the target to be captured can be obtained by collecting charge carriers (e.g., electron-hole pairs) in the photoelectric conversion unit 610 during integration mode that are proportional to the intensity of the incident light passing through the open shutter of the CMOS image sensor.

[0064] The signal generation unit 612 can generate an electrical signal (e.g., an analog pixel signal VP) based on the photocharge generated by the photoelectric conversion operation during readout mode. If the image sensor including pixel 600 is a CMOS image sensor, the shutter can be closed during readout mode after integration mode, and the analog pixel signal VP can be generated based on image information in the form of charge carriers. Figure 4 As shown, in some embodiments, pixel 600 may have a four-transistor structure including four transistors.

[0065] For example, the signal generation unit 612 may include a transmission transistor 620, a reset transistor 640, a drive transistor 650, a selection transistor 660, and a floating diffusion node 630. The transmission transistor 620 may be connected between the photoelectric conversion unit 610 and the floating diffusion node 630, and may include a gate electrode for receiving the transmission signal TX. The reset transistor 640 may be connected between the power supply voltage VDD and the floating diffusion node 630, and may include a gate electrode for receiving the reset signal RX. The drive transistor 650 may be connected between the power supply voltage VDD and the selection transistor 660, and may include a gate electrode connected to the floating diffusion node 630. The selection transistor 660 may be connected between the drive transistor 650 and the output terminal for outputting the analog pixel signal VP, and may include a gate electrode for receiving the selection signal SEL.

[0066] In some example embodiments, a signal generation unit may be shared by multiple photoelectric conversion units to form a single unit pixel with multiple photoelectric conversion units.

[0067] Figure 5A , Figure 5B and Figure 5C It is shown Figure 3 An example diagram of the pixel array included in an autofocus image sensor.

[0068] Figure 5A This is a plan view of an example of a pixel array included in an autofocus image sensor. Figure 5B This is a cross-sectional view of an example of pixels included in a pixel array of an autofocus image sensor. Figure 5C yes Figure 5A A plan view of an example color filter array included in a pixel array.

[0069] refer to Figure 5A The pixel array 512 included in the autofocus image sensor may include multiple pixels PX11, PX12, PX13, PX14, PX15, PX16, PX21, PX22, PX23, PX24, PX25, PX26, PX31, PX32, PX33, PX35, PX36, PX41, PX42, PX43, PX44, PX45, PX46, PX51, PX52, PX53, PX54, PX55, PX56, PX61, PX62, PX63, PX64, PX65, and PX66. The pixel array can be repeated in this manner hundreds, thousands, or more times to be arranged within a larger array forming the image sensor.

[0070] As described above, the plurality of pixels PX11 to PX16, PX21 to PX26, PX31 to PX33, PX35, PX36, PX41 to PX46, PX51 to PX56, and PX61 to PX66 may include a first pixel for detecting phase difference and a second pixel for detecting image. The first pixel may be referred to as an autofocus (AF) pixel, and the second pixel may be referred to as a normal pixel or image pixel. In the pixel array 512 according to the example embodiment, the AF pixel and the normal pixel for obtaining AF information may be arranged in a pixel array 512. Figure 5A In the example, pixel PX33 can be an AF pixel, and the remaining pixels PX11 to PX16, PX21 to PX26, PX31, PX32, PX35, PX36, PX41 to PX46, PX51 to PX56, and PX61 to PX66 can be normal pixels.

[0071] Microlenses (MLs) can be disposed on multiple pixels PX11 to PX16, PX21 to PX26, PX31 to PX33, PX35, PX36, PX41 to PX46, PX51 to PX56, and PX61 to PX66. The size of the microlens of an AF pixel (e.g., pixel PX33) can be larger than the size of the microlens of a normal pixel (e.g., pixel PX35).

[0072] Although Figure 5A An example is shown in which six pixels are arranged in a first direction DR1 in pixel array 512, and six pixels are arranged in a second direction DR2 that intersects (e.g., is substantially perpendicular to) the first direction DR1 in pixel array 512, and pixel array 512 includes one AF pixel and 34 normal pixels, but the example embodiment is not limited thereto. The size of the pixel array and the number of pixels can be varied according to the example embodiment.

[0073] refer to Figure 5B The first pixel (or AF pixel) AFPX may include a first photoelectric conversion area PD1, a second photoelectric conversion area PD2, a first color filter CF1, and a first microlens ML1. For example, the first pixel AFPX can be... Figure 5A The corresponding pixel is PX33.

[0074] A first photoelectric conversion region PD1 and a second photoelectric conversion region PD2 can be formed in the substrate 511. A first color filter CF1 can be formed on the first photoelectric conversion region PD1 and the second photoelectric conversion region PD2 (e.g., along a third direction DR3 that intersects the first direction DR1 and the second direction DR2, e.g., substantially perpendicular to them), and can be shared by the first photoelectric conversion region PD1 and the second photoelectric conversion region PD2. In some example embodiments, the first color filter CF1 may include two color filters having the same color and formed on the first photoelectric conversion region PD1 and the second photoelectric conversion region PD2, respectively. A first microlens ML1 can be formed on the first color filter CF1 and can be shared by the first photoelectric conversion region PD1 and the second photoelectric conversion region PD2.

[0075] The second pixel (or normal pixel) NPX may include a third photoelectric conversion area PD3, a second color filter CF2, and a second microlens ML2. For example, the second pixel NPX can be combined with... Figure 5A The corresponding pixel is PX35.

[0076] The third photoelectric conversion region PD3 can be formed in the substrate 511. The second color filter CF2 can be formed on the third photoelectric conversion region PD3. The second microlens ML2 can be formed on the second color filter CF2.

[0077] In some example embodiments, the first photoelectric conversion region PD1, the second photoelectric conversion region PD2, and the third photoelectric conversion region PD3 have the same dimensions (e.g., the same area in a plan view or on a plane). Therefore, the size (e.g., area in a plan view) of the first microlens ML1, shared by the first photoelectric conversion region PD1 and the second photoelectric conversion region PD2, can be larger than the size of the second microlens ML2. For example, the size of the first microlens ML1 can be approximately twice the size of the second microlens ML2. Terms such as “about” or “approximately” can reflect amounts, sizes, orientations, or layouts that vary only slightly and / or in a manner that does not significantly alter the operation, function, or structure of a particular element. For example, a range from “about 0.1 to about 1” can include ranges such as a deviation of 0%–5% near 0.1 and a deviation of 0% to 5% near 1, especially if such deviations maintain the same effect as the listed ranges.

[0078] Photoelectric conversion regions PD1, PD2, and PD3 can be connected with Figure 4Corresponding to the photoelectric conversion unit 610, it can perform photoelectric conversion operations based on incident light to generate photocharge. For example, each of the photoelectric conversion regions PD1, PD2, and PD3 can generate electron-hole pairs in response to incident light and can collect the electrons and / or holes of the electron-hole pairs. The photoelectric conversion regions PD1, PD2, and PD3 may include photodiodes, phototransistors, photogates, pinned photodiodes (PPDs), and / or combinations thereof.

[0079] In some example embodiments, color filters CF1 and CF2 may include a red color filter, a green color filter, and / or a blue color filter. In other example embodiments, color filters CF1 and CF2 may include a yellow color filter, a magenta color filter, and / or a cyan color filter. Color filters CF1 and CF2 may also include a white color filter.

[0080] Microlenses ML1 and ML2 can adjust the light path entering microlenses ML1 and ML2 so that the light is focused on the corresponding photoelectric conversion areas PD1, PD2 and PD3.

[0081] In some example embodiments, microlenses ML1 and ML2 can form a microlens array, and color filters CF1 and CF2 can form a color filter array.

[0082] In some example embodiments, the pixel array may further include a device isolation region 513 formed between photoelectric conversion regions PD1, PD2, and PD3. The device isolation region 513 may be formed in or disposed in a substrate 511, and may extend substantially vertically from the surface of the substrate 511 in a third direction DR3, thereby penetrating the substrate 511 to isolate or space the photoelectric conversion regions PD1, PD2, and PD3 from each other, and may include an insulating material. The device isolation region 513 may have a mesh structure in a plan view.

[0083] In some example embodiments, the autofocus image sensor may be a front-side illumination image sensor (FIS) that operates in response to incident light passing through the front surface of the substrate 511, or a back-side illumination image sensor (BIS) that operates in response to incident light passing through the rear surface of the substrate 511. Although not shown in detail, it is used to form... Figure 4 The gate electrode and wires of the signal generation unit 612 can be disposed on the front or rear surface of the substrate 511.

[0084] refer to Figure 5C The color filter array CFA1 included in the pixel array 512 may include a red color filter R, a green color filter G, a blue color filter B, and a white color filter W. In other words, the color filter array CFA1 may have an RGBW pattern.

[0085] Pixels PX32, PX36, PX41, and PX45 may include a red filter R. Pixels PX12, PX16, PX21, PX25, PX33, PX43, PX52, PX56, PX61, and PX65 may include a green filter G. Pixels PX14, PX23, PX54, and PX63 may include a blue filter B. Pixels PX11, PX13, PX15, PX22, PX24, PX26, PX31, PX35, PX42, PX44, PX46, PX51, PX53, PX55, PX62, PX64, and PX66 may include a white filter W. Therefore, in some embodiments, the AF pixels and normal pixels included in the pixel array 512 may have the same filter arrangement structure.

[0086] Figure 6A , Figure 6B and Figure 6C It is used to describe the use Figure 3 A diagram illustrating an example of an autofocus image sensor performing automatic focusing.

[0087] refer to Figure 6A , Figure 6B and Figure 6C Light (or incident light) passing through lens 51 of an image capturing device (e.g., a camera) including an autofocus image sensor passes through multiple microlenses (e.g., Figure 5B The microlens array 54 of the microlens ML1 in the lens 51 transmits light to the first AF pixel (R) 57 and the second AF pixel (L) 58. Light input from the pupil 52 located above the optical axis 50 of the lens 51 is sensed by the second AF pixel 58, and light input from the pupil 53 located below the optical axis 50 of the lens 51 is sensed by the first AF pixel 57.

[0088] The consecutive pixel output of the first AF pixel 57 and the second AF pixel 58, which are located at the same positions as the first AF pixel 57 and the second AF pixel 58, is in Figure 6B and Figure 6C As shown in the image. Figure 6B and Figure 6C In each of these, the horizontal axis represents the position of each of the first AF pixel 57 and the second AF pixel 58, and the vertical axis represents the output value of each of the first AF pixel 57 and the second AF pixel 58. (See reference) Figure 6B and Figure 6C The shape of the continuous output values ​​of the first AF pixel 57 is substantially the same as the shape of the continuous output values ​​of the second AF pixel 58. However, the positions (e.g., phases) of the output values ​​of the first AF pixel 57 and the second AF pixel 58 may differ from each other, such as... Figure 6BAs shown. This is because the positions of the light provided by the pupils 52 and 53 of lens 51 to form the image are different from each other. Therefore, if lens 51 is not focused, the phase of the output value of the first AF pixel 57 is different from the phase of the output value of the second AF pixel 58, as shown. Figure 6B As shown. If lens 51 is focused, the image is formed in the same position, as... Figure 6C As shown.

[0089] Furthermore, the direction of the focus difference can be determined based on the difference between the phase of the output value of the first AF pixel 57 and the phase of the output value of the second AF pixel 58. The forward focusing state indicates that the lens 51 is focused in front of the target. In the forward focusing state, the phase of the output value of the first AF pixel 57 shifts to the left from the phase of the focusing state, and the phase of the output value of the second AF pixel 58 shifts to the right from the phase of the focusing state. On the other hand, the rear focusing state indicates that the lens 51 is focused behind the target. In the rear focusing state, the phase of the output value of the first AF pixel 57 shifts to the right from the phase of the focusing state, and the phase of the output value of the second AF pixel 58 shifts to the left from the phase of the focusing state. The offset between the phases of the output values ​​of the first AF pixel 57 and the second AF pixel 58 can be used to obtain the amount of focus deviation.

[0090] Figure 7A , Figure 7B , Figure 7C , Figure 8A , Figure 8B and Figure 9 This is a diagram used to describe the operation of an image processing device according to an example embodiment. Figure 7A , Figure 7B , Figure 7C , Figure 8A , Figure 8B and Figure 9 This shows that when the pixel array of an autofocus image sensor has Figure 5A , Figure 5B and Figure 5C The operation when the structure is shown.

[0091] refer to Figure 7AThe first image frame data FDAT11, provided by the autofocus image sensor 500 and corresponding to an image frame, may include: multiple pixel values ​​PV11W, PV12G, PV13W, PV14B, PV15W, PV16G, PV21G, P ... V22W, PV23B, PV24W, PV25G, PV26W, PV31W, PV32R, PV33G, PV34G, PV35W, PV36R, PV41R, PV42W, PV43G, PV44W, PV45R, PV46W, PV51W, PV52G, PV53W, PV54B, PV55W, PV56G, PV61G, PV62W, PV63B, PV64W, PV65G, and PV66W.

[0092] The pixel values ​​PV32R, PV36R, PV41R, and PV45R obtained from pixels PX32, PX36, PX41, and PX45, which include the red filter R, correspond to red. The pixel values ​​PV12G, PV16G, PV21G, PV25G, PV33G, PV34G, PV43G, PV52G, PV56G, PV61G, and PV65G, which include the green filter G, correspond to green. The pixel values ​​PV14B, PV23B, PV54B, and PV63B, which include the blue filter B, correspond to blue. The pixel values ​​PV11W, PV13W, PV15W, PV22W, PV24W, PV26W, PV31W, PV35W, PV42W, PV44W, PV46W, PX51, PX53, PX55, PX62, PX64, and PX66, which include a white color filter W, can be associated with white. A single pixel value can be generated by a photoelectric conversion area; therefore, two pixel values, PV33G and PV34G, can be obtained from pixel PX33, which is an AF pixel.

[0093] In some embodiments, the color filters disposed on AF pixels are not used to form colors (e.g., not to generate pixel values ​​corresponding to a specific color), and the color filters may be disposed on AF pixels merely for the convenience of manufacturing the color filter array. Therefore, the color filters disposed on AF pixels can be described as dummy color filters and may not correspond to the color patterns of the remaining pixels in the pixel array. Furthermore, for the accuracy of AF information, a color filter with a single color can be disposed on a single AF pixel. However, for the accuracy of image information, it may be useful for the output of a color filter array used for processing image pixels to have the same color filter arrangement structure as when using an image pixel array without AF pixels. Therefore, it may be advantageous to correct pixel values ​​PV33G, PV34G, PV43G, and PV44W to correspond to the WGGW pattern (e.g., changing the pixel value PV33G corresponding to green to correspond to white) for image processing purposes other than autofocus.

[0094] refer to Figure 7B The first correction information CINF11, stored in memory 120 and used to correct the first image frame data FDAT11, may include correction data CI33 used to correct the pixel value PV33G. For example, the pixel value PV33G may be corrected to allow the pixel to be used for both autofocus and as an image pixel.

[0095] In some example embodiments, the correction data CI33 may include a gain (or gain value) used to convert the pixel value PV33G corresponding to green into a corrected pixel value corresponding to white (e.g., Figure 7C The correction data includes the corrected pixel value PV33Wc; and location data representing the location (or region) of pixel PX33. When correcting multiple pixel data for multiple pixels, the correction data may include multiple gains and multiple corresponding location data. For example, different pixels within the same image sensor may have colors corrected using the same or different gains.

[0096] For example, Formula 1 below can be used to convert the pixel value PV33G corresponding to green into the corrected pixel value PV33Wc corresponding to white.

[0097] [Formula 1]

[0098] WV=α1·GV+β1·RV+γ1·BV

[0099] In Formula 1, "WV" represents the corrected pixel value PV33Wc corresponding to white as the result of the correction operation, "GV" represents the pixel value PV33G corresponding to green as the object of the correction operation, "RV" represents the pixel value obtained from the pixel adjacent to pixel PX33 and corresponding to red for the correction operation, and "BV" represents the pixel value obtained from the pixel adjacent to pixel PX33 and corresponding to blue for the correction operation. Values ​​α1, β1, and γ1 represent the gain. Therefore, multiple gains and multiple location data can be used to correct pixel values.

[0100] When performing the correction operation based on Formula 1, not only the pixel value PV33G obtained from pixel PX33, but also the neighboring pixel values ​​obtained from the neighboring pixels adjacent to pixel PX33 can be used together to generate the corrected pixel value PV33Wc. For example, the pixel value PV32R obtained from pixel PX32 adjacent to pixel PX33 and corresponding to red can be used as the value "RV" in Formula 1, and the pixel value PV23B obtained from pixel PX23 adjacent to pixel PX33 and corresponding to blue can be used as the value "BV" in Formula 1. In some example embodiments, the pixel value PV34G obtained from pixel PX34 adjacent to pixel PX33 and corresponding to green and / or the pixel value PV43G obtained from pixel PX43 adjacent to pixel PX33 and corresponding to green can also be used as the neighboring pixel value.

[0101] For example, when the position of pixel PX11 is defined as (1, 1) and the position of pixel PX66 is defined as (6, 6), the position data representing the position of pixel PX33 may include a position value corresponding to (3, 3), which is the position of the photoelectric conversion area that generates the pixel value PV33G to be corrected (e.g., the object of the correction operation) among the pixel values ​​PV33G and PV34G obtained from pixel PX33.

[0102] In other example embodiments, the correction data CI33 may include gain and position data, and may also include a bias (or bias value) used to convert the pixel value PV33G corresponding to green into the corrected pixel value PV33Wc corresponding to white.

[0103] For example, Formula 2 below can be used to convert the pixel value PV33G corresponding to green into the corrected pixel value PV33Wc corresponding to white.

[0104] [Formula 2]

[0105] WV=α2·GV+β2

[0106] In Formula 2, "WV" represents the corrected pixel value PV33Wc corresponding to white as the result of the correction operation, and "GV" represents the pixel value PV33G corresponding to green as the object of the correction operation. The value α2 represents the gain, and the value β2 represents the bias.

[0107] When the correction operation is performed based on Formula 2, only the pixel value PV33G obtained from pixel PX33 is used to generate the corrected pixel value PV33Wc.

[0108] In some other example embodiments, the correction data CI33 may include gain, bias, and position data, and may also include at least one additional data for correcting pixel values.

[0109] exist Figure 7B In the blank areas, the portions indicated, for example, those that do not include or describe correction data, can be regions without correction data, such as regions where correction operations are unnecessary, unrequired, or not used. The first image frame data FDAT11 may include... Figure 7A The pixel values ​​of all pixels are shown; however, the first correction information CINF11 may only include correction data CI33 for the pixels that have undergone the correction operation. Therefore, the size (or data volume) of the first correction information CINF11 may be smaller than the size of the first image frame data FDAT11.

[0110] refer to Figure 7C The first corrected image frame data CFDAT11 can be obtained through a method based on... Figure 7B First calibration information CINF11 calibration Figure 7A The first image frame data CFDAT11 is generated. For example, the first corrected image frame data CFDAT11 can be generated by correcting (or converting) the pixel value PV33G corresponding to green to the corrected pixel value PV33Wc corresponding to white based on the correction data CI33.

[0111] The remaining pixel values ​​PV11W, PV12G, PV13W, PV14B, PV15W, PV16G, PV21G, PV22W, PV23B, PV24W, PV25G, PV26W, PV31W, PV32R, PV34G, PV35W, PV36R, PV41R, PV42W, PV43G, PV44W, PV45R, PV46W, PV51W, PV52G, PV53W, PV54B, PV55W, PV56G, PV61G, PV62W, PV63B, PV64W, PV65G, and PV66W included in the first corrected image frame data CFDAT11 are... The pixel values ​​PV11W, PV12G, PV13W, PV14B, PV15W, PV16G, PV21G, PV22W, PV23B, PV24W, PV25G, PV26W, PV31W, PV32R, PV34G, PV35W, PV36R, PV41R, PV42W, PV43G, PV44W, PV45R, PV46W, PV51W, PV52G, PV53W, PV54B, PV55W, PV56G, PV61G, PV62W, PV63B, PV64W, PV65G, and PV66W are substantially the same as those included in the first image frame data FDAT11. Figure 7C The pixel values ​​shown can therefore represent the pixel values ​​obtained by... Figures 5A-5C The data of the pixel-sensed image (or part of the image).

[0112] refer to Figure 8A In conventional operations, the calibration operation CAL, used to generate correction information, is performed based on the first image frame data FDAT11 to generate calibrated first image frame data FDAT11′. The first corrected image frame data CFDAT11 is then generated by performing a correction operation COR based on the calibrated first image frame data FDAT11′. In this example, two operations, CAL and COR, are performed. Data corresponding to an image frame is calculated and processed for each operation, resulting in significant computational load, computation time, and power consumption.

[0113] refer to Figure 8B In the operation according to the example embodiment, it is possible to obtain and store (e.g., as for) in advance. Figures 7A-7CThe first correction information CINF11 (described in the example) used for the correction operation COR can be loaded when the first image frame data FDAT11 is received. The stored first correction information CINF11 can be loaded, and the first corrected image frame data CFDAT11 can be generated by performing the correction operation COR based on the loaded first correction information CINF11. In this example, only one operation including the correction operation COR is performed, and a relatively small first correction information CINF11 is used during the correction operation COR, thus reducing computational load, computation time, and power consumption.

[0114] refer to Figure 9 An example is shown where multiple corrected image frame data CF1, CF2, CF3, CF4, F5, F6, F7, F8, F9, and F10 are sequentially generated when multiple image frame data F1, F2, F3, F4, F5, F6, F7, F8, F9, and F10 are sequentially received from an autofocus image sensor 500.

[0115] exist Figure 9 In the example, each of the multiple image frame data F1 to F10 can be compared with... Figure 7A and Figure 8B The first image frame data FDAT11 corresponds to the first correction information CINF11, which can be associated with... Figure 7B and Figure 8B The first correction information CINF11 corresponds to this information, and each of the plurality of corrected image frame data CF1 to CF10 can be associated with... Figure 7C and Figure 8B The first corrected image frame data in the image corresponds to CFDAT11.

[0116] Reference Figure 8B Similarly, even when receiving multiple image frame data F1 to F10 sequentially, pre-stored first correction information CINF11 can be loaded, and multiple corrected image frame data CF1 to CF10 can be sequentially generated by performing correction operations COR based on the loaded first correction information CINF11. Therefore, compared with the conventional operation of performing calibration operation CAL and correction operation COR for each frame (or every frame), the amount of computation, computation time, power consumption, etc., can be reduced.

[0117] Although Figure 9An example is shown that includes ten image frame data F1 to F10 and ten corrected image frame data CF1 to CF10, but the example embodiment is not limited thereto. For example, when the first image frame data to the Nth image frame data are received sequentially from the autofocus image sensor 500 (where N is a natural number greater than or equal to 2), the first corrected image frame data to the Nth corrected image frame data can be generated sequentially by correcting the first image frame data to the Nth image frame data based on the first correction information CINF11.

[0118] Although reference Figure 5A , Figure 5B , Figure 5C , Figure 6A , Figure 6B , Figure 6C , Figure 7A , Figure 7B , Figure 7C , Figure 8A , Figure 8B and Figure 9 The example embodiments are described based on specific structures of pixel arrays, specific structures of pixels, and specific colors, but the example embodiments are not limited thereto. For example, the example embodiments can be applied or used when generating pixel values ​​based on AF pixels that share a microlens and a color filter by two or more photoelectric conversion regions, and when at least one pixel value corresponding to any color is corrected to or converted to correspond to any other color.

[0119] Figure 10 This is a block diagram illustrating an image processing apparatus according to an example embodiment. (The symbols "and" are omitted.) Figure 1 Repeated description.

[0120] refer to Figure 10 The image processing device 100a includes a memory 120 and a color correction circuit 140. The image processing device 100a also includes a calibration circuit 110.

[0121] In addition to the calibration circuit 110, the image processing device 100a can also be connected to... Figure 1 The image processing equipment 100 is basically the same.

[0122] The calibration circuit 110 can generate first correction information CINF1 stored in the memory 120. For example, the calibration circuit 110 can generate the first correction information CINF1 based on the first raw image data ORDAT1 corresponding to the reference pattern and the first reference image frame data CRDAT1.

[0123] The reference pattern can represent a reference image used to generate the first correction information CINF1 (e.g., for correcting pixel values ​​for gain, bias, etc.). For example, the reference pattern can be a uniform (or consistent) image (e.g., a white image or a green image) corresponding to a single grayscale value. The reference pattern can include, for example, a specific pattern (e.g., stripes). In some embodiments, the first raw image data ORDAT1 represents the raw data of the reference pattern (e.g., data from a white image or a green image), and the first reference image frame data CRDAT1 represents a frame image obtained by capturing the reference pattern using an autofocus image sensor.

[0124] In some example embodiments, the calibration circuit 110 may generate first calibration information CINF1 in advance and store it in the memory 120 before the autofocus image sensor is operating normally. Therefore, the calibration circuit 110 may be referred to as a preprocessing unit.

[0125] In some example embodiments, the calibration circuit 110 can be disabled or deactivated during normal operation of the autofocus image sensor. For example, the calibration circuit 110 can be enabled or activated during the initial operation of the image processing device 100a or during the manufacturing process of the image processing device 100a to generate first correction information CINF1, and can then be disabled or deactivated and can not operate during normal operation of the autofocus image sensor. For example, the calibration circuit 110 can be enabled or disabled based on the enable signal EN.

[0126] Figure 11 It is shown Figure 10 A block diagram of an example calibration circuit included in an image processing device.

[0127] refer to Figure 11 The calibration circuit 110a may include a comparator 112 and a calculator 114. Each of the comparator 112 and the calculator 114 may be a circuit or may be implemented by computer program code executed by hardware.

[0128] Comparator 112 can receive first raw image data ORDAT1 corresponding to the reference pattern and first reference image frame data CRDAT1 obtained by capturing the reference pattern by an autofocus image sensor, and can compare the first raw image data ORDAT1 with the first reference image frame data CRDAT1 to generate first comparison result data CR1.

[0129] Calculator 114 can calculate first correction information CINF1 based on first comparison result data CR1, which is the result of comparing first original image data ORDAT1 with first reference image frame data CRDAT1. For example, calculator 114 can calculate the relationship between a first color corresponding to the target of the correction operation and a second color corresponding to the result of the image capture operation. For example, calculator 114 can obtain the gain, bias, etc., described in Formulas 1 and 2 as the result of the calculation. For example, the first original image data ORDAT1 can correspond to the target color (e.g., white) to be achieved by color correction, and the first reference image frame data CRDAT1 can correspond to the actual color (e.g., green) received by the image sensor (e.g., by the AF pixel) when the target color is applied to the image sensor (e.g., AF pixel). The gain, bias, etc., between the target color and the actual color can be used later for color correction during operation.

[0130] Figure 12 and Figure 13 This is a block diagram illustrating an image processing apparatus according to an example embodiment. (The symbols "and" are omitted.) Figure 1 Repeated description.

[0131] refer to Figure 12 The image processing device 100b includes a memory 120 and a color correction circuit 140. The image processing device 100b may also include a selector 160. The selector 160 may be a circuit or may be implemented via computer program code executed by hardware.

[0132] Image processing device 100b can be with Figure 1 The image processing device 100b is basically the same as the image processing device 100, but the memory 120 also stores the second correction information CINF2 and the image processing device 100b also includes a selector 160.

[0133] The memory 120 can store and output second correction information CINF2. Similar to the first correction information CINF1, the second correction information CINF2 can be used to correct a first pixel value corresponding to a first color from a plurality of pixel values ​​received from an autofocus image sensor, and can be used to correct the first pixel value to correspond to a second color different from the first color. Furthermore, the second correction information CINF2 can have a value different from that of the first correction information CINF1.

[0134] In some example embodiments, the first correction information CINF1 and the second correction information CINF2 may correspond to different color temperatures. For example, the first correction information CINF1 may correspond to a first color temperature, and the second correction information CINF2 may correspond to a second color temperature different from the first color temperature. However, the example embodiments are not limited thereto, and the first correction information CINF1 and the second correction information CINF2 may correspond to other different parameters associated with or related to color.

[0135] Selector 160 can select one of first correction information CINF1 and second correction information CINF2 based on the first image frame data FDAT1, and can output the selected correction information SINF. Color correction circuit 140 can generate first corrected image frame data CFDAT1 by correcting the first image frame data FDAT1 based on the selected correction information SINF.

[0136] In some example embodiments, when the first correction information CINF1 and the second correction information CINF2 correspond to different color temperatures, the selector 160 may include a color temperature determiner. For example, the color temperature determiner may determine the color temperature of the first image frame data FDAT1, selecting the first correction information CINF1 when the color temperature of the first image frame data FDAT1 matches or is consistent with a first color temperature, and selecting the second correction information CINF2 when the color temperature of the first image frame data FDAT1 matches a second color temperature. As another example, when the color temperature of the first image frame data FDAT1 does not match either the first color temperature or the second color temperature, the color temperature determiner may select the one of the first correction information CINF1 and the second correction information CINF2 that is closer to the color temperature of the first image frame data FDAT1.

[0137] Although Figure 12 An example is shown where memory 120 stores two correction information entries, CINF1 and CINF2, and selector 160 selects one of the two correction information entries, CINF1 and CINF2. However, the example embodiment is not limited to this. For example, memory 120 may store three or more correction information entries, and selector 160 may select one of the three or more correction information entries.

[0138] refer to Figure 13 The image processing device 100c includes a memory 120 and a color correction circuit 140. The image processing device 100c may also include a calibration circuit 110 and a selector 160.

[0139] Image processing device 100c can be with Figure 1The image processing device 100c is basically the same as the image processing device 100, but the memory 120 also stores the second correction information CINF2, and the image processing device 100c also includes a calibration circuit 110 and a selector 160. The calibration circuit 110 can be connected to... Figure 10 The calibration circuit 110 is similar to that in the example. The memory 120 and selector 160 can be respectively connected to... Figure 12 The memory 120 and selector 160 are basically the same.

[0140] The calibration circuit 110 can generate first correction information CINF1 based on first raw image data ORDAT1 and first reference image frame data CRDAT1, and can generate second correction information CINF2 based on second raw image data ORDAT2 and second reference image frame data CRDAT2. For example, when the first correction information CINF1 corresponds to a first color temperature, the first raw image data ORDAT1 can represent a reference pattern corresponding to the first color temperature, and the first reference image frame data CRDAT1 can represent a frame image obtained by capturing the reference pattern corresponding to the first color temperature (or by capturing the reference pattern at the first color temperature). When the second correction information CINF2 corresponds to a second color temperature, the second raw image data ORDAT2 can represent a reference pattern corresponding to the second color temperature, and the second reference image frame data CRDAT2 can represent a frame image obtained by capturing the reference pattern corresponding to the second color temperature (or by capturing the reference pattern at the second color temperature).

[0141] Figure 14 This is a block diagram illustrating an image processing apparatus according to an example embodiment. (The symbols "and" are omitted.) Figure 1 Repeated description.

[0142] refer to Figure 14 The image processing device 300 includes a memory 320 and a color correction circuit 340.

[0143] Image processing device 300 can be with Figure 1 The image processing device 100 is basically the same, but the image processing device 300 performs correction operations on image frames received from an image sensor that is different from an autofocus image sensor.

[0144] Memory 320 stores and outputs first correction information CINFA. The first correction information CINFA is used to correct image sensors located outside the image processing device 300 (e.g., [image sensor]). Figure 15a first pixel value among a plurality of pixel values received by the image sensor 420 in ). Different from an autofocus image sensor, the image sensor only includes a plurality of pixels (e.g., normal pixels) for detecting an image, the first pixel value is obtained from a first pixel among the plurality of pixels and corresponds to a first color, and first correction information CINFA is used to correct the first pixel value to correspond to a second color different from the first color.

[0145] The color correction circuit 340 receives first image frame data FDATA including a plurality of pixel values from the image sensor, loads the first correction information CINFA from the memory 320, and generates first corrected image frame data CFDATA based on the first correction information CINFA and the first image frame data FDATA. For example, the color correction circuit 340 generates the first corrected image frame data CFDATA by correcting the first pixel value included in the first image frame data FDATA to correspond to the second color based on the first correction information CINFA.

[0146] The image processing apparatus 300 according to the example embodiment may correct or convert pixel values of pixels having a specific color included in an image sensor into different colors. The image processing apparatus 300 may obtain correction information CINFA for a correction operation in advance, store the correction information CINFA in the memory 320 in advance, load the correction information CINFA for each image frame, and perform a correction operation based on the loaded correction information CINFA, instead of generating and calculating correction information for a correction operation for each image frame. Therefore, the amount of calculation, calculation time, power consumption and the like for the correction operation can be reduced, and the image processing apparatus 300 can have improved or better performance.

[0147] In some example embodiments, the image processing apparatus 300 may also as described with reference to Figure 10 include a calibration circuit as described, may also as described with reference to Figure 12 include a selector as described, and / or may also as described with reference to Figure 13 include both a calibration circuit and a selector as described. Such a correction operation may be performed on a single pixel or a pixel group within a specific pixel array.

[0148] Figure 15 is a block diagram illustrating an image processing system according to an example embodiment. Descriptions repeated from Figure 1 will be omitted.

[0149] Referring to Figure 15 , the image processing system 400 includes an image sensor 420 and an image processing apparatus 440.

[0150] Image sensor 420 sequentially outputs multiple image frame data (FDAT) based on incident light, including information associated with the image of the target. Image sensor 420 can be implemented with reference to... Figures 3 to 6C The image sensor described is similar, and the structure of the pixel array can be partially changed because the image sensor 420 does not include AF pixels but only normal pixels.

[0151] Image processing apparatus 440 sequentially receives multiple image frame data FDATs and sequentially generates multiple corrected image frame data CFDATs by sequentially correcting the multiple image frame data FDATs. Image processing apparatus 440 can be implemented according to an example embodiment. For example, image processing apparatus 440 could be... Figure 14 Image processing device 300.

[0152] Figure 16 yes Figure 15 A planar diagram of an example pixel array included in an image sensor within an image processing system. (The remaining text will be omitted.) Figure 5A Repeated description.

[0153] refer to Figure 16 The pixel array 514 included in the image sensor may include multiple pixels PXA1, PXA2, PXA3, PXA4, PXA5, PXA6, PXB1, PXB2, PXB3, PXB4, PXB5, PXB6, PXC1, PXC2, PXC3, PXC4, PXC5, PXC6, PXD1, PXD2, PXD3, PXD4, PXD5, PXD6, PXE1, PXE2, PXE3, PXE4, PXE5, PXE6, PXF1, PXF2, PXF3, PXF4, PXF5, and PXF6.

[0154] Multiple pixels PXA1 to PXA6, PXB1 to PXB6, PXC1 to PXC6, PXD1 to PXD6, PXE1 to PXE6, and all pixels PXF1 to PXF6 can be normal pixels of the detected image, and microlenses ML can be placed on them. Each of the multiple pixels PXA1 to PXA6, PXB1 to PXB6, PXC1 to PXC6, PXD1 to PXD6, PXE1 to PXE6, and PXF1 to PXF6 can be implemented as Figure 5B The second pixel NPX in the image.

[0155] Figure 17A , Figure 17B , Figure 17C , Figure 17D , Figure 18A , Figure 18B , Figure 18C and Figure 18DThis is a diagram used to describe the operation of an image processing device according to an example embodiment. (The symbols and symbols are omitted.) Figure 5C , Figure 7A , Figure 7B and Figure 7C Repeated description.

[0156] Figure 17A yes Figure 16 A plan view of an example color filter array included in pixel array 514. Figure 17B , Figure 17C and Figure 17D This shows that when the pixel array of the image sensor has Figure 16 and Figure 17A The operation when the structure is shown. Figure 18A yes Figure 16 A plan view of another example of the color filter array included in the pixel array 514. Figure 18B , Figure 18C and Figure 18D This shows that when the pixel array of the image sensor has Figure 16 and Figure 18A The operation when the structure is shown.

[0157] refer to Figure 17A The color filter array CFAA1 included in pixel array 514 may include a red color filter R, a green color filter G, a blue color filter B, and a white color filter W. Pixels PXA2, PXA4, PXA6, PXC2, PXC4, PXC6, PXE2, PXE4, and PXE6 may include a red color filter R. Pixels PXA1, PXA3, PXA5, PXC1, PXC3, PXC5, PXE1, PXE3, and PXE5 may include a green color filter G. Pixels PXB1, PXB3, PXB5, PXD1, PXD3, PXD5, PXF1, PXF3, and PXF5 may include a blue color filter B. Pixels PXB2, PXB4, PXB6, PXD2, PXD4, PXD6, PXF2, PXF4, and PXF6 may include a white color filter W.

[0158] refer to Figure 17BThe first image frame data FDATA1, provided by the image sensor 420 and corresponding to an image frame, may include: pixel values ​​PVA2R, PVA4R, PVA6R, PVC2R, PVC4R, PVC6R, PVE2R, PVE4R, and PVE6R obtained from pixels PXA2, PXA4, PXA6, PXC2, PXC4, PXE2, PXE4, and PXE6 including the red filter R and corresponding to red; and pixel values ​​PVA1G, PVA3G, PVA5G, PVC1G, PVC3G, and PVC5G obtained from pixels PXA1, PXA3, PXA5, PXC1, PXC3, PXC5, PXE1, PXE3, and PXE5 including the green filter G and corresponding to green. PVE1G, PVE3G, and PVE5G; pixel values ​​PVB1B, PVB3B, PVB5B, PVD1B, PVD3B, PVD5B, PVF1B, PVF3B, and PVF5B corresponding to blue, obtained from pixels PXB1, PXB3, PXB5, PXD1, PXD3, PXD5, PXF1, PXF3B, and PVF5B, including the blue filter B; and pixel values ​​PVB2W, PVB4W, PVB6W, PVD2W, PVD4W, PVD6W, PVF2W, PVF4W, and PVF6W, corresponding to white, obtained from pixels PXB2, PXB4, PXB6, PXD2, PXD4, PXD6, PXF2, PXF4W, and PXF6, including the white filter W.

[0159] refer to Figure 17C The first correction information CINFA1 stored in memory 320 and used to correct the first image frame data FDATA1 may include correction data CIB2, CIB4, CIB6, CID2, CID4, CID6, CIF2, CIF4, and CIF6 for correcting pixel values ​​PVB2W, PVB4W, PVB6W, PVD2W, PVD4W, PVD6W, PVF2W, PVF4W, and PVF6W.

[0160] refer to Figure 17D It can be based on Figure 17C The first correction information CINFA1 will Figure 17BThe pixel values ​​PVB2W, PVB4W, PVB6W, PVD2W, PVD4W, PVD6W, PVF2W, PVF4W, and PVF6W included in the first image frame data FDATA1 and corresponding to white are corrected to the corrected pixel values ​​PVB2Gc, PVB4Gc, PVB6Gc, PVD2Gc, PVD4Gc, PVD6Gc, PVF2Gc, PVF4Gc, and PVF6Gc corresponding to green, to generate the first corrected image frame data CFDATA1. For example, an RGBW pattern can be converted into a Bayer pattern.

[0161] refer to Figure 18A The color filter array CFAA2 included in pixel array 514 may include a red color filter R, a green color filter G, a blue color filter B, and a white color filter W. Pixels PXC2, PXC6, PXD1, and PXD5 may include the red color filter R. Pixels PXA2, PXA6, PXB1, PXB5, PXC4, PXD3, PXE2, PXE6, PXF1, and PXF5 may include the green color filter G. Pixels PXA4, PXB3, PXE4, and PXF3 may include the blue color filter B. Pixels PXA1, PXA3, PXA5, PXB2, PXB4, PXB6, PXC1, PXC3, PXC5, PXD2, PXD4, PXD6, PXE1, PXE3, PXE5, PXF2, PXF4, and PXF6 may include a white color filter W.

[0162] refer to Figure 18BThe first image frame data FDATA2, provided by the image sensor 420 and corresponding to an image frame, may include: pixel values ​​PVJ2R, PVJ6R, PVK1R, and PVK5R obtained from pixels PXC2, PXC6, PXD1, and PXD5 including the red filter R and corresponding to red; pixel values ​​PVG2G, PVG6G, PVH1G, PVH5G, PVJ4G, PVK3G, PVL2G, PVL6G, PVM1G, and PVM5G obtained from pixels PXA2, PXA6, PXB1, PXB5, PXC4, PXD3, PXE2, PXE6, PXF1, and PXF5 including the green filter G and corresponding to green; and pixel values ​​PVG2G, PVG6G, PVH1G, PVH5G, PVJ4G, PVK3G, PVL2G, PVL6G, PVM1G, and PVM5G obtained from pixels PXA4, PXB3, PXC6, PXC6, PXC7, PXC8, PXC9, PXC1, PXC9 ... Pixel values ​​PVG4B, PVH3B, PVL4B, and PVM3B, obtained from E4 and PXF3 and corresponding to blue; and pixel values ​​PVG1W, PVG3W, PVG5W, PVH2W, PVH4W, PVH6W, PVJ1W, PVJ3W, PVJ5W, PVK2W, PVK4W, PVK6W, PVL1W, PVL3W, PVL5W, PVM2W, PVM4W, and PVM6W, obtained from pixels including the white filter W, PXA1, PXA3, PXA5, PXB2, PXB4, PXB6, PXC1, PXC3, PXC5, PXD2, PXD4, PXD6, PXE1, PXE3, PXE5, PXF2, PXF4, and PXF6 and corresponding to white.

[0163] refer to Figure 18C The first correction information CINFA2, stored in memory 320 and used to correct the first image frame data FDATA2, may include: correction data CIG1, CIG3, CIG5, CIH2, CIH4, CIH6, CIJ1, CIJ3, CIJ5, CIK2, CIK4, CIK6, CIM2, CIM4, and CIM6 for correcting pixel values ​​PVG1W, PVG3W, PVG5W, PVH2W, PVH4W, PVH6W, PVM2W, PVM4W, and PVM6W.

[0164] refer to Figure 18D , can be based on Figure 18C The first correction information CINFA2, through Figure 18BThe pixel values ​​PVJ1W, PVJ5W, PVK2W, and PVK6W included in the first image frame data FDATA2 and corresponding to white are corrected to the corrected pixel values ​​PVJ1Rc, PVJ5Rc, PVK2Rc, and PVK6Rc corresponding to red, by... Figure 18B The pixel values ​​PVG1W, PVG5W, PVH2W, PVH6W, PVJ3W, PVK4W, PVL1W, PVL5W, PVM2W, and PVM6W included in the first image frame data FDATA2 and corresponding to white are corrected to the corrected pixel values ​​PVG1Gc, PVG5Gc, PVH2Gc, PVH6Gc, PVJ3Gc, PVK4Gc, PVL1Gc, PVL5Gc, PVM2Gc, and PVM6Gc corresponding to green, and by... Figure 18B The pixel values ​​PVG3W, PVH4W, PVL3W, and PVM4W included in the first image frame data FDATA2 and corresponding to white are corrected to the corrected pixel values ​​PVG3Bc, PVH4Bc, PVL3Bc, and PVM4Bc corresponding to green, to generate the first corrected image frame data CFDATA2. For example, an RGBW pattern can be converted into a quaternion pattern.

[0165] Although reference Figure 16 , Figure 17A , Figure 17B , Figure 17C , Figure 17D , Figure 18A , Figure 18B , Figure 18C and Figure 18D The example embodiments are described based on the specific structure of the pixel array, the specific structure of the pixels, and the specific color, but the example embodiments are not limited thereto. For example, the example embodiments can be applied or used when the image sensor includes a pixel array of arbitrary structure and when at least one pixel value corresponding to any color is corrected or converted to correspond to any other color.

[0166] Figure 19 This is a flowchart illustrating an image processing method according to an example embodiment.

[0167] refer to Figure 1 , Figure 2 , Figure 10 and Figure 19In the image processing method according to the example embodiment, first correction information CINF1 is generated (step S100). As described above, the first correction information CINF1 is used to correct a first pixel value among a plurality of pixel values ​​received from the autofocus image sensor 220. The autofocus image sensor 220 includes a first pixel for detecting a phase difference and a second pixel for detecting an image. The first pixel value is obtained from the first pixel and corresponds to a first color, and the first correction information CINF1 is used to correct the first pixel value (e.g., at least one pixel value) to correspond to a second color different from the first color. Step S100 can be performed by a calibration circuit 110 disposed inside or outside the image processing device.

[0168] First correction information CINF1 is stored in memory 120 (step S200). First image frame data FDAT1, including multiple pixel values, is received from autofocus image sensor 220 (step S300). First correction information CINF1 is loaded from memory 120 (step S400). First corrected image frame data CFDAT1 is generated by correcting the first pixel values ​​included in the first image frame data FDAT1 to correspond to the second color based on the first correction information CINF1 (step S500). Steps S300, S400, and S500 can be performed by color correction circuit 140.

[0169] Figure 20 It is shown Figure 19 The flowchart shows an example of generating the first correction information.

[0170] refer to Figure 2 , Figure 10 , Figure 11 , Figure 19 and Figure 20 When the first correction information CINF1 is generated (step S100), the first reference image frame data CRDAT1 obtained by capturing a reference pattern by the autofocus image sensor 220 can be received (step S110). The first reference image frame data CRDAT1 can be compared with the first original image data ORDAT1 corresponding to the reference pattern (step S120). The first correction information CINF1 can be calculated based on the result of comparing the first original image data ORDAT1 with the first reference image frame data CRDAT1 (step S130). Steps S110 and S120 can be performed by the comparator 112, and step S130 can be performed by the calculator 114.

[0171] Figure 21 and Figure 22 This is a flowchart illustrating an image processing method according to an example embodiment. (The remaining text is incomplete and likely refers to a separate topic.) Figure 19 Repeated description.

[0172] refer to Figure 1 , Figure 2 and Figure 21 In the image processing method according to the example embodiment, steps S100, S200, and S400 can be respectively connected with... Figure 19 Steps S100, S200 and S400 are basically the same.

[0173] The first image frame data to the Nth image frame data are received sequentially from the autofocus image sensor 220 (step S350). The first corrected image frame data to the Nth corrected image frame data are generated sequentially by correcting the first pixel values ​​included in the first image frame data to the Nth image frame data to correspond to the second color based on the first correction information CINF11 (step S550). Figure 21 An example can also be described as, in Figure 19 After step S300, the second image frame data to the Nth image frame data are received sequentially, and through... Figure 19 After step S500, the second image frame data to the Nth image frame data are corrected sequentially to generate the second corrected image frame data to the Nth corrected image frame data.

[0174] refer to Figure 2 , Figure 12 , Figure 13 and Figure 22 In the image processing method according to the example embodiment, first correction information CINF1 and second correction information CINF2 are generated (step S150). The first correction information CINF1 and the second correction information CINF2 are used to correct a first pixel value corresponding to a first color to correspond to a second color, and they are different from each other. The first correction information CINF1 and the second correction information CINF2 are stored in memory 120 (step S250). Steps S150 and S250 can be respectively connected to… Figure 19 Steps S100 and S200 are similar.

[0175] First image frame data FDAT1 is received from the autofocus image sensor 220 (step S300). First correction information CINF1 and second correction information CINF2 are loaded from the memory 120 (step S450). One of the first correction information CINF1 and the second correction information CINF2 is selected based on the first image frame data FDAT1 (step S600). First corrected image frame data CFDAT1 is generated by correcting the first pixel value included in the first image frame data FDAT1 to correspond to the second color based on the selected correction information CINF (step S700). Steps S300, S450 and S700 can be respectively connected to... Figure 19Steps S300, S400, and S500 are similar. Step S600 can be performed by selector 160.

[0176] Figure 23 This is a flowchart illustrating an image processing method according to an example embodiment. (The remaining text is incomplete and likely refers to a separate topic.) Figure 19 Repeated description.

[0177] refer to Figure 14 , Figure 15 and Figure 23 In the image processing method according to the example embodiment, first correction information CINFA is generated (step S1100). As described above, the first correction information CINFA is used to correct a first pixel value among a plurality of pixel values ​​received from the image sensor 420. The image sensor 420 includes only a plurality of pixels for detecting an image, the first pixel value is obtained from a first pixel among the plurality of pixels and corresponds to a first color, and the first correction information CINFA is used to correct the first pixel value to correspond to a second color different from the first color. Step S1100 can be performed by a calibration circuit disposed inside or outside the image processing device.

[0178] The first correction information CINFA is stored in memory 320 (step S1200). First image frame data FDATA, including multiple pixel values, is received from image sensor 420 (step S1300). The first correction information CINFA is loaded from memory 320 (step S1400). First corrected image frame data CFDATA is generated by correcting the first pixel values ​​included in the first image frame data FDATA to correspond to the second color based on the first correction information CINFA (step S1500). Steps S1300, S1400, and S1500 can be performed by color correction circuit 340.

[0179] As those skilled in the art will recognize, the inventive concept can be implemented using systems, methods, computer program products, and / or computer program products embodied in one or more computer-readable media, on which computer-readable program code is embodied. The computer-readable program code can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be any tangible medium that can contain or store a program used by or associated with an instruction execution system, apparatus, or device. For example, a computer-readable medium can be a non-transitory computer-readable medium. The computer program product can interact with hardware to implement, for example, functions as described by… Figure 11 Comparator 112 and calculator 114 Figure 12 and Figure 13The selector 160, and the other functions performed by the components described herein.

[0180] Figure 24 This is a block diagram illustrating an electronic system including an image processing apparatus according to an example embodiment.

[0181] refer to Figure 24 The electronic system 1000 can be implemented as a data processing device using or supporting a Mobile Industrial Processor Interface (MIPI) interface. The electronic system 1000 may include an application processor 1110, an image sensor 1140, a display device 1150, etc. The electronic system 1000 may also include a radio frequency (RF) chip 1160, a global positioning system (GPS) 1120, a storage device 1170, a microphone (MIC) 1180, dynamic random access memory (DRAM) 1185, and a speaker 1190. Furthermore, the electronic system 1000 can perform communication using ultra-wideband (UWB) 1210, wireless local area network (WLAN) 1220, and Global Microwave Access Interoperability (WIMAX) 1230, etc.

[0182] Application processor 1110 may be a controller or processor that controls the operation of image sensor 1140 and display device 1150.

[0183] Application processor 1110 may include: a Display Serial Interface (DSI) master controller 1111, which performs serial communication with the DSI device 1151 of display device 1150; a Camera Serial Interface (CSI) master controller 1112, which performs serial communication with the CSI device 1141 of image sensor 1140; a Physical Layer (PHY) 1113, which performs data communication with the PHY 1161 of RF chip 1160 based on MIPI DigRF; and a DigRF master device 1114, which controls the data communication of physical layer 1161. The DigRF slave device 1162 of RF chip 1160 can be controlled by DigRF master device 1114.

[0184] In some example embodiments, the DSI master controller 1111 may include a serializer (SER), and the DSI device 1151 may include a deserializer (DES). In some example embodiments, the CSI master controller 1112 may include a deserializer (DES), and the CSI device 1141 may include a serializer (SER).

[0185] CSI master controller 1112 may include an image processing device according to an example embodiment, and CSI master controller 1112 and image sensor 1140 may form an image processing system according to an example embodiment. Image sensor 1140 may be an autofocus image sensor and / or an image sensor included in the image processing system according to an example embodiment. CSI master controller 1112 and image sensor 1140 may operate based on a method according to an example embodiment.

[0186] This invention concept can be applied to a wide range of devices and systems, including image processing equipment and image sensors. For example, it can be applied to systems such as: personal computers (PCs), server computers, data centers, workstations, mobile phones, smartphones, tablets, laptops, personal digital assistants (PDAs), portable multimedia players (PMPs), digital cameras, portable game consoles, music players, camcorders, video players, navigation devices, wearable devices, Internet of Things (IoT) devices, Internet of Everything (IoE) devices, e-book readers, virtual reality (VR) devices, augmented reality (AR) devices, robotic devices, drones, etc.

[0187] The foregoing is illustrative of exemplary embodiments and should not be construed as limiting them. Although some exemplary embodiments have been described, those skilled in the art will readily understand that various modifications can be made to the exemplary embodiments without substantially departing from the novel doctrine and advantages of the exemplary embodiments. Therefore, all such modifications are intended to be included within the scope of the exemplary embodiments as defined in the claims. Accordingly, it should be understood that the foregoing is illustrative of various exemplary embodiments and should not be construed as limiting to the specific exemplary embodiments disclosed, and modifications to the disclosed exemplary embodiments and other exemplary embodiments are intended to be included within the scope of the appended claims.

Claims

1. An image processing apparatus, comprising: A memory is configured to store first correction information for correcting a first pixel value among a plurality of pixel values, the plurality of pixel values ​​being received from an autofocus image sensor including a first pixel configured to detect a phase difference and a second pixel configured to detect an image, the first pixel value being obtained from the first pixel and corresponding to a first color, the first correction information including information used in a calculation to correct the first pixel value to correspond to a second color different from the first color, the memory being configured to further store second correction information different from the first correction information, the second correction information being used to correct the first pixel value to correspond to the second color, wherein the first correction information corresponds to a first color temperature, and the second correction information corresponds to a second color temperature different from the first color temperature; A selector, configured to select one of the first correction information and the second correction information based on a first image frame data including the plurality of pixel values; and A color correction circuit is configured to receive first image frame data from the autofocus image sensor, load first correction information and second correction information from the memory, and, after the selector selects one of the first correction information and the second correction information based on the first image frame data, generate first corrected image frame data by performing the calculation using the first pixel value and the selected correction information as input to correct the first pixel value included in the first image frame data to correspond to the second color. The selector is configured to: determine the color temperature of the first image frame data when performing the selection, and select correction information that matches the color temperature of the first image frame data from the first correction information and the second correction information. The first correction information and the second correction information include: Multiple gains and multiple biases are used to convert a first pixel value corresponding to the first color into a first corrected pixel value corresponding to the second color; and Multiple location data points represent the position of the first pixel. Wherein, the first pixel value is a unique pixel value used to generate the first corrected pixel value, and wherein the first corrected pixel value is generated using the first pixel value, the plurality of gains, and the plurality of biases.

2. The image processing apparatus according to claim 1, further comprising: A calibration circuit is configured to generate the first calibration information stored in the memory.

3. The image processing apparatus according to claim 2, wherein, The calibration circuit includes: A comparator configured to receive reference image frame data obtained by capturing a reference pattern using the autofocus image sensor, and to compare raw image data corresponding to the reference pattern with the reference image frame data; and The calculator is configured to calculate the first correction information based on the result of comparing the original image data with the reference image frame data.

4. The image processing apparatus according to claim 2, wherein, The calibration circuit is configured to generate the first calibration information and store it in the memory before the autofocus image sensor is in normal operation.

5. The image processing apparatus according to claim 4, wherein, The calibration circuit is configured to be disabled when the autofocus image sensor is operating normally.

6. The image processing apparatus according to claim 1, wherein, The color correction circuit is configured as follows: The autofocus image sensor sequentially receives image frame data from the first image frame data, followed by the second image frame data, up to the Nth image frame data, where N is a natural number greater than or equal to 2; and By sequentially correcting the second image frame data to the Nth image frame data based on the selected correction information, the second corrected image frame data to the Nth corrected image frame data are generated sequentially.

7. The image processing apparatus according to claim 1, wherein, Each of the first pixels includes: The first and second photoelectric conversion regions in the substrate; A first color filter, located in the first photoelectric conversion region and the second photoelectric conversion region, and having the first color; and The first microlens is located on the first color filter and is shared by the first photoelectric conversion region and the second photoelectric conversion region.

8. The image processing apparatus according to claim 7, wherein, Each of the second pixels includes: The third photoelectric conversion region in the substrate; A second color filter is located in the third photoelectric conversion region; and The second microlens is located on the second color filter.

9. The image processing apparatus according to claim 8, wherein: The first photoelectric conversion region, the second photoelectric conversion region, and the third photoelectric conversion region have the same size, and The size of the first microlens is larger than the size of the second microlens.

10. An image processing method, comprising: First correction information is generated for correcting a first pixel value among a plurality of pixel values ​​received from an autofocus image sensor including a first pixel configured to detect a phase difference and a second pixel configured to detect an image. The first pixel value is obtained from the first pixel and corresponds to a first color. The first correction information is used to correct the first pixel value to correspond to a second color different from the first color. Second correction information, different from the first correction information, is also generated for correcting the first pixel value to correspond to the second color. The first correction information corresponds to a first color temperature, and the second correction information corresponds to a second color temperature different from the first color temperature. The first correction information and the second correction information are stored in the memory; Receive first image frame data including the plurality of pixel values ​​from the autofocus image sensor; Load the first correction information and the second correction information from the memory; Based on the first image frame data, select one of the first correction information and the second correction information; and By performing calculations using the first pixel value and selected correction information as input, the first pixel value included in the first image frame data is corrected to correspond to the second color, thereby generating first corrected image frame data. The first correction information and the second correction information include: Multiple gains and multiple biases are used to convert a first pixel value corresponding to the first color into a first corrected pixel value corresponding to the second color; and Multiple location data represent the position of the first pixel, wherein the selection includes: determining the color temperature of the first image frame data, and selecting correction information from the first correction information and the second correction information that matches the color temperature of the first image frame data. Wherein, the first pixel value is a unique pixel value used to generate the first corrected pixel value, and wherein the first corrected pixel value is generated using the first pixel value, the plurality of gains, and the plurality of biases.

11. The image processing method according to claim 10, wherein, Generating the first correction information includes: Receive reference image frame data obtained by capturing a reference pattern using the autofocus image sensor; The original image data corresponding to the reference pattern is compared with the reference image frame data; and The first correction information is calculated based on the result of comparing the original image data with the reference image frame data.

12. The image processing method according to claim 10, further comprising: The autofocus image sensor sequentially receives image frame data from the first image frame data to the second image frame data up to the Nth image frame data, where N is a natural number greater than or equal to 2; as well as By sequentially correcting the second image frame data to the Nth image frame data based on the selected correction information, the second corrected image frame data to the Nth corrected image frame data are generated sequentially.

13. An image processing system, comprising: An autofocus image sensor, including a first pixel configured to detect a phase difference and a second pixel configured to detect an image; as well as An image processing device is configured to perform image processing operations on image frame data provided from the autofocus image sensor. The image processing device includes: A calibration circuit is configured to receive reference image frame data obtained by capturing a reference pattern using the autofocus image sensor, compare raw image data corresponding to the reference pattern with the reference image frame data, and calculate first correction information and second correction information based on the result of comparing the raw image data with the reference image frame data. The first correction information and the second correction information are used to correct a first pixel value among a plurality of pixel values ​​received from the autofocus image sensor. The first pixel value is obtained from a first pixel and corresponds to a first color. The first correction information and the second correction information are used to correct the first pixel value to correspond to a second color different from the first color. The first correction information corresponds to a first color temperature, and the second correction information corresponds to a second color temperature different from the first color temperature. A memory configured to receive the first correction information and the second correction information from the calibration circuit and to store the first correction information and the second correction information; A selector, configured to select one of the first correction information and the second correction information based on a first image frame data including the plurality of pixel values; and A color correction circuit is configured to receive first image frame data from the autofocus image sensor, load first correction information and second correction information from the memory, and, after the selector selects one of the first correction information and the second correction information based on the first image frame data, generate first corrected image frame data by performing a calculation using the first pixel value and the selected correction information as input to correct the first pixel value included in the first image frame data to correspond to the second color. The first correction information and the second correction information include: Multiple gains and multiple biases are used to convert a first pixel value corresponding to the first color into a first corrected pixel value corresponding to the second color; and Multiple location data represent the position of the first pixel, and The calibration circuit and the memory are configured to generate and store the first correction information and the second correction information before the autofocus image sensor operates normally. The selector is configured to: determine the color temperature of the first image frame data when performing the selection, and select correction information that matches the color temperature of the first image frame data from the first correction information and the second correction information. Wherein, the first pixel value is a unique pixel value used to generate the first corrected pixel value, and wherein the first corrected pixel value is generated using the first pixel value, the plurality of gains, and the plurality of biases.

Images