Image Processing Apparatus, Image Processing Method, and Image Processing System for Color Correction

Abstract

To provide an image processing device, an image processing method and an image processing system, capable of efficiently correcting pixel values of a specific color output from an auto-focus image sensor.SOLUTION: An AF pixel AFPX includes a first photoelectric conversion region PD1, a second photoelectric conversion region PD2, a first color filter CF1 and a first micro lens ML1. A normal NPX includes a third photoelectric conversion region PD3, a second color filter CF2 and a second micro lens ML2. The first color filter CF1 is a green filter, and the second color filter CF2 is a white filter. Pixel values corresponding to a green color in the first photoelectric conversion region are converted into corrected pixel values corresponding to a white color using pixel values corresponding to a white color in the second photoelectric conversion region.SELECTED DRAWING: Figure 5b

Description

Technical Field

[0001] The present invention relates to a semiconductor integrated circuit, and more particularly to an image processing apparatus for color correction, an image processing method, and an image processing system including the image processing apparatus.

Background Art

[0002] A CMOS (Complementary Metal-Oxide Semiconductor) image sensor is an image pickup device manufactured using a CMOS process. The CMOS image sensor has advantages such as lower manufacturing cost, smaller pixel size, and less power consumption compared to a CCD (charge-coupled device) image sensor including a high-voltage analog circuit. Also, as the performance of the CMOS image sensor improves, the CMOS image sensor is widely used in mobile electronic devices such as smartphones, tablet PCs, or digital cameras.

[0003] In recent years, in a digital image pickup device such as a camera, an autofocus function is provided, and in order to realize this, it is necessary to detect the focus adjustment state of a photographing lens. For the autofocus function, when an element for focus detection is included separately from the image sensor, there is a problem that the cost increases or the overall device size becomes large in manufacturing a digital image pickup device. Therefore, an autofocus image sensor capable of performing both an image pickup function and an autofocus function has been studied. Also, various methods for processing an image signal output from the autofocus image sensor have been studied.

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of the present invention is to provide an image processing apparatus capable of efficiently correcting pixel values of a specific color output from an autofocus image sensor or an image sensor.

[0005] Another object of the present invention is to provide an image processing method capable of efficiently correcting specific color pixel values output from an autofocus image sensor or an image sensor.

[0006] Still another object of the present invention is to provide an image processing system including an image processing apparatus.

Means for Solving the Problems

[0007] To achieve the above object, an image processing apparatus according to an embodiment of the present invention includes a memory and a color correction unit. The memory stores first correction information used to correct a first pixel value corresponding to a first color, which is obtained from a first pixel among a plurality of pixel values received from an autofocus image sensor including a first pixel used for detecting a phase difference and a second pixel used for detecting an image, to correspond to a second color different from the first color. The color correction unit receives first image frame data including the plurality of pixel values from the autofocus image sensor, loads the first correction information from the memory, and corrects the first pixel value included in the first image frame data to correspond to the second color based on the first correction information, thereby generating first corrected image frame data.

[0008] To achieve the other object, the image processing method according to an embodiment of the present invention generates first correction information used to correct a first pixel value obtained from the first pixel and corresponding to a first color to correspond to a second color different from the first color, among a plurality of pixel values received from an autofocus image sensor including the first pixel used for detection of a phase difference and the second pixel used for detection of an image. The first correction information is stored in a memory. The first image frame data including the plurality of pixel values is received from the autofocus image sensor. The first correction information is loaded from the memory. Based on the first correction information, the first pixel value included in the first image frame data is corrected to correspond to the second color, and first corrected image frame data is generated.

[0009] In order to achieve the above and other objects, an image processing system according to an embodiment of the present invention includes an autofocus image sensor and an image processing device. The autofocus image sensor includes a first pixel used for detecting a phase difference and a second pixel used for detecting an image. The image processing device performs image processing on image frame data provided from the autofocus image sensor. The image processing device includes a calibration unit, a memory, and a color correction unit. The calibration unit receives reference image frame data obtained by the autofocus image sensor capturing a reference pattern, compares the original image data corresponding to the reference pattern with the reference image frame data, calculates first correction information based on the comparison result between the original image data and the reference image frame data, and generates the first correction information used to correct the first pixel value corresponding to the first color, obtained from the first pixel among a plurality of pixel values received from the autofocus image sensor, to correspond to a second color different from the first color. The memory receives and stores the first correction information from the calibration unit. The color correction unit receives first image frame data including the plurality of pixel values from the autofocus image sensor, loads the first correction information from the memory, corrects the first pixel value included in the first image frame data to correspond to the second color based on the first correction information, and generates first corrected image frame data. The first correction information includes a plurality of gain values used to convert the first pixel value corresponding to the first color to a first corrected pixel value corresponding to the second color, a plurality of offset values, and a plurality of position data representing the position of the first pixel. The calibration unit and the memory generate and store the first correction information in advance before the autofocus image sensor operates normally.

Effects of the Invention

[0010] According to the image processing apparatus, image processing method, and image processing system according to an embodiment of the present invention, pixel values of phase detection pixels included in an autofocus image sensor can be corrected with different colors. Here, instead of generating / calculating correction information for performing a correction operation for each image frame, correction information used for performing the correction operation is acquired in advance, stored in a memory, and loaded for each image frame, and based on this, the correction operation can be performed. Further, pixel values received from an image sensor, rather than an autofocus image sensor, can also be corrected with different colors based on the correction information acquired and stored in advance. Therefore, the calculation amount, calculation time, power consumption amount, etc. for the correction operation are reduced, and thus the operation performance of the image processing apparatus can be improved.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Best Mode for Carrying Out the Invention

[0012] Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention will be described in more detail. For the same components in the drawings, the same reference numerals are assigned, and duplicate descriptions of the same components are omitted.

[0013] FIG. 1 is a block diagram showing an image processing apparatus according to an embodiment of the present invention.

[0014] As shown in FIG. 1, the image processing apparatus 100 includes a memory 120 and a color correction unit 140.

[0015] The memory 120 stores and outputs first correction information (CINF1). The first correction information (CINF1) is used for correcting the first pixel value among a plurality of pixel values received from an external autofocus image sensor (for example, 220 in FIG. 2). As will be described later with reference to FIG. 5, the autofocus image sensor includes a first pixel used for phase difference detection and a second pixel used for image detection. The first pixel value is obtained from the first pixel and corresponds to the first color. The first correction information (CINF1) is used to correct the first pixel value corresponding to the first color to correspond to a second color different from the first color. In other words, the first correction information (CINF1) includes data for correcting the pixel value of the phase detection pixel with a heterogeneous color. For example, each of the plurality of pixel values includes a gradation value, a luminance value, and / or a brightness value of one of the plurality of pixels.

[0016] In one embodiment, the memory 120 includes non-volatile memory such as EEPROM, flash memory, PRAM (Phase Change Random Access Memory), RRAM (Resistance Random Access Memory), NFGM (Nano Floating Gate Memory), PoRAM (Polymer Random Access Memory), MRAM (Magnetic Random Access Memory), FRAM (Ferroelectric Random Access Memory), and / or volatile memory such as DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory).

[0017] In one embodiment, the first correction information (CINF1) is acquired in advance before the image processing apparatus 100 operates normally and / or before the autofocus image sensor operates normally, and is stored in the memory 120 in advance.

[0018] The first correction information (CINF1) is generated by a calibration unit (for example, 110 in FIG. 10) disposed inside or outside the image processing apparatus 100. The generation operation of the first correction information (CINF1) will be described later with reference to FIGS. 10 and 11.

[0019] The color correction unit 140 receives first image frame data (FDAT1) including a plurality of pixel values from the autofocus image sensor, loads first correction information (CINF1) from the memory 120, and generates first corrected image frame data (CFDAT1) based on the first correction information (CINF1) and the first image frame data (FDAT1). For example, the color correction unit 140 corrects the first pixel values included in the first image frame data (FDAT1) corresponding to a second color based on the first correction information (CINF1), and generates first corrected image frame data (CFDAT1). For example, the first image frame data (FDAT1) and the first corrected image frame data (CFDAT1) correspond to one image frame provided from the autofocus image sensor. The operation of the color correction unit 140 will be described in detail later with reference to FIGS. 7 to 9.

[0020] The image processing apparatus 100 according to an embodiment of the present invention can correct pixel values of phase detection pixels included in an autofocus image sensor with different colors. Here, instead of generating and calculating correction information for performing a correction operation for each image frame, the correction information (CINF1) used for executing the correction operation is acquired in advance and stored in the memory 120, and the correction information (CINF1) is loaded for each image frame, and based on this, the correction operation can be performed. Therefore, the amount of calculation, calculation time, power consumption amount, etc. for the correction operation are reduced, and the operation performance of the image processing apparatus 100 is improved.

[0021] FIG. 2 is a block diagram showing an image processing system according to an embodiment of the present invention.

[0022] As shown in FIG. 2, the image processing system 200 includes an autofocus image sensor 220 and an image processing apparatus 240.

[0023] The autofocus image sensor 220 sequentially outputs a plurality of image frame data (FDAT) including information on an image of a subject based on incident light. Each of the plurality of image frame data (FDAT) includes a plurality of pixel values obtained from a plurality of pixels included in the autofocus image sensor 220. For example, the first image frame data (FDAT1) in FIG. 1 is included in the plurality of image frame data (FDAT). The structure and operation of the autofocus image sensor 220 will be described in detail later with reference to FIGS. 3 to 6.

[0024] The image processing apparatus 240 sequentially receives the plurality of image frame data (FDAT), sequentially corrects the plurality of image frame data (FDAT), and sequentially generates a plurality of corrected image frame data (CFDAT). The image processing apparatus 240 is implemented according to an embodiment of the present invention, pre-acquires and stores correction information (e.g., CINF1 in FIG. 1) used for performing a correction operation, and performs a correction operation based on the correction information stored in advance for each image frame.

[0025] In one embodiment, the image processing apparatus 240 may be the image processing apparatus 100 in FIG. 1. In other embodiments, the image processing apparatus 240 may be any one of the image processing apparatuses 100a, 100b, and 100c in FIGS. 10, 12, and 13. The image processing apparatus 240 is called an Image Signal Processor (ISP).

[0026] According to an embodiment, the image processing apparatus 240 may further perform at least one image processing operation such as image interpolation, demosaicing, white balance, gamma correction, color conversion, and the like.

[0027] FIG. 3 is a block diagram showing an example of an autofocus image sensor included in an image processing system according to an embodiment of the present invention.

[0028] As shown in FIG. 3, the autofocus image sensor 500 includes a pixel array 510, a correlated double sampling (CDS) block 530, and an analog-to-digital conversion (ADC) block 540. The autofocus image sensor 500 further includes a row driver 520, a digital signal processing (DSP) unit 550, a ramp signal generator 560, and a timing controller 580.

[0029] The pixel array 510 includes a plurality of pixels (or unit pixels) (PX) arranged in a matrix. Each of the plurality of pixels (PX) is connected to one of a plurality of rows (RW1, RW2,..., RWX; X is a natural number of 2 or more) and one of a plurality of columns (CL1, CL2,..., CLY; Y is a natural number of 2 or more). The pixel array 510 senses incident light and generates a plurality of analog pixel signals (VP1, VP2,..., VPY) corresponding to the incident light.

[0030] The plurality of pixels (PX) includes a first pixel used for detecting a phase difference and a second pixel used for detecting an image. The structure of each of the plurality of pixels (PX) and the structure of the pixel array 510 will be described in detail later in FIGS. 4 to 5.

[0031] The row driver 520 is connected to the plurality of rows (RW1 to RWX) of the pixel array 510 and generates drive signals for driving the plurality of rows (RW1 to RWX). For example, the row driver 520 drives the plurality of pixels (PX) included in the pixel array 510 in units of rows.

[0032] The correlated double sampling block 530 includes a plurality of correlated double sampling circuits 530a, 530b,..., 530c. The plurality of correlated double sampling circuits 530a to 530c are connected to the plurality of columns (CL1 to CLY) of the pixel array 510 and perform a CDS operation on the plurality of analog pixel signals (VP1 to VPY) output from the pixel array 510.

[0033] The analog-to-digital conversion block 540 includes a plurality of analog-to-digital converters 540a, 540b, ..., 540c. The plurality of analog-to-digital converters 540a to 540c are connected to a plurality of columns (CL1 to CLY) of the pixel array 510 via a plurality of correlated double sampling circuits 530a to 530c, and perform a column ADC that converts a plurality of analog pixel signals (VP1 to VPY) output from the pixel array 510 (i.e., the CDSed analog pixel signals output from the plurality of correlated double sampling circuits 530a to 530c) in parallel (i.e., simultaneously) into a plurality of digital signals (CNT1, CNT2, ..., CNTY).

[0034] Each of the plurality of analog-to-digital converters 540a to 540c includes one of a plurality of comparators 542a, 542b, ..., 542c and one of a plurality of counters (CNT) 544a, 544b, ..., 544c. For example, the first analog-to-digital converter 540a includes a first comparator 542a that compares a first analog pixel signal (VP1) with a ramp signal (VRAMP) to generate a first comparison signal (CS1), and a first counter 544a that counts the level transition time of the first comparison signal (CS1) to generate a first digital signal (CNT1).

[0035] The operations of the above-described correlated double sampling block 530 and analog-to-digital conversion block 540 are performed in units of rows of the pixel array 510.

[0036] The plurality of correlated double sampling circuits 530a to 530c and the plurality of analog-to-digital converters 540a to 540c form a plurality of column drive circuits. For example, the first correlated double sampling circuit 530a and the first analog-to-digital converter 540a form a first column drive circuit.

[0037] The digital signal processing unit 550 performs digital signal processing based on a plurality of digital signals (CNT1 to CNTY). For example, an autofocus information acquisition operation using a first pixel and an image frame acquisition operation using a second pixel are performed by the digital signal processing unit 550. The digital signal processing unit 550 sequentially outputs a plurality of image frame data (FDAT) generated by digital signal processing.

[0038] According to an embodiment, the digital signal processing unit 550 can be omitted. In this case, the above-described digital signal processing is performed by an external signal processing device (for example, the image processing device 240 in FIG. 2).

[0039] The lamp signal generator 560 generates a lamp signal (VRAMP). The timing controller 580 controls the overall operation timing of the autofocus image sensor 500 and generates control signals such as a count enable signal (CNT_EN) and a clock signal (not shown).

[0040] FIG. 4 is a circuit diagram showing an example of a pixel included in the pixel array of the autofocus image sensor in FIG. 3.

[0041] As shown in FIG. 4, a pixel (or unit pixel) 600 includes a photoelectric conversion unit 610 and a signal generation unit 612. Among the plurality of pixels (PX) included in the pixel array 510 of FIG. 3, the first pixel used for phase difference detection and the second pixel used for image detection have the same pixel structure and circuit structure except that the configurations of the microlens and the color filter are different, as will be described later with reference to FIG. 5.

[0042] The photoelectric conversion unit 610 performs photoelectric conversion. That is, the photoelectric conversion unit 610 converts incident light in the integration mode to generate photocharges. When the image sensor including the pixel 600 is a CMOS (Complementary Metal-Oxide Semiconductor) image sensor, the shutter of the CMOS image sensor is opened in the integration mode, and charge carriers such as electron-hole pairs are generated in the photoelectric conversion unit 610 by the incident light, and information regarding the image of the subject is collected.

[0043] The signal generation unit 612 generates an analog pixel signal (VP) based on the photocharges generated by photoelectric conversion in the readout mode. When the image sensor including the pixel 600 is a CMOS image sensor, in the readout mode after the integration mode, the shutter is closed, and an analog pixel signal (VP) is generated based on the information regarding the image of the subject collected in the form of charge carriers. FIG. 4 shows the signal generation unit 612 having a 4-transistor structure.

[0044] Specifically, the signal generation unit 612 includes a transfer transistor 620, a reset transistor 640, a drive transistor 650, and a selection transistor 660, and includes a floating diffusion node 630. The transfer transistor 620 is connected between the photoelectric conversion unit 610 and the floating diffusion node 630 and includes a gate to which a transmission signal (TX) is applied. The reset transistor 640 is connected between the power supply voltage (VDD) and the floating diffusion node 630 and includes a gate to which a reset signal (RX) is applied. The drive transistor 650 is connected between the power supply voltage (VDD) and the selection transistor 660 and includes a gate connected to the floating diffusion node 630. The selection transistor 660 is connected between the drive transistor 650 and an output terminal that outputs an analog pixel signal (VP) and includes a gate to which a selection signal (SEL) is applied.

[0045] According to the embodiment, it is possible to implement a structure in which a plurality of photoelectric conversion units share one signal generation unit.

[0046] FIGS. 5A to 5C are diagrams showing an example of a pixel array of the autofocus image sensor in FIG. 3.

[0047] FIG. 5A is a plan view showing an example of a pixel array of the autofocus image sensor. FIG. 5B is a cross-sectional view showing an example of a pixel included in the pixel array of the autofocus image sensor. FIG. 5C is a plan view showing an example of a color filter array included in the pixel array of the autofocus image sensor in FIG. 5A.

[0048] As shown in FIG. 5A, the pixel array 512 of the autofocus image sensor includes a plurality of pixels (PX11, PX12, PX13, PX14, PX15, PX16, PX21, PX22, PX23, PX24, PX25, PX26, PX31, PX32, PX33, PX35, PX36, PX41, PX42, PX43, PX44, PX45, PX46, PX51, PX52, PX53, PX54, PX55, PX56, PX61, PX62, PX63, PX64, PX65, PX66).

[0049] As described above, the plurality of pixels (PX11 to PX16, PX21 to PX26, PX31 to PX33, PX35, PX36, PX41 to PX46, PX51 to PX56, PX61 to PX66) include a first pixel used for detecting a phase difference and a second pixel used for detecting an image. The first pixel is also called an Auto-Focus (AF) pixel, and the second pixel is also called a normal pixel. In other words, in the auto-focus image sensor according to an embodiment of the present invention, pixels for obtaining AF information and normal pixels can be arranged within one pixel array. In the example of FIG. 5a, the pixel (PX33) is an AF pixel, and the remaining pixels (PX11 to PX16, PX21 to PX26, PX31, PX32, PX35, PX36, PX41 to PX46, PX51 to PX56, PX61 to PX66) are normal pixels.

[0050] Micro-lenses (ML) are arranged on the plurality of pixels (PX11 to PX16, PX21 to PX26, PX31 to PX33, PX35, PX36, PX41 to PX46, PX51 to PX56, PX61 to PX66). The micro-lens of the AF pixel (for example, the pixel (PX33)) is larger in size than the micro-lens of the normal pixel (for example, the pixel (PX35)).

[0051] In FIG. 5a, six pixels are arranged in each of a first direction (DR1) and a second direction (DR2) that intersect (for example, are orthogonal) to each other within the pixel array 512, and it is shown that the pixel array 512 includes one AF pixel and 34 normal pixels. However, the present invention is not limited to this, and the size of the pixel array and the number of pixels can be changed according to the embodiment.

[0052] As shown in FIG. 5b, the first pixel (or AF pixel) (AFPX) includes a first photoelectric conversion region (PD1), a second photoelectric conversion region (PD2), a first color filter (CF1), and a first microlens (ML1). For example, the first pixel (AFPX) corresponds to the pixel (PX33) in FIG. 5a.

[0053] The first and second photoelectric conversion regions (PD1, PD2) are formed in the substrate 511. The first color filter (CF1) is formed on the first and second photoelectric conversion regions (PD1, PD2) (i.e., along a third direction (DR3) intersecting (e.g., orthogonal) the first and second directions (DR1, DR2)) and is shared by the first and second photoelectric conversion regions (PD1, PD2). According to an embodiment, the first color filter (CF1) includes two color filters of the same color formed on the first and second photoelectric conversion regions (PD1, PD2). The first microlens (ML1) is formed on the first color filter (CF1) and is shared by the first and second photoelectric conversion regions (PD1, PD2).

[0054] The second pixel (or general pixel) (NPX) includes a third photoelectric conversion region (PD3), a second color filter (CF2), and a second microlens (ML2). For example, the second pixel (NPX) corresponds to the pixel (PX35) in FIG. 5a.

[0055] The third photoelectric conversion region (PD3) is formed in the substrate 511. The second color filter (CF2) is formed on the third photoelectric conversion region (PD3). The second microlens (ML2) is formed on the second color filter (CF2).

[0056] In one embodiment, the sizes (i.e., the areas on the plane) of the first, second, and third photoelectric conversion regions (PD1, PD2, PD3) are all substantially the same. Accordingly, the size (i.e., the area on the plane) of the first microlens (ML1) shared by the first and second photoelectric conversion regions (PD1, PD2) is larger than the size of the second microlens (ML2). For example, the size of the first microlens (ML1) is about twice the size of the second microlens (ML2).

[0057] The photoelectric conversion regions (PD1, PD2, PD3) correspond to the photoelectric conversion unit 610 in FIG. 4, perform photoelectric conversion based on incident light, and generate photocurrent. For example, electron-hole pairs corresponding to the incident light are generated in the photoelectric conversion regions (PD1, PD2, PD3), and the photoelectric conversion regions (PD1, PD2, PD3) collect such electrons or holes respectively. For convenience, the photoelectric conversion regions (PD1, PD2, PD3) are described as photodiodes, but the photoelectric conversion regions (PD1, PD2, PD3) can include photodiodes, phototransistors, photogates, pinned photo diodes (PPD), or combinations thereof.

[0058] According to an embodiment, the color filters (CF1, CF2) include a red filter, a green filter, and a blue filter, also include a yellow filter, a magenta filter, and a cyan filter, and can further include a white filter (or a transparent filter).

[0059] The microlenses (ML1, ML2) can adjust the path of the incident light so that the incident light incident on the microlenses (ML1, ML2) is condensed on the photoelectric conversion regions (PD1, PD2, PD3).

[0060] In one embodiment, the microlenses (ML1, ML2) form a microlens array, and the color filters (CF1, CF2) form a color filter array.

[0061] In one embodiment, it further includes an element isolation part 513 formed between the photoelectric conversion regions (PD1, PD2, PD3). The element isolation part 513 extends in a third direction (DR3) so as to penetrate the substrate 511, separates the photoelectric conversion regions (PD1, PD2, PD3) from each other, and has a mesh structure on a plane.

[0062] According to the embodiment, the autofocus image sensor is a frontside illuminated image sensor (FIS) that operates in response to incident light incident through the front surface of the substrate 511, or a backside illuminated image sensor (BIS) that operates in response to incident light incident through the back surface. Although not shown in detail, gate electrodes and wirings for forming the signal generation part 612 of FIG. 4 are arranged on the front surface or the back surface of the substrate 511.

[0063] As shown in FIG. 5c, the color filter array (CFA1) included in the pixel array 512 includes a red filter (R), a green filter (G), a blue filter (B), and a white filter (W). In other words, the color filter array (CFA1) has an RGBW pattern.

[0064] Pixels (PX32, PX36, PX41, PX45) include a red filter (R). Pixels (PX12, PX16, PX21, PX25, PX33, PX43, PX52, PX56, PX61, PX65) include a green filter (G). Pixels (PX14, PX23, PX54, PX63) include a blue filter (B). Pixels (PX11, PX13, PX15, PX22, PX24, PX26, PX31, PX35, PX42, PX44, PX46, PX51, PX53, PX55, PX62, PX64, PX66) include a white filter (W). In other words, the AF pixels and general pixels included in the pixel array 512 have the same color filter arrangement structure.

[0065] Figures 6a to 6c are diagrams for explaining the operation of performing autofocus using the autofocus image sensor of FIG. 3.

[0066] As shown in FIGS. 6a to 6c, incident light that has passed through the lens 51 of an image pickup apparatus (for example, a camera) including an autofocus image sensor passes through a microlens array 54 including a plurality of microlenses (for example, the microlens (ML1) of FIG. 5b) and is guided to a first AF pixel (R) 57 and a second AF pixel (L) 58. Then, of the pupils of the lens 51, light from a pupil 52 that is on a first side (for example, above) of the optical axis 50 of the lens 51 is guided to the second AF pixel (L), and light from a pupil 53 that is on a second side (for example, below) of the optical axis 50 of the lens 51 is guided to the first AF pixel (R).

[0067] When an array of the first AF pixel (R) and the second AF pixel (L) depicts continuous outputs, it becomes as shown in FIGS. 6b and 6c. In FIGS. 6b and 6c, the horizontal axis is the positions of the first AF pixel (R) and the second AF pixel (L), and the vertical axis is the output values of the first AF pixel (R) and the second AF pixel (L). Looking at the output values of the first AF pixel (R) and the second AF pixel (L), it can be seen that they have the same shape as each other. However, the positions, that is, the phases are different from each other. This is because the imaging positions of the light from the decentered pupils 52, 53 of the lens 51 are different. Therefore, when out of focus, the phase is shifted as shown in FIG. 6b, and when in focus, the image is formed at the same position as shown in FIG. 6c. Also, from this, the direction of the focus difference can be determined. When in focus in front of the object, it is called front-focusing, and in the case of front-focusing, the phase of the output value of the first AF pixel (R) is shifted to the left from the in-focus phase, and the phase of the output value of the second AF pixel (L) is shifted to the right from the in-focus phase. Conversely, when in focus behind the object, it is called back-focusing, and in the case of back-focusing, the phase of the output value of the first AF pixel (R) is shifted to the right from the in-focus phase, and the phase of the output value of the second AF pixel (L) is shifted to the left from the in-focus phase. The shift amount between the phases of the output values of the first AF pixel (R) and the second AF pixel (L) can be converted by the deviation amount between the foci.

[0068] FIGS. 7a to 7c, 8a, 8b, and 9 are diagrams for explaining the operation of the image processing apparatus according to an embodiment of the present invention. FIGS. 7a to 7c, 8a, 8b, and 9 show the operation when the pixel array of the autofocus image sensor has the structure described above with reference to FIGS. 5a to 5c.

[0069] As shown in FIG. 7a, the first image frame data (FDAT11) provided from the autofocus image sensor 500 and corresponding to one image frame includes a plurality of pixel values (PV11W, PV12G, PV13W, PV14B, PV15W, PV16G, PV21G, PV22W, PV23B, PV24W, PV25G, PV26W, PV31W, PV32R, PV33G, PV34G, PV35W, PV36R, PV41R, PV42W, PV43G, PV44W, PV45R, PV46W, PV51W, PV52G, PV53W, PV54B, PV55W, PV56G, PV61G, PV62W, PV63B, PV64W, PV65G, PV66W) obtained from a plurality of pixels (PX11~PX16, PX21~PX26, PX31~PX33, PX35, PX36, PX41~PX46, PX51~PX56, PX61~PX66).

[0070] Pixel values (PV32R, PV36R, PV41R, PV45R) obtained from pixels (PX32, PX36, PX41, PX45) including a red filter (R) correspond to red. Pixel values (PV12G, PV16G, PV21G, PV25G, PV33G, PV34G, PV43G, PV52G, PV56G, PV61G, PV65G) obtained from pixels (PX12, PX16, PX21, PX25, PX33, PX43, PX52, PX56, PX61, PX65) including a green filter (G) correspond to green. Pixel values (PV14B, PV23B, PV54B, PV63B) obtained from pixels (PX14, PX23, PX54, PX63) including a blue filter (B) correspond to blue. Pixel values (PV11W, PV13W, PV15W, PV22W, PV24W, PV26W, PV31W, PV35W, PV42W, PV44W, PV46W, PV51W, PV53W, PV55W, PV62W, PV64W, PV66W) obtained from pixels (PX11, PX13, PX15, PX22, PX24, PX26, PX31, PX35, PX42, PX44, PX46, PX51, PX53, PX55, PX62, PX64, PX66) including a white filter (W) correspond to white. One pixel value is generated from one photoelectric conversion region, and thus, two pixel values (PV33G, PV34G) are obtained from the pixel (PX33) which is an AF pixel.

[0071] The color filter disposed on the AF pixel is not used to embody a color. Instead, for the convenience of the color filter array manufacturing process, the color filter is disposed on the AF pixel. Here, for the accuracy of the AF information, the same color color filter can be disposed on one AF pixel. However, for the accuracy of the image information, it is effective for the color filter array to have the same color filter arrangement structure. Therefore, it is necessary to correct the pixel value (PV33G) corresponding to green so that the pixel values (PV33G, PV34G, PV43G, PV44W) correspond to the WGGW structure, that is, corresponding to white.

[0072] As shown in FIG. 7b, the first correction information (CINF11) stored in the memory 120 and used for correcting the first image frame data (FDAT11) includes correction data (CI33) used for correcting the pixel value (PV33G).

[0073] In one embodiment, the correction data (CI33) includes a gain value (gain) used to convert the pixel value (PV33G) corresponding to green to a corrected pixel value corresponding to white (e.g., PV33Wc in FIG. 7c), and position data indicating the position of the pixel (PX33).

[0074] For example, the following Equation 1 is used to convert the pixel value (PV33G) corresponding to green to the corrected pixel value (PV33Wc) corresponding to white.

[0075]

Equation

[0076] When the correction operation is performed using the above Equation 1, in order to generate the corrected pixel value (PV33Wc), not only the pixel value (PV33G) obtained from the pixel (PX33) but also the adjacent pixel values obtained from the adjacent pixels adjacent to the pixel (PX33) are both used. For example, the pixel value (PV32R) corresponding to red, which is obtained from the pixel (PX32) adjacent to the pixel (PX33), is used as the RV value in Equation 1, and the pixel value (PV23B) corresponding to blue, which is obtained from the pixel (PX23) adjacent to the pixel (PX33), is used as the BV value in Equation 1. According to an embodiment, the pixel value (PV34G) corresponding to green, which is obtained from the pixel (PX34) adjacent to the pixel (PX33), and the pixel value (PV43G) corresponding to green, which is obtained from the pixel (PX43) adjacent to the pixel (PX33), can be further used as the adjacent pixel values.

[0077] For example, when the position of pixel (PX11) is defined as (1, 1) and the position of pixel (PX66) is defined as (6, 6), the position data representing the position of pixel (PX33) can include a position value representing (3, 3), which is the position of the photoelectric conversion region that generates the pixel value (PV33G) to be corrected among the pixel values (PV33G, PV34G) obtained from pixel (PX33).

[0078] In another embodiment, the correction data (CI33) includes a gain value and position data, and further includes an offset value used for converting the pixel value (PV33G) corresponding to green to the corrected pixel value (PV33Wc) corresponding to white.

[0079] For example, the following Equation 2 is used to convert the pixel value (PV33G) corresponding to green to the corrected pixel value (PV33Wc) corresponding to white.

[0080]

Equation

[0081] When the correction operation is performed using the above Equation 2, only the pixel value (PV33G) obtained from pixel (PX33) is used to generate the corrected pixel value (PV33Wc).

[0082] Also, in another embodiment, the correction data (CI33) can include a gain value, an offset value, and position data, and can further include at least one additional data for correcting the pixel value.

[0083] On the one hand, in FIG. 7b, the portion shown as a blank, that is, the portion where correction data is not described, is an area without correction data, that is, an area where a correction operation is not required. As shown in FIG. 7a, the first image frame data (FDAT11) includes pixel values for all pixels, but the first correction information (CINF11) includes only correction data (CI33) for pixels that require a correction operation. Therefore, the size (or data amount) of the first correction information (CINF11) is smaller than the size of the first image frame data (FDAT11).

[0084] As shown in FIG. 7c, based on the first correction information (CINF11) in FIG. 7b, the first image frame data (FDAT11) in FIG. 7a is corrected to generate first corrected image frame data (CFDAT11). For example, based on the correction data (CI33), the pixel value (PV33G) corresponding to green is corrected to the corrected pixel value (PV33Wc) corresponding to white to generate the first corrected image frame data (CFDAT11).

[0085] On the one hand, the remaining pixel values (PV11W, PV12G, PV13W, PV14B, PV15W, PV16G, PV21G, PV22W, PV23B, PV24W, PV25G, PV26W, PV31W, PV32R, PV34G, PV35W, PV36R, PV41R, PV42W, PV43G, PV44W, PV45R, PV46W, PV51W, PV52G, PV53W, PV54B, PV55W, PV56G, PV61G, PV62W, PV63B, PV64W, PV65G, PV66W) included in the first corrected image frame data (CFDAT11) are substantially the same as the pixel values (PV11W, PV12G, PV13W, PV14B, PV15W, PV16G, PV21G, PV22W, PV23B, PV24W, PV25G, PV26W, PV31W, PV32R, PV34G, PV35W, PV36R, PV41R, PV42W, PV43G, PV44W, PV45R, PV46W, PV51W, PV52G, PV53W, PV54B, PV55W, PV56G, PV61G, PV62W, PV63B, PV64W, PV65G, PV66W) included in the first image frame data (FDAT11).

[0086] As shown in FIG. 8a, conventionally, based on the first image frame data (FDAT11), a calibration operation (CAL) for generating correction information is performed to generate calibrated first image frame data (FDAT11'), and based on the calibrated first image frame data (FDAT11'), a correction operation (COR) is performed to generate the first corrected image frame data (CFDAT11). In this case, two operations of the calibration operation (CAL) and the correction operation (COR) are performed, and data corresponding to one image frame is calculated / processed for each operation, whereby the calculation amount, calculation time, power consumption amount, etc. are relatively large.

[0087] As shown in FIG. 8b, according to an embodiment of the present invention, first correction information (CINF11) used for performing a correction operation (COR) is acquired and stored in advance. When first image frame data (FDAT11) is received, the stored first correction information (CINF11) is loaded, and a correction operation (COR) is performed based on the loaded first correction information (CINF11) to generate first corrected image frame data (CFDAT11). In this case, only one operation of the correction operation (COR) is performed. By using the relatively small-sized first correction information (CINF11) during the correction operation (COR), it is possible to reduce the amount of calculation, calculation time, power consumption, and the like.

[0088] As shown in FIG. 9, when a plurality of image frame data (F1, F2, F3, F4, F5, F6, F7, F8, F9, F10) are sequentially received from the autofocus image sensor 500, a case where a plurality of corrected image frame data (CF1, CF2, CF3, CF4, CF5, CF6, CF7, CF8, CF9, CF10) are sequentially generated is shown.

[0089] In FIG. 9, each of the plurality of image frame data (F1 to F10) corresponds to the first image frame data (FDAT11) in FIGS. 7a and 8b, the first correction information (CINF11) corresponds to the first correction information (CINF11) in FIGS. 7b and 8b, and each of the plurality of corrected image frame data (CF1 to CF10) corresponds to the first corrected image frame data (CFDAT11) in FIGS. 7c and 8b.

[0090] Similar to the foregoing described with reference to FIG. 8b, when a plurality of image frame data (F1 to F10) are sequentially received, the stored first correction information (CINF11) is loaded, and correction operations (COR) are sequentially performed based on the loaded first correction information (CINF11) to sequentially generate a plurality of corrected image frame data (CF1 to CF10). Therefore, compared with the prior art in which calibration operations (CAL) and correction operations (COR) are performed for each image frame, it is possible to reduce the amount of calculation, calculation time, power consumption, and the like.

[0091] On the other hand, FIG. 9 shows ten image frame data (F1 to F10) and ten corrected image frame data (CF1 to CF10), but the present invention is not limited thereto. For example, when the first to Nth (N is a natural number of 2 or more) image frame data are sequentially received from the autofocus image sensor 500, based on the first correction information (CINF11), the first to Nth image frame data can be sequentially corrected to sequentially generate the first to Nth corrected image frame data.

[0092] On the other hand, with reference to FIGS. 5 to 9, embodiments of the present invention have been described based on a specific pixel array structure, pixel structure, specific color, etc., but the present invention is not limited thereto. For example, among the pixel values generated by an AF pixel structure in which two or more photoelectric conversion regions share one microlens and one color filter, when at least one pixel value corresponding to an arbitrary color is corrected to correspond to an arbitrary other color, the embodiments of the present invention are applicable.

[0093] FIG. 10 is a block diagram showing an image processing apparatus according to an embodiment of the present invention. Hereinafter, descriptions overlapping with those of FIG. 1 will be omitted.

[0094] As shown in FIG. 10, the image processing apparatus 100a includes a memory 120 and a color correction unit 140, and may further include a calibration unit 110.

[0095] Except for further including the calibration unit 110, the image processing apparatus 100a is substantially the same as the image processing apparatus 100 of FIG. 1.

[0096] The calibration unit 110 generates first correction information (CINF1) stored in the memory 120. For example, the calibration unit 110 generates the first correction information (CINF1) based on the first original image data (ORDAT1) corresponding to the reference pattern and the first reference image frame data (CRDAT1).

[0097] The reference pattern represents a reference image for generating first correction information (CINF1) (i.e., gain values, offset values, etc. used for correcting pixel values), and is, for example, a uniform (or flat) image that displays only one gradation value that is the same throughout, such as a white image, a green image, etc., or an image that includes a specific pattern such as a stripe. The first original image data (ORDAT1) represents the raw data for the reference pattern itself, and the first reference image frame data (CRDAT1) represents the frame image obtained by the autofocus image sensor capturing the reference pattern.

[0098] In one embodiment, the calibration unit 110 can generate the first correction information (CINF1) in advance and store it in the memory 120 in advance before the autofocus image sensor operates normally. Accordingly, the calibration unit 110 is called a preprocessing unit.

[0099] In one embodiment, the calibration unit 110 is deactivated while the autofocus image sensor is operating normally. In other words, the calibration unit 110 is activated at the initial stage of operation or during manufacturing to generate the first correction information (CINF1), and thereafter is deactivated and does not operate. For example, the calibration unit 110 is activated / deactivated based on an enable signal (EN).

[0100] FIG. 11 is a block diagram showing an example of the calibration unit included in the image processing apparatus of FIG. 10.

[0101] As shown in FIG. 11, the calibration unit 110a includes a comparison unit 112 and a calculation unit 114.

[0102] The comparison unit 112 receives the first original image data (ORDAT1) corresponding to the reference pattern and the first reference image frame data (CRDAT1) obtained by imaging the reference pattern by the autofocus image sensor, compares the first original image data (ORDAT1) with the first reference image frame data (CRDAT1), and generates first comparison result data (CR1).

[0103] Based on the first comparison result data (CR1), which is the comparison result between the first original image data (ORDAT1) and the first reference image frame data (CRDAT1), the arithmetic unit 114 calculates first correction information (CINF1). For example, the arithmetic unit 114 calculates the relationship between the first color to be corrected and the second color as the correction result. For example, the arithmetic unit 114 can obtain the gain value, offset value, etc. described above as calculation results by referring to Equation 1 and Equation 2.

[0104] FIGS. 12 and 13 are block diagrams showing an image processing apparatus according to an embodiment of the present invention. Hereinafter, descriptions overlapping with FIG. 1 are omitted.

[0105] As shown in FIG. 12, the image processing apparatus 100b includes a memory 120 and a color correction unit 140, and may further include a selection unit 160.

[0106] The image processing apparatus 100b is substantially the same as the image processing apparatus 100 of FIG. 1, except that the memory 120 further stores second correction information (CINF2) and further includes a selection unit 160.

[0107] The memory 120 stores and outputs second correction information (CINF2). Similar to the first correction information (CINF1), the second correction information (CINF2) is used to correct the first pixel value corresponding to the first color among the plurality of pixel values received from the autofocus image sensor to correspond to the second color. However, the second correction information (CINF2) can have a value different from that of the first correction information (CINF1).

[0108] In one embodiment, the first and second correction information (CINF1, CINF2) can correspond to different color temperatures. For example, the first correction information (CINF1) corresponds to the first color temperature, and the second correction information (CINF2) corresponds to a second color temperature different from the first color temperature. However, the present invention is not limited thereto, and the first and second correction information (CINF1, CINF2) can also correspond to parameters regarding other different colors.

[0109] The selection unit 160 selects one of the first correction information (CINF1) and the second correction information (CINF2) based on the first image frame data (FDAT1), and outputs the selected correction information (SINF). The color correction unit 140 corrects the first image frame data (FDAT1) based on the selected correction information (SINF) to generate first corrected image frame data (CFDAT1).

[0110] In one embodiment, when the first and second correction information (CINF1, CINF2) correspond to different color temperatures, the selection unit 160 includes a color temperature determination unit. For example, the color temperature determination unit determines the color temperature of the first image frame data (FDAT1), and when the color temperature of the first image frame data (FDAT1) matches the first color temperature, selects the first correction information (CINF1), and when the color temperature of the first image frame data (FDAT1) matches the second color temperature, selects the second correction information (CINF2). In another example, when the color temperature of the first image frame data (FDAT1) does not match the first and second color temperatures, the color temperature determination unit can select the correction information corresponding to the closer color temperature among the first and second color temperatures.

[0111] On the other hand, although the case where the memory 120 stores two pieces of correction information (CINF1, CINF2) and the selection unit 160 selects one of the two pieces of correction information (CINF1, CINF2) is shown, the present invention is not limited thereto, and the memory 120 can store three or more pieces of correction information, and the selection unit 160 can also select one of the three or more pieces of correction information.

[0112] As shown in FIG. 13, the image processing apparatus 100c includes a memory 120 and a color correction unit 140, and may further include a calibration unit 110 and a selection unit 160.

[0113] Except that the memory 120 further stores second correction information (CINF2) and further includes a calibration unit 110 and a selection unit 160, the image processing apparatus 100c is substantially the same as the image processing apparatus 100 of FIG. 1. The calibration unit 110 is the same as the calibration unit 110 of FIG. 10. The memory 120 and the selection unit 160 are substantially the same as the memory 120 and the selection unit 160 of FIG. 12.

[0114] The calibration unit 110 generates first correction information (CINF1) based on first original image data (ORDAT1) and first reference image frame data (CRDAT1), and generates second correction information (CINF2) based on second original image data (ORDAT2) and second reference image frame data (CRDAT2). For example, when the first correction information (CINF1) corresponds to a first color temperature and the second correction information (CINF2) corresponds to a second color temperature, the first original image data (ORDAT1) and the first reference image frame data (CRDAT1) represent a reference pattern corresponding to the first color temperature and a frame image obtained by imaging this (or imaging at the first color temperature), and the second original image data (ORDAT2) and the second reference image frame data (CRDAT2) represent a reference pattern corresponding to the second color temperature and a frame image obtained by imaging this (or imaging at the second color temperature).

[0115] FIG. 14 is a block diagram showing an image processing apparatus according to an embodiment of the present invention. Hereinafter, descriptions overlapping with those of FIG. 1 are omitted.

[0116] As shown in FIG. 14, the image processing apparatus 300 includes a memory 320 and a color correction unit 340.

[0117] Except for performing a correction operation on the image frame received from the image sensor instead of the autofocus image sensor, the image processing apparatus 300 is substantially the same as the image processing apparatus 100 of FIG. 1.

[0118] The memory 320 stores and outputs first correction information (CINFA). The first correction information (CINFA) is used to correct the first pixel value among a plurality of pixel values received from an external image sensor (for example, 420 in FIG. 15). Different from the autofocus image sensor, the image sensor includes only a plurality of pixels (that is, general pixels) used to detect an image, and the first pixel value is obtained from the first pixel among the plurality of pixels, corresponds to the first color, and the first correction information (CINFA) is used to correct the first pixel value corresponding to the first color to correspond to a second color different from the first color.

[0119] The color correction unit 340 receives first image frame data (FDATA) including a plurality of pixel values from the image sensor, loads the first correction information (CINFA) from the memory 320, and generates first corrected image frame data (CFDATA) based on the first correction information (CINFA) and the first image frame data (FDATA). For example, the color correction unit 340 corrects the first pixel value included in the first image frame data (FDATA) to correspond to the second color based on the first correction information (CINFA), and generates first corrected image frame data (CFDATA).

[0120] According to an embodiment of the present invention, an image processing apparatus 300 can correct pixel values of pixels of a specific color included in an image sensor with a different color. At this time, instead of generating / calculating correction information for performing a correction operation for each image frame, correction information (CINFA) used for performing a correction operation is acquired in advance and stored in a memory 320, and the correction information (CINFA) is loaded for each image frame, and based on this, a correction operation can be performed. Therefore, the amount of calculation, calculation time, power consumption amount, etc. for the correction operation are reduced, and thereby, the operation performance of the image processing apparatus 300 can be improved.

[0121] On the other hand, according to an embodiment, the image processing apparatus 300 can further include a calibration unit as described above with reference to FIG. 10, can further include a selection unit as described above with reference to FIG. 12, and can further include both a calibration unit and a selection unit as described above with reference to FIG. 13.

[0122] FIG. 15 is a block diagram showing an image processing system according to an embodiment of the present invention. Hereinafter, descriptions overlapping with those of FIG. 2 will be omitted.

[0123] As shown in FIG. 15, an image processing system 400 includes an image sensor 420 and an image processing apparatus 440.

[0124] The image sensor 420 sequentially outputs a plurality of image frame data (FDAT) including information on an image of a subject based on incident light. The image sensor 420 is implemented in the same manner as described above with reference to FIGS. 3 to 6, but does not include AF pixels and includes only general pixels, so that the structure of the pixel array can be partially changed.

[0125] The image processing apparatus 440 sequentially receives a plurality of image frame data (FDAT), sequentially corrects the plurality of image frame data (FDAT), and sequentially generates a plurality of corrected image frame data (CFDAT). The image processing apparatus 440 is implemented according to an embodiment of the present invention and can be, for example, the image processing apparatus 300 of FIG. 14.

[0126] FIG. 16 is a plan view showing an example of a pixel array of an image sensor included in the image processing system of FIG. 15. Hereinafter, descriptions overlapping with those of FIG. 5a are omitted.

[0127] As shown in FIG. 16, the pixel array 514 of the image sensor includes a plurality of pixels (PXA1, PXA2, PXA3, PXA4, PXA5, PXA6, PXB1, PXB2, PXB3, PXB4, PXB5, PXB6, PXC1, PXC2, PXC3, PXC4, PXC5, PXC6, PXD1, PXD2, PXD3, PXD4, PXD5, PXD6, PXE1, PXE2, PXE3, PXE4, PXE5, PXE6, PXF1, PXF2, PXF3, PXF4, PXF5, PXF6).

[0128] All of the plurality of pixels (PXA1 to PXA6, PXB1 to PXB6, PXC1 to PXC6, PXD1 to PXD6, PXE1 to PXE6, PXF1 to PXF6) are general pixels used for image detection, and a microlens (ML) is disposed on the upper part. Each of the plurality of pixels (PXA1 to PXA6, PXB1 to PXB6, PXC1 to PXC6, PXD1 to PXD6, PXE1 to PXE6, PXF1 to PXF6) is implemented like the second pixel (NPX) of FIG. 5b.

[0129] FIGS. 17a to d and 18a to d are diagrams for explaining the operation of the image processing apparatus according to an embodiment of the present invention. Hereinafter, descriptions overlapping with those of FIGS. 5c and 7a to c are omitted.

[0130] FIG. 17a is a plan view showing an example of a color filter array included in the pixel array 514 of the image sensor of FIG. 16, and FIGS. 17b to 17d show operations when the pixel array of the image sensor has the structures of FIGS. 16 and 17a. FIG. 18a is a plan view showing another example of the color filter array included in the pixel array 514 of the image sensor of FIG. 16, and FIGS. 18b to 18d show operations when the pixel array of the image sensor has the structures of FIGS. 16 and 18a.

[0131] As shown in FIG. 17a, the color filter array (CFAA1) included in the pixel array 514 includes a red filter (R), a green filter (G), a blue filter (B), and a white filter (W). Pixels (PXA2, PXA4, PXA6, PXC2, PXC4, PXC6, PXE2, PXE4, PXE6) include the red filter (R). Pixels (PXA1, PXA3, PXA5, PXC1, PXC3, PXC5, PXE1, PXE3, PXE5) include the green filter (G). Pixels (PXB1, PXB3, PXB5, PXD1, PXD3, PXD5, PXF1, PXF3, PXF5) include the blue filter (B). Pixels (PXB2, PXB4, PXB6, PXD2, PXD4, PXD6, PXF2, PXF4, PXF6) include the white filter (W).

[0132] As shown in FIG. 17b, the first image frame data (FDATA1) provided from the image sensor 420 and corresponding to one image frame is obtained from pixels (PXA2, PXA4, PXA6, PXC2, PXC4, PXE2, PXE4, PXE6) including a red filter (R), and pixel values corresponding to red (PVA2R, PVA4R, PVA6R, PVC2R, PVC4R, PVC6R, PVE2R, PVE4R, PVE6R), obtained from pixels (PXA1, PXA3, PXA5, PXC1, PXC3, PXC5, PXE1, PXE3, PXE5) including a green filter (G), and pixel values corresponding to green (PVA1G, PVA3G, PVA5G, PVC1G, PVC3G, PVC5G, PVE1G, PVE3G, PVE5G), obtained from pixels (PXB1, PXB3, PXB5, PXD1, PXD3, PXD5, PXF1, PXF3, PXF5) including a blue filter (B), and pixel values corresponding to blue (PVB1B, PVB3B, PVB5B, PVD1B, PVD3B, PVD5B, PVF1B, PVF3B, PVF5B), and obtained from pixels (PXB2, PXB4, PXB6, PXD2, PXD4, PXD6, PXF2, PXF4, PXF6) including a white filter (W), and includes pixel values corresponding to white (PVB2W, PVB4W, PVB6W, PVD2W, PVD4W, PVD6W, PVF2W, PVF4W, PVF6W).

[0133] As shown in FIG. 17c, the first correction information (CINFA1) stored in the memory 320 and used to correct the first image frame data (FDATA1) includes correction data (CIB2, CIB4, CIB6, CID2, CID4, CID6, CIF2, CIF4, CIF6) used to correct the pixel values (PVB2W, PVB4W, PVB6W, PVD2W, PVD4W, PVD6W, PVF2W, PVF4W, PVF6W).

[0134] As shown in FIG. 17d, based on the first correction information (CINFA1) in FIG. 17c, the pixel values (PVB2W, PVB4W, PVB6W, PVD2W, PVD4W, PVD6W, PVF2W, PVF4W, PVF6W) corresponding to white and included in the first image frame data (FDATA1) in FIG. 17b are corrected to the corrected pixel values (PVB2Gc, PVB4Gc, PVB6Gc, PVD2Gc, PVD4Gc, PVD6Gc, PVF2Gc, PVF4Gc, PVF6Gc) corresponding to green, thereby generating the first corrected image frame data (CFDATA1). In other words, the RGBW pattern can be changed to the Bayer pattern.

[0135] As shown in FIG. 18a, the color filter array (CFAA2) included in the pixel array 514 includes a red filter (R), a green filter (G), a blue filter (B), and a white filter (W). Pixels (PXC2, PXC6, PXD1, PXD5) include the red filter (R). Pixels (PXA2, PXA6, PXB1, PXB5, PXC4, PXD3, PXE2, PXE6, PXF1, PXF5) include the green filter (G). Pixels (PXA4, PXB3, PXE4, PXF3) include the blue filter (B). Pixels (PXA1, PXA3, PXA5, PXB2, PXB4, PXB6, PXC1, PXC3, PXC5, PXD2, PXD4, PXD6, PXE1, PXE3, PXE5, PXF2, PXF4, PXF6) include the white filter (W).

[0136] As shown in FIG. 18b, the first image frame data (FDATA2) provided from the image sensor 420 and corresponding to one image frame is obtained from pixels (PXC2, PXC6, PXD1, PXD5) including a red filter (R), and pixel values (PVJ2R, PVJ6R, PVK1R, PVK5R) corresponding to red, from pixels (PXA2, PXA6, PXB1, PXB5, PXC4, PXD3, PXE2, PXE6, PXF1, PXF5) including a green filter (G), and pixel values (PVG2G, PVG6G, PVH1G, PVH5G, PVJ4G, PVK3G, PVL2G, PVL6G, PVM1G, PVM5G) corresponding to green, from pixels (PXA4, PXB3, PXE4, PXF3) including a blue filter (B), and pixel values (PVG4B, PVH3B, PVL4B, PVM3B) corresponding to blue, and from pixels (PXA1, PXA3, PXA5, PXB2, PXB4, PXB6, PXC1, PXC3, PXC5, PXD2, PXD4, PXD6, PXE1, PXE3, PXE5, PXF2, PXF4, PXF6) including a white filter (W), and includes pixel values (PVG1W, PVG3W, PVG5W, PVH2W, PVH4W, PVH6W, PVJ1W, PVJ3W, PVJ5W, PVK2W, PVK4W, PVK6W, PVL1W, PVL3W, PVL5W, PVM2W, PVM4W, PVM6W) corresponding to white.

[0137] As shown in FIG. 18c, the first correction information (CINFA2) stored in the memory 320 and used to correct the first image frame data (FDATA2) includes correction data (CIG1, CIG3, CIG5, CIH2, CIH4, CIH6, CIJ1, CIJ3, CIJ5, CIK2, CIK4, CIK6, CIL1, CIL3, CIL5, CIM2, CIM4, CIM6) used to correct the pixel values (PVG1W, PVG3W, PVG5W, PVH2W, PVH4W, PVH6W, PVJ1W, PVJ3W, PVJ5W, PVK2W, PVK4W, PVK6W, PVL1W, PVL3W, PVL5W, PVM2W, PVM4W, PVM6W).

[0138] As shown in FIG. 18d, based on the first correction information (CINFA2) in FIG. 18c, among the pixel values (PVG1W, PVG3W, PVG5W, PVH2W, PVH4W, PVH6W, PVJ1W, PVJ3W, PVJ5W, PVK2W, PVK4W, PVK6W, PVL1W, PVL3W, PVL5W, PVM2W, PVM4W, PVM6W) included in the first image frame data (FDATA2) in FIG. 18b and corresponding to white, the pixel values (PVJ1W, PVJ5W, PVK2W, PVK6W) are corrected to the corrected pixel values (PVJ1Rc, PVJ5Rc, PVK2Rc, PVK6Rc) corresponding to red, the pixel values (PVG1W, PVG5W, PVH2W, PVH6W, PVJ3W, PVK4W, PVL1W, PVL5W, PVM2W, PVM6W) are corrected to the corrected pixel values (PVG1Gc, PVG5Gc, PVH2Gc, PVH6Gc, PVJ3Gc, PVK4Gc, PVL1Gc, PVL5Gc, PVM2Gc, PVM6Gc) corresponding to green, and the pixel values (PVG3W, PVH4W, PVL3W, PVM4W) are corrected to the pixel values (PVG3Bc, PVH4Bc, PVL3Bc, PVM4Bc) corresponding to blue, whereby the first corrected image frame data (CFDATA2) can be generated. In other words, the RGBW pattern can be changed to the Tetra pattern.

[0139] On the other hand, with reference to FIGS. 16 to 18, embodiments of the present invention have been described based on a specific pixel array structure, a specific color, etc., but the present invention is not limited thereto. For example, in any pixel array structure of an image sensor, when at least one pixel value corresponding to an arbitrary color is corrected corresponding to an arbitrary other color, the embodiments of the present invention are applicable.

[0140] FIG. 19 is a sequence diagram showing an image processing method according to an embodiment of the present invention.

[0141] As shown in FIGS. 1, 2, 10, and 19, in the image processing method according to an embodiment of the present invention, among a plurality of pixel values received from an autofocus image sensor 220 including a first pixel used for detecting a phase difference and a second pixel used for detecting an image, a first pixel value obtained from the first pixel and corresponding to a first color is corrected to correspond to a second color different from the first color. A first correction information (CINF1) used for this is generated (step S100). Step S100 is performed by a calibration unit 110 disposed inside or outside the image processing apparatus.

[0142] The first correction information (CINF1) is stored in a memory 120 (step S200). A first image frame data (FDAT1) including a plurality of pixel values is received from the autofocus image sensor 220 (step S300), the first correction information (CINF1) is loaded from the memory 120 (step S400), and based on the first correction information (CINF1), the first pixel value included in the first image frame data (FDAT1) is corrected to correspond to the second color, and a first corrected image frame data (CFDAT1) is generated (step S500). Steps S300, S400, and S500 are performed by a color correction unit 140.

[0143] FIG. 20 is a sequence diagram showing an example of a step of generating the first correction information in FIG. 19.

[0144] As shown in FIGS. 2, 10, 11, 19, and 20, when generating the first correction information (CINF1) (step S100), the autofocus image sensor 220 receives the first reference image frame data (CRDAT1) obtained by imaging the reference pattern (step S110), compares the first original image data (ORDAT1) corresponding to the reference pattern with the first reference image frame data (CRDAT1) (step S120), and calculates the first correction information (CINF1) based on the comparison result between the first original image data (ORDAT1) and the first reference image frame data (CRDAT1) (step S130). Steps S110 and S120 are performed by the comparison unit 112, and step S130 is performed by the calculation unit 114.

[0145] FIGS. 21 and 22 are sequence diagrams showing an image processing method according to an embodiment of the present invention. Hereinafter, descriptions overlapping with FIG. 19 are omitted.

[0146] As shown in FIGS. 1, 2, and 21, in the image processing method according to an embodiment of the present invention, steps S100, S200, and S400 are substantially the same as steps S100, S200, and S400 in FIG. 19, respectively.

[0147] The first to Nth image frame data are sequentially received from the autofocus image sensor 220 (step S350), and based on the first correction information (CINF1), the first pixel values included in the first to Nth image frame data are sequentially corrected corresponding to the second color, and the first to Nth corrected image frame data are sequentially generated (step S550). In other words, the embodiment of FIG. 21 can also be described as sequentially receiving the second to Nth image frame data after step S300 of FIG. 19 and sequentially correcting the second to Nth image frame data after step S500 to sequentially generate the second to Nth corrected image frame data.

[0148] As shown in FIGS. 2, 12, 13, and 22, in the image processing method according to an embodiment of the present invention, the first pixel value corresponding to the first color is used to correct the first pixel value corresponding to the second color, and different first and second correction information (CINF1, CINF2) is generated (step S150), and the first and second correction information (CINF1, CINF2) is stored in the memory 120 (step S250). Steps S150 and S250 are the same as steps S100 and S200 in FIG. 19, respectively.

[0149] The first image frame data (FDAT1) is received from the autofocus image sensor 220 (step S300), the first and second correction information (CINF1, CINF2) is loaded from the memory 120 (step S450), and based on the first image frame data (FDAT1), one of the first and second correction information (CINF1, CINF2) is selected (step S600), and based on the selected correction information (SINF), the first pixel value included in the first image frame data (FDAT1) is corrected corresponding to the second color to generate the first corrected image frame data (CFDAT1) (step S700). Steps S300, S450, and S700 are the same as S300, S400, and S500 in FIG. 19, respectively. Step S600 is performed by the selection unit 160.

[0150] FIG. 23 is a sequence diagram showing an image processing method according to an embodiment of the present invention. Hereinafter, the description overlapping with FIG. 19 will be omitted.

[0151] As shown in FIGS. 14, 15, and 23, in the image processing method according to an embodiment of the present invention, among the plurality of pixel values received from the image sensor 420 including a plurality of pixels used for image detection, the first pixel value corresponding to the first color is used to correct the first pixel value corresponding to the second color different from the first color, and the first correction information (CINFA) is generated (step S1100). Step S1100 is performed by a calibration unit disposed inside or outside the image processing apparatus.

[0152] Store the first correction information (CINFA) in the memory 320 (step S1200). Receive first image frame data (FDATA) including a plurality of pixel values from the image sensor 420 (step S1300), load the first correction information (CINFA) from the memory 320 (step S1400), and based on the first correction information (CINFA), correct the first pixel values included in the first image frame data (FDATA) corresponding to the second color to generate first corrected image frame data (CFDATA) (step S1500). Steps S1300, S1400, and S1500 are performed by the color correction unit 340.

[0153] On the other hand, embodiments of the present invention can also be embodied in the form of a product including computer-readable program code stored in a computer-readable medium. The computer-readable program code is provided by a processor of various computers or other data processing devices. The computer-readable medium is either a computer-readable signal medium or a computer-readable recording medium. The computer-readable recording medium can be any type of medium that stores or includes a program in or connected to a command execution system, apparatus, or device. For example, the computer-readable medium is provided in the form of a non-transitory storage medium. Here, non-transitory only means that the storage medium does not contain a signal and is tangible, and does not distinguish whether the data is stored semi-permanently or temporarily in the storage medium.

[0154] FIG. 24 is a block diagram showing an electronic system including an image processing apparatus according to an embodiment of the present invention.

[0155] As shown in FIG. 24, the electronic system 1000 is embodied as a data processing device that uses or supports a MIPI interface, and includes an application processor 1110, an image sensor 1140, a display 1150, and the like. The electronic system 1000 further includes an RF chip 1160, a GPS 1120, a storage 1170, a microphone 1180, a DRAM 1185, and a speaker 1190, and can communicate using UWB 1210, WLAN 1220, WiMAX 1230, and the like.

[0156] The application processor 1110 represents a controller or processor that controls the operations of the image sensor 1140 and the display 1150.

[0157] The application processor 1110 includes a DSI host 1111 that communicates with the DSI device 1151 of the display 1150, a CSI host 1112 that communicates with the CSI device 1141 of the image sensor 1140, a PHY 1113 that transmits and receives data with the PHY 1161 of the RF chip 1160 via DigRF, and a DigRF master 1114 that controls the DigRF slave 1162 of the RF chip 1160.

[0158] In one embodiment, the DSI host 1111 includes an optical serializer (SER), and the DSI device 1151 includes an optical deserialzier (DES). In one embodiment, the CSI host 1112 includes an optical deserialzier (DES), and the CSI device 1141 includes an optical serializer (SER).

[0159] The CSI host 1112 includes an image processing device according to an embodiment of the present invention, and the CSI host 1112 and the image sensor 1140 form an image processing system according to an embodiment of the present invention. The image sensor 1140 is an autofocus image sensor and / or an image sensor included in the image processing system according to an embodiment of the present invention. The CSI host 1112 and the image sensor 1140 operate based on a driving method according to an embodiment of the present invention.

Industrial Applicability

[0160] Embodiments of the present invention are usefully applied to an image processing apparatus and any electronic device and system including an image sensor. For example, embodiments of the present invention are more usefully applicable to electronic systems such as a PC, a workstation, a notebook computer, a mobile phone, a smartphone, an MP3 player, a PDA (Personal Digital Assistant), a PMP (Portable Multimedia Player), a digital TV, a digital camera, a portable game console, a navigation device, a wearable device, an IoT (Internet of Things) device, an IoE (Internet of Everything) device, an e-book, a VR device, an AR device, a drone, and the like.

[0161] As described above with reference to the preferred embodiments of the present invention, those skilled in the relevant art will understand that the present invention can be variously modified and changed without departing from the spirit and scope of the present invention described in the following claims.

Claims

1. Of the plurality of pixel values received from an autofocus image sensor including a first pixel used for detecting a phase difference and a second pixel used for detecting an image, a first pixel value obtained from the first pixel and corresponding to a first color, a memory for storing first correction information used to correct the first pixel value corresponding to the first color to correspond to a second color different from the first color, further storing second correction information used to correct the first pixel value corresponding to the first color to correspond to the second color, the first correction information corresponding to a first color temperature, and the second correction information corresponding to a second color temperature different from the first color temperature, and a memory, Receiving first image frame data including the plurality of pixel values from the autofocus image sensor, and after a selection unit selects one of the first correction information and the second correction information based on the first image frame data, loading, from the memory, third correction information that is one of the first correction information and the second correction information selected by the selection unit, and based on the third correction information, correcting the first pixel value included in the first image frame data to correspond to the second color to generate first corrected image frame data, and a color correction unit An image processing apparatus comprising:

2. The image processing apparatus according to claim 1, further comprising a calibration unit that generates the first correction information stored in the memory.

3. The calibration unit is A comparison unit that receives reference image frame data obtained by the autofocus image sensor imaging a reference pattern and compares the original image data corresponding to the reference pattern with the reference image frame data; The image processing apparatus according to claim 2, further comprising an arithmetic unit that calculates the first correction information based on a comparison result between the original image data and the reference image frame data.

4. The calibration unit is The image processing apparatus according to claim 2, wherein the first correction information is generated in advance and stored in the memory in advance before the autofocus image sensor operates normally.

5. The calibration unit is The image processing apparatus according to claim 4, wherein the autofocus image sensor is deactivated while operating normally.

6. The color correction unit sequentially receives second to N (N is a natural number of 2 or more) image frame data from the autofocus image sensor after the first image frame data, sequentially corrects the second to N image frame data based on the first correction information, and sequentially generates second to N corrected image frame data, the image processing apparatus according to any one of claims 1 to 5.

7. The first correction information includes a plurality of gain values used to convert the first pixel value corresponding to the first color into the first corrected pixel value corresponding to the second color, and a plurality of position data representing the position of the first pixel, the image processing apparatus according to any one of claims 1 to 6.

8. The image processing apparatus according to claim 7, wherein an adjacent pixel value obtained from an adjacent pixel adjacent to the first pixel among the first pixel value and the second pixel is used to generate the first corrected pixel value.

9. The image processing apparatus according to claim 7, wherein the first correction information further includes a plurality of offset values used to convert the first pixel value into the first corrected pixel value.

10. The image processing apparatus according to claim 9, wherein only the first pixel value is used to generate the first corrected pixel value.

11. Each of the first pixels includes a first photoelectric conversion region and a second photoelectric conversion region formed in the substrate, a first color filter formed on the first photoelectric conversion region and the second photoelectric conversion region and having the first color, and a first microlens formed on the first color filter and shared by the first photoelectric conversion region and the second photoelectric conversion region, the image processing apparatus according to any one of claims 1 to 10.

12. Each of the second pixels includes a third photoelectric conversion region formed in the substrate, a second color filter formed on the third photoelectric conversion region, The image processing apparatus according to claim 11, further comprising a second microlens formed on the second color filter.

13. The sizes of the first photoelectric conversion region, the second photoelectric conversion region, and the third photoelectric conversion region are the same. The image processing apparatus according to claim 12, wherein the size of the first microlens is larger than the size of the second microlens.

14. A step of generating first correction information used to correct a first pixel value obtained from a first pixel and corresponding to a first color among a plurality of pixel values received from an autofocus image sensor including the first pixel used for detecting a phase difference and the second pixel used for detecting an image, to correspond to a second color different from the first color, further generating second correction information used to correct the first pixel value corresponding to the first color to correspond to the second color, the first correction information corresponding to a first color temperature, and the second correction information corresponding to a second color temperature different from the first color temperature. A step of storing the first correction information and the second correction information in a memory. A step of receiving first image frame data including the plurality of pixel values from the autofocus image sensor. A step of selecting one of the first correction information and the second correction information based on the first image frame data. A step of loading third correction information, which is the one selected in the selecting step, from among the first correction information and the second correction information, from the memory. An image processing method, comprising: a step of correcting the first pixel value included in the first image frame data to correspond to the second color based on the third correction information, and generating first corrected image frame data.

15. The step of generating the first correction information includes: A step of the autofocus image sensor imaging a reference pattern and receiving acquired reference image frame data. A step of comparing original image data corresponding to the reference pattern with the reference image frame data. The image processing method according to claim 14, further comprising the step of calculating the first correction information based on a comparison result between the original image data and the reference image frame data.

16. Furthermore, a step of sequentially receiving second to N (N is a natural number of 2 or more) image frame data from the autofocus image sensor after the first image frame data; The image processing method according to claim 14, further comprising the step of sequentially correcting the second to N image frame data based on the first correction information to sequentially generate second to N corrected image frame data.

17. An autofocus image sensor including a first pixel used for detecting a phase difference and a second pixel used for detecting an image; An image processing apparatus that performs image processing on image frame data provided from the autofocus image sensor, The image processing apparatus includes: A calibration unit that receives reference image frame data obtained by the autofocus image sensor imaging a reference pattern, compares the original image data corresponding to the reference pattern with the reference image frame data, and generates first correction information and second correction information based on a comparison result between the original image data and the reference image frame data. The first correction information and the second correction information are obtained from the first pixel among a plurality of pixel values received from the autofocus image sensor and are used to correct a first pixel value corresponding to a first color to correspond to a second color different from the first color. The first correction information corresponds to a first color temperature, and the second correction information corresponds to a second color temperature different from the first color temperature. A memory that receives and stores the first correction information and the second correction information from the calibration unit; Receiving first image frame data including the plurality of pixel values from the autofocus image sensor, after a selection unit selects one of the first correction information and the second correction information based on the first image frame data, loading, from the memory, third correction information that is one of the first correction information and the second correction information selected by the selection unit, and correcting the first pixel values included in the first image frame data corresponding to the second color based on the third correction information to generate first corrected image frame data, and a color correction unit; The first correction information and the second correction information are; a plurality of gain values, a plurality of offset values, and a plurality of position data representing positions of the first pixels, which are used to convert the first pixel values corresponding to the first color into first corrected pixel values corresponding to the second color; The calibration unit and the memory are characterized in that the first correction information and the second correction information are generated and stored in advance before the autofocus image sensor operates normally. An image processing system.

Images