Imaging apparatus

Abstract

To provide a technique for adjusting the focal position of a camera module via phase difference detection autofocus pixels.SOLUTION: An imaging apparatus can include: a pixel array PA including a plurality of image detection pixels and a plurality of phase difference detection pixels PX1 to PX4; a position determination part 210 which determines the position of each unit pixel in the pixel array; a weight setting part 220 which sets a different weight for the position of each phase difference detection pixel on the basis of an output from the position determination part; a signal blending part 230 which adds the set weight to each phase difference detection pixel to create a plurality of phase images IMG1 to IMG4; a parallax calculation part 240 which calculates parallax for at least any one direction of parallax in a first direction and parallax in a second direction from the center point of an optical axis in the plurality of phase images; and a focal position detection part 250 which creates a driving signal for adjusting the position of a lens on the basis of the calculated parallax.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an imaging device, and is a technology for adjusting the focus position of a camera module via a Phase Detection Auto Focus (PDAF) pixel. [Background technology]

[0002] Image sensing devices are devices that capture optical images using the properties of photosensitive semiconductor materials that react to light. With the development of industries such as automobiles, medicine, computers, and communications, there is an increasing demand for high-performance image sensing devices in various fields, such as smartphones, digital cameras, game consoles, the Internet of Things, robots, security cameras, and medical microcameras.

[0003] 2. Description of the Related Art In a device for photographing an object (for example, a camera), it is important to accurately focus on the object in order to capture a clear image (for example, a still image) or video (for example, a video).

[0004] Image sensing devices include a phase-difference detection autofocus (PDAF) function that automatically focuses by detecting the phase difference between adjacent pixels. The phase-difference detection autofocus method uses the phase difference between two or more different focus points to measure the offset direction and amount from the center image acquired through the image sensing device. In recent years, to improve PDAF functionality, image sensing devices have adopted a structure in which multiple pixels of the same color are arranged adjacent to each other, with one microlens applied to each of the multiple pixels.

[0005] However, in an image sensing device with a structure in which one microlens is applied to multiple pixels, the parallax varies due to the influence of lens astigmatism, etc., which can make it difficult to perform accurate autofocus. Summary of the Invention [Problem to be solved by the invention]

[0006] An embodiment of the present invention provides an imaging device that allows for accurate adjustment of the focus position in an image sensing device including a Phase Detection Auto Focus (PDAF) pixel by measuring parallax taking into account central symmetry around the optical axis. [Means for solving the problem]

[0007] An imaging device according to an embodiment of the present invention may include a pixel array including a plurality of image detection pixels and a plurality of phase difference detection pixels; a position determination unit that determines a position of each unit pixel in the pixel array; a weight setting unit that sets different weights for each position of each phase difference detection pixel based on an output of the position determination unit; a signal blending unit that generates a plurality of phase images by adding the weights set by the weight setting unit to each phase difference detection pixel; a parallax calculation unit that calculates a parallax in at least one of a first direction and a second direction from a center point of an optical axis in the plurality of phase images; and a focus position detection unit that generates a drive signal for adjusting a lens position based on the parallax calculated by the parallax calculation unit.

[0008] An imaging device according to another embodiment of the present invention may include a pixel group including a plurality of unit pixels arranged in an N×N matrix, and an image signal processor that generates a plurality of phase images by adding weights differently according to positions of the plurality of unit pixels, and calculates parallax for the plurality of phase images in at least one of a first direction from a center point of an optical axis and a second direction perpendicular to the first direction. [Effects of the Invention]

[0009] SUMMARY OF THE INVENTION Embodiments of the present invention provide the advantage of enabling precise adjustment of focus position in an image sensing device that includes Phase Detection Auto Focus (PDAF) pixels.

[0010] It should be noted that the embodiments of the present invention are for illustrative purposes only, and that those skilled in the art may make various modifications, changes, substitutions, and additions within the technical spirit and scope of the appended claims, and such modifications, changes, etc. should be considered to fall within the scope of the following claims. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an image capturing apparatus according to an embodiment of the present invention; [Figure 2] 2 is a diagram illustrating an example of the structure of a pixel array included in the image sensor of FIG. 1; [Figure 3] FIG. 2 is a detailed block diagram of the image signal processor of FIG. 1. [Figure 4] 4 is a diagram illustrating an example of a pixel array for explaining the operation of the position determination unit of FIG. 3. FIG. [Figure 5] 10 is another embodiment for explaining the operation of the position determination unit in FIG. 3. [Figure 6] 4 is a diagram illustrating an example of a pixel array for explaining the operation of the weight setting unit of FIG. 3. FIG. [Figure 7] 4 is a diagram illustrating an example of a phase image for explaining the operation of the signal blending unit of FIG. 3. FIG. [Figure 8] 4 is a diagram for explaining the parallax in the radial direction and the circumferential direction in the parallax calculation unit of FIG. 3. FIG. [Figure 9] 4 is a diagram for explaining an operation for calculating radial parallax in the parallax calculation unit in FIG. 3. FIG. [Figure 10] 4 is a diagram for explaining an operation for calculating radial parallax in the parallax calculation unit in FIG. 3. FIG. [Figure 11] 4 is a diagram for explaining the operation of the parallax calculation unit in FIG. 3 for calculating parallax in the circumferential direction. FIG. [Figure 12] 4 is a diagram for explaining the operation of the parallax calculation unit in FIG. 3 for calculating parallax in the circumferential direction. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0012] Various embodiments will be described below with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to specific embodiments and includes various modifications, equivalents, and / or alternatives of the embodiments. The embodiments of the present disclosure can provide various effects that can be recognized directly or indirectly by the present disclosure.

[0013] 1 is a block diagram showing an image capturing device according to an embodiment of the present invention. The image capturing device 1 (e.g., a camera) according to the embodiment can capture a photograph (or video) by collecting light reflected from a subject. A method for performing an AF (Auto-Focus) function in the image capturing device 1 will be described with reference to FIG. 1.

[0014] 1, the photographing device 1 can be realized in various forms, such as a digital still camera for capturing still images or a digital video camera for capturing moving images. The photographing device 1 can also include a digital single-lens reflex camera (DSLR), a mirrorless camera, or a smartphone. However, the photographing device 1 according to an embodiment is not limited thereto, and can include a device equipped with multiple camera modules, each including a lens and an image sensor, that can capture an object and generate an image.

[0015] Such an image capturing device 1 may include an image capturing unit 10 , an image sensor 100 , an image signal processor 200 , and a memory 210 . Here, the image capturing unit 10 may be a component that receives light. Specifically, the image capturing unit 10 may include a lens 11, a lens driving unit 12, an aperture 13, and an aperture driving unit 140.

[0016] The lens 11 can collect light that has reflected from the object S and reached the image capture device 1. The lens 11 can refer not only to a single lens, but also to a configuration including multiple lenses (lens assembly) aligned in the optical axis direction. By adjusting the position of the lens 11, the focus on the object S can be changed. The position of the lens 11 can be based on signals generated from pixels of the image sensor 100.

[0017] The lens driver 12 can control the position of the lens 11 in response to a control signal from the image signal processor 200. That is, the lens driver 12 can adjust the position of the lens 11 to adjust the focal length and perform autofocus, zoom change, and focus change operations. By adjusting the position of the lens 11, the distance between the lens 11 and the object S can be adjusted. For example, the lens driver 12 can move the lens 11 parallel to the optical axis direction.

[0018] The aperture 13 can adjust the degree of opening and closing under the control of the aperture driver 14, thereby adjusting the amount of light entering the lens 11. By adjusting the amount of light (amount of received light) entering the lens 11 through the aperture 13, the magnitude of the signal generated from the image sensor 100 can be adjusted.

[0019] The aperture driver 14 controls the aperture 13 and can adjust the amount of light entering through the lens 11. For example, the aperture driver 14 can move the lens 11 parallel to the light emission direction. An optical signal transmitted through the lens 11 and the aperture 13 reaches the light receiving surface of the image sensor 100 and can form an image of the subject.

[0020] The image sensor 100 may include a pixel array (described later) in which a plurality of unit pixels are arranged two-dimensionally in a grid pattern. The image sensor 100 may generate a digital signal (or an electrical signal) based on light reflected from an object, and may generate digital image data (referred to as "image data") based on the electrical signal. Incident light (optical signal) passing through the lens 11 and the aperture 13 may be focused on the pixel array and converted into an electrical signal. Each unit pixel (described later) may generate an electrical signal corresponding to an external object S.

[0021] According to various embodiments, the image sensor 100 may include a photodiode (PD), a transfer transistor, a reset transistor, and a floating diffusion node (FD). The photodiode PD may generate and accumulate photocharges corresponding to an optical image of an object. The transfer transistor may transfer photocharges collected in the photodiode PD to the floating diffusion node FD in response to a transfer signal. The reset transistor may discharge charges stored in the floating diffusion node FD in response to a reset signal. Charges stored in the floating diffusion node FD before the reset signal is applied may be output. At this time, correlated double sampling (CDS) processing may be performed, and the CDS-processed analog signal may be converted into a digital signal via an analog-to-digital circuit (ADC) and / or an analog front end (AFE). The image sensor 100 disclosed herein may, for example, have four photodiodes in a unit pixel corresponding to one microlens (e.g., a 4PD pixel).

[0022] The unit pixels may be arranged in a matrix in a pixel array. An electrical signal generated from the unit pixel may include an image signal and a phase signal for the object S. Here, the image signal is a signal generated in response to light incident on the image sensor 100 from the object S and can be used as a signal for generating an image of the object S. The phase signal is a signal generated in response to light incident on the image sensor 100 from the object S and can be used as a signal for adjusting the distance between the object S and the lens 11. The unit pixel may be classified as a phase difference detection pixel or an image detection pixel depending on the signal it outputs.

[0023] According to an embodiment of the present invention, the phase difference detection pixels may be arranged in an N×N matrix (N is a natural number equal to or greater than 2). The image detection pixels may be arranged adjacent to the phase difference detection pixels. The structure of such a unit pixel will be described in more detail with reference to FIG. 2 below.

[0024] The image signal processor 200 can obtain image information, etc. based on the signals output from the image detection pixels. The image signal processor 200 can obtain phase information, etc. based on the signals output from the phase difference detection pixels.

[0025] That is, the image signal processor 200 can receive image data from the image sensor 100 and generate phase data (phase image). The image signal processor 200 can also process phase difference calculations used for autofocus operations based on the phase data. The image signal processor 200 can determine the position and direction of the focus, the distance between the object S and the photographing device 1, and the like through the phase difference calculations. The image signal processor 200 can provide a driving signal to the aperture driver 14 for adjusting the aperture value 13 based on the result of the phase difference calculations. The image signal processor 200 can also provide a driving signal to the lens driver 12 for adjusting the position of the lens 11 based on the result of the phase difference calculations.

[0026] The image signal processor 200 may perform image data processing for improving image quality, such as noise reduction, gain adjustment, waveform shaping, analog-to-digital conversion, interpolation, white balance, gamma processing, edge enhancement, etc. The image signal processor 200 may also change a region of interest (ROI) for an image based on information about the detected focus and information about the image of the subject.

[0027] Although the embodiment of FIG. 1 shows the image signal processor 200 as being external to the image sensor 100, the image signal processor 200 may be provided inside the image sensor 100 or may be provided separately outside the photographing device 1.

[0028] The memory 210 may store at least a portion of an image acquired via the image sensor 100 for subsequent image processing, or may store instructions or data related to the image signal processor 200. According to an embodiment, the memory 210 may store at least one correction data (e.g., white balance correction data, gamma correction data, knee correction data, parallax, lens drive amount, etc.). For example, the at least one correction data may be stored in a look-up table (LUT) format.

[0029] For ease of explanation, FIG. 1 shows memory 210 as being external to image signal processor 200, but this is not necessarily limited to what is shown; for example, memory 210 may be internal to image signal processor 200.

[0030] When there is no phase difference between signals generated from phase difference detection pixels included in the image sensor 100, the distance between the lens 11 and the object S may be an in-focus position. When the distance between the lens 11 and the object S is an in-focus position, the magnitude of incident light passing through one microlens and reaching each unit pixel may be the same, and therefore the magnitude of signals detected from each unit pixel sharing one microlens may be the same. For example, the position of the lens 11 where there is no phase difference between phase signals detected from phase difference detection pixels included in the pixel array may be the in-focus position.

[0031] On the other hand, if the distance between the lens 11 and the object S is not at a focal point, a difference may occur between the signals generated from the phase difference detection pixels. The magnitude of incident light reaching each unit pixel may differ depending on the position of the unit pixel within the pixel group. This is because a path difference may occur in the incident light passing through the microlens.

[0032] Therefore, if the distance between the lens 11 and the object S is not at a focus position, the magnitude of the phase signal of each unit pixel collected by the image signal processor 200 may differ. If the distance between the lens 11 and the object S is not at a focus position, the image signal processor 200 can calculate the difference in magnitude between the phase signals to generate phase data.

[0033] Based on the phase data, the image signal processor 200 can provide a drive signal to the lens driver 12. Based on the drive signal provided from the image signal processor 200, the lens driver 12 can move the lens 11 so that the distance between the lens 11 and the object S is at a correct focus position.

[0034] To improve autofocus performance in high-resolution images, image sensing devices have been developed with a structure in which one microlens is applied to multiple pixels (described later in Figure 2). Such image sensing devices focus on the parallax at a selected region of interest position, as the focal position varies depending on the object position.

[0035] However, due to factors such as lens astigmatism, disparity can vary, making accurate autofocus difficult. Astigmatism is the phenomenon or degree of difference between the image point where vertically diverging light beams converge and the image point where horizontally diverging light beams converge when light beams from an object point pass through an imaging optical system (lens) and reconverge to form an image. This results in the image of the object point appearing as a shape rather than a point (stigmatic). This astigmatism ultimately occurs when the rotational symmetry of the imaging optical system is broken, which has two causes. First, the imaging optical system itself is rotationally symmetric, but the imaging conditions break that rotational symmetry. Second, the imaging optical system itself is not rotationally symmetric.

[0036] That is, because the focus position varies depending on the texture direction of the image, it is difficult to determine the focus position where the parallax becomes "0" within the region of interest. Furthermore, because the focus is adjusted using the parallax in the same direction (for example, the left-right direction of the image) regardless of the position of the image, the central symmetry of the imaging system around the optical axis may be lost. In this case, the focus adjustment performance may become uneven in regions of interest that are far from the center point of the image.

[0037] Therefore, in this embodiment, by calculating the parallax taking into account the central symmetry with respect to the optical axis, it is possible to detect focus in each of the two directions of astigmatism (the radial direction and the circumferential direction, which will be described later), and it is possible to improve the spatial uniformity of the focus adjustment performance around the optical axis. In this embodiment, the operation of calculating the parallax taking into account the central symmetry will be described in more detail with reference to FIGS. 2 to 12, which will be described later.

[0038] FIG. 2 is a diagram illustrating an example of the structure of a pixel array included in the image sensor of FIG. 2, 16 unit pixels are shown arranged in a matrix including four rows and four columns. As an example, the 16 unit pixels may be repeated in the row and column directions as the minimum unit of the pixel array PA, but the scope of the present invention is not limited thereto.

[0039] The pixel array PA of the image sensor 100 may include pixel groups (PG1 to PG4). The pixel groups (PG1 to PG4) may be arranged in an N×N matrix (N is a natural number greater than or equal to 2). The pixel groups PG1 and PG4 may be arranged diagonally opposite each other, and the pixel groups PG2 and PG3 may be arranged diagonally opposite each other.

[0040] One microlens ML (corresponding to lens 11 in FIG. 1) may be formed in each pixel group (PG1 to PG4). For example, four unit pixels (PX1 to PX4) share one microlens ML, which may be referred to as an A4C (all 4-coupled) structure. The microlens ML can adjust the path of incident light entering the image sensor 100.

[0041] The pixel groups (PG1 to PG4) may be configured such that four unit pixels (PX1 to PX4) having the same color are adjacently arranged in an NxN matrix (N is a natural number equal to or greater than 2). When four light receiving elements are arranged in each of the pixel groups (PG1 to PG4), they may be arranged point-symmetrically in the upper right, upper left, lower right, and lower left directions based on the center of each pixel group. The four light receiving elements may correspond to the four unit pixels (PX1 to PX4), respectively.

[0042] For example, pixel group PG1 may include four unit pixels with green (Gr) color filters, pixel group PG2 may include four unit pixels with red (R) color filters, pixel group PG3 may include four unit pixels with blue (Gr) color filters, and pixel group PG4 may include four unit pixels with green (Gb) color filters.

[0043] That is, each pixel group (PG1 to PG4) may include four unit pixels (PX1 to PX4). Each unit pixel (PX1 to PX4) may include one photoelectric conversion element (not shown) corresponding to the unit pixel. The colors (red (R), green (Gr, Gb), and blue (B)) corresponding to the pixel groups (PG1 to PG4) may be arranged in a Bayer pattern. Therefore, raw images captured and generated by the image sensor 100 may include color image pixels arranged in a structure similar to the pixel array PA described above. The repeating arrangement structure and pattern of the pixel array PA are not limited thereto and may vary depending on the embodiment.

[0044] Within each pixel group (PG1-PG4), the positions at which the red, blue, or green pixels are located may be the same to reduce the amount of calculation required for image signal processing by the image signal processor 200, but the scope of the present invention is not limited to this. Also, in the embodiment of the present invention, each pixel group (PG1-PG4) is illustrated as including four unit pixels (PX1-PX4), but the number of unit pixels included in each pixel group is not limited to this.

[0045] A unit pixel included in the pixel array PA generates a signal corresponding to the object S in FIG. 1, which can be used to capture an image of the object S and generate a phase signal for autofocus. The phase signal can include information about the position on the pixel array PA of the unit pixel that generated the phase signal. The phase signal is transmitted to the image signal processor 200 and can be used to detect the distance between the object S and the lens 11. That is, autofocus operation using phase detection refers to detecting a phase difference between images generated by the unit pixels, calculating a lens movement distance from the detected phase difference, and adjusting the lens position to obtain a focused image.

[0046] In one embodiment, among the pixel groups (PG1 to PG4) of the pixel array PA, the pixel group PG1 may be set as the phase difference detection pixels (PX1 to PX4), but the positions of the phase difference detection pixels are not limited thereto. And, among the pixel groups (PG1 to PG4) of the pixel array PA, the remaining pixel groups (PG2 to PG4) may be set as the image detection pixels, but the positions of the image detection pixels are not limited thereto.

[0047] In Figure 2, the English letters (LT, LB, RT, RB) written on each unit pixel (PX1 to PX4) may indicate the position of the unit pixel within the pixel group PG. For example, in each pixel group (PG1 to PG4), the position of unit pixel PX1 located at the top left may be referred to as "LT (Left Top)," and the position of unit pixel PX2 located at the bottom left may be referred to as "LB (Left Bottom)." Furthermore, in each pixel group (PG1 to PG4), the position of unit pixel PX3 located at the top right based on the center point may be referred to as "RT (Right Top)," and the position of unit pixel PX4 located at the bottom right may be referred to as "RB (Right Bottom)."

[0048] FIG. 3 is a detailed block diagram of the image signal processor of FIG. Referring to FIG. 3, the image signal processor 200 may include a position determination unit 210, a weight setting unit 220, a signal blending unit 230, a parallax calculation unit 240, and a focus position determination unit 250.

[0049] Here, the position determination unit 210 may determine the position of each pixel group (PG1 to PG4) in the pixel array PA. The position determination unit 210 may then determine at what position on the image a pixel group (e.g., PG1) including a phase difference detection pixel (PX1 to PX4) is located among the pixel groups (PG1 to PG4) in the pixel array PA.

[0050] For example, the position determination unit 210 can detect the pixel positions while counting all unit pixels PX in the pixel array PA using the counter 211. The operation of the position determination unit 210 to determine the pixel positions will be described in more detail with reference to FIG.

[0051] According to another embodiment, the position determination unit 210 may use angle information between the optical axis of the pixel array PA and a target pixel group when determining the position of a pixel. For example, if pixel values ​​in the radial and circumferential directions are used as a linear sum of phase difference detection pixels, the position information may be expressed as a function of angle. Therefore, the position determination unit 210 may output angle information as pixel position information. The operation of the position determination unit 210 to detect the pixel position using angle information will be described in more detail with reference to FIG. 5, which will be described later.

[0052] The weight setting unit 220 may determine weights for each position (LT, LB, RT, RB) of the phase difference detection pixels (PX1 to PX4) based on position information of the pixel group detected by the position determination unit 210. That is, the weights applied to the phase difference detection pixels (PX1 to PX4) may be set differently for each position (LT, LB, RT, RB) of the phase difference detection pixels (PX1 to PX4). According to another embodiment, the weights applied to the phase difference detection pixels (PX1 to PX4) may be set differently depending on angle information of the phase difference detection pixels (PX1 to PX4) with respect to the optical axis. The weight setting operation of the weight setting unit 220 will be described in more detail with reference to FIG. 6, which will be described later.

[0053] The signal blending unit 230 may generate a plurality of phase images by blending the weights determined by the weight setting unit 220 with the pixel values ​​of the phase difference detection pixels PX1 to PX4. For example, the signal blending unit 230 may perform an operation of adding weights to input pixel data. The operation of the signal blending unit 230 will be described in more detail with reference to FIG. 7.

[0054] The disparity calculation unit 240 may calculate a parallax from a plurality of phase images generated by the signal blending unit 230 and output a disparity value corresponding to the calculated parallax. For example, the disparity calculation unit 240 may calculate the parallax using a cross correlation method. The disparity calculation unit 240 may then calculate the parallax in a radial direction and / or a circumferential direction from the center point of the optical axis.

[0055] In this embodiment, the parallax calculation is described for the radial direction and the circumferential direction, but this is merely an example, and the parallax can also be calculated based on other directions. The operation of such parallax calculation unit 240 will be described in more detail with reference to Figures 8 to 12 described later.

[0056] The focal position determination unit 250 can output a drive signal for determining the focal position of the lens 11 to the lens driver 12 based on the parallax value calculated by the parallax calculation unit 240. The focal position determination unit 250 can adjust the position of the lens 11 so as to set the disparity to "0" based on the parallax value calculated by the parallax calculation unit 240. For example, the focal position determination unit 250 can store the parallax information calculated by the parallax calculation unit 240, the lens driving amount, and the like in a lookup table in the memory 210 described above, and control the lens driver 12 using the data stored in the lookup table.

[0057] FIG. 4 is a diagram illustrating an example of a pixel array for explaining the operation of the position determination unit of FIG. 4, the pixel array PA may have raw image data arranged in a Bayer pattern, and pixel groups PG, each including a unit pixel (red pixel, blue pixel, or green pixel), may be repeatedly arranged in the row and column directions in the pixel array PA, as described above with reference to FIG.

[0058] In the pixel array PA, the index of a pixel group arranged in the row direction can be set to i, and the index of a pixel group arranged in the column direction can be set to j. The index i can be greater than or equal to "0" and can be less than or equal to h-1. Here, "h" can indicate the number of pixels in the horizontal direction (row direction) (horizontal cell size) in the pixel array PA. The index j can be greater than or equal to "0" and less than or equal to v-1. Here, "v" can indicate the number of pixels in the vertical direction (column direction) (vertical cell size) in the pixel array PA. The pixel groups arranged in the row direction can be set as i0, i1, i2, i3, etc., and the pixel groups arranged in the column direction can be set as j0, j1, j2, j3, etc.

[0059] The position determination unit 210 detects the position of raw image data in the pixel array PA and determines the position of a pixel group (e.g., PG1 in FIG. 2) including a phase difference detection pixel (e.g., PX1 to PX4 in FIG. 2) among the pixel groups (PG1 to PG4). The position determination unit 210 can determine the position by reading out data for all unit pixels in the pixel array PA.

[0060] For example, when the count value of the counter 211 is "0", the position determination unit 210 can output the position value (i, j) of the pixel group corresponding to i0, j0. When the count value of the counter 211 is "1", the position determination unit 210 can output the position value (i, j) of the pixel group corresponding to i0, j1. In this way, when the read operation for one i0 line (row line) is completed, a pixel position detection operation can be performed for the next i1 line (row line).

[0061] In this way, the position determination unit 210 can sequentially read out the data of all pixel groups PG and output the coordinate values ​​(i, j) for the position values ​​while incrementing the count value of the counter 211 by "1." When the counting operation for all pixel groups PG in the pixel array PA is completed, the counter 211 can be reset to the value "0" again.

[0062] FIG. 5 shows another embodiment for explaining the operation of the position determination unit of FIG. Referring to FIG. 5, the position determination unit 210 determines the position (e.g., (i, j) coordinates) of the pixel group for every pixel PX in the pixel array PA, and can output angle information between the optical axis and the pixel group (PG1 to PG4) corresponding to the target area as position information of the pixel group.

[0063] For example, in the pixel array PA, a line that crosses the center point CP of the optical axis in the horizontal direction can be defined as a horizontal line h, and a line that crosses the center point CP in the vertical direction can be defined as a vertical line v.

[0064] When the pixel array PA is divided by a vertical line v and a horizontal line h based on the center point CP of the optical axis, the pixel array PA can be divided into four regions (Q1 to Q4), i.e., four quadrants. The first region Q1 may be located in the first quadrant at the lower right side based on the center point CP. The second region Q2 may be located in the second quadrant at the lower left side based on the center point CP. The third region Q3 may be located in the third quadrant at the upper left side based on the center point CP. The fourth region Q4 may be located in the fourth quadrant at the upper right side based on the center point CP. In this embodiment, the positions of the four regions (Q1 to Q4) have been described above, but the positions of the four regions (Q1 to Q4) corresponding to the first to fourth quadrants may be different, and the number of divided regions may also be different.

[0065] Each pixel group PG may include a plurality of unit pixels (PX1 to PX4) that share one microlens. When the plurality of unit pixels (PX1 to PX4) are divided by position, they can be defined as an LT pixel region, an LB pixel region, an RT pixel region, and an RB pixel region, as described above with reference to FIG. 2.

[0066] In the embodiment of Fig. 5, the pixel pair that is closest in the vertical direction relative to the horizontal line h in the pixel region (LT, LB, RT, RB) is indicated by a solid line (Vertically Close; VC), and the pixel pair that is closest in the horizontal direction relative to the vertical line v in the pixel region (LT, LB, RT, RB) is indicated by a dotted line (Horizontally Close; HC).

[0067] In each region (Q1 to Q4), the position of the pixel region (LT, LB, RT, RB) closest or furthest from the optical axis will be explained with reference to Table 1 below.

[0068] [Table 1]

[0069] For example, "nhnv" can indicate the pixel area located closest to the optical axis in the horizontal and vertical directions. That is, the nhnv position corresponds to the LT pixel area in the Q1 area, the RT pixel area in the Q2 area, the RB pixel area in the Q3 area, and the LB pixel area in the Q4 area.

[0070] "nhfv" can refer to the pixel area that is closest to the optical axis horizontally and farthest from the optical axis vertically. That is, the LB pixel area in the Q1 area, the RB pixel area in the Q2 area, the RT pixel area in the Q3 area, and the LT pixel area in the Q4 area correspond to the nhfv position.

[0071] In addition, "fhnv" can refer to the pixel area located farthest from the optical axis horizontally and closest to the optical axis vertically. That is, the RT pixel area in the Q1 area, the LT pixel area in the Q2 area, the LB pixel area in the Q3 area, and the RB pixel area in the Q4 area correspond to the fhnv position.

[0072] "fhfv" can indicate the pixel area located farthest from the optical axis in the horizontal and vertical directions. That is, the RB pixel area in the Q1 area, the LB pixel area in the Q2 area, the LT pixel area in the Q3 area, and the RT pixel area in the Q4 area correspond to the fhfv position.

[0073] The angle at which each pixel group (PG1 to PG4) is positioned based on the center line CP of the optical axis (horizontal line h or vertical line v) of the image is defined as "θ." Therefore, the position determination unit 210 can calculate the angle θ(i, j) according to the pixel position (i, j) and output it to the weight setting unit 220.

[0074] FIG. 6 is a diagram illustrating an example of a pixel array for explaining the operation of the weight setting unit of FIG. 6, the weight setting unit 220 may determine a weight for each position (LT, LB, RT, RB) of a unit pixel based on the position value (i, j) of a pixel group PG in a pixel array PA. That is, the weight setting unit 220 may set different weights for each position of each unit pixel to calculate a disparity in a specific direction (e.g., a radial direction or a circumferential direction).

[0075] For example, the pixel array PA may be divided into four regions (region 1, region 2, region 3, and region 4), and different weights may be set for each unit pixel position (LT, LB, RT, and RB) in each region. Here, region 1 and region 3 may refer to the region above the center position in the horizontal direction h. Region 2 and region 4 may refer to the region below the center position in the horizontal direction h. Region 1 and region 2 may refer to the region to the left of the center position in the vertical direction v. Region 3 and region 4 may refer to the region to the right of the center position in the vertical direction v.

[0076] The method by which the weight setting unit 220 determines the weights wmn (where m and n are set to 1, 2, 3, and 4, respectively) can be set as shown in the following [Equation 1] to [Equation 4]. Hereinafter, as described in the embodiment of FIG. 4, a method of setting the weights by receiving the input of coordinates (i, j) for the position value from the position determination unit 210 will be described as an example.

[0077]

number

[0078] The weight setting unit 220 can set the weight wmn for the position value (i, j) in the region circle 1 as shown in the above [Equation 1].

[0079]

number

[0080] The weight setting unit 220 can set the weight wmn for the position value (i, j) in the area circle 2 as shown in the above [Equation 2].

[0081]

number

[0082] The weight setting unit 220 can set the weight wmn for the position value (i, j) in the area circle 3 as shown in the above [Equation 3].

[0083]

number

[0084] The weight setting unit 220 can set the weight wmn for the position value (i, j) in the area circle 4 as shown in the above [Equation 4]. The matrix of weights wmn set by the weight setting unit 220 can be expressed in the order shown in the following [Equation 5].

[0085]

number

[0086] According to another embodiment, the method by which the weight setting unit 220 determines the weights wmn (where m and n are set to 1, 2, 3, and 4, respectively) can be set as shown in the following [Equation 6] to [Equation 9]. Hereinafter, as described in the embodiment of FIG. 5, a method of setting the weights by receiving the angle θ(i, j) relative to the position value from the position determination unit 210 will be described as an example.

[0087]

number

[0088] The above [Equation 6] can represent the method by which the weight setting unit 220 determines the weights W11 to W14.

[0089]

number

[0090] The above [Equation 7] can represent the method by which the weight setting unit 220 determines the weights W21 to W24.

[0091]

number

[0092] The above [Equation 8] can represent the method by which weight setting section 220 determines the weights W31 to W34.

[0093]

number

[0094] The above [Equation 9] can represent the method by which the weight setting unit 220 determines the weights W41 to W44.

[0095] FIG. 7 is a diagram illustrating an example of a phase image for explaining the operation of the signal blending unit of FIG. Referring to FIG. 7, the signal blending unit 230 may generate a plurality of phase images (IMG1 to IMG4) by adding the weights wmn determined by the weight setting unit 220 to the pixel values ​​of the phase difference detection pixels (PX1 to PX4). Each of the phase images (IMG1 to IMG4) may include a plurality of unit pixels arranged in a 3×4 matrix. Here, the number of phase images (IMG1 to IMG4) may be equal to or less than the number of phase difference detection pixels (PX1 to PX4). For example, if the number of phase difference detection pixels (PX1 to PX4) (pixel regions (LT, LB, RT, RB)) is four, four phase images (IMG1 to IMG4) may be generated.

[0096] The blending operation in the signal blending unit 230 may be performed as shown in the following Equation 10. The following Equation 10 illustrates an example of a method of performing a signal blending operation based on the weight wmn calculated by the above-mentioned Equation 5.

[0097]

number

[0098] As shown in the above [Equation 10], the signal blending unit 230 can perform a blending operation on the phase images (IMG1, IMG2) in the radial direction by adding the weights set by the above [Equation 5] to the positions (LT, LB, RT, RB) of each unit pixel (PX1 to PX4).The signal blending unit 230 can also perform a blending operation on the phase images (IMG3, IMG4) in the tangential direction by adding the weights set by the above [Equation 5] to the positions (LT, LB, RT, RB) of each unit pixel (PX1 to PX4).

[0099] That is, the signal blending unit 230 may add weights (W11, W12, W13, W14) to the positions (LT, LB, RT, RB) of each unit pixel (PX1 to PX4) to generate a phase image IMG1(r+(i, j)). The signal blending unit 230 may add weights (W21, W22, W23, W24) to the positions (LT, LB, RT, RB) of each unit pixel (PX1 to PX4) to generate a phase image IMG2(r-(i, j)). The signal blending unit 230 may add weights (W31, W32, W33, W34) to the positions (LT, LB, RT, RB) of each unit pixel (PX1 to PX4) to generate a phase image IMG3(t+(i, j)). The signal blending unit 230 can add weights (W41, W42, W43, W44) to the positions (LT, LB, RT, RB) of each unit pixel (PX1 to PX4) to generate a phase image IMG4(t-(i, j)).

[0100] The phase difference data (e.g., luminance data) of the phase images (r+(i, j), r-(i, j), t+(i, j), t-(i, j)) calculated by the signal blending unit 230 can be stored in the aforementioned memory 210 (e.g., a buffer).

[0101] According to another embodiment, the blending operation in the signal blending unit 230 may be performed as shown in the following Equation 11. The following Equation 11 illustrates an example of a method for performing a signal blending operation based on the weights wmn calculated according to the above-described Equations 6 to 9.

[0102]

number

[0103] As shown in the above [Equation 11], the signal blending unit 230 can perform a blending operation on the phase images (IMG1, IMG2) in the radial direction by adding the weights (W11 to W24) set using the above [Equation 6] to [Equation 9] to the positions (nhnv, nhfv, fhnv, fhfv) of each unit pixel (PX1 to PX4).The signal blending unit 230 can perform a blending operation on the phase images (IMG3, IMG4) in the tangential direction by adding the weights (W31 to W44) set using the above [Equation 6] to [Equation 9] to the positions (nhnv, nhfv, fhnv, fhfv) of each unit pixel (PX1 to PX4).

[0104] FIG. 8 is a diagram for explaining the parallax in the radial direction and the circumferential direction in the parallax calculation unit of FIG. As shown in FIG. 8(a), phase image IMG1 and phase image IMG2 may have radial disparity (RD) due to radial parallax. Here, the radial direction may refer to the direction from the center point CP of the optical axis of an image (e.g., a circular image) toward the edge of the image. Radial parallax may indicate meridional distortion of the lens with respect to the image. That is, radial parallax may indicate disparity RD with respect to the radial distance from the center point CP of the optical axis of the image to the edge of the image. For example, radial parallax may occur due to non-uniformity in the refractive index of a lens.

[0105] A phase image IMG1 with a relatively large radial disparity RD can be defined as an "r+" image having positive radial distortion. For example, the "r+" image may have a relatively long radial length based on the center point CP. A phase image IMG2 with a relatively small radial disparity RD can be defined as an "r-" image having negative radial distortion. For example, the "r-" image may have a relatively short radial length based on the center point CP.

[0106] As shown in FIG. 8(b), the phase images IMG3 and IMG4 may have tangential disparity (TD) due to tangential parallax. Here, the tangential direction may be perpendicular to the radial direction. The tangential parallax may indicate decentering distortion of an image (e.g., a circular image) in the sagittal direction of the lens. That is, the tangential parallax may indicate disparity TD, in which the images are not parallel to each other but are offset at a certain angle based on a vertical line VL that vertically intersects the center point CP of the optical axis of the image and a horizontal line HL that horizontally intersects the center point CP of the optical axis of the image. For example, the tangential parallax may occur due to misalignment of the optical axes of the lens and the image sensor.

[0107] The phase image IMG3, which has a relatively large tangential disparity TD, can be defined as a "t+" image with positive tangential distortion. For example, the "t+" image can be rotated rightward around the center point CP. The phase image IMG4, which has a relatively small tangential disparity TD, can be defined as a "t-" image with negative tangential distortion. For example, the "t-" image can be rotated leftward around the center point CP.

[0108] 9 and 10 are diagrams for explaining the operation for calculating the parallax in the radial direction in the parallax calculation unit of FIG. Referring to FIG. 9, (a) may show the phase images (IMG1, IMG2) of FIG. 8 described above. The rectangular box-shaped areas may indicate focus detection areas, i.e., regions of interest (ROIs). In the embodiment of FIG. 9, an image of a portion of the area in the upper left corner of the circular image is set as the region of interest (ROI). However, this is merely an example, and the number of regions of interest (ROIs) and the size of each region of interest (the number of pixels included in each ROI) may be changed as needed.

[0109] 9(b), when the region of interest ROI in (a) is enlarged, it can be seen that the phase images (IMG1(r+), IMG2(r-)) have parallax in the radial direction. Therefore, the parallax calculation unit 240 can calculate the parallax between the phase images (IMG1(r+), IMG2(r-)) and output a disparity value corresponding to the calculated parallax.

[0110] In one embodiment, the disparity calculation unit 240 may calculate the disparity by performing a data correlation calculation. The correlation calculation method may be to calculate a data correlation value between a pixel to be subjected to the correlation calculation (e.g., a pixel of the image IMG2) and a reference pixel (e.g., a pixel of the image IMG1) while shifting the position of the pixel in the radial direction.

[0111] Here, the correlation calculation method may be, for example, SAD (sum of absolute difference), SSD (sum of squared difference), NCC (normalized cross-correlation), ZNCC (zero-mean normalized cross-correlation), census transform, and absolute differences census transform (AD-Census), or various other methods.

[0112] In the embodiment of FIG. 10, calculation of disparity using the SAD method will be described as an example. (a) of FIG. 10 may represent a pixel shift value where the position of the region of interest ROI in the phase image IMG2 is shifted. (b) of FIG. 10 may represent a state where the position of the region of interest ROI in the phase image IMG2 is shifted. (c) of FIG. 10 may represent a state where the position of the region of interest ROI in the phase image IMG1 is fixed without shifting. The disparity calculation unit 240 can extract a correlation calculation graph based on the SAD value, as shown in (d) of FIG. 10.

[0113] That is, the disparity calculation unit 240 calculates the disparity value for the object by calculating the SAD value based on the difference between the shift values ​​while shifting the region of interest ROI of the target phase image IMG2 in the radial direction based on the phase image IMG1. Here, the sum of the absolute values ​​of the difference values ​​of the two image data that are the subject of the correlation calculation can be defined as the SAD value.

[0114] According to one embodiment, the disparity calculation unit 240 may calculate a disparity (e.g., "3") as a pixel shift amount that minimizes the SAD in the correlation calculation graph. In addition, the disparity calculation unit 240 may calculate disparity by obtaining an offset of a peak position in a radial direction or a circumferential direction using a cross-correlation method. The disparity value calculated by the disparity calculation unit 240 may be used for adjusting a focus position, detecting depth information, etc.

[0115] That is, when the photographing apparatus 1 performs the initial focus detection process, if the disparity is determined to be 0, it can determine that the correct focus has been detected. On the other hand, if the disparity is determined to be not 0, the photographing apparatus 1 can control the addition range of the phase difference pixels.

[0116] 11 and 12 are diagrams for explaining the operation of the parallax calculation unit in FIG. 3 for calculating parallax in the circumferential direction. Referring to FIG. 11, (a) may show the phase images (IMG3, IMG4) of FIG. 7 described above. The areas indicated by dotted lines may indicate focus detection areas, i.e., regions of interest (ROIs). In the embodiment of FIG. 11, an image of a portion of the area present in the upper left corner of the circular image is set as the region of interest (ROI). However, this is merely an example, and the number of regions of interest (ROIs) and the size of each region of interest (the number of pixels included in each region of interest) may be changed as needed.

[0117] 11(b), when the region of interest ROI in (a) is enlarged, it can be seen that the phase images (IMG3(t+), IMG4(t-)) have parallax in the circumferential direction. Therefore, the parallax calculation unit 240 can calculate the parallax between the phase images (IMG3(t+), IMG4(t-)) and output a disparity value corresponding to the calculated parallax.

[0118] In one embodiment, the disparity calculation unit 240 can calculate a data correlation value with a reference pixel (e.g., a pixel of image IMG3) while shifting the position of the pixel (e.g., a pixel of image IMG4) that is the subject of the correlation calculation in the circumferential direction.

[0119] In the embodiment of FIG. 12, calculation of disparity using the SAD method described above will be described as an example. (a) of FIG. 12 may represent a pixel shift value obtained by shifting the position of the region of interest ROI in the phase image IMG4. (b) of FIG. 12 may represent a state in which the position of the region of interest ROI in the phase image IMG4 has shifted. (c) of FIG. 12 may represent a state in which the position of the region of interest ROI in the phase image IMG3 has been fixed without shifting. The disparity calculation unit 240 can extract a correlation calculation graph based on the SAD value, as shown in (d) of FIG. 12.

[0120] That is, the disparity calculation unit 240 can calculate a disparity value for the object by shifting the region of interest ROI of the target phase image IMG4 in the circumferential direction based on the phase image IMG3, obtaining an SAD value based on the difference in the shift values. According to one embodiment, the disparity calculation unit 240 can calculate the value of "2" that minimizes the SAD in the correlation calculation graph as the disparity value. As an example, the minimum SAD value in the correlation calculation graph may vary depending on the noise in the input image.

Claims

1. a pixel array including a plurality of image detection pixels and a plurality of phase difference detection pixels; a position determination unit that determines a position of each unit pixel in the pixel array; a weight setting unit that sets different weights for each position of each phase difference detection pixel based on an output of the position determination unit; a signal blending unit that generates a plurality of phase images by adding the weights set by the weight setting unit to the phase difference detection pixels; a parallax calculation unit that calculates a parallax in at least one direction from a center point of the optical axis in a first direction and a parallax in a second direction in the plurality of phase images; a focus position detection unit that generates a drive signal for adjusting a lens position based on the parallax calculated by the parallax calculation unit; an imaging device,

2. the pixel array includes a plurality of pixel groups; Each of the plurality of pixel groups The image sensor includes a plurality of unit pixels arranged in an N×N matrix, The imaging device according to claim 1 , wherein the plurality of unit pixels have color filters of the same color and share one microlens.

3. The position determination unit 2. The imaging device according to claim 1, wherein position values ​​of pixels corresponding to indexes of unit pixels arranged in a row direction and indexes of unit pixels arranged in a column direction in the pixel array are counted and sequentially read out.

4. The position determination unit The imaging device according to claim 1 , further comprising: detecting angle information between the optical axis and each of the unit pixels on the pixel array to determine a position of each of the unit pixels.

5. The position determination unit The pixel array is divided into four quadrants based on a horizontal line crossing a center point of the optical axis in a horizontal direction and a vertical line crossing a center point of the optical axis in a vertical direction; determining a position of a target pixel group according to a distance between each of the unit pixels located on the four quadrants and the center point; The imaging device according to claim 4 , wherein the angle information is calculated in accordance with the optical axis and the position of the pixel group.

6. The pixel array The optical axis is divided into four quadrants based on a horizontal line crossing the center point of the optical axis in a horizontal direction and a vertical line crossing the center point of the optical axis in a vertical direction, The weight setting unit The imaging device according to claim 1 , wherein the weights are set differently for each of four quadrants of the pixel array.

7. The imaging device according to claim 1 , wherein the number of the plurality of phase images is equal to or less than the number of the plurality of phase difference detection pixels.

8. The signal blending unit a first phase image having a positive radial distortion in the first direction; and The imaging device of claim 1 , wherein the imaging device produces a second phase image having negative radial distortion in the first direction.

9. The first direction is The imaging device of claim 8 , wherein the direction is a radial direction from a center point of the optical axis toward an edge side of the first phase image and the second phase image.

10. The radial distortion is The imaging device of claim 8 , having a disparity with respect to a radial distance from a center point of the optical axis to an edge of the first phase image and the second phase image.

11. The parallax calculation unit The imaging device according to claim 8 , wherein the position of the second phase image is shifted in the first direction, and the parallax is calculated according to a data correlation value with the first phase image.

12. The signal blending unit a third phase image having a positive tangential strain in the second direction; and The imaging device of claim 1 , wherein a fourth phase image is produced having a negative tangential distortion in the second direction.

13. The second direction is The imaging device according to claim 12 , wherein the first direction is a circumferential direction perpendicular to the first direction.

14. The tangential strain is The imaging device of claim 12 , wherein the first phase image and the second phase image have a disparity such that the first phase image and the second phase image are offset at a certain angle from the center point of the optical axis.

15. The parallax calculation unit The imaging device according to claim 12 , wherein the position of the fourth phase image is shifted in the second direction, and the parallax is calculated according to a data correlation value with the third phase image.

16. The parallax calculation unit 2. The imaging apparatus according to claim 1, wherein a data correlation value between two of the plurality of phase images is calculated, and a value that minimizes the correlation value is calculated as the disparity.

17. a pixel group including a plurality of unit pixels arranged in an N×N matrix; an image signal processor that generates a plurality of phase images by adding weights differently for each position of the plurality of unit pixels, and calculates parallax for the plurality of phase images in at least one of a first direction from a center point of the optical axis and a second direction perpendicular to the first direction; an imaging device,

18. 20. The imaging device of claim 17, wherein the pixel groups have color filters of the same color and share a single microlens.

19. The image signal processor The imaging device according to claim 17 , wherein the disparity relative to the radial distance from the center point of the optical axis to the edges of the plurality of phase images is calculated as the parallax.

20. The image signal processor The imaging device according to claim 17 , wherein the disparity calculated as the parallax is a disparity at which the plurality of phase images are shifted at a certain angle from the center point of the optical axis.

Images