Image processing apparatus and photographing apparatus including the same

Abstract

To provide an image processing device for generating a parallax image for AutoFocus (AutoFocus) in an image processing device and a photographing device including the image processing device.SOLUTION: The image processing device 200 may include an image combination unit 310 configured to generate a plurality of color images IMG1 to IMG4 by combining pixel data of unit pixels disposed at the same position in each of the plurality of pixel groups PG1 to PG4, and a brightness image processing unit 320 configured to calculate a weight value based on texture amplitudes of the plurality of color images IMG1 to IMG4 and a texture correlation value between the plurality of color images IMG1 to IMG4 and apply the weight value to the plurality of color images IMG1 to IMG4 to generate a brightness image AF_IMG.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device and a photographing device including the same, and more particularly to a technology relating to an image processing device for generating a parallax image for autofocus. [Background technology]

[0002] Image sensing devices are devices that capture optical images using the properties of photosensitive semiconductor materials that react to light.With the development of industries such as automobiles, medicine, computers, and communications, there is an increasing demand for high-performance image sensing devices in various fields such as smartphones, digital cameras, game consoles, the Internet of Things, robots, security cameras, and medical micro cameras.

[0003] 2. Description of the Related Art In a device (for example, a camera) for photographing an object, it is important to accurately focus on the object in order to capture a clear image (for example, a still image) or video (for example, a video). Summary of the Invention [Problem to be solved by the invention]

[0004] An embodiment of the present invention provides an image processing device and an imaging device including the same that can accurately adjust the focus position by analyzing texture and adding an appropriate weight when synthesizing pixel values ​​of each color pixel to generate a parallax image for autofocus. [Means for solving the problem]

[0005] An image processing device according to an embodiment of the present invention may include an image combination unit that combines pixel data of unit pixels arranged at the same position in each of a plurality of pixel groups to generate a plurality of color images; and a luminance image processing unit that calculates weights based on texture amplitudes of the plurality of color images and texture correlation values ​​between the plurality of color images and applies the weights to the plurality of color images to generate a luminance image.

[0006] According to another embodiment of the present invention, an imaging apparatus may include a first texture amplitude calculation unit that calculates a texture amplitude of a first color image; a first noise amount calculation unit that calculates a noise amount of the first color image; a calculation unit that generates an averaged image by averaging a second color image and a third color image having the same hue as the second color image; a first texture correlation determination unit that determines a texture correlation value between the first color image and the averaged image; a second texture amplitude calculation unit that calculates a texture amplitude of the averaged image; a second noise amount calculation unit that calculates a noise amount of the averaged image; a first weight calculation unit that calculates a weight based on outputs of the first and second texture amplitude calculation units, outputs of the first and second noise amount calculation units, and output of the first texture correlation determination unit; and a synthesis unit that generates a luminance image for autofocus by applying the weight to the first color image and the averaged image. [Effects of the Invention]

[0007] The embodiment of the present invention provides the effect of reducing texture noise and improving autofocus accuracy by controlling weights to preserve edge regions when synthesizing pixel values ​​of each color pixel.

[0008] Furthermore, the embodiments of the present invention are for illustrative purposes only, and those skilled in the art may make various modifications, changes, substitutions, and additions within the technical spirit and scope of the appended claims, and such modifications and variations should be considered to fall within the scope of the following claims. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an image capturing apparatus according to an embodiment of the present invention; [Figure 2] 2 is a diagram illustrating an example of the structure of a pixel array included in the image sensor of FIG. 1; [Figure 3] FIG. 2 is a detailed configuration diagram of a focus control unit in FIG. [Figure 4] FIG. 4 is a detailed configuration diagram of a luminance image processing unit in FIG. 3. [Figure 5] 4 is a flowchart for explaining the operation of the luminance image processing unit in FIG. 3. [Figure 6] 5 is a diagram for explaining the operation of the texture correlation determining unit in FIG. 4. FIG. [Figure 7] 5 is a diagram for explaining the operation of the weight value calculation unit in FIG. 4. FIG. [Figure 8] 5 is a diagram for explaining the operation of the weight value calculation unit in FIG. 4. FIG. [Figure 9] 5 is a diagram for explaining the operation of the weight value calculation unit in FIG. 4. FIG. [Figure 10] 5 is a diagram for determining when the absolute value of a weight is adjusted to be smaller in the weight calculation unit of FIG. 4. FIG. [Figure 11] FIG. 2 is a block diagram illustrating an example of a computing device corresponding to the image processing device of FIG. 1. DETAILED DESCRIPTION OF THE INVENTION

[0010] Various embodiments will be described below with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to specific embodiments and includes various modifications, equivalents, and / or alternatives of the embodiments. The embodiments of the present disclosure may provide various advantages that can be recognized directly or indirectly through the present disclosure.

[0011] 1 is a block diagram illustrating a photographing device according to an embodiment of the present invention. The photographing device 1 (e.g., a camera) according to the embodiment can capture a photograph (or a video) by collecting light reflected from a subject. A method for performing an autofocus (AF) function in the photographing device 1 can be described with reference to FIG. 1.

[0012] 1, the photographing device 1 may be implemented in various forms, such as a digital still camera for capturing still images or a digital video camera for capturing moving images. The photographing device 1 may also include a digital single-lens reflex camera (DSLR), a mirrorless camera, or a smartphone. However, the photographing device 1 according to an embodiment may include a device equipped with multiple camera modules, each including a lens and an image sensor, for capturing an image of a subject and generating an image.

[0013] Such an image capturing device 1 may include an image capturing unit 10, an image sensor 100, and an image processing device 200.

[0014] Here, the imaging unit 10 may be a component that receives light. Specifically, the imaging unit 10 may include a lens 11 and a lens driving unit 12.

[0015] The lens 11 can collect light reflected from the object (S) and reaching the image capturing device 1. The lens 11 can refer to a configuration including not only a single lens but also multiple lenses (lens assembly) aligned in the optical axis direction. By adjusting the position of the lens 11, the focus on the object (S) can be changed. The position of the lens 11 can be based on a signal generated by a pixel of the image sensor 100.

[0016] The lens driver 12 can control the position of the lens 11 in response to a control signal from the image processing device 200. That is, the lens driver 12 can adjust the position of the lens 11 to adjust the focal length and perform operations such as autofocusing, zoom change, and focus change. By adjusting the position of the lens 11, the distance between the lens 11 and the object (S) can be adjusted. For example, the lens driver 12 can move the lens 11 parallel to the optical axis direction.

[0017] The image sensor 100 may include a pixel array (described below) in which a plurality of unit pixels are arranged two-dimensionally in a grid pattern. The image sensor 100 may generate a digital signal (or an electrical signal) based on light reflected from an object, and may generate digital image data (referred to as "image data") based on the electrical signal. Incident light (optical signal) passing through the lens 11 may be focused on the pixel array and converted into an electrical signal. The unit pixels (described below) may each generate an electrical signal corresponding to an external object (S).

[0018] According to various embodiments, the image sensor 100 may include a photodiode (PD), a transfer transistor, a reset transistor, and a floating diffusion node (FD). The photodiode (PD) generates and accumulates photocharges corresponding to the optics of an object. The transfer transistor transfers photocharges collected in the photodiode (PD) to the floating diffusion node (FD) in response to a transfer signal. The reset transistor discharges charges stored in the floating diffusion node (FD) in response to a reset signal. Charges stored in the floating diffusion node (FD) before the reset signal is applied are output. At this time, correlated double sampling (CDS) processing may be performed, and the CDS-processed analog signal may be converted to a digital signal via an analog-to-digital circuit (ADC) and / or an analog front end (AFE). The image sensor 100 of the present disclosure may be, for example, a pixel having four photodiodes per pixel (e.g., a 4PD pixel) corresponding to one microlens.

[0019] The unit pixels may be arranged in a matrix shape in a pixel array. Electrical signals generated in the unit pixels may include an image signal and a phase signal for the object (S). Here, the image signal is a signal generated in response to light incident on the image sensor 100 from the object (S) and may be used as a signal for generating an image of the object (S). The phase signal is a signal generated in response to light incident on the image sensor 100 from the object (S) and may be used as a signal for adjusting the distance between the object (S) and the lens 11. The unit pixels may be classified as phase difference detection pixels or image detection pixels depending on the signals they output.

[0020] According to an embodiment of the present invention, the phase difference detection pixels may be arranged in an N×N matrix (N is a natural number equal to or greater than 2). The image detection pixels may be arranged adjacent to the phase difference detection pixels. The structure of such a unit pixel will be described in more detail with reference to FIG. 2, which will be described later.

[0021] The image processing device 200 can obtain image information, etc. based on the signals output from the image detection pixels. The image processing device 200 can obtain phase information, etc. based on the signals output from the phase difference detection pixels.

[0022] That is, the image processing device 200 can receive image data from the image sensor 100 and generate phase data (phase image). The image processing device 200 can also process a phase difference calculation used for an autofocus operation based on the phase data. The image processing device 200 can calculate the position and direction of the focus, and the distance between the object (S) and the photographing device 1, etc., through the phase difference calculation. The image processing device 200 can provide a driving signal to the lens driver 12 for adjusting the position of the lens 11 based on the result of the phase difference calculation.

[0023] The image processing device 200 may perform image data processing for improving image quality, such as, for example, re-mosaic processing, noise reduction processing, gain adjustment, waveform stylization processing, analog-to-digital conversion processing, interpolation processing, white balance processing, gamma processing, edge enhancement processing, etc. The image processing device 200 may also change a region of interest (ROI) for an image based on information about the detected focus and information about the image of the subject.

[0024] The image processing device 200 may also include a focus control unit 300. In another embodiment, the focus control unit 300 may be implemented separately from the image processing device 200.

[0025] The focus control unit 300 may determine weights to preserve edge regions when generating a luminance image used for autofocus adjustment by combining pixel values ​​of pixels. The luminance image for autofocus generated by the focus control unit 300 may be output to the lens driver 12 and used as a driving signal for determining a focus position. The focus control unit 300 may adjust the position of the lens 11 to set disparity to "0" based on the luminance image for autofocus. For example, the image processing device 200 may store the luminance image for autofocus generated by the focus control unit 300, disparity information, and lens driving amount in a lookup table in a memory, and control the lens driver 12 using the data stored in the lookup table.

[0026] In the embodiment of FIG. 1, the image processing device 200 is shown to be provided outside the image sensor 100, but the image processing device 200 may be provided inside the image sensor 100 or separately provided outside the photographing device 1.

[0027] The image sensing device includes a phase-difference detection autofocusing (PDAF) function that automatically focuses by detecting the phase difference between adjacent pixels. The phase-difference detection autofocusing method uses the phase difference between two or more different focus points to measure the offset direction and amount from the center image acquired through the image sensing device.

[0028] When there is no phase difference in signals generated by phase difference detection pixels included in the image sensor 100, the distance between the lens 11 and the object (S) may be the in-focus position. When the distance between the lens 11 and the object (S) is the in-focus position, the size of incident light passing through one microlens and reaching each unit pixel may be the same, and therefore the size of signals detected from each unit pixel sharing one microlens may be the same. For example, the position of the lens 11 where there is no phase difference in phase signals detected by phase difference detection pixels included in the pixel array may be the in-focus position.

[0029] Meanwhile, if the distance between the lens 11 and the object (S) is not at the exact focal point, a difference may occur between the signals generated by the phase difference detection pixels. The size of the incident light reaching each unit pixel may vary depending on the position of the unit pixel within the pixel group. This is because a path difference may occur in the incident light passing through the microlens.

[0030] Therefore, if the distance between the lens 11 and the object S is not at a focal point, the size of the phase signal of each unit pixel collected by the image processing device 200 may differ. If the distance between the lens 11 and the object S is not at a focal point, the image processing device 200 can generate phase data by calculating the difference in size between the phase signals.

[0031] The image processing device 200 can provide a driving signal to the lens driving unit 12 based on the phase data. The lens driving unit 12 can move the lens 11 based on the driving signal provided by the image processing device 200 so that the distance between the lens 11 and the object (S) becomes a precise focal position.

[0032] Meanwhile, image sensing devices have been developed that have a structure in which one microlens is applied to multiple pixels to improve autofocus performance in high-resolution images. Figure 2 below will explain in more detail the pixel array structure in which multiple pixels of the same color are arranged adjacent to each other and one microlens is applied to each of the multiple pixels.

[0033] Such an image sensing device focuses on the parallax at a selected region of interest, since the focal position varies depending on the position of the object (S). In an image sensing device having a structure in which one microlens is applied to multiple pixels, a luminance image for autofocusing can be generated by combining pixel values ​​of each color pixel. However, when detecting changes in pixel values ​​at edge regions between each color to generate a parallax image for autofocusing, the edge regions may not be preserved depending on the color of each color pixel. In such cases, it may be difficult to perform autofocusing accurately.

[0034] Therefore, the focus control unit 300 according to this embodiment may determine weights so that edge areas are preserved when generating a luminance image for autofocus by combining pixel values ​​(e.g., R (Red), G (Green), and B (Blue)). In this embodiment, the operation of generating a texture signal by controlling weights so that edge areas are preserved will be described in more detail with reference to FIGS. 2 to 10.

[0035] FIG. 2 is a diagram illustrating an example of the structure of a pixel array included in the image sensor of FIG.

[0036] 2, 16 unit pixels are shown arranged in a matrix including four rows and four columns. As an example, the 16 unit pixels may be repeated in the row and column directions as a minimum unit of a pixel array (PA), but the scope of the present invention is not limited thereto.

[0037] The pixel array (PA) of the image sensor 100 may include pixel groups (PG1 to PG4). The pixel groups (PG1 to PG4) may be arranged in an N×N (N is a natural number greater than or equal to 2) matrix. The pixel groups (PG1) and (PG4) may be arranged diagonally opposite each other, and the pixel groups (PG2) and (PG3) may be arranged diagonally opposite each other.

[0038] Each pixel group (PG1 to PG4) may have one microlens (ML) (corresponding to lens 11 in FIG. 1) formed therein. For example, four unit pixels (PX1 to PX4) may share one microlens (ML), which may be referred to as an A4C (all 4-coupled) structure. The microlens (ML) can adjust the path of incident light entering the image sensor 100.

[0039] The pixel groups (PG1 to PG4) may be configured such that four unit pixels (PX1 to PX4) having the same color are adjacently arranged in an N×N (N is a natural number equal to or greater than 2) matrix. When four light receiving elements are arranged in each of the pixel groups (PG1 to PG4), they may be arranged point-symmetrically in the upper right, upper left, lower right, and lower left directions based on the center of each pixel group. The four light receiving elements may correspond to the four unit pixels (PX1 to PX4), respectively.

[0040] For example, pixel group (PG1) may include four unit pixels with green (Gr) color filters, pixel group (PG2) may include four unit pixels with red (R) color filters, pixel group (PG3) may include four unit pixels with blue (B) color filters, and pixel group (PG4) may include four unit pixels with green (Gb) color filters.

[0041] That is, each pixel group (PG1 to PG4) may include four unit pixels (PX1 to PX4). Each unit pixel (PX1 to PX4) may include one photoelectric conversion element (not shown) corresponding to the unit pixel. The colors (red (R), green (Gr, Gb), and blue (B)) corresponding to the pixel groups (PG1 to PG4) may be arranged in a Bayer pattern. Therefore, raw images captured and generated by the image sensor 100 may include color image pixels arranged in a structure similar to the pixel array (PA) described above. The repeating arrangement structure and pattern of the pixel array (PA) are not limited thereto and may vary depending on the embodiment.

[0042] The positions of the red, blue, or green pixels in each pixel group (PG1 to PG4) may be the same to reduce the amount of calculation required for image signal processing in the image processing device 200, but the scope of the present invention is not limited thereto. Also, although the present invention has been described in the embodiment in which each pixel group (PG1 to PG4) includes four unit pixels (PX1 to PX4), the number of unit pixels included in each pixel group is not limited thereto.

[0043] The pixel array (PA) may include phase difference detection pixels and image detection pixels. Here, the image detection pixels may generate a signal corresponding to the object (S) in FIG. 1. The phase difference detection pixels may be used to capture an image of the object (S) and generate a phase signal for autofocusing. The phase signal may include information about the position on the pixel array (PA) of the phase difference detection pixel that generated the phase signal. The phase signal may be transmitted to the image processing device 200 and used to detect the distance between the object (S) and the lens 11.

[0044] In other words, autofocusing using phase detection means detecting the phase difference between a first image generated by some of the phase difference detection pixels and a second image generated by some of the other pixels, calculating the lens movement distance from the detected phase difference, and adjusting the lens position to obtain a focused image.

[0045] In one embodiment, the pixel group PG1 of the pixel groups PG1 to PG4 of the pixel array PA may be set as the phase difference detection pixels PX1 to PX4, but the locations of the phase difference detection pixels are not limited thereto. The remaining pixel groups PG2 to PG4 of the pixel groups PG1 to PG4 of the pixel array PA may be set as the image detection pixels, but the locations of the image detection pixels are not limited thereto.

[0046] 2, the English letters (TL, BL, TR, BR) written on each unit pixel (PX1 to PX4) may indicate the position of the unit pixel within the corresponding pixel group (PG). For example, in each pixel group (PG1 to PG4), the position of the unit pixel (PX1) located at the top left of the center point (center point of the microlens) may be referred to as "TL (Top Left)," and the position of the unit pixel (PX2) located at the bottom left of the center point may be referred to as "BL (Bottom Left)." Furthermore, in each pixel group (PG1 to PG4), the position of the unit pixel (PX3) located at the top right of the center point may be referred to as "TR (Top Right)," and the position of the unit pixel (PX4) located at the bottom right of the center point may be referred to as "BR (Bottom Right)."

[0047] FIG. 3 is a detailed configuration diagram of the focus control unit in FIG.

[0048] Referring to FIG. 3, the focus control unit 300 may include an image combining unit 310 and a luminance image processing unit 320 .

[0049] Here, the image combination unit 310 can output a plurality of color images (IMG1 to IMG4) by combining image data (IDATA) applied from the image sensor 100. The image data (IDATA) applied to the image combination unit 310 may have a form corresponding to the pixel array (PA) shown in FIG.

[0050] That is, the image combining unit 310 separates the image data (IDATA) into pixel groups (PG1 to PG4), and then combines the pixel data by collecting unit pixels arranged at the same position to generate four color images (IMG1 to IMG4). The pixel group (PG1) and the pixel group (PG4) may have the same green color (G), but may be arranged at different positions within one Bayer pattern.

[0051] For example, in this embodiment, green (G) may be divided into Gr and Gb, and Gr and Gb may be divided and treated as different colors. Also, in the embodiment of the present invention, the image combining unit 310 separates four channels to generate four color images (IMG1 to IMG4), but the number of color images can be changed depending on the position of each unit pixel and is not limited to this.

[0052] For example, color image (IMG1) can be generated by gathering pixels (TL) located in the upper left corner of each pixel group (PG1 to PG4) of image data (IDATA). Color image (IMG2) can be generated by gathering pixels (TR) located in the upper right corner of each pixel group (PG1 to PG4) of image data (IDATA). Color image (IMG3) can be generated by gathering pixels (BL) located in the lower left corner of each pixel group (PG1 to PG4) of image data (IDATA). Color image (IMG4) can be generated by gathering pixels (BR) located in the lower right corner of each pixel group (PG1 to PG4) of image data (IDATA).

[0053] The luminance image processor 320 processes the four color images IMG1 to IMG4 applied from the image combiner 310 using luminance data to generate an autofocus luminance image AF_IMG.

[0054] According to an embodiment, a photoelectric conversion signal of each light receiving element included in each unit pixel (PX1 to PX4) may include phase difference information of the unit pixel. The photoelectric conversion signals output when the light receiving elements arranged in the unit pixel are divided into upper, lower, left, and right groups may indicate the phase difference of light incident at different incident angles to the individual light receiving elements. In order to reduce the amount (or size) of data storage, the phase difference data including the phase difference information may be output in a form of YCbCr data representing a color space, including only luminance (Y) information, excluding Cb / Cr hue information. An image including this luminance information may be output as an autofocus luminance image (AF_IMG).

[0055] For example, the luminance image processor 320 may convert data of a color image (IMG1) into luminance data to generate one TL luminance image. The luminance image processor 320 may convert data of a color image (IMG2) into luminance data to generate one TR luminance image. The luminance image processor 320 may convert data of a color image (IMG3) into luminance data to generate one BL luminance image. The luminance image processor 320 may convert data of a color image (IMG4) into luminance data to generate one BR luminance image. The TL luminance image, TR luminance image, BL luminance image, and BR luminance image may be output to an autofocus luminance image (AF_IMG).

[0056] In the embodiment of the present invention, the luminance image processor 320 generates the luminance image (AF_IMG) by separating and combining unit pixels arranged at the same position, but the present invention is not limited to this and the luminance image (AF_IMG) can be generated by various methods.

[0057] However, when an edge region (or texture) exists between two different color images, the edges of the luminance image (AF_IMG) may not be preserved when the two color images are combined to generate the luminance image (AF_IMG). Therefore, the luminance image processor 320 according to this embodiment can determine the texture of the image and control the weighting value so that the edges are preserved when combining color images having different colors to generate the luminance image (AF_IMG) for autofocus. The operation of the luminance image processor 320 will be described in more detail with reference to FIGS. 4 to 10.

[0058] Fig. 4 is a detailed configuration diagram of the luminance image processing unit of Fig. 3. Fig. 5 is a flowchart for explaining the operation of the luminance image processing unit of Fig. 3. The operation of the luminance image processing unit 320 of Fig. 4 will be explained with reference to the flowchart of Fig. 5.

[0059] Referring to FIG. 4, the luminance image processing unit 320 may include a calculation unit 321, a plurality of texture amplitude calculation units 322 to 324, a plurality of noise amount calculation units 325 to 327, a plurality of texture correlation determination units 328 and 329, a plurality of weight calculation units 330 and 331, and a synthesis unit 332.

[0060] As described above with reference to FIG. 3, four color images (IMG1 to IMG4) may be input to the luminance image processor 320. In the embodiment of FIG. 4, the TL luminance image is generated by receiving the color image (IMG1) among the four color images (IMG1 to IMG4). The configuration and operation of receiving the remaining color images (IMG2 to IMG4) and generating the TR, BL, and BR luminance images are the same as those in FIG. 4, and therefore, redundant description will be omitted.

[0061] The luminance image processor 320 may receive a first image (R_IMG), a second image (G1_IMG), a third image (G2_IMG), and a fourth image (B_IMG). Here, the first image (R_IMG), the second image (G1_IMG), the third image (G2_IMG), and the fourth image (B_IMG) may correspond to a color image (IMG1). The first to fourth images (R_IMG, G1_IMG, G2_IMG, B_IMG) may further include, in addition to a target pixel (TL) at a processing target position corresponding to the color image (IMG1), surrounding pixels located around the target pixel and having the same color.

[0062] For example, the first image (R_IMG) may have red color TL pixels arranged in a 5x5 matrix. The second image (G1_IMG) may have green color TL pixels arranged in a 5x5 matrix. The third image (G2_IMG) may have green color TL pixels arranged in a 5x5 matrix. The fourth image (B_IMG) may have blue color TL pixels arranged in a 5x5 matrix.

[0063] In this embodiment, the first to fourth images (R_IMG, G1_IMG, G2_IMG, B_IMG) may be set as regions of interest (ROIs). (Step S1) The regions of interest may be arbitrarily set at various positions on the image sensor. In this embodiment, the number of regions of interest, the size (number of pixels included in each region of interest), and the position of each region of interest may be changed as necessary and are not limited thereto.

[0064] The calculation unit 321 may generate an averaged image by calculating pixel values ​​of the second image (G1_IMG) and the third image (G2_IMG). For example, the calculation unit 321 may average pixel values ​​of the second image (G1_IMG) and the third image (G2_IMG) to output an average value of green pixels.

[0065] Each of the texture amplitude calculation units 322 to 324 may calculate an amplitude for a luminance level of a texture image. (Step S2) The texture amplitude calculation unit 322 may calculate a texture amplitude of a red pixel included in the first image (R_IMG). The texture amplitude calculation unit 323 may calculate a texture amplitude of a green pixel included in the output of the calculation unit 321. The texture amplitude calculation unit 324 may calculate a texture amplitude of a blue pixel included in the fourth image (B_IMG).

[0066] Here, the "amplitude" of a texture may be an index indicating the degree of difference in brightness between two adjacent regions of the texture that differ in brightness. For example, if the texture is an edge region, the amplitude of the texture can be calculated based on the absolute value of the difference between the bright and dark regions. As another example, in the case of a complex texture, the amplitude of the texture can be calculated using the difference between the maximum and minimum values ​​within the texture region. As yet another example, in order to improve noise resistance, the texture amplitude can be calculated by finding the difference between the maximum and minimum values ​​using an average filter or the like. As yet another example, the brightness of the texture can be calculated using the variance of pixel values ​​or the like as an index.

[0067] The plurality of noise amount calculation units 325 to 327 may calculate the amount of noise of each pixel included in the first to fourth images (R_IMG, G1_IMG, G2_IMG, B_IMG). (Step S3) The noise amount calculation unit 325 may calculate the amount of noise of the first image (R_IMG). The noise amount calculation unit 326 may calculate the amount of noise in the output image of the calculation unit 321. The noise amount calculation unit 327 may calculate the amount of noise of the fourth image (B_IMG).

[0068] Here, the amount of noise in the texture region of the first to fourth images (R_IMG, G1_IMG, G2_IMG, B_IMG) may indicate noise variance or noise standard deviation. For example, the average pixel value in the texture region may be calculated and the sensor gain at the time of shooting may be additionally set. In this manner, information regarding the level of noise variance relative to the average pixel value, gain value, etc. may be obtained, thereby estimating the noise variance. The noise standard deviation may be obtained by calculating the standard deviation of image data generated in a state where light incident on the image sensing device is completely blocked during a test of the image sensing device.

[0069] In addition, the plurality of texture correlation determination units 328 and 329 can calculate a texture correlation value between two images based on the first image (R_IMG), the fourth image (B_IMG), and the output image of the calculation unit 321. (Step S4) That is, the texture correlation determination unit 328 can calculate a texture correlation value between the first image (R_IMG) and the output image of the calculation unit 321. And the texture correlation determination unit 329 can calculate a texture correlation value between the fourth image (B_IMG) and the output image of the calculation unit 321.

[0070] For example, the first image (R_IMG) may correspond to red pixels, and the output image of the calculation unit 321 may correspond to green pixels. The texture correlation determination unit 328 may determine whether the textures of the red pixels and green pixels have a positive (+) correlation or a negative (-) correlation. The fourth image (B_IMG) may correspond to blue pixels, and the output image of the calculation unit 321 may correspond to green pixels. The texture correlation determination unit 329 may determine whether the textures of the blue pixels and green pixels have a positive (+) correlation or a negative (-) correlation. The operation of determining the texture correlation value in the texture correlation determination units 328 and 329 will be described in more detail with reference to FIG. 6, which will be described later.

[0071] Furthermore, the weight calculation units 330 and 331 may calculate weights based on outputs from the texture amplitude calculation units 322 to 324, the noise amount calculation units 325 to 327, and the texture correlation determination units 328 and 329. (Step S5) The weight calculation unit 330 may calculate weights based on outputs from the texture amplitude calculation units 322 and 323, the noise amount calculation units 325 and 326, and the texture correlation determination unit 328. The weight calculation unit 331 may calculate weights based on outputs from the texture amplitude calculation units 323 and 324, the noise amount calculation units 326 and 327, and the texture correlation determination unit 329. The operation of calculating weights in the weight calculation units 330 and 331 will be described in more detail with reference to FIGS. 7 to 10, which will be described later.

[0072] In addition, the synthesis unit 332 synthesizes the first image (R_IMG), the output image of the calculation unit 321, and the fourth image (B_IMG) with the weights calculated by the weight calculation units 330 and 331 to generate a luminance image for autofocus (AF_IMG) (Step S6).

[0073] FIG. 6 is a diagram for explaining the operation of the texture correlation determining unit in FIG.

[0074] Referring to FIG. 6, the texture correlation determination units 328 and 329 may calculate a texture correlation value by determining whether pixel values ​​increase or decrease in edge regions of textures having two different hues.

[0075] In the case of an edge or gradation where black (BK) changes to yellow (Y) as in (a) of Figure 6, there may be a correlation in which the R pixel value also increases as the G pixel value increases. Here, when the correlation value between two adjacent regions with different hues is a positive number, this may be referred to as a "positive correlation."

[0076] On the other hand, in the case of an edge or gradation where red (R) changes to green (G) as shown in (b) of Figure 6, there may be a negative correlation where the R pixel value decreases as the G pixel value increases. Here, when the correlation value between two adjacent regions with different hues is negative, this may be referred to as a "negative correlation."

[0077] In this way, the texture correlation determination units 328 and 329 can use the correlation value (correlation coefficient) as an index that quantifies the correlation. The texture correlation determination units 328 and 329 can determine the correlation of the textures by determining whether the correlation coefficient is positive or negative.

[0078] For example, correlation can be determined by calculating the pixel average values ​​of the R image and G image within the region of interest, calculating the deviation (difference from the average value) of each pixel for the R image and G image, and determining the sign.

[0079]

number

[0080] When this calculation is expressed as a formula, it can be defined as shown in [Formula 1] above.

[0081]

number

[0082] As another example, correlation can be determined by using a majority vote of the signs of each pixel of the R image and the G image as in [Equation 2].

[0083] 7 to 9 are diagrams for explaining the operation of the weight value calculation unit in FIG.

[0084] The image (a) in FIG. 7 may include a red region and a green region. When the red region and the green region are displayed by color, they can be divided into an R image and a G image. In the R image, the pixel value may decrease from 200 to 0 from left to right. On the other hand, in the G image, the pixel value may increase from 0 to 200 from left to right. In other words, the pixel value decreases in the R image and increases in the G image, so the increase and decrease in pixel value may be opposite to each other in the R image and G image.

[0085] As shown in Figure 7(b), when the pixel values ​​of the R image are added to the pixel values ​​of the G image, the R+G pixel values ​​all become 200, so the difference in pixel values ​​on the left and right of the edge area can become "0." In such a case, the difference in brightness between the left and right texture areas becomes "0," making it impossible to perform autofocus.

[0086] On the other hand, if the pixel value of the R image is subtracted from the pixel value of the G image, as in Figure 7(c), the GR pixel value on the left becomes -200 and the GR pixel value on the right becomes 200, so the difference in pixel values ​​on the left and right of the edge area increases to 400. In this case, the brightness difference between the left and right texture areas increases to 400, which can be used for autofocus operations.

[0087] In this way, the weight calculation units 330 and 331 can calculate weights by performing a subtraction operation when the pixel values ​​of the R image and the G image increase or decrease in opposite directions. In other words, when the pixel values ​​of the R image and the G image increase or decrease in opposite directions in an edge region, if the weight of the G image is positive, the weight calculation units 330 and 331 can calculate weights by setting the weight of the R image to a negative value. In other words, when the brightness change of the R image is opposite to that of the G image, the weight calculation units 330 and 331 can calculate weights by subtracting the weight.

[0088] The image (a) in FIG. 8 may include a yellow area and a black area. When the yellow area and the black area are displayed by color, they can be divided into an R image and a G image. In the R image, the pixel value may decrease from 200 to 0 from left to right. Similarly, in the G image, the pixel value may decrease from 200 to 0 from left to right. In other words, since the pixel values ​​decrease in both the R image and the G image, the increase and decrease in pixel values ​​may be the same.

[0089] As shown in (b) of Figure 8, when the pixel value of the R image is added to the pixel value of the G image, the R+G pixel value on the left side becomes 400 and the R+G pixel value on the right side becomes 0, so the difference in pixel values ​​between the left and right of the edge area increases to '400'. In this case, the brightness difference between the left and right texture areas increases to '400', which can be used for autofocus operations. In this way, the weight calculation units 330 and 331 can calculate weights by performing an addition operation when the increase or decrease in pixel values ​​of the R image and G image is the same.

[0090] On the other hand, if the pixel values ​​of the R image are subtracted from the pixel values ​​of the G image, as in Figure 8(c), all GR pixel values ​​will be 0, so the difference in pixel values ​​on the left and right of the edge area may be "0." In such a case, the difference in brightness between the left and right texture areas will be "0," making it impossible to perform autofocus.

[0091] The image (a) in Figure 9 may include a green area and a black area. When the green area and the black area are displayed by color, they can be divided into an R image and a G image. In the R image, the pixel value remains unchanged at 0 from left to right. In the G image, the pixel value may decrease from 200 to 0 from left to right. In other words, since the pixel value remains unchanged in the R image and decreases in the G image, the increase and decrease in pixel value are not opposite between the R image and the G image.

[0092] As shown in Figure 9(b), when the pixel values ​​of the R image are added to the pixel values ​​of the G image, the R+G pixel value on the left side becomes 200, and the R+G pixel value on the right side becomes 0, so there is no change in the difference in pixel values ​​on the left and right sides of the edge area compared to when there is only the G image.

[0093] When the pixel value of the R image is subtracted from the pixel value of the G image, as shown in Figure 9(c), the GR pixel value on the left side becomes 200 and the GR pixel value on the right side becomes 0, so there is no change in the difference in pixel values ​​on the left and right sides of the edge area compared to when there is only a G image.

[0094] In this case, the signal-to-noise ratio (SNR) of the texture increases, making it impossible to perform autofocus. A small SNR of the texture may mean that the amplitude of the texture (e.g., the brightness difference of an edge) is small compared to the size of the noise. When the increase and decrease in pixel values ​​of the R image and the G image are the same and the difference in pixel values ​​between the addition operation and the subtraction operation is the same, the weight calculation units 330 and 331 may set a small absolute value of the weight to reduce noise.

[0095] The method for calculating the weights in the weight calculation units 330 and 331 will now be described in detail.

[0096] According to an embodiment, the following [Equation 3] to [Equation 5] may show an example of setting a weight when the texture amplitude is calculated using the difference between the maximum and minimum values ​​in the texture region and the texture amplitude does not have a sign.

[0097]

number

[0098] In Equation 3, WR may represent a weighting value in the R image, TR may represent the texture amplitude in the R image, and VR may represent the standard deviation of noise in the R image. CR may represent a parameter value that is 1 when the R image and the G image have a positive correlation and is -1 when they have a negative correlation. TG may represent the texture amplitude in the G image, and VG may represent the standard deviation of noise in the G image. According to Equation 3, when the signal-to-noise ratio of the texture is small, that is, when TR is small and VR is large, the weighting value may be small.

[0099]

number

[0100] In Equation 4, WG represents a weight value in the G image, and the weight value in the G image can be set to a reference value of 1.

[0101]

number

[0102] In Equation 5, WB may represent a weighting value for the B image, TB may represent the amplitude of the texture in the B image, and VB may represent the standard deviation of the noise in the B image. CB may represent a parameter value that is '1' when the B image and the G image have a positive correlation and '-1' when they have a negative correlation. According to Equation 5, when the signal-to-noise ratio of the texture is small, i.e., when TB is small and VB is large, the weighting value may be small.

[0103] When the texture has a negative correlation as in Equation 3 to Equation 5, the weights may have a negative sign. A negative weight may mean that the weights are calculated by performing a subtraction operation. That is, as described in FIG. 7, the weight calculation units 330 and 331 may calculate the weights by performing a subtraction operation when the pixel values ​​of the R image (or B image) and the G image increase or decrease in opposite directions.

[0104] In another embodiment, the following [Equation 6] to [Equation 8] may show an example of setting a weight when the texture amplitude is calculated using the absolute value of the difference between the bright and dark parts in the texture area and the texture amplitude has a sign.

[0105]

number

[0106] In Equation 6, WR represents the weighting value of the R image, TR represents the texture amplitude of the R image, and VR represents the standard deviation of the noise in the R image. According to Equation 6, when the signal-to-noise ratio of the texture is small, that is, when TR is small and VR is large, the weighting value may be small. That is, the greater the absolute value of the texture amplitude (TR) of the R image relative to the G image, the greater the absolute value of the weighting value, and the greater the noise standard deviation (VR) of the R image, the smaller the weighting value may tend to be.

[0107]

number

[0108] In Equation 7, WG represents a weight value in the G image, and the weight value in the G image can be set to a reference value of 1.

[0109]

number

[0110] In Equation 8, WB may represent a weighting value in the B image, TB may represent the amplitude of the texture in the B image, and VB may represent the standard deviation of the noise in the B image. According to Equation 8, when the signal-to-noise ratio of the texture is small, that is, when TB is small and VB is large, the weighting value may be small.

[0111] Hereinafter, the operation of the composition unit 332 to compose the weights (WR, WG, WB) calculated by the above mathematical formula with the first to fourth images (R_IMG, G1_IMG, G2_IMG, B_IMG) to generate the autofocus brightness image (AF_IMG) will be described.

[0112] The synthesis unit 332 can apply weights to the first image (R_IMG), the average image of the second image (G1_IMG) and the third image (G2_IMG), and the fourth image (B_IMG) as shown in the following [Equation 9].

[0113]

number

[0114] In Equation 9, O' may represent an autofocus luminance image (AF_IMG), i.e., an output signal, output from the composition unit 332. The output signal (O') may be calculated by adding a first value obtained by multiplying the coordinates of the R image (R(x, y)) by a weight (WR), a second value obtained by multiplying the coordinates of the G image (G(x, y)) by a weight (WG), and a third value obtained by multiplying the coordinates of the B image (B(x, y)) by a weight (WB).

[0115] If the weight value is negative, the output signal (O') may also be negative. Depending on the embodiment, if subsequent processing requires a positive number, a positive constant (Z) can be added to the entire equation to prevent the output signal (O') from becoming negative.

[0116]

number

[0117] As shown in Equation 10, a positive output signal (O) can be obtained by adding a constant (Z) to the output signal (O'). The constant (Z) can be a positive value if it is greater than the inverted sign of the minimum value of the output signal (O').

[0118] If the constant (Z) is predetermined, the weighting value can be set as shown in the following [Equation 11] and [Equation 12].

[0119]

number

[0120]

number

[0121] The synthesis unit 332 synthesizes the weights (W'R, W'G, W'B) calculated by [Equation 11] with the R image, G image, and B image, respectively, to calculate an output signal (O'').

[0122] FIG. 10 is a diagram for determining when the absolute value of the weight is adjusted to a smaller value in the weight calculation unit of FIG.

[0123] As shown in (a) of Figure 10, when the standard deviation of the noise is 40 and the brightness difference in the edge region is 10, the edge between two different colors is blurred, reducing the color edge contrast of the image. As another example, as shown in (b) of Figure 10, when the standard deviation of the noise is 10 and the brightness difference in the edge region is 10, i.e., when the standard deviation of the noise and the brightness difference are the same, the edge between two textures may appear blurred rather than sharp. As another example, as shown in (c) of Figure 10, when the standard deviation of the noise is 1, even if the brightness difference in the edge region is 10, the edge is preserved and appears clear.

[0124] Therefore, the weight calculation units 330 and 331 according to this embodiment can determine colors with a small signal-to-noise ratio and adjust the absolute value of the weight to be smaller.

[0125]

number

[0126] For example, assume that the signal-to-noise ratio (SNR) value is small, about 1 to 2. As shown in Equation 13, the SNR can be calculated by dividing the texture amplitude (TR) by the noise standard deviation (VR). For example, if the SNR is smaller than a preset value of 1, the weight calculation units 330 and 331 can set the weight to 0, which is lower than 1. That is, the weight calculation units 330 and 331 can detect the texture signal-to-noise ratio and adjust the weight in the R image to be smaller than the preset value if the SNR is smaller than a preset value.

[0127] FIG. 11 is a block diagram illustrating an example of a computing device corresponding to the image processing device of FIG.

[0128] Referring to FIG. 11, a computing device 1000 may represent one embodiment of a hardware configuration for performing the operations of the image processing device 200 of FIG.

[0129] The computing device 1000 may be mounted on a chip separate from the chip on which the image sensing device is mounted. According to an embodiment, the chip on which the image sensing device is mounted and the chip on which the computing device 1000 is mounted may be embodied in a single package, for example, a multi-chip package (MCP), although the scope of the present invention is not limited thereto.

[0130] The computing device 1000 may include a processor 1010, a memory 1020, an input / output interface 1030, and a communication interface 1040.

[0131] The processor 1010 is capable of processing data and / or instructions necessary to perform the operations of the components 310, 320 of the image processing device 200 described in FIG.

[0132] The memory 1020 can store data and / or instructions necessary to perform the operations of the components 310, 320 of the image processing device 200, and can be accessed by the processor 1010. For example, the memory 1020 can be embodied as a volatile memory (e.g., a dynamic random access memory (DRAM), a static random access memory (SRAM), etc.) or a non-volatile memory (e.g., a programmable read only memory (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, etc.).

[0133] That is, a computer program for performing the operations of the image processing device 200 disclosed herein may be recorded in the memory 1020 and executed and processed by the processor 1010 to implement the operations of the image processing device 200.

[0134] The input / output interface 1030 may provide an interface for connecting an external input device (e.g., a keyboard, a mouse, a touch panel, etc.) and / or an external output device (e.g., a display) to the processor 1010 to allow data to be sent and received.

[0135] The communication interface 1040 is configured to be capable of transmitting and receiving various data to and from an external device (for example, an application processor, an external memory, etc.), and may be a device capable of supporting wired or wireless communication. [Explanation of symbols]

[0136] 1. Imaging device 10. Imaging unit 11 Lens 12 Lens drive unit 100 image sensors 200 Image Processing Device 300 Focus control unit 310 Image Association Department 320 Luminance Image Processing Unit 321 Arithmetic section 322, 323, 324 Texture amplitude calculation section 325, 326, 327 Noise amount calculation section 328,329 Texture correlation judgment unit 330,331 Weighted value calculation unit 332 Synthesis Section 1000 computing devices 1010 processor 1020 memory 1030 Input / Output Interface 1040 Communication Interface

Claims

1. an image combining unit that combines pixel data of unit pixels arranged at the same position in each of the plurality of pixel groups to generate a plurality of color images; a luminance image processing unit that calculates weights based on texture amplitudes of the plurality of color images and texture correlation values ​​between the plurality of color images, and applies the weights to the plurality of color images to generate a luminance image.

2. The luminance image processing unit The image processing apparatus according to claim 1 , further comprising a calculation unit that generates an averaged image by averaging pixel values ​​of some of the plurality of color images.

3. The luminance image processing unit 3. The image processing apparatus of claim 2, further comprising a plurality of texture correlation determiners for determining the texture correlation value between each of the plurality of color images and the averaged image.

4. The plurality of texture correlation determination units include:

4. The image processing apparatus according to claim 3, further comprising: determining a deviation of each pixel between two color images having different hues from among the plurality of color images; determining a sign of the deviation; and calculating the texture correlation value.

5. The plurality of texture correlation determination units include: The image processing apparatus according to claim 3 , wherein the texture correlation value is calculated by determining whether pixel values ​​increase or decrease in edge regions of two color images having different hues from among the plurality of color images.

6. The luminance image processing unit The image processing device of claim 1 , further comprising: a plurality of texture amplitude calculation units configured to calculate the texture amplitude based on an absolute value of a brightness difference between two adjacent regions of the plurality of color images, the brightness of which differs from one another.

7. The luminance image processing unit The image processing apparatus according to claim 1 , further comprising a plurality of noise amount calculation units that calculate the amount of noise in each of the plurality of color images.

8. The luminance image processing unit The image processing apparatus according to claim 7 , further comprising a plurality of weight calculation units that calculate the weights based on the texture amplitude, the noise amount, and the texture correlation value.

9. The plurality of weight value calculation units The image processing device of claim 8 , wherein the weighting value is calculated by addition when the texture correlation value has a positive correlation, and the weighting value is calculated by subtraction when the texture correlation value has a negative correlation.

10. The plurality of weight value calculation units The image processing apparatus of claim 8 , wherein the absolute value of the weight is set to be smaller than a preset value when the texture correlation value does not increase or decrease.

11. The plurality of weight value calculation units The image processing apparatus of claim 8, wherein if a signal-to-noise ratio based on the amount of noise is smaller than a preset value, the absolute value of the weight is set to be smaller than a preset value.

12. a first texture amplitude calculation unit for calculating a texture amplitude of the first color image; a first noise amount calculation unit that calculates the amount of noise in the first color image; a calculation unit that averages the second color image and a third color image having the same hue as the second color image to generate an averaged image; a first texture correlation determining unit for determining a texture correlation value between the first color image and the averaged image; a second texture amplitude calculation unit for calculating a texture amplitude of the averaged image; a second noise amount calculation unit that calculates the amount of noise in the averaged image; a first weight calculation unit that calculates a weight based on outputs of the first and second texture amplitude calculation units, outputs of the first and second noise amount calculation units, and an output of the first texture correlation determination unit; a synthesis unit that applies the weighting value to the first color image and the averaged image to generate a luminance image for autofocus.

13. Each of the first texture amplitude calculation unit and the second texture amplitude calculation unit The imaging device of claim 12 , wherein the textures of the first color image and the second color image have different brightness levels, and the texture amplitude is calculated based on an absolute value of a brightness difference between two adjacent regions.

14. The first texture correlation determination unit The imaging device according to claim 12 , further comprising: determining a deviation of each pixel between the first color image and the averaged image; and determining a sign of the deviation to calculate the texture correlation value.

15. The first texture correlation determination unit The image capturing apparatus of claim 12 , wherein the texture correlation value is calculated by determining whether pixel values ​​increase or decrease in edge regions between the first color image and the averaged image.

16. The first weight value calculation unit The imaging device according to claim 15 , wherein the weighting value is calculated by addition when the texture correlation value has a positive correlation, and the weighting value is calculated by subtraction when the texture correlation value has a negative correlation.

17. The imaging device of claim 12 , wherein the first color image has a red color, and the second color image and the third color image have a green color.

18. The first weight value calculation unit 18. The imaging device according to claim 17, wherein the weighting value is calculated so that a subtraction is made when the first color image has an opposite change in brightness to the averaged image.

19. an image sensor including a plurality of pixel groups for generating image data; The imaging device of claim 12 , further comprising an image combining unit that generates the first color image and the second color image by combining pixel data of unit pixels arranged at the same position in each of the plurality of pixel groups.

20. Each of the plurality of pixel groups The pixel array includes a plurality of unit pixels arranged in an N×N matrix shape, where N is a constant equal to or greater than 2; The imaging device according to claim 19 , wherein the plurality of unit pixels have color filters of the same color and share one microlens.

Images