Image processing apparatus and imaging apparatus including same

Abstract

Disclosed are an image processing apparatus and an imaging apparatus including the same, which relate to a technique relating to an image processing apparatus for generating a parallax image required for autofocus. The image processing apparatus includes: an image combiner configured to generate a plurality of color images by combining pixel data of unit pixels arranged at the same position in each of a plurality of pixel groups; and a luminance image processor configured to calculate a weight based on a texture amplitude of the plurality of color images and a texture correlation value between the plurality of color images, and generate a luminance image by applying the weight to the plurality of color images.

Description

Technical Field

[0001] The technologies and implementations disclosed in this patent document generally relate to image processing apparatuses and imaging apparatuses including the same, and more specifically, to technologies related to image processing apparatuses for generating parallax images required for autofocus. Background Technology

[0002] Image sensing devices are devices that capture optical images by converting light into electrical signals using photosensitive semiconductor materials that react to light. With the development of the automotive, medical, computer, and communications industries, the demand for high-performance image sensing devices is constantly increasing in various fields such as smartphones, digital cameras, game consoles, IoT (Internet of Things), robotics, security cameras, and medical miniature cameras.

[0003] In a device (e.g., a camera) used to photograph a target object, it may be important to focus accurately on the target object in order to capture a clear image (e.g., a still image) or video (e.g., a moving image). Summary of the Invention

[0004] Various embodiments of the disclosed technology relate to an image processing apparatus and an imaging apparatus that can accurately adjust the focus position by analyzing texture and adding appropriate weights when synthesizing pixel values ​​of corresponding color pixels to generate a parallax image for autofocus.

[0005] According to an embodiment of the disclosed technology, an image processing apparatus may include: an image combiner configured to generate a plurality of color images by combining pixel data of unit pixels arranged in the same position in each of a plurality of pixel groups of a pixel array; and a luminance image processor configured to calculate weights based on the texture amplitude of the plurality of color images and texture correlation values ​​between the plurality of color images, and to generate a luminance image by applying the weights to the plurality of color images.

[0006] According to another embodiment of the disclosed technology, an imaging apparatus may include: a first texture amplitude calculator configured to calculate the texture amplitude of a first color image; a first noise level calculator configured to calculate the noise level of the first color image; a calculator configured to generate an average image by averaging a second color image and a third color image having the same color as the second color image; a first texture correlation determiner configured to determine a texture correlation value between the first color image and the average image; a second texture amplitude calculator configured to calculate the texture amplitude of the average image; a second noise level calculator configured to calculate the noise level of the average image; a first weight calculator configured to calculate weights based on the output signals of the first and second texture amplitude calculators, the output signals of the first and second noise level calculators, and the output signal of the first texture correlation determiner; and a synthesizer configured to generate a brightness image for autofocus by applying weights to the first color image and the average image.

[0007] It will be understood that both the above general description and the following detailed description of the disclosed technology are illustrative and explanatory, and are intended to provide further explanation of the claimed disclosure. Attached Figure Description

[0008] The above and other features and advantages of the disclosed technology will become readily apparent when considered in conjunction with the accompanying drawings, with reference to the following detailed description.

[0009] Figure 1 This is a block diagram illustrating examples of imaging devices based on some implementations of the disclosed technology.

[0010] Figure 2 These are examples of some implementations based on the disclosed technologies. Figure 1 A diagram showing an example structure of the pixel array included in an image sensor.

[0011] Figure 3 These are examples of some implementations based on the disclosed technologies. Figure 1 A schematic diagram of an example focus controller is shown.

[0012] Figure 4 These are examples of some implementations based on the disclosed technologies. Figure 3 A schematic diagram of an example of a brightness image processor is shown.

[0013] Figure 5 These are examples of some implementations based on the disclosed technologies. Figure 3 The flowchart shows an example operation of the brightness image processor.

[0014] Figure 6 (a) and Figure 6 (b) illustrates some implementations based on the disclosed technology. Figure 3 A diagram illustrating an example operation of the texture-dependent determiner.

[0015] Figure 7 (a) to Figure 7 (c) illustrates some implementations based on the disclosed techniques when the image includes both red and green areas, such as Figure 4 The diagram shows an example operation of the weight calculator.

[0016] Figure 8 (a) to Figure 8 (c) illustrates some implementations based on the disclosed techniques when the image includes yellow and black areas, such as Figure 4 The diagram shows an example operation of the weight calculator.

[0017] Figure 9 (a) to Figure 9 (c) illustrates some implementations based on the disclosed techniques when the image includes green and black areas, such as Figure 4 The diagram shows an example operation of the weight calculator.

[0018] Figure 10 (a) to Figure 10 (c) is an example of how the determination of some implementation methods based on the disclosed technology is determined by Figure 4 A diagram illustrating how the weight calculator adjusts the absolute values ​​of each weight to a smaller example case.

[0019] Figure 11 These are examples of some implementations based on the disclosed technology. Figure 1 A block diagram of an example computing device corresponding to an image processing device. Detailed Implementation

[0020] This patent document provides implementations and examples of an image processing apparatus and an imaging apparatus including the same, and more specifically, relates to technology related to an image processing apparatus for generating a parallax image required for autofocus, which can be used in a configuration that substantially solves one or more technical or engineering problems and mitigates limitations or disadvantages encountered in some image processing apparatuses in the art. Some implementations of the disclosed technology relate to an image processing apparatus and an imaging apparatus including the same that can accurately adjust the focus position by analyzing texture and adding appropriate weights when synthesizing pixel values ​​of corresponding color pixels to generate a parallax image for autofocus. Recognizing the above-mentioned problems, the image processing apparatus and the imaging apparatus including the same based on some implementations of the disclosed technology can reduce texture noise and improve autofocus accuracy by controlling weights to preserve edge regions when pixel values ​​of corresponding color pixels are synthesized.

[0021] Reference will now be made in detail to some embodiments of the disclosed technology, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to denote the same or similar parts. Although this disclosure is readily available in various modifications and alternatives, specific embodiments are shown in the drawings as examples. However, this disclosure should not be construed as limiting itself to the embodiments set forth herein.

[0022] Various embodiments will now be described with reference to the accompanying drawings. However, it should be understood that the disclosed technology is not limited to specific embodiments, but includes various modifications, equivalents, and / or substitutions of the embodiments. Embodiments of the disclosed technology can provide various effects that can be directly or indirectly identified by the disclosed technology.

[0023] Figure 1 This is a block diagram illustrating an example of an imaging device 1 based on some implementations of the disclosed technology. The imaging device 1 (e.g., a camera) captures a light image (or moving image) of a target object by collecting light reflected from the target object. The following will refer to… Figure 1 A method is described for performing an autofocus (AF) function by an imaging device 1.

[0024] Reference Figure 1 Imaging device 1 can refer to, for example, a digital still camera for capturing still images or a digital video camera for capturing moving images. For example, imaging device 1 can be implemented as a digital single-lens reflex (DSLR) camera, a mirrorless camera, or a smartphone. Imaging device 1 may include a device with multiple camera modules, each having both a lens and an image pickup element, enabling the device to capture (or photograph) a target object, thereby creating an image of the target object.

[0025] Reference Figure 1 The imaging device 1 may include an imaging circuit 10, an image sensor 100, and an image processing device 200.

[0026] In this case, the imaging circuit 10 may be a light-receiving component. More specifically, the imaging circuit 10 may include a lens 11 and a lens driver 12.

[0027] Lens 11 can converge the light reflected from the target object S and reaching the imaging device 1. Although Figure 1 A single lens is shown, but other implementations are possible. For example, in some implementations, a lens assembly may be provided comprising multiple lenses aligned along the optical axis. As the position of lens 11 is adjusted, the focal point on the target object S can be changed. The position of lens 11 may be based on signals generated from pixels of image sensor 100.

[0028] The lens driver 12 can control the position of the lens 11 based on control signals from the image processing device 200. In some implementations, the lens driver 12 can adjust the focal length by adjusting the position of the lens 11, and can perform operations such as autofocus, zoom variation, and focus variation. As the position of the lens 11 is adjusted, the distance between the lens 11 and the target object S can be adjusted. For example, the lens driver 12 can move the lens 11 in a direction parallel to the optical axis.

[0029] Image sensor 100 may include a pixel array (described later) in which multiple unit pixels are arranged in rows and columns. For example, multiple unit pixels may be arranged in a grid shape. Image sensor 100 may generate digital signals (or electrical signals) based on light reflected from a target object, and may generate digital image data (hereinafter referred to as "image data") based on the electrical signals. Incident light that has passed through lens 11 (i.e., optical signals) may be imaged in the pixel array and may be converted into electrical signals. Each unit pixel (described later) may generate an electrical signal corresponding to an external object S.

[0030] In various embodiments, the image sensor 100 may include a photodiode (PD), a transfer transistor, a reset transistor, and a floating diffusion node (FD). The photodiode (PD) generates and accumulates photocharge corresponding to an optical image of the target object S. Although the photodiode (PD) is mentioned as a component of the image sensor 100, any photoelectric conversion element, not limited to a photodiode, may be used, as long as the photoelectric conversion element converts an optical signal or incident light into an electrical signal. For example, the photoelectric conversion element may include, for example, a photodiode, a phototransistor, an optical gate, or other photosensitive circuitry capable of converting light into pixel signals (e.g., charge, voltage, or current). The transfer transistor may transfer the photocharge focused on the photodiode (PD) to the floating diffusion node (FD) in response to a transfer signal. The reset transistor may discharge the charge stored in the floating diffusion node (FD) in response to a reset signal. The charge stored in the floating diffusion node (FD) may be output before the reset signal is applied to the image sensor. At this time, correlated double sampling (CDS) processing may be performed, and the analog signal processed by CDS may be converted into a digital signal by analog-to-digital conversion (ADC) processing and / or analog front-end (AFE) processing. An example of the image sensor 100 according to the disclosed technology can be configured such that four photodiodes are assigned to a unit pixel corresponding to a single microlens (e.g., a four-photodiode (4PD) pixel structure).

[0031] Unit pixels can be arranged in a matrix form as a pixel array. The individual electrical signals generated by each unit pixel can include an image signal and a phase signal for the target object S. In this case, the image signal can be a signal generated in response to light incident on the image sensor 100 from the target object S, and can be used as a signal to generate an image of the target object S. Additionally, the phase signal can be a signal generated in response to light incident on the image sensor 100 from the target object S, and can be used as a signal to adjust the distance between the target object S and the lens 11. Unit pixels can be classified as phase difference detection pixels or image detection pixels according to their output signals. For example, a phase difference detection pixel refers to a unit pixel that outputs a phase signal, and an image detection pixel refers to a unit pixel that outputs an image signal.

[0032] Phase difference detection pixels can be arranged in an (N×N) matrix shape (where "N" is a natural number or positive integer of 2 or greater). Image detection pixels can be arranged adjacent to phase difference detection pixels. The following will refer to... Figure 2 A more detailed description of the structure of a unit pixel.

[0033] The image processing apparatus 200 can obtain image information, etc., based on signals output from image detection pixels. The image processing apparatus 200 can also obtain phase information, etc., based on signals output from phase difference detection pixels.

[0034] In some implementations, the image processing device 200 can receive image data from the image sensor 100 and generate phase data (phase image). Furthermore, the image processing device 200 can process phase difference calculations used in autofocus operations based on the phase data. The image processing device 200 can obtain the position and direction of the focal point, as well as the distance between the target object S and the imaging device 1, through this phase difference calculation. Based on the result of the phase difference calculation, the image processing device 200 can provide a drive signal to the lens driver 12 for adjusting the position of the lens 11.

[0035] The image processing apparatus 200 can perform various image data processing operations to improve image quality, such as demosaicing, noise reduction, gain adjustment, waveform shaping, analog-to-digital conversion (ADC), interpolation, white balance processing, gamma processing, and / or edge sharpening. In some implementations, the image processing apparatus 200 can modify the region of interest (ROI) of the image based on information about the detected focus and information about the image of the target object.

[0036] In some implementations, the image processing apparatus 200 may include a focus controller 300. In other implementations, the focus controller 300 may be implemented independently of the image processing apparatus 200.

[0037] The focus controller 300 can determine weights to preserve edge regions when generating a brightness image for autofocus adjustment by synthesizing pixel values. The brightness image for autofocus generated by the focus controller 300 can be output to the lens driver 12 and used as a drive signal to determine the focus position. The focus controller 300 can adjust the position of the lens 11 based on the brightness image for autofocus to make the parallax "0". For example, the image processing device 200 can store the brightness image for autofocus, parallax information, and lens drive amount generated by the focus controller 300 in a lookup table in its memory, and can use the data stored in the lookup table to control the lens driver 12.

[0038] although Figure 1 The embodiment shown depicts the image processing device 200 disposed outside the image sensor 100, but other implementations are also possible. For example, the image processing device 200 may be disposed inside the image sensor 100 or disposed separately outside the image sensor 100.

[0039] Image sensing devices may include a phase difference detection autofocus (PDAF) function, which autofocuses based on operations using phase difference detection to detect the phase difference in light received by adjacent phase difference detection pixels. The phase difference detection autofocus (PDAF) method measures the offset direction and amount relative to a central image acquired by the image sensing device by using the phase difference between two or more different measurement points.

[0040] When there is no phase difference between the signals generated by the phase difference detection pixels included in the image sensor 100, the distance between the lens 11 and the target object S can be referred to as being in a "focused position". When the distance between the lens 11 and the target object S is in a focused position, the magnitudes of the incident light reaching the unit pixel after passing through a microlens can be equal, so that the magnitudes of the signals detected from the unit pixels sharing a microlens can also be equal. For example, when the positions of the lens 11 and the target object S are in a focused position, the phase difference between the phase signals detected by the phase difference detection pixels included in the pixel array becomes zero.

[0041] If the distance between lens 11 and the target object S is not at the focusing position, differences may occur between the signals generated by the phase difference detection pixels. The magnitude of the incident light reaching each unit pixel can vary depending on the position of the unit pixel within the pixel group. This is because path differences may occur in the incident light passing through the microlens.

[0042] Therefore, when the distance between lens 11 and target object S is not in focus, the magnitudes of the phase signals of each unit pixel collected by image processing device 200 can be different from each other. When the distance between lens 11 and target object S is not in focus, image processing device 200 can generate phase data by calculating the magnitude differences between the phase signals.

[0043] The image processing device 200 can provide a drive signal to the lens driver 12 based on this phase data. Based on the drive signal provided from the image processing device 200, the lens driver 12 can move the lens 11 so that the distance between the lens 11 and the target object S is in a focused position.

[0044] To improve autofocus (AF) performance in high-resolution images, image sensing devices with structures each having a microlens applied to multiple pixels are being developed. The following will refer to... Figure 2 A more detailed description is given of a pixel array structure in which multiple pixels of the same color are arranged adjacent to each other and a microlens is applied to the multiple pixels.

[0045] Since the focal position of an image sensing device changes according to the position of the target object (S), the image sensing device can use parallax obtained from the selected ROI (Region of Interest) to perform focusing. An image sensing device with a structure having a microlens applied to multiple pixels can generate a brightness image for autofocus adjustment by synthesizing the pixel values ​​of corresponding color pixels. However, when detecting changes in pixel values ​​between corresponding colors in edge regions to generate a parallax image for autofocus adjustment, edge regions may not be preserved depending on the color of each individual color pixel. In this case, it may be difficult to perform autofocus accurately.

[0046] Therefore, the focus controller 300, based on some implementations of the disclosed technology, can determine weights such that edge regions are preserved when generating a brightness image for autofocus adjustment using pixel values ​​of synthesized pixels (e.g., R (red) pixels, G (green) pixels, and B (blue) pixels). (Refer to...) Figures 2 to 10 A method for generating texture signals by controlling weights to preserve edge regions is described in more detail.

[0047] Figure 2 This demonstrates some implementation methods based on the disclosed technology. Figure 1 A diagram showing an example structure of the pixel array included in the image sensor 100.

[0048] Reference Figure 2 The diagram illustrates 16 unit pixels arranged in a matrix comprising 4 rows and 4 columns. For example, 16 unit pixels is the smallest unit of a pixel array (PA), and these 16 unit pixels can be repeated in both the row and column directions. However, pixel arrangements are not limited to this, and other implementations are possible.

[0049] The pixel array (PA) of the image sensor 100 may include pixel groups (PG1 to PG4). The pixel groups (PG1 to PG4) may be arranged in an (N×N) matrix (where "N" is a natural number of 2 or greater). Pixel groups (PG1, PG4) may be arranged diagonally opposite each other, and pixel groups (PG2, PG3) may be arranged diagonally opposite each other.

[0050] A microlens ML can be formed in each pixel group (PG1~PG4) (which can correspond to...) Figure 1 (Lens 11). For example, since four unit pixels (PX1 to PX4) share a single microlens (ML), this shared structure can be called an A4C (full 4-connection) structure. The microlens (ML) can adjust the path of light incident on the image sensor 100.

[0051] Each pixel group (PG1 to PG4) may have four unit pixels (PX1 to PX4) of the same color, and these four unit pixels (PX1 to PX4) are arranged adjacent to each other in an (N×N) matrix (where "N" is a natural number of 2 or greater). When four light-receiving elements are placed in each pixel group (PG1 to PG4), these four light-receiving elements can be arranged symmetrically in the upper right / upper left / lower right / lower left directions based on the center of each pixel group. These four light-receiving elements can each correspond to four unit pixels (PX1 to PX4).

[0052] For example, pixel group PG1 may include four unit pixels, each with a green (Gr) color filter. Pixel group PG2 may include four unit pixels, each with a red (R) color filter. Additionally, pixel group PG3 may include four unit pixels, each with a blue (B) color filter. Pixel group PG4 may include four unit pixels, each with a green (Gb) color filter.

[0053] That is, each pixel group (PG1 to PG4) may include four unit pixels (PX1 to PX4). Each unit pixel (PX1 to PX4) may include a photoelectric conversion element (not shown) corresponding to each unit pixel. The colors corresponding to the pixel groups (PG1 to PG4) (i.e., red (R), green (Gr, Gb), and blue (B)) may be arranged in a Bayer pattern. Therefore, the original image generated by capturing an image by the image sensor 100 may include color image pixels arranged in a structure similar to the pixel array (PA) described above. The repeating arrangement structure and pattern of the pixel array (PA) are not limited to this and may vary depending on the implementation.

[0054] Within each pixel group (PG1 to PG4), the positions of red, blue, and / or green pixels can be equal to each other to reduce the computational load required for image signal processing by the image processing apparatus 200, but the scope or spirit of the disclosed technology is not limited thereto. Furthermore, although this embodiment discloses that each pixel group (PG1 to PG4) includes four unit pixels (PX1 to PX4) for ease of description, the number of unit pixels included in each pixel group is not limited to this.

[0055] The pixel array (PA) may include phase difference detection pixels and image detection pixels. Here, the image detection pixels can generate... Figure 1 The signal corresponds to the target object (S). Additionally, phase difference detection pixels can be used to capture an image of the target object (S) and generate a phase signal for autofocus. Each phase signal can include information about the position of a unit pixel on the pixel array (PA) (where the phase signal is generated). The phase signals can be sent to the image processing device 200 and can be used to detect the distance between the target object S and the lens 11.

[0056] In some implementations, the phase difference between a first image generated by some pixels among the phase difference detection pixels and a second image generated by some other pixels can be detected by phase detection autofocus operation. The lens movement distance can be calculated based on the detected phase difference, and the lens position can be adjusted based on the calculated movement distance, thus obtaining a focused image.

[0057] In some implementations, pixel group (PG1) within pixel group (PG1-PG4) of the pixel array (PA) can be designated as phase difference detection pixels (PX1-PX4), but the position of the phase difference detection pixels is not limited to this. Additionally, the remaining pixel group (PG2-PG4) within pixel group (PG1-PG4) of the pixel array (PA) can be designated as image detection pixels, but the position of the image detection pixels is not limited to this.

[0058] exist Figure 2 In this diagram, each pixel group (PG1 to PG4) includes unit pixels (PX1 to PX4), which are also labeled TL, BL, TR, BR, ... The labels TL, BL, TR, and BR indicate the positions of the four unit pixels within the corresponding pixel group (PG). For example, in each pixel group (PG1 to PG4), the position below the top-left unit pixel (PX1) will be referred to as "TL (top left)", and the position below the bottom-left unit pixel (PX2) will be referred to as "BL (bottom left)". Similarly, the position below the top-right unit pixel (PX3) in each pixel group (PG1 to PG4) will be referred to as "TR (top right)", and the position below the bottom-right unit pixel (PX4) will be referred to as "TR (top right)".

[0059] Figure 3 These are examples of some implementations based on the disclosed technologies. Figure 1 A schematic diagram of an example of a focus controller 300 is shown.

[0060] Reference Figure 3 The focus controller 300 may include an image combiner 310 and a brightness image processor 320.

[0061] Here, the image combiner 310 can combine the image data (IDATA) received from the image sensor 100 to output multiple color images (IMG1 to IMG4). The image data (IDATA) applied to the image combiner 310 may have the same characteristics as described above. Figure 2 The format corresponding to the pixel array (PA) shown.

[0062] In some implementations, the image combiner 310 can separate the image data (IDATA) of each pixel group (PG1 to PG4), collect unit pixels arranged in the same position, and combine the pixel data of the collected unit pixels. Therefore, it can generate four color images (IMG1 to IMG4) based on the result of combining the pixel data. In some implementations, pixel group PG1 and pixel group PG4 may have the same green (G), while pixel group PG1 and pixel group PG4 may be arranged in different positions within a single Bayer pattern.

[0063] For example, in this embodiment, green (G) can be divided into a first green (Gr) and a second green (Gb), and the Gr and Gb colors derived from green (G) can be considered as different colors. Furthermore, in some implementations of the disclosed technology, the image combiner 310 is described as an example of separating four channels to generate four color images (IMG1 to IMG4), but the number of color images can vary depending on the position of each unit pixel, and is not limited thereto.

[0064] For example, a color image (IMG1) can be generated by collecting pixels (TL) in the upper left (TL) region of the corresponding pixel group (PG1 to PG4) of the image data (IDATA). A color image (IMG2) can be generated by collecting pixels (TR) in the upper right (TR) region of the corresponding pixel group (PG1 to PG4) of the image data (IDATA). A color image (IMG3) can be generated by collecting pixels (BL) in the lower left (BL) region of the corresponding pixel group (PG1 to PG4) of the image data (IDATA). A color image (IMG4) can be generated by collecting pixels (BR) in the lower right (BR) region of the corresponding pixel group (PG1 to PG4) of the image data (IDATA).

[0065] Additionally, the luminance image processor 320 can process the four color images (IMG1 to IMG4) received from the image combiner 310 into luminance data to generate a luminance image (AF_IMG) for autofocus adjustment.

[0066] In some implementations, the photoelectric conversion signal of each photoreceiving element in each unit pixel (PX1 to PX4) may include phase difference information of each unit pixel. The photoelectric conversion signal output when the photoreceiving elements of a unit pixel are arranged, for example, along different parts of the pixel group in the up, down, left, and right directions, may represent the phase difference of light incident on each photoreceiving element at different incident angles. To reduce data storage (or data storage size), the phase difference data including phase difference information may be output as only including luminance (Y) information from the YCbCr data type used to represent the color space, excluding Cb / Cr color information. An image including this luminance information may be output as a luminance image (AF_IMG) to be used for autofocus adjustment.

[0067] For example, the luminance image processor 320 can convert data from a color image (IMG1) into luminance data to generate a TL luminance image. The luminance image processor 320 can convert data from a color image (IMG2) into luminance data to generate a TR luminance image. The luminance image processor 320 can convert data from a color image (IMG3) into luminance data to generate a BL luminance image. The luminance image processor 320 can convert data from a color image (IMG4) into luminance data to generate a BR luminance image. The aforementioned TL, TR, BL, and BR luminance images can be output as a luminance image (AF_IMG) for autofocus (hereinafter referred to as the autofocus luminance image AF_IMG).

[0068] In some implementations, the luminance image processor 320 may generate a luminance image (AF_IMG), for example, by separating and combining unit pixels at the same position in corresponding pixel groups arranged in a pixel array. However, the scope of this embodiment is not limited to this, and luminance images (AF_IMG) may be generated in various other ways.

[0069] However, when edge regions (or textures) exist between two different color images, the edges of the luminance image (AF_IMG) may not be preserved when the two color images are combined to generate a luminance image (AF_IMG). Therefore, the luminance image processor 320 according to this embodiment can control the weights so that edges are preserved by analyzing the texture of the image when combining color images with different colors to generate a luminance image (AF_IMG) to be used for autofocus adjustment. The following will refer to... Figures 4 to 10 The operation of the luminance image processor 320 is described in more detail.

[0070] Figure 4 These are examples of some implementations based on the disclosed technologies. Figure 3 A schematic diagram of an example of a brightness image processor 300 is shown. Figure 5 This demonstrates some implementation methods based on the disclosed technology. Figure 3 The flowchart illustrates an example operation of the brightness image processor 300. The following will refer to... Figure 5 To describe using a flowchart Figure 4 The operation of the brightness image processor 320 is shown.

[0071] Reference Figure 4 The luminance image processor 320 may include a calculator 321, multiple texture amplitude calculators (322-324), multiple noise level calculators (325-327), multiple texture correlation determiners (328, 329), multiple weight calculators (330, 331), and a synthesizer 332.

[0072] As shown above (refer to the reference) Figure 3 As described, the four color images (IMG1 to IMG4) can be input to the luminance image processor 320. Figure 4 In the implementation method, the following will refer to Figure 4 This describes an example of receiving one of the four color images (IMG1 to IMG4) and generating a TL luminance image. The configuration and operation for receiving the remaining color images (IMG2 to IMG4) and generating TR, BL, and BR luminance images are described below. Figure 4 The same applies to those, so for the sake of brevity, redundant descriptions will be omitted in this article.

[0073] The luminance image processor 320 can receive a first image (R_IMG), a second image (G1_IMG), a third image (G2_IMG), and a fourth image (B_IMG). Here, the first image (R_IMG), the second image (G1_IMG), the third image (G2_IMG), and the fourth image (B_IMG) can correspond to the color image (IMG1). The first to fourth images (R_IMG, G1_IMG, G2_IMG, B_IMG) can include not only the target pixel (TL) located at the processing target and corresponding to the color image (IMG1), but also peripheral pixels located around the target pixel (TL) and having the same color.

[0074] For example, the first image (R_IMG) may include TL pixels of red (R) arranged in a (5×5) matrix. The second image (G1_IMG) may include TL pixels of green (G) arranged in a (5×5) matrix. The third image (G2_IMG) may include TL pixels of green arranged in a (5×5) matrix. The fourth image (B_IMG) may include TL pixels of blue arranged in a (5×5) matrix.

[0075] In some implementations, the first to fourth images (R_IMG, G1_IMG, G2_IMG, B_IMG) can be set as ROIs (Regions of Interest) (Operation S1). ROIs can be arbitrarily set at various locations on the image sensor. The number of ROIs, the size of each ROI (i.e., the number of pixels included in each ROI), and the position of each ROI can be varied, and are not limited to these.

[0076] Calculator 321 can generate an average image by calculating the pixel values ​​of the second image (G1_IMG) and the third image (G2_IMG). For example, calculator 321 can average the pixel values ​​of the second image (G1_IMG) and the third image (G2_IMG) and output the average value of the green pixels.

[0077] Each of the multiple texture amplitude calculators (322-324) can calculate the amplitude of the brightness level of the texture image (operation S2). Texture amplitude calculator 322 can calculate the amplitude of the texture of the red pixels included in the first image (R_IMG). Texture amplitude calculator 323 can calculate the amplitude of the texture of the green pixels included in the output image of calculator 321. Texture amplitude calculator 324 can calculate the amplitude of the texture of the blue pixels included in the fourth image (B_IMG).

[0078] Here, the "amplitude" of a texture can be an indicator (metric) representing the degree of contrast difference between two regions with different brightness levels that are in contact with each other within multiple textures. For example, if the texture is an edge region, the amplitude of the texture can be calculated based on the absolute value of the difference between the bright and dark regions. In another example, if the texture is a complex texture, the amplitude of the texture can be calculated using the difference between the maximum and minimum values ​​within the texture region. In yet another example, to improve noise resistance, the amplitude of the texture can be calculated by using an averaging filter or similar method to obtain the difference between the maximum and minimum values. In yet another example, the variance of pixel values ​​or similar methods can be used as an indicator (metric) to calculate the brightness (contrast) of the texture.

[0079] Additionally, multiple noise level calculators (325-327) can calculate the noise level of each pixel included in the first to fourth images (R_IMG, G1_IMG, G2_IMG, B_IMG) (operation S3). Noise level calculator 325 can calculate the noise level of the first image (R_IMG). Noise level calculator 326 can calculate the noise level from the output image of calculator 321. Noise level calculator 327 can calculate the noise level of the fourth image (B_IMG).

[0080] Here, the amount of noise in the textured regions within the first to fourth images (R_IMG, G1_IMG, G2_IMG, B_IMG) can represent noise variance or noise standard deviation, etc. For example, the average pixel value in the textured region can be obtained, and the sensor gain during image capture (shooting) can be set separately. In this way, the noise variance can be estimated by obtaining information about the range of noise variance for the average pixel value, gain value, etc. Furthermore, the noise standard deviation can be obtained by calculating the standard deviation of image data generated under conditions where the light incident on the image sensing device is completely blocked during the testing of the image sensing device.

[0081] Additionally, multiple texture correlation determiners 328 and 329 can calculate the texture correlation value between two images based on the first image (R_IMG), the fourth image (B_IMG), and the output image of the calculator 321 (operation S4). In some implementations, texture correlation determiner 328 can calculate the texture correlation value between the first image (R_IMG) and the output image of the calculator 321. Furthermore, texture correlation determiner 329 can calculate the texture correlation value between the fourth image (B_IMG) and the output image of the calculator 321.

[0082] For example, the first image (R_IMG) may correspond to red pixels, and the output image of calculator 321 may correspond to green pixels. Texture correlation determiner 328 can determine whether the textures of the red and green pixels have positive (+) or negative (-) correlation values. Additionally, the fourth image (B_IMG) may correspond to blue pixels, and the output image of calculator 321 may correspond to green pixels. Texture correlation determiner 329 can determine whether the textures of the blue and green pixels have positive (+) or negative (-) correlation values. The operation of determining texture correlation values ​​by texture correlation determiners (328, 329) will refer to... Figure 6 To describe in more detail.

[0083] In some implementations, multiple weight calculators (330, 331) can calculate weights based on the output signals of a texture amplitude calculator (322-324), a noise level calculator (325-327), and a texture correlation determiner (328, 329) (operation S5). Weight calculator 330 can calculate weights based on the output signals of the texture amplitude calculator (322, 323), the noise level calculator (325, 326), and the texture correlation determiner 328. Weight calculator 331 can calculate weights based on the output signals of the texture amplitude calculator (323, 324), the noise level calculator (326, 327), and the texture correlation determiner 329. The operation of calculating weights by weight calculators (330, 331) will refer to... Figures 7 to 10 To describe in more detail.

[0084] In some implementations, the synthesizer 332 can generate a brightness image (AF_IMG) for autofocus by combining the first image (R_IMG), the output image of the calculator 321, and the fourth image (B_IMG) using weights calculated by the weight calculators (330, 331) (operation S6).

[0085] Figure 6 (a) and Figure 6 (b) illustrates some implementations based on the disclosed technology. Figure 3 A diagram illustrating an example operation of the texture-dependent determiner.

[0086] Reference Figure 6 (a) and Figure 6 (b) The texture-dependent determinant (328, 329) can calculate the texture-dependent value by determining whether the pixel value increases or decreases in the edge region of a texture with two different colors.

[0087] Figure 6 (a) shows an example where the edge or gradient changes from black (BK) to yellow (Y). In such... Figure 6 In the example shown in (a), as the value of pixel G increases, the value of pixel R also increases simultaneously, which allows edges or gradients to have a positive (+) correlation. Here, the example case where the correlation between two regions with different colors and in contact with each other is positive (+) will be referred to as "positive correlation" below.

[0088] Figure 6 (b) shows another example where the edge or gradient changes from red (R) to green (G). Figure 6 In the example shown in (b), the value of the G pixel increases as the value of the R pixel decreases, which allows for negative (-) correlations between edges or gradients. Here, the example case of a negative (-) correlation between two regions with different colors that are in contact with each other will be referred to as "negative correlation" below.

[0089] Thus, the texture correlation determiner (328, 329) can use the correlation value (or correlation coefficient) obtained by quantifying the correlation as an indicator. The texture correlation determiner (328, 329) can determine the texture correlation by determining whether the correlation coefficient has a positive (+) value or a negative (-) value.

[0090] For example, methods for determining such correlation may include calculating the average pixel values ​​of the R and G images within the ROI (region of interest), calculating the deviation of each pixel relative to the R and G images (the difference from the average value), and determining the sign of the calculated deviation.

[0091] [Formula 1]

[0092]

[0093] The above operation can be represented by Equation 1.

[0094] [Equation 2]

[0095]

[0096] In another example, the correlation can also be determined by a majority vote of the symbols of the corresponding pixels of the R and G images, as in Equation 2 above.

[0097] Figure 7 (a) to Figure 9 (c) illustrates some implementations based on the disclosed technology. Figure 4 The diagram shows an example operation of the weight calculator.

[0098] Figure 7 Image (a) may include red and green areas. When the red and green areas are displayed by color, they can be distinguished as a red (R) image and a green (G) image, respectively. In the case of the R image, the pixel value can decrease from 200 to 0 in the direction from left to right. In the case of the G image, the pixel value can increase from 0 to 200 in the direction from left to right. Therefore, the pixel value of the R image decreases, and the pixel value of the G image increases, indicating that the trends of the pixel values ​​of the R and G images are opposite to each other. The trends of the pixel values ​​of the R and G images can refer to whether the pixel values ​​of the R and G images increase or decrease, respectively.

[0099] like Figure 7 As shown in (b), when the pixel values ​​of the R image are added to the pixel values ​​of the G image, the pixel values ​​of both the R and G pixels (i.e., the R+G pixel value) are 200, so that the difference between the pixel values ​​in the left and right directions of the edge region can be zero. In this case, the contrast difference between the left and right texture regions becomes zero, so that autofocus cannot be performed.

[0100] On the other hand, when Figure 7 As shown in (c), when the pixel values ​​of the G image are subtracted from the pixel values ​​of the R image, the GR pixel values ​​on the left become "-200" and the GR pixel values ​​on the right become "200", so that the difference between the pixel values ​​on the left and right sides of the edge region can be increased to "400". In this case, the contrast difference between the left and right texture regions can be increased to "400", so that this calculation result can be used for autofocus operation.

[0101] Thus, when the trend of pixel values ​​in image R, which indicates whether pixel values ​​in image R are increasing or decreasing, is opposite to the trend of pixel values ​​in image G, which indicates whether pixel values ​​in image G are increasing or decreasing, the weight calculator (330, 331) can calculate the weight by performing a subtraction operation. In other words, when the trends of pixel values ​​in the edge regions of images R and G are different from each other—for example, i) pixel values ​​in the edge regions of image R are increasing, but pixel values ​​in the edge regions of image G are decreasing, or ii) pixel values ​​in the edge regions of image R are decreasing, but pixel values ​​in the edge regions of image G are increasing—if the weight of image G is positive (+), the weight of image R can be set to negative (-) so that the weights of images R and G can be calculated. Therefore, weights can be calculated such that subtraction is performed when the brightness change of image R is opposite to the brightness change of image G.

[0102] Figure 8 The image in (a) may include yellow and black areas. The pixel values ​​of both R and G pixels in the yellow area are 200, and the pixel values ​​of both R and G pixels in the black area are 0. When the yellow and black areas are displayed by color, they can be distinguished as R-image and G-image, respectively. In the case of the R-image, the pixel value can decrease from 200 to 0 in the direction from left to right. Similarly, in the case of the G-image, the pixel value can decrease from 200 to 0 in the direction from left to right. Therefore, the pixel values ​​of both the R-image and the G-image decrease, indicating that the trends of the pixel values ​​of the R-image and the G-image are equal. The trends of the pixel values ​​of the R-image and the G-image can refer to whether the pixel values ​​of the R-image and the G-image are increasing or decreasing, respectively.

[0103] like Figure 8 As shown in (b), when the pixel values ​​of the R image are added to the pixel values ​​of the G image, the R+G pixel value on the left becomes "400" and the R+G pixel value on the right becomes zero "0", so that the difference between the pixel values ​​on the left and right sides of the edge region can be increased to "400". In this case, the contrast difference between the left and right texture regions increases to "400", so that the resulting value can be used for autofocus operation. Thus, when the pixel values ​​of the R image and the G image have the same trend, for example, when the pixel values ​​in the R image decrease and the pixel values ​​in the G image decrease, the weight calculators (330, 331) can calculate the weights by performing an addition operation.

[0104] On the other hand, when Figure 8 In (c), when the pixel values ​​of the R image are subtracted from the pixel values ​​of the G image, all the GR pixel values ​​become zero, so that the difference between the pixel values ​​on the left and right sides of the edge region becomes zero. In this case, the contrast difference between the left and right texture regions becomes zero, making it impossible to perform autofocus operation.

[0105] Figure 9 The image in (a) may include green and black areas. The R and G pixels in the green area have pixel values ​​of 0 and 200, respectively, while the R and G pixels in the black area both have pixel values ​​of 0. When the green and black areas are displayed in color, they can be distinguished as R-image and G-image, respectively. In the case of the R-image, the pixel value remains at zero "0," with no change in the direction from left to right. In the case of the G-image, the pixel value can decrease from 200 to 0 in the direction from left to right. Since the pixel value of the R-image does not change and the pixel value of the G-image decreases, the trends of the pixel values ​​of the R-image and G-image are different from each other, but not opposite. The trends of the pixel values ​​of the R-image and G-image can refer to whether the pixel values ​​of the R-image and G-image are increasing or decreasing, respectively.

[0106] like Figure 9 As shown in (b), when the pixel values ​​of the R image are added to the pixel values ​​of the G image, the R+G pixel value on the left becomes "200" and the R+G pixel value on the right becomes zero "0", so that the difference between the pixel values ​​on the left and right sides of the edge region does not change compared to other cases where only the G image is used.

[0107] like Figure 9 As shown in (c), when the pixel values ​​of the R image are subtracted from the pixel values ​​of the G image, the GR pixel values ​​on the left become “200” and the GR pixel values ​​on the right become zero “0”, so that the difference between the pixel values ​​on the left and right sides of the edge region does not change compared to other cases where only the G image is used.

[0108] In this situation, the signal-to-noise ratio (SNR) of the texture may increase, making it difficult or impossible to perform autofocus operations. A lower SNR of the texture may mean that the magnitude of the texture (e.g., the difference in contrast at edges) is smaller compared to the magnitude of the noise. According to another implementation, when the increase and decrease of pixel values ​​in the R image are equal to those in the G image, and the difference in pixel values ​​between the addition and subtraction operations in the R image is equal to that in the G image, the weight calculators (330, 331) may set the absolute values ​​of the weights to smaller absolute values ​​in order to reduce noise.

[0109] The following describes the method for calculating weights using a weight calculator (330, 331) with reference to the accompanying drawings.

[0110] In some implementations, Equations 3, 4, and 5 may represent examples of setting weights in some cases where the texture amplitude is calculated by using the difference between the maximum and minimum values ​​in the texture region and the texture amplitude is unsigned.

[0111] [Formula 3]

[0112]

[0113] In Equation 3, w R T can represent the weights in an R-image. R V can represent the magnitude of texture in an R image. R This can represent the standard deviation of noise in an R image. Additionally, C... R This can represent a parameter value that becomes "1" when the R-image and G-image have a positive (+) correlation and "-1" when they have a negative (-) correlation. Additionally, T... G V can represent the magnitude of texture in a G image. G This can represent the standard deviation of noise in the G image. In Equation 3, when the signal-to-noise ratio (SNR) of the texture is small, that is, when T... R Smaller and V R When the value is large, the weight can be reduced.

[0114] [Formula 4]

[0115] w G =1

[0116] In Equation 4, W G The weights in the G image can be represented by a value of 1, which can be set as a reference value.

[0117] [Formula 5]

[0118]

[0119] In Equation 5, w B T can represent the weights in the B-image. B V can represent the magnitude of texture in a B-image. B This can represent the standard deviation of the noise in image B. Additionally, C... B This can be represented by a parameter value that becomes "1" when the B-image and G-image have a positive (+) correlation and "-1" when they have a negative (-) correlation. In Equation 5, when the signal-to-noise ratio (SNR) of the texture is small, i.e., when T... B Smaller and V B When the value is large, the weight can be reduced.

[0120] As in Equations 3, 4, and 5, when the texture has a negative (-) correlation, the sign of the weight can be negative (-). A negative (-) sign for the weight means that the weight can be calculated by performing a subtraction operation. That is, as referred to above... Figure 7 As described, when the increase and decrease of pixel values ​​in the R image (or B image) are opposite to the increase and decrease of pixel values ​​in the G image, the weight calculator (330, 331) can perform a subtraction operation to calculate the weights.

[0121] According to another embodiment, Equations 6, 7 and 8 may represent examples of setting weights in some cases where the absolute value of the difference between the bright and dark parts within the texture region is used to calculate the texture amplitude and the texture amplitude has a sign.

[0122] [Formula 6]

[0123]

[0124] In Equation 6, w R T can represent the weights in an R-image. R V can represent the magnitude of texture in an R image. R This can represent the standard deviation of noise in the R image. In Equation 6, when the signal-to-noise ratio (SNR) of the texture is small, i.e., when T... R Smaller and V R When the value is large, the weights can be reduced. In other words, based on the G image, the absolute value of the weights increases with the texture amplitude (T) of the R image. R The absolute value of ) increases, and it also increases with the noise standard deviation (V) of the R image. R As the weight increases, the weight tends to decrease.

[0125] [Formula 7]

[0126] w G =1

[0127] In Equation 7, W G The weights in the G image can be represented by a value of 1, which can be set as a reference value.

[0128] [Formula 8]

[0129]

[0130] In Equation 8, w B T can represent the weights in the B-image. B V can represent the magnitude of texture in a B-image. B This can represent the standard deviation of noise in the B image. In Equation 8, when the signal-to-noise ratio (SNR) of the texture is small, that is, when T... B Smaller and V B When the value is large, the weight can be reduced.

[0131] The following will describe, with reference to the attached figures, the weights (W) calculated using the above formula. R W G W B The operation of synthesizer 332, which combines the first to fourth images (R_IMG, G1_IMG, G2_IMG, B_IMG) to generate an autofocus brightness image (AF_IMG).

[0132] The synthesizer 332 can apply weights not only to the average image of the first image (R_IMG), the second image (G1_IMG), and the third image (G2_IMG), but also to the fourth image (B_IMG), as shown in Equation 9 below.

[0133] [Formula 9]

[0134] O ′ (x,y)=w R R(x,y)+w G G(x,y)+w B B(x,y)

[0135] In Equation 9, O' can represent the autofocused brightness image (AF_IMG) output from synthesizer 332 (i.e., the output signal of synthesizer 332). The output signal (O') can be calculated by summing the first and third values, where the first value is obtained by multiplying the coordinates "R(x,y)" of the R image by a weight (w). R The third value is obtained by multiplying the coordinates "B(x,y)" of the B image by the weight (w). B It was obtained through this process.

[0136] When the weight is negative (-), the output signal (O') may also become negative (-). In some implementations, when post-processing requires a positive (+) number, a positive (+) constant (Z) can be added to the entire expression so that the output signal (O') does not become negative (-).

[0137] [Formula 10]

[0138] O ′ (x,y)=O ′ (x,y)+Z=w R R(x,y)+w G G(x,y)+w B B(x,y)+Z

[0139] In Equation 10, the constant (Z) can be added to the output signal (O') to obtain a positive (+) output signal (O). If the constant (Z) is greater than the value obtained by reversing the sign of the minimum value of the output signal (O'), then the constant (Z) can be changed to a positive (+) value.

[0140] If the constant (Z) is predetermined, the weights can be set in Equations 11 and 12 as follows.

[0141] [Equation 11]

[0142]

[0143] [Equation 12]

[0144] O"(x,y)=w ′ R R+w ′ G G+w ′ B B

[0145] Synthesizer 332 can calculate the weights (w) using Equation 11 ′ R ), weight (w) ′ G ) and weights (w ′ B It can be synthesized with R, G and B images respectively, and the output signal (O”) can be calculated and output based on the synthesis result.

[0146] Figure 10 (a) to Figure 10 (c) is an example of how the determination of some implementation methods based on the disclosed technology is determined by Figure 4 A diagram illustrating how the weight calculator adjusts the absolute values ​​of each weight to a smaller example case.

[0147] like Figure 10 As shown in (a), when the standard deviation of the noise is 40 and the brightness difference in the edge region is 10, edges across two different colors may be blurred, thus reducing the color edge contrast of the image. In another example, as Figure 10 As shown in (b), when the standard deviation of the noise is 10 and the brightness difference in the edge region is 10, that is, if the standard deviation of the noise and the brightness difference are the same, the edge between the two textures may not be clearly visible and may appear blurry. In another example, as Figure 10 As shown in (c), when the standard deviation of the noise is 1, the edge can be preserved and clearly seen even when the brightness difference in the edge region is 10.

[0148] Therefore, according to the weight calculator (330, 331) of this embodiment, a color with a small signal-to-noise ratio (SNR) can be determined, and the absolute value of the weight can be adjusted to be small.

[0149] [Equation 13]

[0150] SN R =|T R | / sqrt(V R )

[0151] For example, assume a low signal-to-noise ratio (SNR), such as 1 to 2. In Equation 13, this can be achieved by adjusting the amplitude (T) of the texture. R Divide by the standard deviation of the noise (V) RThe weight calculator (330, 331) calculates the signal-to-noise ratio (SNR). For example, when the SNR is less than a preset value "1", the weight calculator (330, 331) can set the weights to zero "0", which is less than "1". That is, the weight calculator (330, 331) can detect the SNR of the texture, and when the SNR is less than a preset value, the weight calculator (330, 331) can adjust the weights in the R image to be less than the preset value.

[0152] Figure 11 It is shown that... Figure 1 A block diagram of an example of a computing device 1000 corresponding to an image processing device 200.

[0153] Reference Figure 11 The computing device 1000 can represent a device for performing... Figure 1 An implementation of the hardware configuration for operating the image processing device 200.

[0154] The computing device 1000 can be mounted on a chip independent of the chip on which the image sensing device is mounted. According to one embodiment, the chip on which the image sensing device is mounted and the chip on which the computing device 1000 is mounted can be implemented in a single package (e.g., a multi-chip package (MCP)), but the scope of the disclosed technology is not limited thereto.

[0155] The computing device 1000 may include a processor 1010, a memory 1020, an input / output (I / O) interface 1030, and a communication interface 1040.

[0156] Processor 1010 can handle execution Figure 1 The processor 1010 refers to the image processing apparatus 200 and the data and / or instructions required for the operation of its components (310, 320). That is, the processor 1010 may refer to the image processing apparatus 200, but the scope of the disclosed technology is not limited thereto.

[0157] The memory 1020 may store instructions and / or data required to perform the operation of the components (310, 320) of the image processing apparatus 200, and may be accessed by the processor 1010. For example, the memory 1020 may be volatile memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM), etc.) or non-volatile memory (e.g., programmable read-only memory (PROM), erasable PROM (EPROM), etc.), EEPROM (electrically erasable PROM), flash memory, etc.).

[0158] That is, the computer program for performing the operation of the image processing apparatus 200 disclosed in this document is recorded in the memory 1020 and executed and processed by the processor 1010, thereby realizing the operation of the image processing apparatus 200.

[0159] Input / output (I / O) interface 1030 is an interface for connecting external input devices (e.g., keyboard, mouse, touch panel, etc.) and / or external output devices (e.g., display) to processor 1010 to allow sending and receiving data.

[0160] The communication interface 1040 is a component that can send and receive various data with external devices (e.g., application processors, external memory, etc.) and can be a device that supports wired or wireless communication.

[0161] It is evident from the above description that an image processing apparatus based on some implementations of the disclosed technology and an imaging apparatus including the image processing apparatus can reduce texture noise and improve autofocus accuracy by controlling weights to preserve edge regions when pixel values ​​of corresponding color pixels are synthesized.

[0162] The implementation of the disclosed technology can provide various effects that can be directly or indirectly understood through the aforementioned patent documents.

[0163] Although several exemplary embodiments have been described, it should be understood that modifications and enhancements to the disclosed embodiments and other embodiments can be conceived based on the descriptions and / or illustrations in this patent document. Therefore, the scope of the disclosed technology should not be limited to the above-described embodiments, but should include their equivalents.

[0164] Cross-reference to related applications

[0165] This patent document claims priority and benefit to Korean Patent Application No. 10-2024-0110633, filed on August 19, 2024, the disclosure of which in its entirety is incorporated herein by reference as part of the disclosure of this patent document.

Claims

1. An image processing apparatus, the image processing apparatus comprising: An image combiner that generates multiple color images by combining pixel data of unit pixels arranged in the same position in each of multiple pixel groups in a pixel array. as well as A luminance image processor calculates weights based on the texture amplitudes of the plurality of color images and the texture correlation values ​​between the plurality of color images, and generates a luminance image by applying the weights to the plurality of color images.

2. The image processing apparatus according to claim 1, wherein, The brightness image processor includes: A calculator that generates an average image by averaging the pixel values ​​of some of the plurality of color images.

3. The image processing apparatus according to claim 2, wherein, The brightness image processor includes: Multiple texture correlation determiners determine the texture correlation value between each of the multiple color images and the average image.

4. The image processing apparatus according to claim 3, wherein, The plurality of texture-related determiners also: Calculate the deviation of each pixel relative to two color images with different colors among the plurality of color images; Determine the sign of the deviation; and The texture correlation value is calculated based on the sign of the deviation.

5. The image processing apparatus according to claim 3, wherein, The plurality of texture-related determiners: The texture correlation value is calculated by determining whether the pixel value increases or decreases in the edge region of two color images with different colors among the plurality of color images.

6. The image processing apparatus according to claim 1, wherein, The brightness image processor includes: Multiple texture amplitude calculators calculate the texture amplitude based on the absolute value of the contrast difference between two regions with different brightness levels and in contact with each other in the texture of each of the multiple color images.

7. The image processing apparatus according to claim 1, wherein, The brightness image processor includes: Multiple noise level calculators are used to calculate the noise level of the multiple color images.

8. The image processing apparatus according to claim 7, wherein, The brightness image processor includes: Multiple weight calculators calculate the weights based on the texture amplitude, the noise level, and the texture correlation value.

9. The image processing apparatus according to claim 8, wherein, The multiple weight calculators: When the texture correlation values ​​have a positive (+) correlation, the weights are calculated by performing addition; and When the texture correlation value has a negative (-) correlation, the weight is calculated by performing subtraction.

10. The image processing apparatus according to claim 8, wherein, The multiple weight calculators: When the texture-related value does not increase or decrease, the absolute value of each weight is determined to be less than a preset value.

11. The image processing apparatus according to claim 8, wherein, The multiple weight calculators: When the signal-to-noise ratio (SNR) is less than a preset value based on the noise level, the absolute value of each weight is determined to be less than the preset value.

12. An imaging apparatus, the imaging apparatus comprising: A first texture amplitude calculator calculates the texture amplitude of a first color image; A first noise level calculator calculates the noise level of the first color image; A calculator that generates an average image by averaging a second color image and a third color image having the same color as the second color image; A first texture correlation determiner determines a texture correlation value between the first color image and the average image; A second texture amplitude calculator calculates the texture amplitude of the average image; A second noise level calculator calculates the noise level of the average image; A first weight calculator calculates weights based on the output signals of the first texture amplitude calculator and the second texture amplitude calculator, the output signals of the first noise amount calculator and the second noise amount calculator, and the output signal of the first texture correlation determiner. as well as A synthesizer that generates a brightness image to be used for autofocus adjustment by applying the weights to the first color image and the average image.

13. The imaging apparatus according to claim 12, wherein, Each of the first texture amplitude calculator and the second texture amplitude calculator: The texture amplitude is calculated based on the absolute value of the contrast difference between two regions with different brightness levels and in contact with each other in the respective textures of the first color image and the second color image.

14. The imaging apparatus according to claim 12, wherein, First texture correlation determiner: The texture correlation value is calculated by calculating the deviation of each pixel relative to the first color image and the average image and determining the sign of the deviation.

15. The imaging apparatus according to claim 12, wherein, First texture correlation determiner: The texture correlation value is calculated by determining whether the pixel value increases or decreases in the edge regions of the first color image and the average image.

16. The imaging apparatus according to claim 15, wherein, First weight calculator: When the texture correlation values ​​have a positive (+) correlation, the weights are calculated by performing addition; and When the texture correlation value has a negative (-) correlation, the weight is calculated by performing subtraction.

17. The imaging apparatus according to claim 12, wherein, The first color image has red, and each of the second and third color images has green.

18. The imaging apparatus according to claim 17, wherein, First weight calculator: The weights are calculated such that subtraction is performed when the change in brightness of the first color image is opposite to the change in brightness of the average image.

19. The imaging apparatus according to claim 12, further comprising: An image sensor that generates image data comprising multiple groups of pixels; as well as An image combiner that generates the first color image and the second color image by combining pixel data of unit pixels arranged in the same position in each of the plurality of pixel groups.

20. The imaging apparatus according to claim 19, wherein, Each of the plurality of pixel groups includes: Multiple unit pixels, arranged in an (N×N) matrix shape, where "N" is an integer greater than or equal to 2. The multiple unit pixels have color filters of the same color and share a single microlens.

Images