Apparatus and method for obtaining image using lens array

Abstract

An image obtaining apparatus includes: an image sensor including a microlens and a plurality of adjacent pixels sharing the microlens; and a processor configured to: generate a plurality of parallax images having the same parallax based on an input image received from the image sensor; dividing first parallax images with similar image characteristics in the plurality of parallax images into one group; executing at least one image processing operation on the first parallax image; and generating an output image by combining the first parallax images on which the at least one image processing operation has been performed.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims the benefit of priority from Korean Patent Application No. 10-2023-0099003 filed on July 28, 2023, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to a device and method for obtaining an image using a lens array. Background Art

[0004] The image sensor may include red, green, and blue color filters arranged in a regular pattern on the photosensitive pixels for imaging color images. For example, the color filters may include a quad-Bayer pattern, in which 2×2 pixels of the same color are arranged under one microlens. The quad-Bayer pattern may use the parallax of light passing through the same microlens to autofocus on a target object.

[0005] An image signal processor (ISP) may perform image processing on an input image generated by an image sensor. The ISP may generate an output image from the input image based on various registers (or parameters), and the quality, characteristics, etc. of the image may vary depending on the parameter values ​​applied to the ISP.

[0006] As the number of pixels in image sensors increases, devices that acquire high-resolution images may require additional logic and memory to perform image processing. Summary of the invention

[0007] Provided are an apparatus and method for obtaining an image, in which an increase in logic and memory in an apparatus performing image processing is minimized.

[0008] Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the embodiments presented in the disclosure.

[0009] According to one aspect of the present disclosure, an image obtaining device may include: an image sensor including a microlens and a plurality of adjacent pixels sharing the microlens; and a processor configured to: generate a plurality of disparity images having the same disparity, respectively, based on an input image received from the image sensor; group first disparity images having similar image characteristics among the plurality of disparity images; perform at least one image processing operation on the first disparity images; and generate an output image by combining the first disparity images on which at least one image processing operation has been performed.

[0010] The first parallax image may include a first parallax image group and a second parallax image group, and the processor may be configured to perform at least one image processing operation on the first parallax image by performing image processing on the first parallax image group and the second parallax image group using context switching.

[0011] The processor may include: two logic circuits connected in parallel and configured to simultaneously perform image processing on the first parallax image group and the second parallax image group; and a memory configured to be shared and used by the two logic circuits.

[0012] The memory may include a random access memory (RAM) having a kernel memory area, and the processor may be configured to perform at least one image processing operation on the first disparity image group and the second disparity image group using context switching by preloading a register corresponding to the second disparity image group into a process control block of the kernel memory area during performance of at least one image processing operation on the first disparity image group.

[0013] The image sensor may include a quad Bayer pattern array in which pixels arranged in a 2×2 matrix include color filters of the same color, or a quad Bayer pattern array in which pixels arranged in a 4×4 matrix include color filters of the same color.

[0014] The processor can also be configured to generate an A parallax image, a B parallax image, a C parallax image, and a D parallax image based on an input image received from the image sensor, wherein the multiple adjacent pixels sharing a microlens include four pixels, and wherein the A parallax image corresponds to a combination of data values ​​of pixels arranged in the second quadrant, the B parallax image corresponds to a combination of data values ​​of pixels arranged in the first quadrant, the C parallax image corresponds to a combination of data values ​​of pixels arranged in the third quadrant, and the D parallax image corresponds to a combination of data values ​​of pixels arranged in the fourth quadrant.

[0015] A quad Bayer pattern array or a quad Bayer pattern array may include a GRBG pattern or an RGGB pattern, wherein the GRBG pattern includes a red pixel in a first quadrant of a first unit pixel, a green pixel in a second quadrant and a fourth quadrant of the first unit pixel, and a blue pixel in a third quadrant of the first unit pixel; and the RGGB pattern includes a green pixel in a first quadrant and a third quadrant of a second unit pixel, a red pixel in a second quadrant of the second unit pixel, and a blue pixel in a fourth quadrant of the second unit pixel.

[0016] A quad Bayer pattern array or a quad Bayer pattern array may include a GRBG pattern, wherein a first parallax image group includes an A parallax image and a D parallax image, a second parallax image group includes a B parallax image and a C parallax image, and wherein a color shift of the first parallax image group is less than a color shift of the second parallax image group.

[0017] A quad Bayer pattern array or a quad Bayer pattern array may include an RGGB pattern, wherein a first disparity image group includes a B disparity image and a C disparity image, a second disparity image group includes an A disparity image and a D disparity image, and wherein a noise level of the first disparity image group is less than a noise level of the second disparity image group.

[0018] The image obtaining apparatus may be configured to operate in a low power mode, and the processor may be configured to perform at least one image processing operation only on the second parallax image group in the low power mode.

[0019] The at least one image processing operation may include at least one of color shading correction, image processing correction, and noise correction.

[0020] The at least one image processing operation may include color shading correction and color shift correction, and the processor may be configured to perform the color shading correction and the color shift correction by aligning a red channel and a blue channel based on a green channel of the first parallax image group.

[0021] The at least one image processing operation may include performing white balancing on the input image, and the processor may be configured to perform white balancing on the input image by applying the same parameters to the first parallax image group and the second parallax image group in groups.

[0022] In the binning mode, the processor can also be configured to: generate a first combined image by combining the A disparity image and the D disparity image per unit pixel, wherein the A disparity image and the D disparity image are the same color; generate a second combined image by combining the B disparity image and the C disparity image after color shift correction is performed on the B disparity image and the C disparity image, wherein the B disparity image and the C disparity image are the same color; and perform weighted summation of the first combined image and the second combined image.

[0023] The processor may also be configured to generate an output image by combining the A parallax image, the B parallax image, the C parallax image, and the D parallax image, and wherein image processing is performed on the A parallax image, the B parallax image, the C parallax image, and the D parallax image in groups.

[0024] The similar image characteristics may include at least one of noise level, color shift, and color shading.

[0025] According to one aspect of the present disclosure, a method for obtaining an image by an image obtaining device, the image obtaining device including an image sensor, the image sensor including a plurality of adjacent pixels sharing a microlens, the method may include: generating a plurality of disparity images having the same disparity based on an input image received from the image sensor; grouping first disparity images having similar image characteristics among the plurality of disparity images; performing at least one image processing operation on the first disparity images; and generating an output image by combining the first disparity images on which the at least one image processing operation has been performed.

[0026] The first parallax image may include a first parallax image group and a second parallax image group, and performing at least one image processing operation on the first parallax image may include performing image processing on the first parallax image group and the second parallax image group using context switching.

[0027] The first disparity image may include a first disparity image group and a second disparity image group, at least one image processing operation may include a white balance operation, and performing at least one image processing operation may include: performing white balance on the input image by applying the same parameters to the first disparity image group and the second disparity image group as a group.

[0028] The similar image characteristics may include at least one of noise level, color shift, and color shading.

[0029] According to one aspect of the present disclosure, a non-temporary computer-readable storage medium storing instructions, when the instructions are executed by at least one processor, causes the at least one processor to: generate multiple disparity images with the same disparity based on an input image received from an image sensor; group first disparity images with similar image characteristics among the multiple disparity images; perform at least one image processing operation on the first disparity images; and generate an output image by combining the first disparity images on which the at least one image processing operation has been performed.

[0030] The first parallax image may include a first parallax image group and a second parallax image group, and when the instruction is executed, it may also enable at least one processor to perform at least one image processing operation on the first parallax image by performing image processing on the first parallax image group and the second parallax image group using context switching.

[0031] When the instructions are executed, at least one processor may also generate an A parallax image, a B parallax image, a C parallax image, and a D parallax image based on an input image received from the image sensor, wherein the A parallax image corresponds to a combination of data values ​​of pixels arranged in the second quadrant, the B parallax image corresponds to a combination of data values ​​of pixels arranged in the first quadrant, the C parallax image corresponds to a combination of data values ​​of pixels arranged in the third quadrant, and the D parallax image corresponds to a combination of data values ​​of pixels arranged in the fourth quadrant.

[0032] The at least one image processing operation may include at least one of color shading correction, image processing correction, and noise correction.

[0033] At least one image processing operation may include performing white balancing on the input image, and when the instructions are executed, the at least one processor may also be caused to perform white balancing on the input image by applying the same parameters to the first parallax image group and the second parallax image group in groups. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and other aspects, features and advantages of some embodiments of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings, in which:

[0035] Figure 1 is a diagram showing an image acquisition apparatus according to an embodiment;

[0036] Figure 2 and Figure 3 is a diagram showing an image sensor according to an embodiment;

[0037] Figure 4 is a diagram showing a pixel array of an image sensor according to an embodiment;

[0038] Figure 5A is a plan view showing an area of ​​a quad-Bayer pattern according to an embodiment;

[0039] Figure 5B is a plan view showing a region of a quadrilateral Bayer pattern according to an embodiment;

[0040] Figure 5C is a diagram showing a method according to an embodiment of the present invention. Figure 5A and Figure 5B A cross-sectional view of a region of the pixel array shown;

[0041] Fig. 6A is a plan view showing a color arrangement represented by a pixel array of an image sensor according to an embodiment;

[0042] Figure 6B and Figure 6C is a diagram showing a method according to an embodiment of the present invention. Fig. 6AA plan view of a metal prism array and a sensor substrate provided in a pixel array of the image sensor shown;

[0043] Figure 7 is a block diagram illustrating image processing performed by a processor according to an embodiment;

[0044] Fig. 8A , Figure 8B , Figure 8C and Fig.8D is a diagram showing a method of performing image processing in an image acquisition apparatus including a quad-Bayer pattern according to an embodiment;

[0045] Fig. 9 is a diagram of a color shift effect occurring at a step edge according to an embodiment;

[0046] Fig.10 is a diagram showing a simulated merging mode of the image acquisition device according to the embodiment;

[0047] Fig.11 is a flowchart showing an image processing method of an image acquisition device according to an embodiment;

[0048] Fig.12 is a diagram of an electronic device including an image sensor according to an embodiment; and

[0049] Fig.13 is a block diagram of a camera module provided in an electronic device according to an embodiment. DETAILED DESCRIPTION

[0050] Reference will now be made in detail to embodiments, examples of which are shown in the accompanying drawings, wherein similar elements are denoted by similar reference numerals throughout the drawings. In this regard, the embodiments presented may have different forms and should not be construed as being limited to the descriptions set forth herein. Therefore, the embodiments are described below only by reference to the accompanying drawings to explain various aspects.

[0051] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. Expressions such as "at least one of...", when following a list of elements, modify the entire list of elements rather than modifying the individual elements in the list. For example, the expression "at least one of a, b, and c" should be understood to include only a, only b, only c, both a and b, both a and c, both b and c, or all of a, b, and c.

[0052] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. In the accompanying drawings, like reference numerals in the drawings represent like elements, and the sizes of the components in the drawings may be exaggerated for the sake of clarity and ease of description. The embodiments described below are provided only as examples and thus may be embodied in various forms.

[0053] It will be understood that when a component is referred to as being "on" or "on top of" another component, the component can be directly on or above the other component in a non-contact manner. Expressions in the singular encompass plural expressions unless they have a clearly different meaning in the context. Throughout the specification, when a part "includes" an element, unless otherwise stated, another element may also be included without excluding the presence of the other element. In the context of describing embodiments (especially in the context of the appended claims), the use of the terms "a", "an", and "the" and similar indicators should be interpreted as covering both the singular and the plural.

[0054] Some embodiments will be described with reference to the accompanying drawings for functional blocks, parts and / or modules. It will be appreciated by those skilled in the art that these blocks, parts and / or modules are physically implemented by logic circuits, separate components, microprocessors, hard-wired circuits, storage devices, wiring connections and other electronic circuits. This can be formed using semiconductor-based manufacturing techniques or other manufacturing techniques. For blocks, parts and / or modules implemented by microprocessors or other similar hardware, these blocks, parts and / or modules can be programmed and controlled using software to perform the various functions discussed in the present disclosure, and can be operated by firmware and / or software. In addition, each block, part and / or module can be implemented by dedicated hardware, or can be implemented by a combination of dedicated hardware that performs some functions and a processor (e.g., at least one programmed microprocessor and associated circuit) configured to perform functions different from the aforementioned functions. In some embodiments, blocks, parts and / or modules can be physically divided into two or more collaborative separate blocks, parts and / or modules without departing from the conceptual scope of the present disclosure. In addition, in some embodiments, blocks, parts and / or modules can be physically combined into more complex blocks, parts and / or modules without departing from the conceptual scope of the present disclosure.

[0055] Figure 1 is a diagram showing an image obtaining apparatus according to an embodiment.

[0056] Reference Figure 1According to an embodiment, an image acquisition device 100 may include: an image sensor 10, configured to convert an optical image formed in a shooting lens part into an electrical signal to obtain a color image signal, wherein the shooting lens part is used to gather light reflected from an object 30 to form an optical image; and a processor 20, configured to process the color image signal obtained from the image sensor 10 and generate an output image.

[0057] The image sensor 10 may include a pixel array 11, a row driver 12, a readout circuit 13, a column driver 14, a control logic 15, etc. The row driver 12, the readout circuit 13, the column driver 14, and the control logic 15 may be circuits for controlling the pixel array 11 to generate image data, and may be included in a controller.

[0058] The image sensor 10 may generate an input image (or raw data) by converting light transmitted from the object 30 into an electrical signal. The input image may be output to the processor 20. The processor 20 may include an image signal processor (ISP) configured to generate an output image using the input image. According to an embodiment, the image signal processor may be provided in the image sensor 10.

[0059] The pixel array 11 included in the image sensor 10 may include a plurality of pixels PX, and the plurality of pixels PX may include a photoelectric device (e.g., a photodiode) configured to receive light to generate charges. In an embodiment, each of the plurality of pixels PX may include two or more photoelectric devices. Each pixel PX may generate a pixel signal corresponding to light of various wavelength bands, or two or more photoelectric devices may be included in each pixel PX to provide an auto focus (AF) function.

[0060] Each of the plurality of pixels PX may include a pixel circuit for generating a pixel signal according to the charge generated by the photodiode. In an embodiment, the pixel circuit may include a transfer transistor, a drive transistor, a selection transistor, a reset transistor, etc. For example, the pixel circuit may output a reset voltage, and may use the charge generated by the photodiode to generate a pixel voltage. The pixel voltage may be a voltage reflected by the charge generated by the photodiode in each of the plurality of pixels PX. In an embodiment, two or more adjacent pixels PX may constitute a pixel group, and the two or more pixels PX included in the pixel group may share at least some of the transfer transistor, the drive transistor, the selection transistor, and the reset transistor.

[0061] The row driver 12 may drive the pixel array 11 by row. For example, the row driver 12 may generate a transfer control signal for controlling a transfer transistor of a pixel circuit, a reset control signal for controlling a reset transistor, and a selection control signal for controlling a selection transistor.

[0062] The readout circuit 13 may include a correlated double sampler (CDS), an analog-to-digital converter (ADC), etc. The CDS may be connected to the pixels PX included in the row line selected by the row selection signal provided by the row driver 12 through the column line, and detect the reset voltage and the pixel voltage by performing correlated double sampling. The ADC may convert the reset voltage and the pixel voltage detected by the CDS into digital signals and output the digital signals.

[0063] The column driver 14 may include a latch circuit, a buffer, an amplifier circuit, etc., and the processor 20 may store or amplify a digital signal received from the readout circuit 13 to generate image data. The operation timing of the row driver 12, the readout circuit 13, and the column driver 14 may be determined by the control logic 15, and the control logic 15 may operate according to a control command sent by the processor 20. The processor 20 may perform signal processing on an input image output by the column driver 14 or the control logic 15 to generate an output image, and output the output image to a display device, etc., or store the output image in a memory, etc.

[0064] Figure 2 and Figure 3 is a diagram showing an image sensor according to an embodiment.

[0065] Reference Figure 2 , the image sensor 40 according to the embodiment may include a first layer 41, a second layer 42 disposed below the first layer 41, a third layer 43 disposed below the second layer 42, etc. The first layer 41, the second layer 42, and the third layer 43 may be stacked in a vertical direction. In an embodiment, the first layer 41 and the second layer 42 may be stacked at a wafer level, and the third layer 43 may be attached below the second layer 42 at a chip level. The first to third layers 41, 42, and 43 may be provided as one semiconductor package.

[0066] The first layer 41 may include a sensing area SA in which a plurality of pixels PX are disposed, and a first pad area PA1 disposed around the sensing area SA. The first pad area PA1 may include a plurality of upper pads PAD, and the plurality of upper pads PAD may be connected to pads disposed in the second pad area PA2 of the second layer 42 and the logic circuit LC through vias or the like.

[0067] Each of the plurality of pixels PX may include a photodiode configured to receive light and generate charges, a pixel circuit configured to process the charges generated by the photodiode, etc. The pixel circuit may include a plurality of transistors for outputting a voltage corresponding to the charges generated by the photodiode.

[0068] The second layer 42 may include a plurality of elements providing a logic circuit LC. The plurality of elements included in the logic circuit LC may provide a circuit for driving the pixel circuit of the first layer 41 (eg, Figure 1 1, a row driver 12, a readout circuit 13, a column driver 14, a control logic 15, etc. are shown). According to an embodiment, when the processor 20 is provided in the image sensor 40, the processor 20 may be included in the logic circuit LC of the second layer 42. A plurality of elements included in the logic circuit LC may be connected to the pixel circuit through the first pad area PA1 and the second pad area PA2. The logic circuit LC may obtain a reset voltage and a pixel voltage from a plurality of pixels PX and generate a pixel signal.

[0069] In an embodiment, at least one of the plurality of pixels PX may include a plurality of photodiodes arranged at the same level. Pixel signals generated from charges of each of the plurality of photodiodes may have a phase difference from each other, and the logic circuit LC may provide an AF function based on the phase difference between the pixel signals generated from the plurality of photodiodes included in one pixel PX.

[0070] The third layer 43 arranged below the second layer 42 may include a memory chip MC, a dummy chip DC, and a protective layer EN that seals the memory chip MC and the dummy chip DC. The memory chip MC may be a dynamic random access memory (RAM) (DRAM) or a static RAM (SRAM), and the dummy chip DC may not actually have a data storage function. The memory chip MC may be electrically connected to at least some of the elements included in the logic circuit LC of the second layer 42 through a bump, and store information required to provide the AF function. In an embodiment, the bump may be a micro bump.

[0071] Reference Figure 3, the image sensor 50 according to the embodiment may include a first layer 51 and a second layer 52. The first layer 51 may include: a sensing area SA in which a plurality of pixels PX are disposed; a logic circuit LC in which an element for driving the plurality of pixels PX is disposed; and a first pad area PA1 disposed at the periphery of the sensing area SA and the logic circuit LC. According to the embodiment, when the processor 20 is disposed in the image sensor 50, the processor 20 may be included in the logic circuit LC of the first layer 51. The first pad area PA1 may include a plurality of upper pads PAD, and the plurality of upper pads PAD may be connected to a memory chip MC disposed in the second layer 52 through vias or the like. The second layer 52 may include a memory chip MC, a dummy chip DC, and a protective layer EN that seals the memory chip MC and the dummy chip DC.

[0072] Figure 4 is a diagram showing a pixel array of an image sensor according to an embodiment.

[0073] Reference Figure 4 , the pixel array PXA of the image sensor according to the embodiment may include a plurality of pixels PX. The plurality of pixels PX may be connected to a plurality of row lines ROW1 to ROWm (ROW) and a plurality of column lines COL1 to COLn (COL). The image sensor may drive the plurality of pixels PX based on the row lines ROW. For example, the time required for driving the selection drive line among the plurality of row lines ROW and for reading the reset voltage and the pixel voltage from the pixel PX connected to the selection drive line may be defined as a horizontal period. The image sensor may be operated by a rolling shutter method (in which a plurality of pixels PX are sequentially exposed) or a global shutter method (in which a plurality of pixels PX are simultaneously exposed).

[0074] The reset voltage and the pixel voltage output from each of the plurality of pixels PX may be converted into digital data, and then may be processed into an input image through signal processing. An image signal processor provided in an image sensor or in a separate processor communicating with the image sensor may use the input image to generate a result image displayed on a display or a result image stored in a memory. Therefore, depending on the performance of the image signal processor, a tuning method, etc., different result images may be generated from the same input image.

[0075] As the number of pixels PX included in an image sensor increases, a high-resolution image sensor may require more logic and memory to perform image processing.

[0076] Therefore, by minimizing the logic and memory required to perform image processing, the time and cost of image processing can be reduced.

[0077] The pixel array PXA may include a plurality of pixels configured to sense light of different wavelengths. The arrangement of the pixels may be achieved in various ways. For example, Figure 5A and Figure 5B Various arrangements of pixel arrays PXA are shown.

[0078] Figure 5A is a plan view showing an area of ​​a quad-Bayer pattern according to an embodiment. Figure 5B is a plan view showing a region of a quadrilateral Bayer pattern according to an embodiment. Figure 5C is a diagram showing a Figure 5A and Figure 5B A cross-sectional view of a region of a pixel array is shown.

[0079] Reference Figure 5A and Figure 5C , the pixel array PXA may have a quad-Bayer pattern structure. Four pixels arranged in a 2×2 array (including 2 rows and 2 columns) may include color filters of the same color. Therefore, in the pixel array PXA, 4×4 pixels may include a 2×2 array of green pixels G, a 2×2 array of red pixels R, a 2×2 array of blue pixels B, and a 2×2 array of green pixels G.

[0080] Each pixel included in the pixel array PXA may include a color filter and a light receiving element (e.g., a photodiode). For example, the first pixel PX1 may include a first color filter CF1 and a first light receiving element PD1, and the second pixel PX2 may include a second color filter CF2 and a second light receiving element PD2. The first color filter CF1 and the second color filter CF2 may be the same color (e.g., green).

[0081] The first pixel PX1, the second pixel PX2, the third pixel PX3, and the fourth pixel PX4 may be arranged adjacent to each other and share one microlens ML. The microlens ML may have a convex shape to gather light incident on the unit pixel (e.g., the first pixel PX1, the second pixel PX2, the third pixel PX3, and the fourth pixel PX4). The microlens ML may have a hemispherical cross-section. In a plan view, the microlens ML has a circular shape.

[0082] Reference Figure 5B and Figure 5C , the pixel array PXA' may have a quadrilateral Bayer pattern structure. 16 pixels arranged in a 4×4 array (including 4 rows and 4 columns) may include color filters of the same color. Therefore, in the pixel array PXA', the 8×8 pixels may include a 4×4 array of green pixels G, a 4×4 array of red pixels R, a 4×4 array of blue pixels B, and a 4×4 array of green pixels G.

[0083] Each pixel included in the pixel array PXA' may include a color filter and a light receiving element (e.g., a photodiode). For example, the first pixel PX1' may include a first color filter CF1 and a first light receiving element PD1, and the second pixel PX2' may include a second color filter CF2 and a second light receiving element PD2. The first color filter CF1 and the second color filter CF2 may be the same color (e.g., green).

[0084] The first pixel PX1' to the sixteenth pixel PX16' may be arranged adjacently, and the first pixel PX1' to the fourth pixel PX4', the fifth pixel PX5' to the eighth pixel PX8', the ninth pixel PX9' to the twelfth pixel PX12', and the thirteenth pixel PX13' to the sixteenth pixel PX16' may share one microlens ML, respectively. The microlens ML may have a convex shape to gather light incident on a unit pixel (e.g., the first pixel PX1', the second pixel PX2', the third pixel PX3', and the fourth pixel PX4'). The microlens ML may have a hemispherical cross-section. In a plan view, the microlens ML has a circular shape. By including the microlens ML, the sensitivity of the image sensor may be increased. That is, the optical characteristics of the image sensor may be improved.

[0085] Fig. 6A is a plan view showing a color arrangement represented by a pixel array of an image sensor according to an embodiment. Figure 6B and Figure 6C is a diagram showing a Fig. 6A A plan view of a metaprism array and a sensor substrate disposed in a pixel array of an image sensor is shown.

[0086] Fig. 6A The color arrangement shown is a Bayer pattern arrangement. Fig. 6A As shown, a unit pattern may include four quadrants, and the first to fourth quadrants may represent red (R), green (G), blue (B), and green (G), respectively. Such a unit pattern may be repeatedly arranged in a two-dimensional (2D) manner along a first direction (X direction) and a second direction (Y direction). For this color arrangement, two green pixels may be arranged along a diagonal direction in a unit pattern of a 2×2 array, and one blue pixel and one red pixel may be arranged along another diagonal direction. In other words, a first row in which a plurality of green pixels and a plurality of blue pixels are alternately arranged along the first direction and a second row in which a plurality of red pixels and a plurality of green pixels are alternately arranged along the first direction may be repeatedly arranged along the second direction.

[0087] Although an example in which the pixel array PXA″ of the image sensor has a Bayer pattern is described below, its operating principle can be applied to various pixel arrangements other than the Bayer pattern (eg, a quad Bayer pattern, a quad Bayer pattern, etc.).

[0088] The pixel array PXA″ of the image sensor may include a dichroic lens array corresponding to this color arrangement (ie, concentrating light having a color corresponding to a particular pixel).

[0089] Reference Figure 6B , the sensor substrate SUB may include a plurality of pixels that sense incident light. The sensor substrate SUB may include a plurality of unit pixel groups PXA_G. The unit pixel group PXA_G may include a first sub-pixel group 111, a second sub-pixel group 112, a third sub-pixel group 113, and a fourth sub-pixel group 114, each of which is configured to convert incident light into an electrical signal and generate an image signal. The unit pixel group PXA_G may have a Bayer pattern pixel arrangement. That is, the first sub-pixel group 111 and the fourth sub-pixel group 114 may include a green pixel that senses green light, the second sub-pixel group 112 may include a red pixel that senses red light, and the third sub-pixel group 113 may include a blue pixel that senses blue light. The first to fourth sub-pixel groups 111, 112, 113, and 114 may each independently include a plurality of photosensitive units that sense incident light. For example, each of the first to fourth sub-pixel groups 111, 112, 113, and 114 may include first to fourth photosensitive units ce1, ce2, ce3, and ce4. The first to fourth photosensitive cells ce1, ce2, ce3 and ce4 may be arranged in a 2D manner along the first direction (X direction) and the second direction (Y direction). For example, the first to fourth photosensitive cells ce1, ce2, ce3 and ce4 may be arranged in a 2×2 array in the first to fourth sub-pixel groups 111, 112, 113 and 114.

[0090] although Figure 6B It is shown that each of the first to fourth sub-pixel groups 111, 112, 113 and 114 includes four photosensitive units, but four or more independent photosensitive units may be grouped together and arranged in a 2D manner. For example, each of the first to fourth sub-pixel groups 111, 112, 113 and 114 may include a plurality of independent photosensitive units arranged in a 4×4 array, such as Figure 5B For convenience of description, a case where each of the first to fourth sub-pixel groups 111, 112, 113, and 114 includes photosensitive units arranged in a 2×2 array will be described.

[0091] According to an embodiment, a plurality of pixels including a plurality of photosensitive units that sense light of the same color may be used as AF pixels. In the AF pixel, an AF signal may be acquired according to the difference between the output signals of adjacent photosensitive units. For example, an AF signal in a first direction (X direction) may be generated according to the difference between the output signal of the first photosensitive unit ce1 and the output signal of the second photosensitive unit ce2, the difference between the output signal of the third photosensitive unit ce3 and the output signal of the fourth photosensitive unit ce4, or the difference between the sum of the output signals of the first photosensitive unit ce1 and the third photosensitive unit ce3 and the sum of the output signals of the second photosensitive unit ce2 and the fourth photosensitive unit ce4. In addition, an AF signal in a second direction (Y direction) may be generated according to the difference between the output signal of the first photosensitive unit ce1 and the output signal of the third photosensitive unit ce3, the difference between the output signal of the second photosensitive unit ce2 and the output signal of the fourth photosensitive unit ce4, or the difference between the sum of the output signals of the first photosensitive unit ce1 and the second photosensitive unit ce2 and the sum of the output signals of the third photosensitive unit ce3 and the fourth photosensitive unit ce4.

[0092] The sum mode and the full mode can be used as a method of obtaining an image signal. In the sum mode, an image signal can be obtained by combining the output signals of the first to fourth photosensitive cells ce1, ce2, ce3, and ce4. For example, a first green image signal can be generated by combining the output signals of the first to fourth photosensitive cells ce1, ce2, ce3, and ce4 of the first sub-pixel group 111, a red image signal can be generated by combining the output signals of the first to fourth photosensitive cells ce1, ce2, ce3, and ce4 of the second sub-pixel group 112, a blue image signal can be generated by combining the output signals of the first to fourth photosensitive cells ce1, ce2, ce3, and ce4 of the third sub-pixel group 113, and a second green image signal can be generated by combining the output signals of the first to fourth photosensitive cells ce1, ce2, ce3, and ce4 of the fourth sub-pixel group 114. In the full mode, an output signal can be obtained separately from each of the first to fourth photosensitive cells ce1, ce2, ce3, and ce4. In this case, a high-resolution image can be obtained.

[0093] The first to fourth photosensitive units ce1, ce2, ce3 and ce4 may be electrically isolated by an isolation structure DTI. The isolation structure DTI may include a deep trench isolation structure. The deep trench may be filled with an electrically insulating material. After forming the photosensitive layer, a plurality of electrically isolated units may be formed by forming an isolation structure DTI on the photosensitive layer. The isolation structure DTI may electrically isolate the first to fourth sub-pixel groups 111, 112, 113 and 114, and may electrically isolate each of the first to fourth sub-pixel groups 111, 112, 113 and 114 into four regions. The cross-shaped separation line SL indicates the center line of the isolation structure DTI, which is included in the first to fourth sub-pixel groups 111, 112, 113 and 114, and isolates the first to fourth sub-pixel groups 111, 112, 113 and 114 into the first to fourth photosensitive units ce1, ce2, ce3 and ce4, respectively. Although the isolation structure DTI is shown as a line having a thickness close to zero for convenience of explanation, the isolation structure DTI may have a thickness that physically separates adjacent photosensitive units. The isolation structure DTI included in the first to fourth sub-pixel groups 111, 112, 113, and 114 may be an area that cannot sense light, and therefore, when designing a meta prism array described below, a method of reducing performance degradation due to such an isolation structure may be considered.

[0094] Reference Figure 6C , the metaprism array MP may include Figure 6B 1 and 2. The pixel group MP_G may include a plurality of pixel groups MP_G corresponding to the plurality of unit pixel groups PXA_G of the sensor substrate SUB shown in FIG. The pixel group MP_G may include the first to fourth sub-pixel groups 111, 112, 113 and

[0095] 114 corresponding to the first to fourth regions 131, 132, 133 and 134. Each of the first to fourth regions 131, 132, 133 and 134 may include a plurality of nanorods. The nanorods may separate incident light according to wavelengths and focus the separated light on the corresponding first to fourth sub-pixel groups 111, 112, 113 and 114 according to wavelengths. Figure 6B As described above, the first sub-pixel group 111 and the fourth sub-pixel group 114 may be first green pixels and second green pixels, respectively, the second sub-pixel group 112 may be red pixels, and the third sub-pixel group 113 may be blue pixels.

[0096] The shapes and arrangements of the plurality of nanorods included in the metaprism array MP may be determined to form an appropriate phase distribution, thereby achieving separation and collection of light by wavelength.

[0097] Figure 72 is a block diagram illustrating image processing performed by a processor according to an embodiment. The processor 20 may be an image signal processor configured to perform at least one image processing operation on an input image to generate an output image.

[0098] Reference Figure 7 , the processor 20 according to the embodiment may include a first logic 21 and a second logic 22. The first logic 21 and the second logic 22 may be connected in parallel to each other and simultaneously perform image processing (i.e., perform at least one image processing operation). The first logic 21 and the second logic 22 may perform image processing (or pre-processing) on ​​an input image (or raw data) received from the image sensor 10. For example, each of the first logic 21 and the second logic 22 may perform color shift correction, color shading correction, noise correction, etc. on at least a portion of the input image based on image characteristics.

[0099] The first logic 21 and the second logic 22 may share a memory MEM. For example, when performing image processing, the first logic 21 and the second logic 22 may receive the same register (or parameter) from the memory MEM. The memory MEM according to the embodiment may be a DRAM or an SRAM. When the processor 20 is provided in the image sensor 10 (see Figure 1 ), the memory MEM may correspond to Figure 2 and Figure 3 The memory chip MC is shown.

[0100] The processor 20 may perform image processing using a round robin method, which is a preemptive scheduling method.

[0101] The memory MEM may include a kernel memory area, a user memory area, and other memory areas, etc. The kernel memory area can be accessed only in kernel mode and stores code and data that provide key functions of the operating system; the user memory area can be accessed in user mode and stores program code, variable stacks, etc.; other memory areas may include a memory area used in a basic input / output system (BIOS) or a memory area used in a graphics card.

[0102] More specifically, the kernel memory area may include a code area, a data area, a stack area, a heap area, a process control block area, and the like. The code area is an area for storing the code required for executing the kernel. Since the kernel is executed in the kernel memory, the code area may be designated as a read-only area. The data area may be an area for storing data such as global variables and constants required when executing the kernel. The data area may be designated as a read-only area similar to the code area. The stack area may be an area for storing factors, local variables, and the like sent in response to a function call in the kernel. The stack area may have a last-in-first-out (LIFO) structure and generally has a limited size to prevent stack overflow in the kernel. When the kernel dynamically allocates or releases memory, the heap area may be used. The process control block area may be a structure for storing status information of each process. The process control block area can only be used inside the kernel and is mainly used for operations such as generation, termination, and scheduling of processes.

[0103] In the polling scheduling method, the total operation time can be divided at equal intervals and multiple processes can be executed alternately. The processor 20 can store the state of the currently executed process in the process control block area. For example, during the first process, the ID, state value, register, etc. of the first process can be stored in the process control block area, and after using the time allocated to the first process according to the polling scheduling method, the subsequent second process can be executed, and the ID, state value, register, etc. of the second process can be stored in the process control block area accordingly.

[0104] In context switching, when the time allocated for the first process has been used and the operation of the second process is about to be processed, an interruption may occur before the operation of the second process is processed. That is, context switching may refer to the following operation: handing over the control of the processor 20 by storing the state of the current process in the process control block area and preloading the state of the subsequent process from the process control block area.

[0105] Fig. 8A , Figure 8B , Figure 8C and Fig.8D is a diagram illustrating a method of performing image processing in an image obtaining apparatus including a quad Bayer pattern according to an embodiment. Fig. 9 is a diagram showing a color shift effect occurring at a step edge according to an embodiment. Fig. 8A , Figure 8B , Figure 8C and Fig.8D In the embodiment, the case where the image sensor 10 includes the microlens ML is taken as an example to explain the embodiment.

[0106] Reference Figures 7 to 9, the processor 20 according to the embodiment may receive an input image IMG_IN from the image sensor 10. The pixel array PXA of the image sensor 10 according to the embodiment may have a quad-Bayer pattern structure. Therefore, in the input image IMG_IN, the 4×4 sub-images may include 2×2 green G sub-images PA, PB, PC, and PD, 2×2 red R sub-images PA, PB, PC, and PD, 2×2 blue B sub-images PA, PB, PC, and PD, and 2×2 green G sub-images PA, PB, PC, and PD. In this regard, the sub-images PA, PB, PC, and PD of the same color may share one microlens ML (or metaprism array MP), and therefore include four parallaxes. That is, the sub-images PA, PB, PC, and PD may have A parallax, B parallax, C parallax, and D parallax.

[0107] The processor 20 may extract and combine sub-images PA, PB, PC, and PD having the same disparity from the input image IMG_IN to generate a plurality of disparity images IMG_PA, IMG_PB, IMG_PC, and IMG_PD.

[0108] By extracting and combining sub-images PA, PB, PC and PD with A parallax in the input image IMG_IN, an A parallax image IMP_PA can be generated; by extracting and combining sub-images PA, PB, PC and PD with B parallax in the input image IMG_IN, a B parallax image IMG_PB can be generated; by extracting and combining sub-images PA, PB, PC and PD with C parallax in the input image IMG_IN, a C parallax image IMG_PC can be generated; by extracting and combining sub-images PA, PB, PC and PD with D parallax in the input image IMG_IN, a D parallax image IMG_PD can be generated.

[0109] The A parallax image IMP_PA, the B parallax image IMG_PB, the C parallax image IMG_PC, and the D parallax image IMG_PD may have a Bayer pattern structure in which unit pixels having four quadrant areas are repeatedly arranged in a 2D manner along a first direction (e.g., horizontally) and a second direction (e.g., vertically), and the four quadrant areas include red pixels in the first quadrant, green pixels in the second quadrant, blue pixels in the third quadrant, and green pixels in the fourth quadrant. Therefore, the image processing described below may be performed on the parallax images IMG_PA, IMG_PB, IMG_PC, and IMG_PD using an algorithm and logic designed for the Bayer pattern structure. That is, a high-quality output image IMG_OUT may be generated while minimizing the logic for correcting different parallaxes of the four Bayer patterns including the microlens ML (or the metaprism array MP), thereby reducing costs.

[0110] Then, the processor 20 may group disparity images having similar characteristics among the plurality of disparity images IMG_PA, IMG_PB, IMG_PC, IMG_PD into one group and perform at least one image processing operation thereon in the group. The image characteristics may include noise level, color shift, and color shading of the image.

[0111] The plurality of disparity images IMG_PA, IMG_PB, IMG_PC, and IMG_PD may be grouped into a first disparity image group IMG_GR1 and a second disparity image group IMG_GR2.

[0112] According to an embodiment, the quad Bayer pattern array and the quad Bayer pattern array may include a GRBG pattern and an RGGB pattern. In the GRBG pattern, red pixels are arranged in the first quadrant of the unit pixel, green pixels are arranged in the second and fourth quadrants of the unit pixel, and blue pixels are arranged in the third quadrant of the unit pixel (e.g., Fig. 8A and Figure 8B In the RGGB pattern, green pixels are arranged in the first and third quadrants of the unit pixel, red pixels are arranged in the second quadrant of the unit pixel, and blue pixels are arranged in the fourth quadrant of the unit pixel (as shown in FIG. Figure 8C and Fig.8D shown).

[0113] According to an embodiment, when the GRBG pattern is included, the first parallax image group IMG_GR1 may include an A parallax image IMP_PA and a D parallax image IMG_PD, and the second parallax image group IMG_GR2 may include a B parallax image IMG_PB and a C parallax image IMG_PC. Due to the design characteristics of the image sensor 10, among the A parallax image IMP_PA, the B parallax image IMG_PB, the C parallax image IMG_PC, and the D parallax image IMG_PD, the image characteristics of the A parallax image IMP_PA and the D parallax image IMG_PD arranged along a 135° diagonal direction may be similar, and the image characteristics of the B parallax image IMG_PB and the C parallax image IMG_PC arranged along a 45° diagonal direction may be similar.

[0114] The first parallax image group IMG_GR1 may have better / improved image characteristics than the image characteristics of the second parallax image group IMG_GR2. That is, at least one image characteristic of the first parallax image group IMG_GR1 may be greater than, better than, or higher in quality than at least one image characteristic of the second parallax image group IMG_GR2. For example, the color shift of the first parallax image group IMG_GR1 may be less than the color shift of the second parallax image group IMG_GR2. Fig. 9As shown in the graph, the edge transitions of red L2 and blue L3 deviate further from the step edge L0 of the image than green L1, thereby producing color shading at the edge, which may cause color shift. This color shift of the second parallax image group IMG_GR2 may be greater than the color shift of the first parallax image group IMG_GR1. In this regard, the horizontal axis of the step edge L0 represents pixels, and the vertical axis represents RGB pixel values.

[0115] The noise level of the second parallax image group IMG_GR2 may be greater than the noise level of the first parallax image group IMG_GR1 .

[0116] According to another embodiment, when the RGGB pattern is included, the first parallax image group IMG_GR1 may include a B parallax image IMG_PB and a C parallax image IMG_PC, and the second parallax image group IMG_GR2 may include an A parallax image IMP_PA and a D parallax image IMG_PD. Due to the design characteristics of the image sensor 10, among the A parallax image IMP_PA, the B parallax image IMG_PB, the C parallax image IMG_PC, and the D parallax image IMG_PD, the image characteristics of the A parallax image IMP_PA and the D parallax image IMG_PD arranged along the 135° diagonal direction may be similar, and the image characteristics of the B parallax image IMG_PB and the C parallax image IMG_PC arranged along the 45° diagonal direction may be similar.

[0117] The first parallax image group IMG_GR1 may have better / improved image characteristics than the image characteristics of the second parallax image group IMG_GR2. That is, at least one image characteristic of the first parallax image group IMG_GR1 may be greater than, better than, or higher in quality than at least one image characteristic of the second parallax image group IMG_GR2. For example, the color shift of the first parallax image group IMG_GR1 may be less than the color shift of the second parallax image group IMG_GR2. Fig. 9 As shown in the graph, the edge transitions of red L2 and blue L3 deviate further from the step edge L0 of the image than green L1, thereby producing color shadows at the edges, which may cause color shift. This color shift of the second parallax image group IMG_GR2 may be greater than the color shift of the first parallax image group IMG_GR1. The noise level of the second parallax image group IMG_GR2 may be greater than the noise level of the first parallax image group IMG_GR1.

[0118] Conventionally, in order to process multiple disparity images IMG_PA, IMG_PB, IMG_PC, and IMG_PD, logic for each disparity image may be required. However, as described above, when different disparity images have similar image characteristics, the disparity images with similar image characteristics may be grouped together, and then image processing may be performed in groups, which helps to reduce costs. That is, when image processing is performed on pairs of disparity images with similar image characteristics, it is not necessary to include four logics for processing four disparity images IMG_PA, IMG_PB, IMG_PC, and IMG_PD, and only two logics 21 and 22 are required to perform image processing.

[0119] Furthermore, in order for the first logic 21 and the second logic 22 to perform image processing, registers (or parameters) required for image processing should be loaded from the memory MEM, and when image processing is performed on a pair of parallax images having similar image characteristics, the first logic 21 and the second logic 22 may be designed to share one memory MEM.

[0120] Image processing may be performed on the first and second parallax image groups IMG_GR1 and IMG_GR2 using the above-described context switching.

[0121] A first register applied to the first disparity image group IMG_GR1 during a first process of processing the first disparity image group IMG_GR1 may be stored in a process control block area of ​​a kernel memory, and a second register applied to the second disparity image group IMG_GR2 may be preloaded into the process control block area of ​​the kernel memory. Conversely, a second register applied to the second disparity image group IMG_GR2 during a second process of processing the second disparity image group IMG_GR2 may be stored in the process control block area of ​​the kernel memory, and a first register applied to the first disparity image group IMG_GR1 may be preloaded into the process control block area of ​​the kernel memory.

[0122] Then, the processor 20 may combine the A parallax image IMP_PA, the B parallax image IMG_PB, the C parallax image IMG_PC, and the D parallax image IMG_PD processed according to the first parallax image group IMG_GR1 and the second parallax image group IMG_GR2 to generate an output image IMG_OUT.

[0123] The metaprism array MP can operate efficiently for light within a specific angle range incident on the image sensor. However, when the incident angle exceeds the specific angle range, the color separation performance of the metaprism array MP may deteriorate. Thus, when the light efficiency of the metaprism array MP decreases in a region where the chief ray angle (CRA) increases, in order to compensate for the decreased light efficiency, the processor 20 may perform image processing.

[0124] When reading registers (or parameters) by context switching, the processor 20 according to an embodiment may read different registers for the center area and the edge area of ​​the image, rather than reading the same register for the entire image. This may make the context switching rate when performing image processing on the edge area of ​​the image greater than the context switching rate when performing image processing on the center area of ​​the image. Therefore, by increasing the context switching rate, image quality may be improved even in a larger CRA area.

[0125] Refer again Figure 1 , Fig. 8A , Figure 8B , Figure 8C and Fig.8D , the processor 20 may perform white balancing before generating the plurality of disparity images IMG_PA, IMG_PB, IMG_PC, and IMG_PD according to the input image IMG_IN.

[0126] White balance can refer to the following operation: adjusting the gain of R and B based on G so that the RGB ratio of the gray area is consistent under various color temperatures of light. Conceptually, white balance refers to the operation of normalizing the lighting effects of the captured scene so that all objects appear to be captured under ideal "white light".

[0127] In this regard, when performing white balancing, the processor 20 may apply the same (register) parameters to the first parallax image group IMG_GR1 and the second parallax image group IMG_GR2 having similar image characteristics in groups. When the image sensor 10 includes a metaprism array MP, in order to improve the sensitivity of the image sensor, different parameters (or registers) may be applied to the four parallax images IMG_PA, IMG_PB, IMG_PC, and IMG_PD.

[0128] Color shading refers to the color deviation between pixels of different channels in adjacent pixels due to different channel characteristics of color filters. For 4 pixels of the same color arranged in a 2×2 array (comprising 2 rows and 2 columns) such as a quad Bayer pattern, color deviation may occur within the same channel due to various reasons.

[0129] The processor 20 can correct the color deviation between channels by applying a correction gain of different values ​​according to the color channel. For example, a correction gain function having different coefficients according to the color channel can be preset, and the processor 20 can correct the color deviation by applying the correction gain function for each channel to the pixels of each channel. The correction gain function for each channel can be a two-dimensional or multi-dimensional function. In addition to the above-mentioned method of using the correction gain function, various correction methods can also be applied, such as a method of using a lookup table storing correction gain values ​​according to the channel.

[0130] When performing color shading correction and color shift correction, the processor 20 can align the red channel and the blue channel based on the green channel of the first disparity image group IMG_GR1. For a quad-Bayer pattern (i.e., a 4×4 repeating pattern), there are 8 green pixels therein, and a reference value for correction may need to be selected. As described above, in the design of the image sensor, in the GRBG pattern, the green pixel values ​​of the A disparity image IMP_PA and the D disparity image IMG_PD arranged along a 135° diagonal direction in the first disparity image group IMG_GR1 are similar to each other, and their image characteristics are also similar. The red pixel values ​​and the blue pixel values ​​can be aligned based on these green pixel values ​​to perform color shading correction and color shift correction. Conversely, in the RGGB pattern, the green pixel values ​​of the B disparity image IMG_PB and the C disparity image IMG_PC arranged along a 45° diagonal direction in the first disparity image group IMG_GR1 are similar to each other, and their image characteristics are also similar to each other. The red pixel values ​​and the blue pixel values ​​may be aligned based on these green pixel values ​​to perform color shading correction and color shift correction.

[0131] When the image acquisition device 100 operates in a low power mode, the processor 20 according to an embodiment may perform image processing only on the second disparity image group IMG_GR2 of the first disparity image group IMG_GR1 and the second disparity image group IMG_GR2. In the GRBG pattern, the degradation of image quality in the full mode (e.g., a mode that does not use 2×2 merging to process the entire input image) may be caused by the degradation of the B disparity image IMG_PB and the C disparity image IMG_PC due to the design characteristics of the image sensor (e.g., in the RGGB pattern, the degradation of image quality in the full mode may be caused by the degradation of the A disparity image IMP_PA and the D disparity image IMG_PD). ​​Such an operation can reduce the power consumption of the image acquisition device 100 by half.

[0132] Fig.10 1 is a diagram showing a simulated binning mode of an image acquisition device according to an embodiment. In the case of a Bayer pattern, the simulated binning mode may refer to a method capable of achieving the following effects: Figure 5C )After the received light is converted into electric charges, the four charges are combined before being converted by the ADC, just as the light is received by the pixel obtained by combining the pixel pitch of the four pixels.

[0133] Reference Figure 1 , Figure 5A and Fig.10In the simulated merging mode, the processor 20 according to the embodiment may generate a simulated first combined image IMG1 by combining an A parallax image (e.g., a first pixel PX1) and a D parallax image (e.g., a fourth pixel PX4) of the same color (e.g., green G) when including a GRBG pattern; generate a simulated second combined image IMG2 by combining a B parallax image (e.g., a second pixel PX2) and a C parallax image (e.g., a third pixel PX3) of the same color after color shift correction (and / or color shading correction) is performed by logics 23 and 24; and then generate a simulated third combined image IMG3 by weighted summing the first combined image IMG1 and the second combined image IMG2. When generating the third combined image IMG3, the processor 20 according to the embodiment may set the weight applied to the second combined image IMG2 to be lower than the weight applied to the first combined image IMG1. This is to achieve a target quantum efficiency (QE) by applying a higher weight to the A parallax image (e.g., the first pixel PX1) and the D parallax image (e.g., the fourth pixel PX4) (both the A parallax image and the D parallax image have good image characteristics) than to the B parallax image (e.g., the second pixel PX2) and the C parallax image (e.g., the third pixel PX3) (both the B parallax image and the C parallax image have poor image characteristics) based on the design characteristics of the image sensor. The processor 20 can then digitally convert the third combined image IMG3 through the ADC. Although for the convenience of description, Fig.10 The embodiment of the present invention focuses on describing green pixels of a quad-Bayer pattern, but the foregoing can also be applied to red pixels and blue pixels.

[0134] In the case of a four-way Bayer pattern, refer to Figure 5B , an analog first combined image IMG1 is generated by combining the A parallax image (e.g., the first pixel PX1' to the fourth pixel PX4') and the D parallax image (e.g., the thirteenth pixel PX13' to the sixteenth pixel PX16') of the same color; an analog second combined image IMG2 is generated by combining the B parallax image (e.g., the fifth pixel PX5' to the eighth pixel PX8') and the C parallax image (e.g., the ninth pixel PX9' to the twelfth pixel PX12') of the same color after color shift correction (and / or color shading correction) is performed by logic 23 and 24; a weighted summation may be performed on the first combined image IMG1 and the second combined image IMG2 to generate an analog third combined image IMG3. When generating the third combined image IMG3, the processor 20 according to an embodiment may set the weight applied to the second combined image IMG2 to be lower than the weight applied to the first combined image IMG1. Then, the processor 20 may digitally convert the third combined image IMG3 through the ADC.

[0135] Already combined FIG. 8A to FIG. 9 The case where the image sensor 10 includes a quad Bayer pattern (see Figure 5A ). However, the above image processing can also be applied to a four-way Bayer pattern (see Figure 5B ).

[0136] More specifically, in the case of a quad Bayer pattern, in the GRBG pattern, the first disparity image group IMG_GR1 may include an A disparity image IMP_PA and a D disparity image IMG_PD, and the second disparity image group IMG_GR2 may include a B disparity image IMG_PB and a C disparity image IMG_PC, as shown in FIG. Fig. 8A and Figure 8B As shown. In the case of a quad Bayer pattern, each of the disparity images IMG_PA, IMG_PB, IMG_PC, and IMG_PD may correspond to a Bayer pattern arranged in a 2×2 array (including 2 rows and 2 columns). In the case of a quad Bayer pattern, each of the disparity images IMG_PA, IMG_PB, IMG_PC, and IMG_PD may correspond to a quad Bayer pattern arranged in a 4×4 array (including 4 rows and 4 columns). Except for the above differences, other features may be substantially the same.

[0137] According to an embodiment, in the case of a quad Bayer pattern, in the RGGB pattern, the first disparity image group IMG_GR1 may include a B disparity image IMG_PB and a C disparity image IMG_PC, and the second disparity image group IMG_GR2 may include an A disparity image IMP_PA and a D disparity image IMG_PD. In the case of a quad Bayer pattern, each of the disparity images IMG_PA, IMG_PB, IMG_PC, and IMG_PD may correspond to a Bayer pattern arranged in a 2×2 array (including 2 rows and 2 columns). In the case of a quad Bayer pattern, each of the disparity images IMG_PA, IMG_PB, IMG_PC, and IMG_PD may correspond to a quad Bayer pattern arranged in a 4×4 array (including 4 rows and 4 columns). Except for the above differences, other features may be substantially the same.

[0138] Thus, when processing a four-way Bayer pattern, Figure 7 Each of the first logic 21 and the second logic 22 shown may be designed to process four Bayer patterns. However, the present disclosure is not limited thereto, and in order to use the first logic 21 and the second logic 22 designed to process the Bayer pattern, the processor 20 may divide the disparity images IMG_PA, IMG_PB, IMG_PC, and IMG_PD of the four Bayer patterns into four groups and generate disparity images of the Bayer pattern. Then, image processing may be performed on a total of 16 disparity images of the Bayer pattern.

[0139] The following embodiments will focus on the differences and omit or simplify the redundant descriptions of the same features.

[0140] Fig.11 is a flowchart showing an image processing method of an image acquisition device according to an embodiment. Figures 1 to 10 Some features described are not in Fig.11 , but these features can also be applied to Fig.11 Embodiment of the invention.

[0141] Reference Figures 1 to 10 , the method of obtaining an image according to an embodiment may include: in operation S100, generating a plurality of disparity images IMG_PA, IMG_PB, IMG_PC, and IMG_PD having the same disparity respectively according to an input image IMG_IN; in operation S200, grouping disparity images having similar image characteristics among the plurality of disparity images IMG_PA, IMG_PB, IMG_PC, and IMG_PD into one group and performing image processing on them as a group; and in operation S300, generating an output image IMG_OUT by combining the image-processed disparity images.

[0142] Image characteristics may include the noise level, color shift, and color shading of the image.

[0143] The disparity images IMG_PA, IMG_PB, IMG_PC, and IMG_PD may be grouped into a first disparity image group IMG_GR1 and a second disparity image group IMG_GR2, and image processing may be performed on the first disparity image group IMG_GR1 and the second disparity image group IMG_GR2 using context switching.

[0144] According to an embodiment, when the GRBG pattern is included, the first parallax image group IMG_GR1 may include an A parallax image IMP_PA and a D parallax image IMG_PD, and the second parallax image group IMG_GR2 may include a B parallax image IMG_PB and a C parallax image IMG_PC. Due to the design characteristics of the image sensor 10, among the A parallax image IMP_PA, the B parallax image IMG_PB, the C parallax image IMG_PC, and the D parallax image IMG_PD, the image characteristics of the A parallax image IMP_PA and the D parallax image IMG_PD arranged along a 135° diagonal direction may be similar, and the image characteristics of the B parallax image IMG_PB and the C parallax image IMG_PC arranged along a 45° diagonal direction may be similar.

[0145] The first parallax image group IMG_GR1 may have better / improved image characteristics than the image characteristics of the second parallax image group IMG_GR2. That is, at least one image characteristic of the first parallax image group IMG_GR1 may be greater than, better than, or higher in quality than at least one image characteristic of the second parallax image group IMG_GR2. For example, the color shift of the first parallax image group IMG_GR1 may be less than the color shift of the second parallax image group IMG_GR2. For example, referring to Fig. 9 As shown in the graph, the edge transitions of red L2 and blue L3 deviate further from the step edge L0 of the image than green L1, thereby producing color shadows at the edges, which may cause color shift. This color shift of the second parallax image group IMG_GR2 may be greater than the color shift of the first parallax image group IMG_GR1. The noise level of the second parallax image group IMG_GR2 may be greater than the noise level of the first parallax image group IMG_GR1.

[0146] According to another embodiment, when the RGGB pattern is included, the first parallax image group IMG_GR1 may include a B parallax image IMG_PB and a C parallax image IMG_PC, and the second parallax image group IMG_GR2 may include an A parallax image IMP_PA and a D parallax image IMG_PD. Due to the design characteristics of the image sensor 10, among the A parallax image IMP_PA, the B parallax image IMG_PB, the C parallax image IMG_PC, and the D parallax image IMG_PD, the image characteristics of the A parallax image IMP_PA and the D parallax image IMG_PD arranged along the 135° diagonal direction may be similar, and the image characteristics of the B parallax image IMG_PB and the C parallax image IMG_PC arranged along the 45° diagonal direction may be similar.

[0147] The first parallax image group IMG_GR1 may have better / improved image characteristics than the image characteristics of the second parallax image group IMG_GR2. That is, at least one image characteristic of the first parallax image group IMG_GR1 may be greater than, better than, or higher in quality than at least one image characteristic of the second parallax image group IMG_GR2. For example, the color shift of the first parallax image group IMG_GR1 may be less than the color shift of the second parallax image group IMG_GR2. In other words, referring to Fig. 9 As shown in the graph, the edge transitions of red L2 and blue L3 deviate further from the step edge L0 of the image than green L1, thereby producing color shadows at the edges, which may cause color shift. This color shift of the second parallax image group IMG_GR2 may be greater than the color shift of the first parallax image group IMG_GR1. The noise level of the second parallax image group IMG_GR2 may be greater than the noise level of the first parallax image group IMG_GR1.

[0148] Generally speaking, in order to process multiple disparity images IMG_PA, IMG_PB, IMG_PC, and IMG_PD, logic for each disparity image is required. However, as described above, when different disparity images have similar image characteristics, the disparity images with similar image characteristics can be grouped together, and then image processing is performed in groups, which helps to reduce costs. That is, when image processing is performed on pairs of disparity images with similar image characteristics, it is not necessary to include four logics for processing four disparity images IMG_PA, IMG_PB, IMG_PC, and IMG_PD, and only two logics 21 and 22 are required to perform image processing.

[0149] In addition, in order to perform image processing by the first logic 21 and the second logic 22, registers (or parameters) required for image processing can be loaded from the memory MEM, and when image processing is performed on a pair of parallax images having similar image characteristics, the first logic 21 and the second logic 22 can be designed to share a memory MEM.

[0150] Image processing may be performed on the first and second parallax image groups IMG_GR1 and IMG_GR2 using the above-described context switching.

[0151] A first register applied to the first disparity image group IMG_GR1 during a first process of processing the first disparity image group IMG_GR1 may be stored in a process control block area of ​​a kernel memory, and a second register applied to the second disparity image group IMG_GR2 may be preloaded into the process control block area of ​​the kernel memory. Conversely, a second register applied to the second disparity image group IMG_GR2 during a second process of processing the second disparity image group IMG_GR2 may be stored in the process control block area of ​​the kernel memory, and a first register applied to the first disparity image group IMG_GR1 may be preloaded into the process control block area of ​​the kernel memory.

[0152] Then, the processor 20 may combine the A parallax image IMP_PA, the B parallax image IMG_PB, the C parallax image IMG_PC, and the D parallax image IMG_PD processed according to the first parallax image group IMG_GR1 and the second parallax image group IMG_GR2 to generate an output image IMG_OUT.

[0153] The method of obtaining an image according to an embodiment may further include: performing white balancing on the input image by applying the same parameters according to the first parallax image group IMG_GR1 and the second parallax image group IMG_GR2. In this regard, when performing white balancing, the processor 20 may apply the same (register) parameters to the first parallax image group IMG_GR1 and the second parallax image group IMG_GR2 having similar image characteristics in groups. When the image sensor 10 includes a metaprism array MP, in order to improve the sensitivity of the image sensor, different parameters (or registers) may be applied to the four parallax images IMG_PA, IMG_PB, IMG_PC, and IMG_PD.

[0154] Fig.12 is a diagram of an electronic device including an image sensor according to an embodiment. Fig.12 In the network environment ED00, the electronic device ED01 can communicate with another electronic device ED02 through a first network ED98 (short-distance wireless communication network, etc.), or communicate with another electronic device ED04 and / or a server ED08 through a second network ED99 (long-distance wireless communication network, etc.). The electronic device ED01 can communicate with the electronic device ED04 through the server ED08. The electronic device ED01 may include a processor ED20, a memory ED30, an input device ED50, an audio output device ED55, a display device ED60, an audio module ED70, a sensor module ED76, an interface ED77, a tactile module ED79, a camera module ED80, a power management module ED88, a battery ED89, a communication module ED90, a user identification module ED96 and / or an antenna module ED97. In the electronic device ED01, some components (display device ED60, etc.) may be omitted, or other components may be added. Some components may be implemented by an integrated circuit. For example, the sensor module ED76 (fingerprint sensor, iris sensor, illumination sensor, etc.) can be implemented by being embedded in the display device ED60 (display, etc.).

[0155] The processor ED20 can control one or more other components (hardware and software components, etc.) of the electronic device ED01 connected to the processor ED20 by executing software (program ED40, etc.), and perform various data processing or calculations. As part of data processing or calculation, the processor ED20 can load instructions and / or data received from other components (sensor module ED76, communication module ED90, etc.) into the volatile memory ED32, process the instructions and / or data stored in the volatile memory ED32, and store the result data in the non-volatile memory ED34. The processor ED20 may include a main processor ED21 (central processing unit, application processor, etc.), and an auxiliary processor ED23 (graphics processing unit, image signal processor, sensor hub processor, communication processor, etc.) that can operate independently or together with the main processor ED21. The auxiliary processor ED23 can use less power than the main processor ED21 and can perform specialized functions.

[0156] The auxiliary processor ED23 can replace the main processor ED21 when the main processor ED21 is in an inactive state (sleep state), or control functions and / or states related to some components (display device ED60, sensor module ED76, communication module ED90, etc.) of the electronic device ED01 together with the processor ED21 when the main processor ED21 is in an active state (application execution state). The auxiliary processor ED23 (image signal processor, communication processor, etc.) can be implemented as a part of other functionally related components (camera module ED80, communication module ED90, etc.).

[0157] The memory ED30 can store various data required by the components of the electronic device ED01 (processor ED20, sensor module ED76, etc.). The data may include, for example, software (program ED40, etc.) and input data and / or output data related to commands regarding the data. The memory ED30 may include a volatile memory ED32 and / or a non-volatile memory ED34.

[0158] The program ED40 may be stored as software in the memory ED30 , and may include an operating system ED42 , middleware ED44 , and / or application programs ED46 .

[0159] The input device ED50 may receive commands and / or data from outside the electronic device ED01 (user, etc.) for components (processor ED20, etc.) of the electronic device ED01. The input device ED50 may include a microphone, a mouse, a keyboard, and / or a digital pen (stylus, etc.).

[0160] The audio output device ED55 can output audio signals to the outside of the electronic device ED01. The audio output device ED55 may include a speaker and / or an earpiece. The speaker can be used for general purposes such as multimedia playback or recording playback, and the earpiece can be used to receive incoming calls. The earpiece can be implemented by being coupled to the part of the speaker or by an independent separate device.

[0161] The display device ED60 can visually provide information to the outside of the electronic device ED01. The display device ED60 may include a display, a holographic device or a projector, and a control circuit that controls the corresponding device. The display device ED60 may include a touch circuit configured to detect a touch and / or a sensor circuit (pressure sensor, etc.) configured to measure the force generated by the touch.

[0162] The audio module ED70 can convert sound into an electrical signal or convert an electrical signal into sound. The audio module ED70 can obtain sound through the input device ED50 in a wired or wireless manner, or output sound through a speaker and / or earphone connected to the audio output device ED55 and / or another electronic device (electronic device ED02, etc.) of the electronic device ED01.

[0163] The sensor module ED76 can detect the operating state (power, temperature, etc.) or external environment state (user state, etc.) of the electronic device ED01, and generate an electrical signal and / or data value corresponding to the detected state. The sensor module ED76 can include a gesture sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR sensor, a biometric sensor, a temperature sensor, a humidity sensor, and / or an illumination sensor.

[0164] Interface ED77 may support one or more specified protocols for electronic device ED01 to be connected to another electronic device (electronic device ED02, etc.) in a wired or wireless manner. Interface ED77 may include a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, and / or an audio interface.

[0165] The connection terminal ED78 may include a connector for physically connecting the electronic device ED01 to another electronic device (electronic device ED02, etc.) The connection terminal ED78 may include an HDMI connector, a USB connector, an SD card connector, and / or an audio connector (headphone connector, etc.).

[0166] The haptic module ED79 may convert the electrical signal into a mechanical stimulus (vibration, motion, etc.) or an electrical stimulus that the user may sense through a sense of touch or motion. The haptic module ED79 may include a motor, a piezoelectric device, and / or an electrical stimulation device.

[0167] The camera module ED80 can capture still images and videos. The camera module ED80 may include a lens assembly having one or more lenses, an image sensor 10, an image signal processor, and / or a flash. The lens assembly included in the camera module ED80 may collect light emitted from a subject for image capture.

[0168] The power management module ED88 may manage power supplied to the electronic device ED01. The power management module ED88 may be implemented as a part of a power management integrated circuit (PMIC).

[0169] The battery ED89 can supply power to the components of the electronic device ED01. The battery ED89 can include a non-rechargeable primary battery, a rechargeable secondary battery and / or a fuel cell.

[0170] The communication module ED90 can establish a wired communication channel and / or a wireless communication channel between the electronic device ED01 and other electronic devices (electronic device ED02, electronic device ED04, server ED08, etc.), and support communication through the established communication channel. The communication module ED90 can operate independently of the processor ED20 (application processor, etc.), and can include one or more communication processors that support wired communication and / or wireless communication. The communication module ED90 may include a wireless communication module ED92 (cellular communication module, short-range wireless communication module, global navigation satellite system (GNSS) communication module, etc.) and / or a wired communication module ED94 (local area network (LAN) communication module, power line communication module, etc.). In the above-mentioned communication modules, the corresponding communication module can communicate with another electronic device through a first network ED98 (short-range communication network such as Bluetooth, WiFi Direct or Infrared Data Association (IrDA)) or a second network ED99 (long-range communication network such as a cellular network, the Internet or a computer network (LAN, WAN, etc.)). These various types of communication modules may be integrated into one component (single chip, etc.), or may be implemented as multiple separate components (multiple chips). The wireless communication module ED92 may verify and authenticate the electronic device ED01 in a communication network (e.g., first network ED98 and / or second network ED99) by using user information (International Mobile Subscriber Identity (IMSI) etc.) stored in the user identification module ED96.

[0171] The antenna module ED97 can send signals and / or power to the outside (another electronic device, etc.), or receive signals and / or power from the outside. The antenna may include a transmitter formed in a conductive pattern on a substrate (printed circuit board (PCB), etc.). The antenna module ED97 may include one or more antennas. When the antenna module ED97 includes multiple antennas, the communication module ED90 can select an appropriate antenna for a communication method used in a communication network such as a first network ED98 and / or a second network ED99 from the multiple antennas. Signals and / or power can be sent or received between the communication module ED90 and another electronic device through the selected antenna. Other components (RFIC, etc.) other than the antenna may be included as part of the antenna module ED97.

[0172] Some of the constituent elements can be connected to each other via a communication method between peripheral devices (bus, general purpose input and output (GPIO), serial peripheral interface (SPI), mobile industry processor interface (MIPI), etc.), and can exchange signals (commands, data, etc.) with each other.

[0173] Commands or data can be sent or received between the electronic device ED01 and the external electronic device ED04 through the server ED08 connected to the second network ED99. The electronic devices ED02 and ED04 can be of the same or different types as the electronic device ED01. All or part of the operations performed in the electronic device ED01 can be performed in one or more electronic devices (ED02, ED04 and ED08). For example, when the electronic device ED01 needs to perform a function or service, the electronic device ED01 can request one or more electronic devices to perform part of the entire function or service instead of performing the function or service. The one or more electronic devices that receive the request can perform additional functions or services related to the request and send the execution results to the electronic device ED01. To this end, cloud computing, distributed computing and / or client-server computing technologies can be used.

[0174] Fig.13 is a block diagram of a camera module provided in an electronic device according to an embodiment. Fig.13 The camera module ED80 may include a lens assembly 1170 , a flash 1120 , an image sensor 1100 , an image stabilizer 1140 , an AF controller 1130 , a memory 1150 (buffer memory, etc.), an actuator 1180 , and / or an image signal processor (ISP) 1160 .

[0175] The lens assembly 1170 can collect light emitted from the subject for image capture. The lens assembly 1170 may include one or more optical lenses. The lens assembly 1170 may include an optical path changing member configured to change the optical path to point to the image sensor 1000. Depending on whether the optical path changing member is set and its arrangement relative to the optical lens, the camera module ED80 may have a vertical shape or a folded shape. The camera module ED80 may include a plurality of lens assemblies 1170, and in this case, the camera module ED80 may include a dual camera, a 360-degree camera, or a spherical camera. Some lens assemblies 1170 may have the same lens properties (angle of view, focal length, AF, F number, optical zoom, etc.) or different lens properties. The lens assembly 1170 may include a wide-angle lens or a telephoto lens.

[0176] The actuator 1180 may drive the lens assembly 1170. The actuator 180 may move, for example, at least a portion of an optical lens and an optical path changing member constituting the lens assembly 1170. The optical lens may move along the optical axis, and at least a portion of the optical lens included in the lens assembly 1170 may be moved to adjust the distance between adjacent lenses, thereby adjusting the optical zoom ratio accordingly.

[0177] The actuator 1180 may adjust the position of any one optical lens included in the lens assembly 1170 so that the image sensor 1000 is located at the focal length of the lens assembly 1170. The actuator 1180 may drive the lens assembly 1170 according to the AF driving signal transmitted from the AF controller 1130.

[0178] The flash 1120 may emit light for enhancing light emitted or reflected from the subject. The flash 1120 may emit visible light or infrared light. The flash 1120 may include one or more light emitting diodes ((red, green, and blue (RGB) light emitting diodes (LEDs), white light LEDs, infrared LEDs, ultraviolet LEDs, etc.)) and / or xenon lamps. The image sensor 1000 may be related to Figure 1 The described image sensor 1000 may acquire an image corresponding to an object by converting light emitted or reflected from the object and emitted via the lens assembly 1170 into an electric signal.

[0179] The image sensor 1000 may include the metaprism array MP described above, and each pixel may include a plurality of photosensitive units forming a plurality of channels (e.g., a plurality of photosensitive units arranged in a 2×2 array). Such a pixel may be used as an AF pixel, and the image sensor 100 may generate an AF driving signal based on signals of the plurality of channels in the AF pixel.

[0180] The image stabilizer 1140 may move in response to movement of the camera module ED80 or the electronic device ED01 including the camera module ED80, one or more lenses included in the lens assembly 1170, or the image sensor 1000 in a specific direction, or may compensate for the negative effects caused by the movement by controlling the movement characteristics of the image sensor 1000 (adjusting the readout timing, etc.). The image stabilizer 1140 may detect the movement of the camera module ED80 or the electronic device ED01 by using a gyro sensor or an acceleration sensor arranged inside or outside the camera module ED80. The image stabilizer 1140 may be implemented in an optical form.

[0181] The AF controller 1130 may generate an AF driving signal according to a signal value sensed from the AF pixel of the image sensor 1000. The AF controller 1130 may control the actuator 1180 according to the AF driving signal.

[0182] The memory 1150 may store part or all of the data of the image obtained by the image sensor 1000 for subsequent image processing operations. For example, when a plurality of images are obtained at high speed, only the low-resolution image is displayed, and the obtained raw data (data of the Bayer pattern, high-resolution data, etc.) is stored in the memory 1150. The memory 1150 may then be used to send the raw data of the selected (user-selected, etc.) image to the image signal processor 1160. The memory 1150 may be incorporated into the memory ED30 of the electronic device ED01 or may be configured as a separate memory that operates independently.

[0183] The image signal processor 1160 may perform image processing operations on an image obtained by the image sensor 1000 or image data stored in the memory 1150. The image processing operations may include depth map generation, three-dimensional modeling, panorama generation, feature point extraction, image synthesis and / or image compensation (noise reduction, resolution adjustment, brightness adjustment, blur, sharpening, softening, etc.). The image signal processor 1160 may perform control (exposure time control or readout timing control, etc.) for the constituent elements (image sensor 1000, etc.) included in the camera module ED80. The image processed by the image signal processor 1160 may be stored again in the memory 1150 for additional processing or provided to the external constituent elements (memory ED30, display device ED60, electronic device ED02, electronic device ED04, server ED08, etc.) of the camera module ED80. The image signal processor 1160 may be incorporated into the processor ED20, or may be configured as a separate processor that operates independently of the processor ED20. When the image signal processor 1160 is configured as a processor separate from the processor ED20 , the image processed by the image signal processor 1160 may be subjected to additional image processing by the processor ED20 and then displayed by the display device ED60 .

[0184] The AF controller 1130 may be integrated into the image signal processor 1160. The image signal processor 1160 may generate an AF signal by processing a signal from an auto focus pixel of the image sensor 1000, and the AF controller 1130 may convert the AF signal into a driving signal of the actuator 1180 and transmit it to the actuator 1180.

[0185] The image sensor 1000 according to the embodiment may be applied to various electronic devices.

[0186] The image sensor 1000 according to the embodiment can be applied to mobile phones, smart phones, tablet computers, smart tablet computers, digital cameras, video cameras, laptop computers, television sets, smart TVs, etc. For example, a smart phone or smart tablet computer may include multiple high-resolution cameras, each of which is equipped with a high-resolution image sensor. The high-resolution camera can be used to extract depth information of a subject in an image, adjust the defocus of an image, or automatically identify a subject in an image.

[0187] In addition, the image sensor 1000 can also be applied to smart refrigerators, security cameras, robots, medical cameras, etc. For example, a smart refrigerator can use an image sensor to automatically identify food in the refrigerator and notify the user through a smartphone of the presence of specific food, the type of food input or output, etc. Security cameras can provide ultra-high resolution images and can identify objects or people in images in dark environments by using high sensitivity. Robots can be set up at disaster or industrial sites that people cannot directly enter and can provide high-resolution images. Medical cameras can provide high-resolution images for diagnosis or surgery, so the field of view can be adjusted dynamically.

[0188] In addition, the image sensor 1000 can also be applied to a vehicle. The vehicle may include a plurality of vehicle-mounted cameras arranged at different positions, and each vehicle-mounted camera may include an image sensor according to an embodiment. The vehicle may use the vehicle-mounted camera to provide the driver with various information inside or around the vehicle, thereby automatically identifying objects or people in the image and providing information required for autonomous driving.

[0189] In an apparatus and method for obtaining an image according to an embodiment, an increase in logic and memory can be minimized by generating multiple disparity images having the same disparity based on an input image obtained by an image sensor, and grouping disparity images having similar image characteristics into a group to perform image processing thereon in groups.

[0190] When used in conjunction with various embodiments of the present disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms such as logic, logic block, component, or circuit. A module may be a single integrated component or its smallest unit or a portion thereof suitable for performing one or more functions. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0191] Various embodiments described herein may be implemented as software including one or more instructions stored in a machine-readable storage medium. For example, a processor of a machine may call at least one of the one or more instructions stored in a storage medium and use or not use one or more other components to execute the instruction under the control of the processor. This enables the machine to operate to perform at least one function according to at least one instruction called. One or more instructions may include code generated by a compiler or code executable by an interpreter. A machine-readable storage medium may be provided in the form of a non-transitory storage medium. Among them, the term "non-transitory" merely means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between situations where data is semi-permanently stored in a storage medium and situations where data is temporarily stored in a storage medium.

[0192] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or via an application store (e.g., PlayStore). TM ) online distribution (e.g., download or upload), or directly between two user devices (e.g., smart phones). If distributed online, at least a portion of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium (e.g., a manufacturer's server, an application store's server, or a memory of a relay server).

[0193] According to various embodiments, each of the above-mentioned components (e.g., a module or a program) may include a single entity or multiple entities, and some of the multiple entities may be arranged separately in different components. According to various embodiments, one or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform one or more functions of each of the multiple components in the same or similar manner as performed by a corresponding one of the multiple components before integration. According to various embodiments, the operations performed by a module, a program or another component may be performed sequentially, in parallel, repeatedly or heuristically, or one or more of the operations may be performed or omitted in a different order, or one or more other operations may be added.

[0194] The above embodiments include but are not limited to Figure 1 , Figure 7 , Fig.10 , Fig.12 and Fig.13 At least one of the devices, units, components, modules, parts, etc. represented by boxes or equivalent indications may be physically implemented by analog and / or digital circuits including one or more of logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, etc., or may be implemented or driven by software and / or firmware (configured to perform the functions or operations described herein).

[0195] It should be understood that the embodiments described herein should be considered in a descriptive sense only and not for restrictive purposes. The description of features or aspects in each embodiment should generally be considered as applicable to other similar features or aspects in other embodiments. Although one or more example embodiments have been described with reference to the accompanying drawings, it should be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope defined by the appended claims.

Claims

1. An image acquisition device, comprising: An image sensor comprising a microlens and a plurality of adjacent pixels sharing the microlens; as well as The processor is configured as: generating a plurality of parallax images respectively having the same parallax based on an input image received from the image sensor; grouping first disparity images having similar image characteristics among the plurality of disparity images into one group; performing at least one image processing operation on the first disparity image; as well as An output image is generated by combining the first parallax images on which the at least one image processing operation is performed.

2. The image acquisition device according to claim 1, wherein: The first parallax image includes a first parallax image group and a second parallax image group, and The processor is further configured to perform the at least one image processing operation on the first parallax image by performing image processing on the first parallax image group and the second parallax image group using context switching.

3. The image acquisition device according to claim 2, wherein: The processor comprises: two logic circuits connected in parallel, the two logic circuits being configured to simultaneously perform image processing on the first parallax image group and the second parallax image group; and The memory is configured to be shared and used by the two logic circuits.

4. The image acquisition device according to claim 3, wherein: The memory includes a random access memory RAM having a kernel memory area, and The processor is further configured to perform the at least one image processing operation on the first parallax image group and the second parallax image group using context switching by: During the execution of the at least one image processing operation on the first parallax image group, registers corresponding to the second parallax image group are preloaded into a process control block of the kernel memory area.

5. The image acquisition device according to claim 2, wherein: The image sensor comprises: a quad-Bayer pattern array in which pixels arranged in a 2×2 matrix include color filters of the same color; or A quad Bayer pattern array in which pixels arranged in a 4×4 matrix include color filters of the same color.

6. The image acquisition device according to claim 5, wherein: The processor is further configured to generate an A parallax image, a B parallax image, a C parallax image, and a D parallax image based on the input image received from the image sensor, wherein the plurality of adjacent pixels sharing the microlens include four pixels, and Among them, the A parallax image corresponds to a combination of data values ​​of pixels arranged in the second quadrant, the B parallax image corresponds to a combination of data values ​​of pixels arranged in the first quadrant, the C parallax image corresponds to a combination of data values ​​of pixels arranged in the third quadrant, and the D parallax image corresponds to a combination of data values ​​of pixels arranged in the fourth quadrant.

7. The image acquisition device according to claim 6, wherein: The quad Bayer pattern array or the quad Bayer pattern array comprises: GRBG patterns, including: a red pixel in a first quadrant of a first unit pixel; green pixels in the second and fourth quadrants of the first unit pixel; and a blue pixel in the third quadrant of the first unit pixel; or RGGB pattern, including: green pixels in the first and third quadrants of the second unit pixel; a red pixel in a second quadrant of the second unit pixel; and A blue pixel in a fourth quadrant of the second unit pixel.

8. The image acquisition device according to claim 7, wherein: The quad Bayer pattern array or the quad Bayer pattern array includes the GRBG pattern, The first parallax image group includes the A parallax image and the D parallax image. The second disparity image group includes the B disparity image and the C disparity image, and The color shift of the first parallax image group is smaller than the color shift of the second parallax image group.

9. The image acquisition device according to claim 7, wherein: The quad Bayer pattern array or the quad Bayer pattern array includes the RGGB pattern, The first disparity image group includes the B disparity image and the C disparity image. wherein the second parallax image group includes the A parallax image and the D parallax image, and The noise level of the first parallax image group is smaller than the noise level of the second parallax image group.

10. The image acquisition device according to claim 8, wherein: The image acquisition device is configured to operate in a low power mode, and The processor is configured to: perform the at least one image processing operation only on the second parallax image group in the low power mode.

11. The image acquisition device according to claim 8, wherein: The at least one image processing operation includes at least one of color shading correction, color shift correction, and noise correction.

12. The image acquisition device according to claim 11, wherein: The at least one image processing operation includes the color shading correction and the color shift correction, and The processor is further configured to perform the color shading correction and the color shift correction by aligning a red channel and a blue channel based on a green channel of the first parallax image group.

13. The image acquisition device according to claim 8, wherein: The at least one image processing operation comprises: performing white balancing on the input image, and The processor is further configured to perform the white balance on the input image by applying the same parameters to the first parallax image group and the second parallax image group in groups.

14. The image acquisition device according to claim 8, wherein: In the merge mode, the processor is further configured to: generating a first combined image by combining the A parallax image and the D parallax image, the A parallax image and the D parallax image having the same color, per unit pixel; generating a second combined image by combining the B parallax image and the C parallax image after performing color shift correction on the B parallax image and the C parallax image, the B parallax image and the C parallax image having the same color; and A weighted sum is performed on the first combined image and the second combined image.

15. The image acquisition device according to claim 6, wherein: The processor is further configured to generate the output image by combining the A parallax image, the B parallax image, the C parallax image, and the D parallax image, and Here, image processing is performed on the A parallax image, the B parallax image, the C parallax image, and the D parallax image in groups.

16. The image acquisition device according to claim 1, wherein: The similar image characteristics include at least one of noise level, color shift, and color shading.

17. A method for obtaining an image by an image obtaining device, the image obtaining device comprising an image sensor, the image sensor comprising a plurality of adjacent pixels sharing a microlens, the method comprising: generating a plurality of parallax images having the same parallax based on an input image received from the image sensor; grouping first disparity images having similar image characteristics among the plurality of disparity images into one group; performing at least one image processing operation on the first disparity image; as well as An output image is generated by combining the first parallax images on which the at least one image processing operation is performed.

18. The method according to claim 17, wherein: The first parallax image includes a first parallax image group and a second parallax image group, and The performing of the at least one image processing operation on the first parallax image comprises: performing image processing on the first parallax image group and the second parallax image group using context switching.

19. The method according to claim 17, wherein: The first parallax image includes a first parallax image group and a second parallax image group, wherein the at least one image processing operation comprises a white balance operation, and The performing of the at least one image processing operation comprises: performing white balancing on the input image by applying the same parameters to the first disparity image group and the second disparity image group in groups.

20. The method according to claim 17, wherein: The similar image characteristics include at least one of noise level, color shift, and color shading.

Images