Image acquisition apparatus and method using lens array

Abstract

An image acquisition apparatus includes: an image sensor including a microlens, and a plurality of pixels arranged adjacent to each other and sharing the microlens; and a processor configured to process an input image acquired by the image sensor, where the processor may further be configured to: generate a plurality of parallax images by extracting and combining sub-images having the same parallax from the input image; generating an RGB demosaiced image by interpolating color values of null pixels of each of the plurality of parallax images using color values of surrounding pixels; generating a plurality of up-sampled images by up-sampling each of the RGB demosaiced images to an input image size; and generating an output image by weighted summation of one reference up-sampled image selected from the up-sampled images and other up-sampled images selected from the up-sampled images, where the one reference up-sampled image and the other up-sampled images are selected based on their similarity.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims the benefit of priority from Korean Patent Application No. 10-2023-0097034 filed on July 25, 2023, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to an image acquisition device and method using a lens array. Background Art

[0004] The image sensor may include red, green and blue filters arranged in a predetermined pattern on the light sensing pixels for imaging of color images. For example, the color filters may be configured as a quad-Bayer pattern with 2×2 pixels of the same color located under one microlens.

[0005] The four-bayer pattern can be used to automatically focus on a photographic target by using the parallax of light passing through the same microlens. The image sensor requires a demosaicing process to include all red, green, and blue information in one pixel. However, in the case of the four-bayer pattern, parallax occurs between pixels of the same color, and therefore, the demosaicing process requires a parallax restoration process.

[0006] A high-definition image acquisition device may have a problem in that as the number of pixels in a color filter array included in a conventional image sensor increases, more logic and memory are required for a parallax restoration process. Summary of the invention

[0007] Provided are an image acquisition apparatus and method capable of minimizing an increase in logic and memory for a parallax restoration process in a demosaic process.

[0008] According to one aspect of the present disclosure, an image acquisition device includes: an image sensor, including a microlens, and a plurality of pixels arranged adjacent to each other and sharing the microlens; and a processor, configured to process an input image acquired by the image sensor, wherein the processor can also be configured to: generate a plurality of disparity images by extracting and combining sub-images with the same disparity from the input image; generate an RGB demosaiced image by interpolating the color values ​​of empty pixels of each of the plurality of disparity images using the color values ​​of surrounding pixels; generate a plurality of upsampled images by upsampling each of the RGB demosaiced images to the size of the input image; and generate an output image by weighted summing a reference upsampled image selected from the upsampled images and the other upsampled images based on the similarity between the one reference upsampled image and the other upsampled images.

[0009] The processor may also be configured to generate the RGB demosaiced image by using one of average value interpolation, linear interpolation, cubic interpolation, and directional bilinear interpolation.

[0010] The image sensor may include: a quad Bayer pattern array in which pixels arranged in a 2×2 matrix include corresponding color filters having the same color; or a quad Bayer pattern array in which pixels arranged in a 4×4 matrix include corresponding color filters having the same color.

[0011] The processor may also be configured to generate a parallax A image, a parallax B image, a parallax C image, and a parallax D image as a plurality of parallax images based on the input image.

[0012] The processor may include: multiple logics configured to simultaneously interpolate each of the parallax A image, the parallax B image, the parallax C image, and the parallax D image; or a single logic configured to sequentially interpolate the parallax A image, the parallax B image, the parallax C image, and the parallax D image.

[0013] The processor may also be configured to generate an upsampled image by using one of bilinear interpolation, bicubic interpolation, and a deep learning network.

[0014] The processor may also be configured to determine similarity between the reference upsampled image and the other upsampled images by using one of a sum of absolute differences (SAD), a sum of squared differences (SSD), a normalized cross correlation (NCC), and a zero-mean normalized cross correlation (ZNCC).

[0015] The processor may also be configured to detect aliased regions of the upsampled image.

[0016] The processor may also be configured to increase the weight in the aliasing region when the similarity between the reference up-sampled image and the other up-sampled images in the aliasing region is lower than a preset value.

[0017] The processor may also be configured to: output an edge result value by using a directional filter; output an edge absolute value by performing absolute value processing of the edge result value; and generate an edge map by using an input value determined based on the edge absolute value and a control signal.

[0018] The directional filters include a horizontal directional filter, a vertical directional filter, a 45 degree directional filter, and a 135 degree directional filter.

[0019] The control signal may include one of the following: a first control signal, used to generate an edge map by using the maximum value of the following as an input value: the absolute edge value of the horizontal direction filter, the absolute edge value of the vertical direction filter, the absolute edge value of the 45 degree direction filter, and the absolute edge value of the 135 degree direction filter; a second control signal, used to generate an edge map by using the average value of the following as an input value: the absolute edge value of the horizontal direction filter, the absolute edge value of the vertical direction filter, the absolute edge value of the 45 degree direction filter, and the absolute edge value of the 135 degree direction filter; and a third control signal, used to generate an edge map by using the maximum value of the following as an input value: the weighted average of the absolute edge value of the horizontal direction filter and the absolute edge value of the vertical direction filter, and the weighted average of the absolute edge value of the 45 degree direction filter and the absolute edge value of the 135 degree direction filter.

[0020] The processor may also be configured to: determine that the input value is not an edge based on the input value being less than a first threshold; determine that the input value is an edge based on the input value being greater than a second threshold; and perform linear interpolation based on determining that the input value is between the first threshold and the second threshold.

[0021] The processor may also be configured to perform white balance processing on the input image before generating the plurality of parallax images.

[0022] The processor may also be configured to perform color shading compensation on the input image prior to the white balance process.

[0023] The processor may also be configured to crop only the target area from the input image between the color shading compensation and the white balance processing.

[0024] According to one aspect of the present disclosure, an image acquisition method used by an image acquisition device including an image sensor, the image sensor including a microlens and a plurality of pixels arranged adjacent to each other and sharing the microlens, the image acquisition method including: generating a plurality of disparity images by extracting and combining sub-images having the same disparity from an input image acquired by the image sensor; demosaicing the image by interpolating the color values ​​of empty pixels in each of the plurality of disparity images using the color values ​​of surrounding pixels; generating a plurality of upsampled images by upsampling each of the interpolated disparity images to the size of the input image; and image synthesis, including generating an output image by weighted summing an upsampled image selected from the upsampled images and the other upsampled images based on the similarity between the one upsampled image and the other upsampled images.

[0025] The image sensor may include: a quad Bayer pattern array in which pixels arranged in a 2×2 matrix include corresponding color filters having the same color; or a quad Bayer pattern array in which pixels arranged in a 4×4 matrix include corresponding color filters having the same color.

[0026] The image synthesis may further include: detecting an aliasing region of the upsampled image.

[0027] In this image synthesis, when the similarity is lower than a preset value in the aliasing region, the weight is increased in the aliasing region. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and / or other aspects will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0029] Figure 1 Schematically shows the configuration of an image acquisition device according to an embodiment;

[0030] Figure 2 is a schematic block diagram of an image sensor according to an embodiment;

[0031] Figure 3A is a plan view corresponding to one area of ​​a four-Bayer pattern;

[0032] Figure 3B is a plan view corresponding to a region of a quadrilateral Bayer pattern;

[0033] Figure 3C is with Figure 3A and Figure 3B A cross-sectional view corresponding to a region of the pixel array shown;

[0034] Figure 3D is a plan view showing a color arrangement of a pixel array of an image sensor according to an embodiment;

[0035] Figure 3E It is used to describe the settings Figure 3D A plan view of a sensor substrate and a metaprism array in a pixel array of an image sensor shown;

[0036] Figure 3F It is used to describe the settings Figure 3D A plan view of a sensor substrate and a metaprism array in a pixel array of an image sensor shown;

[0037] Figure 4A is a diagram showing a method of performing a demosaic process in an image acquisition device including a quad-Bayer pattern;

[0038] Figure 4B It is shown by Figure 4A A diagram of an RGB demosaiced image generated by a demosaicing unit for disparity A is shown;

[0039] Figure 4C It is shown by Figure 4A The diagram shows an upsampled image generated by an upsampling unit for disparity A;

[0040] Figure 5 is a schematic block diagram for describing an edge detection unit according to an embodiment;

[0041] Figure 6 is a graph illustrating a method of generating an edge map;

[0042] Figure 7 is a diagram showing a method of performing a demosaic process in an image acquisition device including a color shading compensation unit;

[0043] Figure 8 is a diagram for describing a method of performing a demosaic process in an image acquisition device including a cropped image generation unit;

[0044] Fig.9A is a diagram showing a method of performing a demosaic process in an image acquisition device including a quad Bayer pattern;

[0045] Fig. 9B It is shown by Fig.9A A diagram of an RGB demosaiced image generated by a demosaicing unit for disparity A is shown;

[0046] Fig. 9C It is shown by Fig.9A The diagram shows an upsampled image generated by an upsampling unit for disparity A;

[0047] Fig.10 is a flow chart showing a demosaicing method of an image acquisition device including a color separation lens array;

[0048] Fig.11 is a block diagram schematically illustrating an electronic device including an image sensor according to an embodiment; and

[0049] Fig.12 It shows the settings in Fig.11 Block diagram of a camera module in an electronic device. DETAILED DESCRIPTION

[0050] Hereinafter, example embodiments will be described in detail with reference to the accompanying drawings. In the following drawings, the same reference numerals refer to the same components, and the size of each component in the drawings may be exaggerated for the sake of clarity and ease of description. The embodiments described below are merely examples, and various modifications may be made based on these embodiments.

[0051] Hereinafter, the term "upper" or "on..." may also include "existing above based on non-contact" and "in direct contact with..." on the top. Unless the context clearly indicates otherwise, a singular expression includes a plural expression. In addition, unless otherwise specifically stated, when a part "includes" a component, it means that the part may also include other components, rather than excluding other components. The use of the term "the" and similar indicative terms may correspond to both the singular and the plural.

[0052] In addition, some embodiments are described in the accompanying drawings for functional blocks, units and / or modules. It will be understood by those skilled in the art that these blocks, units and / or modules are physically implemented by logic circuits, discrete components, microprocessors, hard-wired circuits, storage devices, line connections and other electronic circuits. This can be formed using semiconductor-based manufacturing techniques or other manufacturing techniques. For blocks, units and / or modules implemented by microprocessors or other similar hardware, software can be used to program and control software programs to perform various functions discussed herein, and can be optionally driven by firmware and / or software. In addition, each block, unit and / or module can be implemented by dedicated hardware, or can be implemented as a combination of processors (e.g., one or more programmed microprocessors and associated circuits) that perform functions different from dedicated hardware that performs some functions. In addition, in some embodiments, without departing from the scope of the present invention, blocks, units and / or modules can be physically divided into two or more separate blocks, units and / or modules that interact within the scope. In addition, in some embodiments, without departing from the scope of the present invention, blocks, units and / or modules can be physically more complex blocks, units and / or modules within the scope.

[0053] Figure 1 The configuration of the image acquisition device according to the embodiment is schematically shown.

[0054] Reference Figure 1, the image acquisition device 100 may include: an image sensor 200 that acquires individual color image signals by converting an optical image formed by focusing light reflected from the object OBJ by the photographing lens unit 120 into an electrical signal; and a processor 250 that generates an output signal by processing the individual color image signals acquired by the image sensor 200. The image acquisition device 100 may also include a memory 180, etc., for storing the individual color image signals and the image formed by the processor 250. An additional optical element (e.g., an infrared blocking filter) may be further arranged between the image sensor 200 and the photographing lens unit 120.

[0055] Figure 2 is a schematic block diagram of an image sensor 200 according to an embodiment.

[0056] Reference Figure 2 , the image sensor 200 may include a pixel array 210, a timing controller 240 (also referred to as “T / C”), a row decoder 220, and an output circuit 230. The image sensor 200 may be a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor.

[0057] The pixel array 210 includes pixels arranged in two dimensions along a plurality of rows and columns. The row decoder 220 selects one of the rows of the pixel array 210 in response to a row address signal output from the timing controller 240. The output circuit 230 outputs a light sensing signal in columns from a plurality of pixels arranged along the selected row. To this end, the output circuit 230 may include a column decoder and an analog-to-digital converter (ADC). For example, the output circuit 230 may include a plurality of ADCs placed on each column between the column decoder and the pixel array 210, or an ADC placed on the output end of the column decoder. The timing controller 240, the row decoder 220, and the output circuit 230 may be implemented as one chip or separate chips. Figure 1 The processor 250 for processing an image signal output through the output circuit 230 may be implemented as a single chip together with the timing controller 240 , the row decoder 220 , and the output circuit 230 .

[0058] The pixel array 210 may include a plurality of pixels that sense light of different wavelengths. The arrangement of the pixels may be implemented in various ways. For example, Figure 3A and Figure 3B Various arrangements of pixel array 210 are shown.

[0059] Figure 3A is a plan view corresponding to one area of ​​a four-Bayer pattern. Figure 3B is a plan view corresponding to one area of ​​a quadrilateral Bayer pattern. Figure 3C is with Figure 3A and Figure 3B A cross-sectional view corresponding to a region of the pixel array is shown.

[0060] Reference Figure 3A and Figure 3C , the pixel array PXA may have a quad Bayer pattern structure. Four pixels arranged in 2×2 (two rows and two columns) may include corresponding color filters of the same color. Therefore, the pixel array PXA may include 4×4 pixels consisting of 2×2 green pixels G, 2×2 red pixels R, 2×2 blue pixels B, and 2×2 green pixels G.

[0061] Each pixel included in the pixel array PXA may include a color filter and a light receiving element (e.g., a photodiode). For example, the first pixel PX1 may include a first color filter CF1 and a first light receiving element PD1, and the second pixel PX2 may include a second color filter CF2 and a second light receiving element PD2. The first color filter CF1 and the second color filter CF2 may have the same color (e.g., green).

[0062] The first pixel PX1, the second pixel PX2, the third pixel PX3, and the fourth pixel PX4 are arranged adjacent to each other and may share one microlens ML. The microlens ML may have a convex shape to converge light incident on the unit pixel (e.g., the first pixel PX1, the second pixel PX2, the third pixel PX3, and the fourth pixel PX4). The microlens ML may have a hemispherical cross-section. From a planar perspective, the microlens ML may have a circular shape. When the microlens ML is included, the sensitivity of the image sensor 200 may be improved. That is, the optical characteristics of the image sensor 200 may be improved.

[0063] Reference Figure 3B and Figure 3C , the pixel array PXA' may have a quadrilateral Bayer pattern structure. 16 pixels arranged in 4×4 (four rows and four columns) may include corresponding color filters of the same color. Therefore, the pixel array PXA' may include 8×8 pixels consisting of 4×4 green pixels G, 4×4 red pixels R, 4×4 blue pixels B, and 4×4 green pixels G.

[0064] Each pixel included in the pixel array PXA' may include a color filter and a light receiving element (e.g., a photodiode). For example, the first pixel PX1' may include a first color filter CF1 and a first light receiving element PD1, and the second pixel PX2' may include a second color filter CF2 and a second light receiving element PD2. The first color filter CF1 and the second color filter CF2 may have the same color (e.g., green).

[0065] The first pixel PX1' to the sixteenth pixel PX16' are arranged adjacent to each other, and each unit pixel group in the first pixel PX1' to the fourth pixel PX4', the fifth pixel PX5' to the eighth pixel PX8', the ninth pixel PX9' to the twelfth pixel PX12', and the thirteenth pixel PX13' to the sixteenth pixel PX16' may share one microlens ML. The microlens ML may have a convex shape to converge light incident on the unit pixel (e.g., the first pixel PX1', the second pixel PX2', the third pixel PX3', and the fourth pixel PX4'). The microlens ML may have a hemispherical cross-section. From a plane perspective, the microlens ML may have a circular shape.

[0066] Figure 3D is a plan view showing color arrangement indicated by a pixel array of an image sensor according to an embodiment. Figure 3E and Figure 3F It is used to describe the settings Figure 3D A plan view of a sensor substrate and a metaprism array in a pixel array of an image sensor is shown.

[0067] Figure 3D The color arrangement shown is the arrangement of the Bayer pattern. As shown in the figure, one unit pattern includes four quadrant areas, and the first to fourth quadrants can represent red (R), green (G), blue (B) and green (G), respectively. These unit patterns are arranged two-dimensionally and repeatedly in a first direction (X direction) and a second direction (Y direction). For this color arrangement, two green pixels can be placed along a diagonal direction within a unit pattern in a 2×2 array form, and one blue pixel and one red pixel can be placed along another diagonal direction, respectively. In other words, the first row and the second row can be repeatedly arranged along the second direction, in which a plurality of green pixels and a plurality of blue pixels are alternately arranged along the first direction, and in which a plurality of red pixels and a plurality of green pixels are alternately arranged along the first direction.

[0068] Hereinafter, it has been described that the pixel array PXA″ of the image sensor 200 has a Bayer pattern, but the operation principle may also be applied to pixel arrangements of other patterns other than the Bayer pattern (e.g., a quad Bayer pattern, a quad Bayer pattern, etc.).

[0069] The pixel array PXA″ of the image sensor 200 may be provided with a color separation lens array that condenses light of a color corresponding to the color arrangement (ie, light of a color corresponding to a specific pixel).

[0070] Reference Figure 3E, the sensor substrate SUB may include a plurality of pixels that sense incident light. The sensor substrate SUB may include a plurality of unit pixel groups PXA_G. The unit pixel group PXA_G may include a first subgroup pixel 111, a second subgroup pixel 112, a third subgroup pixel 113, and a fourth subgroup pixel 114 that convert incident light into an electrical signal to generate an image signal. The unit pixel group PXA_G may have a pixel arrangement in the form of a Bayer pattern. That is, the first subgroup pixel 111 and the fourth subgroup pixel 114 may be green pixels that detect green light, the second subgroup pixel 112 may be a red pixel that detects red light, and the third subgroup pixel 113 may be a blue pixel that detects blue light. Each of the first subgroup pixel 111, the second subgroup pixel 112, the third subgroup pixel 113, and the fourth subgroup pixel 114 may include a plurality of light sensing units that independently sense incident light. For example, each of the first subgroup pixel 111, the second subgroup pixel 112, the third subgroup pixel 113, and the fourth subgroup pixel 114 may include a first light sensing unit ce1, a second light sensing unit ce2, a third light sensing unit ce3, and a fourth light sensing unit ce4. The first light sensing unit ce1, the second light sensing unit ce2, the third light sensing unit ce3, and the fourth light sensing unit ce4 may be two-dimensionally arranged in a first direction (X direction) and a second direction (Y direction). For example, in each of the first subgroup pixel 111, the second subgroup pixel 112, the third subgroup pixel 113, and the fourth subgroup pixel 114, the first light sensing unit ce1, the second light sensing unit ce2, the third light sensing unit ce3, and the fourth light sensing unit ce4 may be arranged in a 2×2 array.

[0071] Figure 3E Each of the first subgroup pixel 111, the second subgroup pixel 112, the third subgroup pixel 113, and the fourth subgroup pixel 114 is shown to include four light sensing units, but four or more independent light sensing units may be gathered and arranged in two dimensions. For example, each of the first subgroup pixel 111, the second subgroup pixel 112, the third subgroup pixel 113, and the fourth subgroup pixel 114 may include a 4×4 array (e.g., Figure 3B In the following, for convenience, the case where each of the first sub-group pixel 111, the second sub-group pixel 112, the third sub-group pixel 113, and the fourth sub-group pixel 114 includes light sensing units arranged in the form of a 2×2 array will be described.

[0072] According to an embodiment, a plurality of pixels including a plurality of light sensing units sensing light of the same color may be used as autofocus pixels. In the autofocus pixel, an autofocus signal may be obtained according to the difference between the output signals of adjacent light sensing units. For example, an autofocus signal in a first direction (X direction) may be generated according to the following: the difference between the output signal of the first light sensing unit ce1 and the output signal of the second light sensing unit ce2, the difference between the output signal of the third light sensing unit ce3 and the output signal of the fourth light sensing unit ce4, or the difference between the sum of the output signals of the first light sensing unit ce1 and the third light sensing unit ce3 and the sum of the output signals of the second light sensing unit ce2 and the fourth light sensing unit ce4. For example, an autofocus signal in a second direction (Y direction) may be generated according to the following: the difference between the output signal of the first light sensing unit ce1 and the output signal of the third light sensing unit ce3, the difference between the output signal of the second light sensing unit ce2 and the output signal of the fourth light sensing unit ce4, or the difference between the sum of the output signals of the first light sensing unit ce1 and the second light sensing unit ce2 and the sum of the output signals of the third light sensing unit ce3 and the fourth light sensing unit ce4.

[0073] There are a summing mode and a full mode as methods of obtaining a general image signal. In the summing mode, an image signal may be obtained by summing output signals of the first light sensing unit ce1, the second light sensing unit ce2, the third light sensing unit ce3, and the fourth light sensing unit ce4. For example, the output signals of the first light sensing unit ce1, the second light sensing unit ce2, the third light sensing unit ce3 and the fourth light sensing unit ce4 of the first sub-group pixel 111 are summed to generate a first green image signal, the output signals of the first light sensing unit ce1, the second light sensing unit ce2, the third light sensing unit ce3 and the fourth light sensing unit ce4 of the second sub-group pixel 112 are summed to generate a red image signal, the output signals of the first light sensing unit ce1, the second light sensing unit ce2, the third light sensing unit ce3 and the fourth light sensing unit ce4 of the third sub-group pixel 113 are summed to generate a blue image signal, and the output signals of the first light sensing unit ce1, the second light sensing unit ce2, the third light sensing unit ce3 and the fourth light sensing unit ce4 of the fourth sub-group pixel 114 are summed to generate a second green image signal. In the full mode, each of the first light sensing unit ce1, the second light sensing unit ce2, the third light sensing unit ce3 and the fourth light sensing unit ce4 is used to obtain each output signal as each individual pixel. In this case, a high-resolution image can be obtained.

[0074] The first light sensing unit ce1, the second light sensing unit ce2, the third light sensing unit ce3 and the fourth light sensing unit ce4 can be electrically separated by the isolation structure DTI. The isolation structure can be formed as, for example, a deep trench isolation structure. The deep trench can be filled with air or an electrically insulating material. After forming the light sensing layer, an isolation structure DTI can be formed on the light sensing layer to form a plurality of electrically separated units. The isolation structure electrically separates the first subgroup pixel 111, the second subgroup pixel 112, the third subgroup pixel 113 and the fourth subgroup pixel 114, and also electrically separates each of the first subgroup pixel 111, the second subgroup pixel 112, the third subgroup pixel 113 and the fourth subgroup pixel 114 into four regions. In the isolation structure DTI, specifically, the cross-shaped separation line SL shown in the figure is included in the first sub-group pixel 111, the second sub-group pixel 112, the third sub-group pixel 113 and the fourth sub-group pixel 114, and the cross-shaped separation line SL indicates the DTI center line that separates each of the first sub-group pixel 111, the second sub-group pixel 112, the third sub-group pixel 113 and the fourth sub-group pixel 114 into a plurality of first light sensing units ce1, second light sensing units ce2, third light sensing units ce3 and fourth light sensing units ce4. In the figure, the isolation structure DTI is indicated as a solid line with substantially no thickness, but this is for convenience of explanation, and the isolation structure DTI has a thickness that physically separates adjacent light sensing units. The isolation structure DTI included in the first sub-group pixel 111, the second sub-group pixel 112, the third sub-group pixel 113 and the fourth sub-group pixel 114 is an area where light is not sensed, so a method of reducing performance degradation caused by the isolation structure can be considered in the design of the metaprism array to be described herein.

[0075] Reference Figure 3F The element prism array MP includes a plurality of pixel corresponding areas. The element prism array MP includes Figure 3E The sensor substrate SUB shown in FIG. 1 includes a plurality of pixel corresponding groups MP_G corresponding to a plurality of unit pixel groups PXA_G. The pixel corresponding group MP_G includes first to fourth sub-group pixel corresponding regions 131, 132, 133 and 134 corresponding to first to fourth sub-group pixels 111, 112, 113 and 114, respectively. Each of the first sub-group pixel corresponding region 131, the second sub-group pixel corresponding region 132, the third sub-group pixel corresponding region 133 and the fourth sub-group pixel corresponding region 134 includes a plurality of nano-pillars. The plurality of nano-pillars are configured to separate incident light according to wavelengths and to converge the incident light into the first sub-group pixel 111, the second sub-group pixel 112, the third sub-group pixel 113 and the fourth sub-group pixel 114 corresponding to each wavelength. As Figure 3EAs described in , the first subgroup pixel 111 and the fourth subgroup pixel 114 may be first green pixels and second green pixels, respectively, and the second subgroup pixel 112 may be red pixels, and the third subgroup pixel 113 may be blue pixels.

[0076] The shapes and arrangements of the plurality of nanorods provided in the metaprism array MP may be determined to form a phase distribution suitable for the wavelength separation.

[0077] Figure 4A is a diagram showing a method of performing a demosaic process in an image acquisition device including a quad-Bayer pattern. Figure 4B It is shown by Figure 4A A diagram of an RGB demosaiced image generated by the demosaicing unit for disparity A is shown. Figure 4C It is shown by Figure 4A The figure shows an upsampled image generated by the upsampling unit for disparity A. Figure 5 is a schematic block diagram for describing an edge detection unit according to an embodiment. Figure 6 is a graph showing a method of generating an edge map.

[0078] Reference Figure 1 , Figure 2 , Figure 3A , Figure 3B , Figure 3C , Figure 3D , Figure 3E , Figure 3F , Figure 4A , Figure 4B and Figure 4C , the processor 250 according to an embodiment may include a parallax image generating unit 251 , a demosaicing unit 252 , an up-sampling unit 253 , and an image synthesizing unit 254 .

[0079] The processor 250 may receive an input image IMG_IN from the image sensor 200. The pixel array PXA of the image sensor 200 according to an embodiment may have a quad Bayer pattern structure, such as Figure 3AAs shown. Therefore, the input image IMG_IN may include 4×4 sub-images, which are composed of the following: 2×2 green (G) sub-image PA, sub-image PB, sub-image PC and sub-image PD, 2×2 red (R) sub-image PA, sub-image PB, sub-image PC and sub-image PD, 2×2 blue (B) sub-image PA, sub-image PB, sub-image PC and sub-image PD, and 2×2 green (G) sub-image PA, sub-image PB, sub-image PC and sub-image PD. In this case, since the sub-images PA, sub-image PB, sub-image PC and sub-image PD of the same color share one microlens ML (or metaprism array MP), they may include four parallaxes. That is, the sub-image PA, sub-image PB, sub-image PC and sub-image PD may have parallax A, parallax B, parallax C and parallax D, respectively.

[0080] The disparity image generating unit 251 extracts and combines sub-images PA, PB, PC, and PD having the same disparity from the input image IMG_IN, thereby generating a plurality of disparity images including disparity images IMG_PA, IMG_PB, IMG_PC, and IMG_PD.

[0081] A disparity A image IMG_PA is generated by extracting and combining sub-images PA with different colors in the input image IMG_IN, a disparity B image IMG_PB is generated by extracting and combining sub-images PB with different colors in the input image IMG_IN, a disparity C image IMG_PC is generated by extracting and combining sub-images PC with different colors in the input image IMG_IN, and a disparity D image IMG_PD is generated by extracting and combining sub-images PD with different colors in the input image IMG_IN.

[0082] The disparity A image IMG_PA, the disparity B image IMG_PB, the disparity C image IMG_PC, and the disparity D image IMG_PD may have a Bayer pattern structure in which unit pixels are repeated in two dimensions along a first direction and a second direction in four quadrant regions, and the unit pixels are respectively composed of a blue pixel, a green pixel, a red pixel, and a green pixel. Therefore, the demosaicing unit 252 to be described later can perform demosaicing on the disparity A image IMG_PA, the disparity B image IMG_PB, the disparity C image IMG_PC, and the disparity D image IMG_PD by using a conventional demosaicing method designed for the Bayer pattern structure. In other words, a high-quality output image IMG_OUT can be generated while minimizing the logic for correcting different disparities of the four Bayer patterns including the microlens ML (or the metaprism array MP), thereby expecting a cost saving effect.

[0083] Each of the disparity A image IMG_PA, the disparity B image IMG_PB, the disparity C image IMG_PC, and the disparity D image IMG_PD is an image corresponding to only one of the four disparities. Therefore, the size of each of the disparity A image IMG_PA, the disparity B image IMG_PB, the disparity C image IMG_PC, and the disparity D image IMG_PD may correspond to a quarter size compared to the size of the input image IMG_IN.

[0084] The disparity A image IMG_PA, the disparity B image IMG_PB, the disparity C image IMG_PC, and the disparity D image IMG_PD may be stored in a separate memory for quick access. However, the embodiment is not limited thereto, and for example, the input image IMG_IN may be separated per disparity and directly input to the demosaicing unit 252 without having a process of storing the input image in a separate memory.

[0085] The demosaic unit 252 may generate an RGB demosaic image including a demosaic image IMG_DMR, a demosaic image IMG_DMG, and a demosaic image IMG_DMB by interpolating color values ​​of empty pixels of each of the disparity A image IMG_PA, the disparity B image IMG_PB, the disparity C image IMG_PC, and the disparity D image IMG_PD using color values ​​of adjacent pixels.

[0086] The demosaicing unit 252 may interpolate each of the disparity A image IMG_PA, the disparity B image IMG_PB, the disparity C image IMG_PC, and the disparity D image IMG_PD output in the Bayer pattern. Figure 4B Only the disparity A RGB demosaic image IMG_DMPA including the RGB demosaic images (e.g., the demosaic image IMG_DMR, the demosaic image IMG_DMG, and the demosaic image IMG_DMB) is shown, but the disparity B RGB demosaic image IMG_DMPB, the disparity C RGB demosaic image IMG_DMPC, and the disparity D RGB demosaic image IMG_DMPD may also be generated in the same manner. In other words, each of the disparity B RGB demosaic image IMG_DMPB, the disparity C RGB demosaic image IMG_DMPC, and the disparity D RGB demosaic image IMG_DMPD may include the RGB demosaic image (e.g., the demosaic image IMG_DMR, the demosaic image IMG_DMG, and the demosaic image IMG_DMB).

[0087] The interpolation method may adopt a conventional demosaicing method designed for the Bayer pattern structure, for example, average value interpolation, linear interpolation, cubic interpolation, directional bilinear interpolation, etc., but is not limited thereto.

[0088] In the Bayer pattern, half of the total number of pixels are green, and the remaining pixels are equally assigned to red or blue. Each pixel in this pattern is sensitive to only one color (i.e., R, G, or B). Therefore, the color images obtained using these patterns are interpolated in three color channels to generate a full-color image, which is called demosaicing.

[0089] Demosaicing or color filter array (CFA) interpolation is the process of reconstructing the unknown R, G, and B components to generate a full-color image. The two unknown colors for each pixel are estimated by various demosaicing algorithms. The demosaicing algorithm reconstructs the full-color image from incomplete data (i.e., two missing data in each pixel). The reconstruction algorithm uses the inter-channel and intra-channel correlation of the data available in the RGB channels.

[0090] According to the embodiment, Figure 4A As shown, the demosaic unit 252 is composed of a plurality of logics arranged in parallel, and can simultaneously interpolate the parallax A image IMG_PA, the parallax B image IMG_PB, the parallax C image IMG_PC, and the parallax D image IMG_PD in parallel. In this case, fast demosaic processing can be expected.

[0091] However, the arrangement of the logic constituting the demosaicing unit 252 is not limited thereto. For example, the demosaicing unit 252 may be configured as a single logic, and the parallax A image IMG_PA, the parallax B image IMG_PB, the parallax C image IMG_PC, and the parallax D image IMG_PD may be sequentially interpolated. In this case, the hardware area occupied by the logic is reduced, and product miniaturization and cost reduction effects can be expected.

[0092] The up-sampling unit 253 up-samples the disparity A RGB demosaiced image IMG_DMPA, the disparity B RGB demosaiced image IMG_DMPB, the disparity C RGB demosaiced image IMG_DMPC, and the disparity D RGB demosaiced image IMG_DMPD to the size of the input image IMG_IN, thereby generating up-sampled images IMG_USPA, IMG_USPB, IMG_USPC, and IMG_USPD for corresponding disparities among disparity A, disparity B, disparity C, and disparity D. For example, the up-sampling method may adopt bilinear interpolation, bicubic interpolation, a deep learning network, etc., but is not limited to these methods.

[0093] The reason why upsampling is needed is that the sizes of the RGB demosaiced images IMG_DMPA, IMG_DMPB, IMG_DMPC and IMG_DMPD for the corresponding disparities A, B, C and D are 1 / 4 of the size of the full mode input image IMG_IN.

[0094] The image synthesis unit 254 may generate the output image IMG_OUT by weighted summing the reference upsampled image and the remaining upsampled image based on the similarity between the reference upsampled image (e.g., IMG_USPA selected from the upsampled image IMG_USPB, the upsampled image IMG_USPC, and the upsampled image IMG_USPD for the corresponding disparities A, B, C, and D) and the remaining upsampled images (e.g., the upsampled image IMG_USPB, the upsampled image IMG_USPC, and the upsampled image IMG_USPD). The image synthesis unit 254 may determine the similarity between the reference upsampled image (e.g., the upsampled image IMG_USPA) and the remaining images (e.g., the upsampled image IMG_USPB, the upsampled image IMG_USPC, and the upsampled image IMG_USPD) by using one of the sum of absolute differences (SAD), the sum of squared differences (SSD), the normalized cross correlation (NCC), and the zero-mean normalized cross correlation (ZNCC).

[0095] The image synthesis unit 254 can generate an image by synthesizing the upsampled image IMG_USPA, the upsampled image IMG_USPB, the upsampled image IMG_USPC, and the upsampled image IMG_USPD for disparity A, disparity B, disparity C, and disparity D, respectively. In conventional image synthesis processing, the remaining images are usually synthesized after local / global alignment of specific images. According to the inventive concept of the embodiments herein, the similarity between the reference upsampled image (e.g., the upsampled image IMG_USPA) and the remaining images (e.g., the upsampled image IMG_USPB, the upsampled image IMG_USPC, and the upsampled image IMG_USPD) can be determined without performing such disparity correction. Therefore, when the similarity is low, a method of synthesizing images with a low weight value is used, and when the similarity is high, a method of synthesizing images with a high weight value is used. To this end, a high-quality image can be obtained by removing the mismatching area caused by different disparities while maintaining the resolution of the focus position without considering complex disparity.

[0096] The image synthesis unit 254 may further include an edge detection unit ED that detects aliasing regions of the upsampled images IMG_USPA, IMG_USPB, IMG_USPC, and IMG_USPD for corresponding disparities of disparity A, disparity B, disparity C, and disparity D.

[0097] Reference Figure 5 , the edge detection unit ED according to the embodiment may include a plurality of directional filters DF, an absolute value processing unit ABS and an edge map generating unit EDM.

[0098] The direction filter DF may detect an edge (or edge result value EV) by applying, according to each direction, a high-pass filter (e.g., HPF of 1, -2, and 1) to the upsampled image IMG_USPA, the upsampled image IMG_USPB, the upsampled image IMG_USPC, and the upsampled image IMG_USPD for the corresponding disparity among disparity A, disparity B, disparity C, and disparity D. The direction filter DF includes a horizontal direction filter, a vertical direction filter, a 45 degree direction filter, and a 135 degree direction filter.

[0099] The edge result value EV is calculated by removing the baseband from the signal, and thus may have positive values ​​as well as negative values. Therefore, the absolute value processing unit ABS needs to perform absolute value processing on the edge result value EV to calculate the edge absolute value EV_ABS.

[0100] The edge map generation unit EDM may generate an edge map using an input value determined based on the edge absolute value EV_ABS and the control signal CS.

[0101] In this case, the control signal may include: a first control signal, used to generate an edge map by using the maximum value of the following items as an input value: the absolute edge value EV_ABS of the horizontal direction filter, the absolute edge value EV_ABS of the vertical direction filter, the absolute edge value EV_ABS of the 45 degree direction filter, and the absolute edge value EV_ABS of the 135 degree direction filter; a second control signal, used to generate an edge map by using the average value of the following items as an input value: the absolute edge value EV_ABS of the horizontal direction filter, the absolute edge value EV_ABS of the vertical direction filter, the absolute edge value EV_ABS of the 45 degree direction filter, and the absolute edge value EV_ABS of the 135 degree direction filter; and a third control signal, used to generate an edge map by using the maximum value of the following items as an input value: the weighted average value of the absolute edge value EV_ABS of the horizontal direction filter and the absolute edge value EV_ABS of the vertical direction filter, and the weighted average value of the absolute edge value EV_ABS of the 45 degree direction filter and the absolute edge value EV_ABS of the 135 degree direction filter.

[0102] Reference Figure 6 When the input value is less than the first threshold value TH1, the edge map generation unit EDM can determine that the input value is not an edge (i.e., the α (alpha) value is 0); when the input value is greater than the second threshold value TH2, the edge map generation unit EDM can determine that the input value is an edge (i.e., the α value is 255); and when the input value is between the first threshold value TH1 and the second threshold value TH2, linear interpolation can be performed to prevent sudden changes in the α value.

[0103] Even if the similarity between the reference upsampled image (e.g., upsampled image IMG_USPA) and the remaining upsampled images (e.g., upsampled image IMG_USPB, upsampled image IMG_USPC, upsampled image IMG_USPD) is low, the image synthesis unit 254 may assign a high weight value in the aliasing region detected by the edge detection unit ED. Generally, since a high-frequency region where aliasing occurs may be a region determined to be an edge (i.e., an α value of 255), assigning a high weight value may be effective in generating a high-definition output image IMG_OUT even if the similarity between the images is determined to be low for the region.

[0104] Return to reference Figure 4A The processor 250 may further include a white balance unit 255 that processes white balance of the input image IMG_IN at a front end of the disparity image generating unit 251 .

[0105] White balance is to adjust the gain of R and B around G so that the ratio of RGB (red, green, blue) in the gray area is the same under various color temperature lighting (e.g., R=G=B). Conceptually, white balance aims to normalize the lighting effects of the captured scene so that all objects appear to be captured under ideal "white light".

[0106] In this case, the white balance unit 255 may apply the same parameters to all four parallax images (parallax A image IMG_PA, parallax B image IMG_PB, parallax C image IMG_PC, and parallax D image IMG_PD) during the white balance process. However, when the image sensor 200 includes a metaprism array MP, different parameters may be applied to the four parallax images (parallax A image IMG_PA, parallax B image IMG_PB, parallax C image IMG_PC, and parallax D image IMG_PD) to improve the sensitivity of the image sensor. For example, gains may be applied differently to the four parallax images.

[0107] Hereinafter, other embodiments will be described. In the following embodiments, the same components as those of the above-described embodiments will be omitted or simplified, and differences will be mainly described.

[0108] Figure 7 is a diagram showing a method of performing a demosaic process in an image acquisition device including a color shading compensation unit.

[0109] Reference Figure 1 , Figure 2 , Figure 3A , Figure 3B , Figure 3C , Figure 3D , Figure 3E , Figure 3F , Figure 4A , Figure 4B , Figure 4C , Figure 5 , Figure 6 and Figure 7 , Figure 7 The processor 250 is shown with Figure 4A The illustrated processor 250 is different in that the former further includes a color shading compensation unit 256 that compensates for color shading of the input image IMG_IN at the front end of the white balance unit 255, and the remaining components between the former and the latter are substantially the same.

[0110] In the case of an image sensor in which there are multiple light receiving elements under the microlens ML (or metaprism array MP), color shading correction can be performed first before image processing (e.g., demosaicing) is performed on the four parallax images (parallax A image IMG_PA, parallax B image IMG_PB, parallax C image IMG_PC, and parallax D image IMG_PD).

[0111] Color shading refers to the color deviation between pixels of different channels among adjacent pixels due to different channel characteristics of color filters. Between four pixels of the same color arranged in 2×2 (2 rows and 2 columns) (e.g., a quad Bayer pattern), color variations may exist in the same channel for various reasons.

[0112] Specifically, when the microlens ML (or metaprism array MP) is included, the color deviation may be large within the same color of 2×2, so it may be desirable to process the color shading correction first and perform the demosaicing process later.

[0113] The color shading compensation unit 256 can correct the color deviation between channels by applying a correction gain of different values ​​for each color channel. As an example, a correction gain function with different coefficients for each color channel is pre-set, and the color shading compensation unit 256 can correct the color deviation by applying the correction gain function for each channel to the pixels of each channel. In this case, the correction gain function for each channel can be a function of at least two dimensions or more. In addition to using the correction gain function as described above, various correction methods can be applied, for example, using a lookup table storing the correction gain value for each channel.

[0114] Figure 8 is a diagram for describing a method of performing a demosaic process in an image acquisition device including a cropped image generation unit.

[0115] Reference Figure 1 , Figure 7 and Figure 8 , Figure 8 The processor 250 is shown with Figure 7 The illustrated processor 250 is different in that the former further includes a cropped image generation unit 257 between the color shading compensation unit 256 and the white balance unit 255, and the remaining components between the former and the latter are substantially the same.

[0116] When digital zoom is used in the image acquisition device 100, since the output format size of the image is the same, it is not necessary to process the entire enlarged large image, so only the target area of ​​the user's attention can be cropped to perform image processing (e.g., demosaicing). According to an embodiment, the target area can be determined by the user's input.

[0117] In this way, the effect of reducing power consumption can be expected. In this case, since sensor correction (eg, color shading correction) has difficulty adjusting the address of the lookup table referred to when performing the cropping process, it may be desirable to perform the cropping process after the color shading correction.

[0118] Fig.9A is a diagram showing a method of performing a demosaic process in an image acquisition device including a quad Bayer pattern. Fig. 9B It is shown by Fig.9A A diagram of an RGB demosaiced image generated by the demosaicing unit for disparity A is shown. Fig. 9C It is shown by Fig.9A The figure shows an upsampled image generated by the upsampling unit for disparity A.

[0119] Reference Figure 1 , Figure 2 , Figure 3A , Figure 3B , Figure 3C , Figure 3D , Figure 3E , Figure 3F , Figure 4A , Figure 4B , Figure 4C , Figure 5 , Figure 6 , Figure 7 , Figure 8 , Fig.9A , Fig. 9B and Fig. 9C , Fig.9A The embodiment shown is Figure 4A The embodiment shown is different in that the input image IMG_IN of the former has a quad Bayer pattern structure, while the input image IMG_IN of the latter has a quad Bayer pattern structure, and the remaining components between the former and the latter are substantially the same. Therefore, although not explicitly described, Figure 7 and Figure 8 The embodiments can also be applied to Fig.9A Embodiment of the invention.

[0120] The processor 250 according to an embodiment may include a parallax image generating unit 251 , a demosaicing unit 252 , an up-sampling unit 253 , and an image synthesizing unit 254 .

[0121] The processor 250 may receive an input image IMG_IN from the image sensor 200. The pixel array PXA′ of the image sensor 200 according to an embodiment may have a quad Bayer pattern structure, such as Figure 3BAs shown. Therefore, the input image IMG_IN may include 8×8 sub-images, which are composed of: 4×4 green (G) sub-images ((sub-image PA, sub-image PB, sub-image PC and sub-image PD)×4), 4×4 red (R) sub-images ((sub-image PA, sub-image PB, sub-image PC and sub-image PD)×4), 4×4 blue (B) sub-images ((sub-image PA, sub-image PB, sub-image PC and sub-image PD)×4) and 4×4 green (G) sub-images ((sub-image PA, sub-image PB, sub-image PC and sub-image PD)×4). In this case, since the sub-images PA, sub-image PB, sub-image PC and sub-image PD of the same color share one microlens ML (or meta-prism array MP), they may include four parallaxes. That is, the sub-images PA, sub-image PB, sub-image PC and sub-image PD may have parallax A, parallax B, parallax C and parallax D, respectively.

[0122] The disparity image generating unit 251 extracts and combines sub-images PA, PB, PC, and PD having the same disparity from the input image IMG_IN, thereby generating a plurality of disparity images (eg, disparity A image IMG_PA, disparity B image IMG_PB, disparity C image IMG_PC, and disparity D image IMG_PD).

[0123] A disparity A image IMG_PA is generated by extracting and combining sub-images PA with different colors in the input image IMG_IN, a disparity B image IMG_PB is generated by extracting and combining sub-images PB with different colors in the input image IMG_IN, a disparity C image IMG_PC is generated by extracting and combining sub-images PC with different colors in the input image IMG_IN, and a disparity D image IMG_PD is generated by extracting and combining sub-images PD with different colors in the input image IMG_IN.

[0124] Each of the parallax A image IMG_PA, the parallax B image IMG_PB, the parallax C image IMG_PC, and the parallax D image IMG_PD has a quad-Bayer pattern structure.

[0125] The size of each of the disparity A image IMG_PA, the disparity B image IMG_PB, the disparity C image IMG_PC, and the disparity D image IMG_PD may correspond to a quarter size compared with that of the input image IMG_IN.

[0126] The disparity image generating unit 251 may provide the disparity A image IMG_PA, the disparity B image IMG_PB, the disparity C image IMG_PC, and the disparity D image IMG_PD, each having a Bayer pattern structure instead of a quad-Bayer pattern structure, to the demosaicing unit 252. In this case, the disparity image generating unit 251 may divide the disparity A image IMG_PA, the disparity B image IMG_PB, the disparity C image IMG_PC, and the disparity D image IMG_PD, each having a quad-Bayer pattern structure, into four Bayer patterns, respectively, and provide a total of 16 Bayer patterns to the demosaicing unit 252. Therefore, 16 resizings may be required in the subsequent upsampling unit 253 instead of 4 resizings.

[0127] The demosaicing unit 252 may generate RGB demosaiced images (eg, demosaiced images IMG_DMR, IMG_DMG, and IMG_DMB) by interpolating color values ​​of empty pixels of each of the disparity A image IMG_PA, the disparity B image IMG_PB, the disparity C image IMG_PC, and the disparity D image IMG_PD using color values ​​of adjacent pixels.

[0128] The demosaicing unit 252 may interpolate each of the disparity A image IMG_PA, the disparity B image IMG_PB, the disparity C image IMG_PC, and the disparity D image IMG_PD output in a quad Bayer pattern. Fig. 9B Only the disparity A RGB demosaic image IMG_DMPA including the RGB demosaic image (e.g., the demosaic image IMG_DMR, the demosaic image IMG_DMG, and the demosaic image IMG_DMB) is shown, but the disparity B RGB demosaic image IMG_DMPB, the disparity C RGB demosaic image IMG_DMPC, and the disparity D RGB demosaic image IMG_DMPD may also be generated in the same manner. In other words, each of the disparity B RGB demosaic image IMG_DMPB, the disparity C RGB demosaic image IMG_DMPC, and the disparity D RGB demosaic image IMG_DMPD may include the RGB demosaic image (e.g., the demosaic image IMG_DMR, the demosaic image IMG_DMG, and the demosaic image IMG_DMB).

[0129] The upsampling unit 253 upsamples the disparity A RGB demosaiced image IMG_DMPA, the disparity B RGB demosaiced image IMG_DMPB, the disparity C RGB demosaiced image IMG_DMPC and the disparity D RGB demosaiced image IMG_DMPD to the size of the input image IMG_IN, respectively, thereby generating upsampled images for corresponding disparities among disparity A, disparity B, disparity C and disparity D (for example, upsampled image IMG_USPA, upsampled image IMG_USPB, upsampled image IMG_USPC and upsampled image IMG_USPD). The reason why upsampling is needed is that the size of the RGB demosaiced images (e.g., RGB demosaiced image IMG_DMPA, RGB demosaiced image IMG_DMPB, RGB demosaiced image IMG_DMPC, and RGB demosaiced image IMG_DMPD) for the corresponding disparities (disparity A, disparity B, disparity C, and disparity D) is 1 / 4 of the size of the full mode input image IMG_IN.

[0130] The image synthesis unit 254 may generate the output image IMG_OUT by weighted summing the reference upsampled image and the remaining upsampled images based on the similarity between the reference upsampled image (e.g., IMG_USPA selected from the upsampled image IMG_USPB, the upsampled image IMG_USPC, and the upsampled image IMG_USPD for the corresponding disparity among disparity A, disparity B, disparity C, and disparity D) and the remaining upsampled images (e.g., the upsampled image IMG_USPB, the upsampled image IMG_USPC, and the upsampled image IMG_USPD). The image synthesis unit 254 may generate one image by synthesizing the upsampled images (e.g., the upsampled image IMG_USPA, the upsampled image IMG_USPB, the upsampled image IMG_USPC, and the upsampled image IMG_USPD for the disparity A, disparity B, disparity C, and disparity D, respectively). According to the inventive concept, the similarity between the reference upsampled image (e.g., upsampled image IMG_USPA) and the remaining images (e.g., upsampled image IMG_USPB, upsampled image IMG_USPC, and upsampled image IMG_USPD) can be determined without performing such parallax correction. Therefore, when the similarity is low, a method of synthesizing images with a low weight value is used, and when the similarity is high, a method of synthesizing images with a high weight value is used. To this end, a high-quality image can be obtained by removing mismatched areas caused by different parallaxes while maintaining the resolution of the focus position without considering complex parallax.

[0131] The image synthesis unit 254 may further include an edge detection unit ED that detects aliasing regions of the upsampled images IMG_USPA, IMG_USPB, IMG_USPC, and IMG_USPD for corresponding disparities of disparity A, disparity B, disparity C, and disparity D.

[0132] Even if the similarity between the reference upsampled image (e.g., upsampled image IMG_USPA) and the remaining upsampled images (e.g., upsampled image IMG_USPB, upsampled image IMG_USPC, and upsampled image IMG_USPD) is low, the image synthesis unit 254 may assign a high weight value in the aliasing region detected by the edge detection unit ED. Generally, since a high-frequency region where aliasing occurs may be a region determined to be an edge (i.e., an α value of 255), assigning a high weight value may be effective in generating a high-definition output image IMG_OUT even if the similarity between the images is determined to be low for the region.

[0133] Return to reference Fig.9A The processor 250 may further include a white balance unit 255 that processes white balance of the input image IMG_IN at a front end of the disparity image generating unit 251 .

[0134] Fig.10 is a flowchart showing a demosaicing method of an image acquisition device to which a color separation lens array is applied. In this case, Figure 1 , Figure 2 , Figure 3A , Figure 3B , Figure 3C , Figure 3D , Figure 3E , Figure 3F , Figure 4A , Figure 4B , Figure 4C , Figure 5 , Figure 6 , Figure 7 , Figure 8 , Fig.9A , Fig. 9B and Fig. 9C Some of the configurations described in Fig.10 This is only to avoid redundant description and can be naturally applied to Fig.10 .

[0135] Reference Figure 1 , Figure 2 , Figure 3A , Figure 3B , Figure 3C , Figure 3D , Figure 3E , Figure 3F , Figure 4A , Figure 4B , Figure 4C , Figure 5 , Figure 6 , Figure 7 , Figure 8 , Fig.9A , Fig. 9B , Fig. 9C and Fig.10 The image acquisition method according to the embodiment may include: an operation S10 of generating a plurality of disparity images (e.g., a disparity A image IMG_PA, a disparity B image IMG_PB, a disparity C image IMG_PC, and a disparity D image IMG_PD) by extracting and combining sub-images having the same disparity from an input image IMG_IN acquired by the image sensor 200; a demosaicing operation S20 of interpolating color values ​​of empty pixels of corresponding disparity images among the disparity A image IMG_PA, the disparity B image IMG_PB, the disparity C image IMG_PC, and the disparity D image IMG_PD using color values ​​of adjacent pixels; an upsampling operation S30 of upsampling corresponding interpolated disparity images (e.g., an RGB demosaiced image IMG_DMPA, an RGB demosaiced image IMG_DM PB, RGB demosaic image IMG_DMPC and RGB demosaic image IMG_DMPD) are upsampled to the size of the input image IMG_IN to generate multiple upsampled images (for example, upsampled image IMG_USPA, upsampled image IMG_USPB, upsampled image IMG_USPC and upsampled image IMG_USPD); and an image synthesis operation S40 is performed to generate an output image IMG_OUT by: based on the similarity between an upsampled image selected from the upsampled images (for example, upsampled image IMG_USPA, upsampled image IMG_USPB, upsampled image IMG_USPC and upsampled image IMG_USPD) and the remaining upsampled images, weighted summing the upsampled image and the remaining upsampled images.

[0136] In this case, the image sensor 200 includes: a quad Bayer pattern array in which pixels arranged in a 2×2 matrix include corresponding color filters having the same color; or a quad Bayer pattern array in which pixels arranged in a 4×4 matrix include corresponding color filters having the same color.

[0137] The image synthesis operation S40 may further include an edge detection operation of detecting aliasing regions of the up-sampled image IMG_USPA, the up-sampled image IMG_USPB, the up-sampled image IMG_USPC, and the up-sampled image IMG_USPD.

[0138] In the image synthesis operation, a high weight may be assigned even if the similarity between the reference upsampled image and the remaining upsampled image is low in the aliased region.

[0139] Fig.11 Schematically shows a block diagram of an electronic device including an image sensor according to an embodiment. Fig.11 In the network environment ED00, the electronic device ED01 can communicate with another electronic device ED02 through a first network ED98 (short-distance wireless communication network, etc.), or communicate with another electronic device ED04 and / or a server ED08 through a second network ED99 (long-distance wireless communication network, etc.). The electronic device ED01 can communicate with the electronic device ED04 through the server ED08. The electronic device ED01 may include a processor ED20, a memory ED30, an input device ED50, a sound output device ED55, a display device ED60, an audio module ED70, a sensor module ED76, an interface ED77, a connection terminal ED78, a tactile module ED79, a camera module ED80, a power management module ED88, a battery ED89, a communication module ED90, a subscriber identification module ED96, and / or an antenna module ED97. Some of these components (e.g., the display device ED60) may be omitted from the electronic device ED01, or other components may be added to the electronic device ED01. Some of these components may be implemented as one integrated circuit. For example, the sensor module ED76 (fingerprint sensor, iris sensor, illumination sensor, etc.) can be implemented by being embedded in the display device ED60 (display, etc.).

[0140] The processor ED20 can execute software (program ED40, etc.) to control one or more other components (hardware and software components, etc.) of the electronic device ED01 connected to the processor ED20, and can perform various data processing or operations. As part of the data processing or operation, the processor ED20 can load commands and / or data received from other components (sensor module ED76, communication module (ED90, etc.)), process the commands and / or data stored in the volatile memory ED32, and store the resulting data in the non-volatile memory ED34. The processor ED20 may include a main processor ED21 (central processing unit, application processor, etc.) and an auxiliary processor ED23 (graphics processing unit, image signal processor, sensor hub processor, communication processor, etc.) that can operate independently of the main processor ED21 or together with the main processor ED21. The auxiliary processor ED23 can use less power than the main processor ED21 and perform specialized functions.

[0141] The auxiliary processor ED23 can replace the main processor ED21 when the main processor ED21 is in an inactive state (sleep state), or control functions and / or states related to some of the components of the electronic device ED01 (display device ED60, sensor module ED76, communication module ED90, etc.) together with the main processor ED21 when the main processor ED21 is in an active state (application execution state). The auxiliary processor ED23 (image signal processor, communication processor, etc.) can be implemented as part of other functionally related components (camera module ED80, communication module ED90, etc.).

[0142] The memory ED30 can store various data required by the components (processor ED20 and sensor module ED76) of the electronic device ED01. The data may include, for example, input data and / or output data for software (program ED40, etc.) and related commands. The memory ED30 may include a volatile memory ED32 and / or a non-volatile memory ED34. The non-volatile memory ED34 may include an internal memory ED36 and an external memory ED38.

[0143] The program ED40 may be stored as software in the memory ED30 , and may include an operating system ED42 , middleware ED44 , and / or applications ED46 .

[0144] The input device ED50 may receive commands and / or data from outside the electronic device ED01 (user, etc.) to be used in components (processor ED20, etc.) of the electronic device ED01. The input device ED50 may include a microphone, a mouse, a keyboard, and / or a digital pen (e.g., a stylus, etc.).

[0145] The sound output device ED55 can output sound signals to the outside of the electronic device ED01. The sound output device ED55 may include a speaker and / or an earpiece. The speaker may be used for general purposes such as multimedia playback or recording playback, and the earpiece may be used to receive incoming calls. The earpiece may be coupled as part of the speaker, or may be implemented as an independent separate device.

[0146] The display device ED60 can visually provide information to the outside of the electronic device ED01. The display device ED60 may include a display, a holographic device or a projector, and a control circuit for controlling the corresponding device. The display device ED60 may include a touch circuit configured to sense a touch and / or a sensor circuit (pressure sensor, etc.) configured to measure the strength of the force generated by the touch.

[0147] The audio module ED70 can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. The audio module ED70 can obtain sound through the input device ED50, or output sound through the sound output device ED55 and / or a speaker and / or earphone of another electronic device (e.g., electronic device ED02, etc.) directly or wirelessly connected to the electronic device ED01.

[0148] The sensor module ED76 can detect the operating state (power, temperature, etc.) or external environment state (user state, etc.) of the electronic device ED01, and generate an electrical signal and / or data value corresponding to the sensed state. The sensor module ED76 can include a gesture sensor, a gyroscope sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, and / or an illumination sensor.

[0149] Interface ED77 may support one or more specified protocols, which may be used for electronic device ED01 to directly or wirelessly connect to another electronic device (e.g., electronic device ED02, etc.). Interface ED77 may include a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, and / or an audio interface.

[0150] The connection terminal ED78 may include a connector through which the electronic device ED01 may be physically connected to another electronic device (e.g., the electronic device ED02, etc.). The connection terminal ED78 may include an HDMI connector, a USB connector, an SD card connector, and / or an audio connector (e.g., a headphone connector, etc.).

[0151] The haptic module ED79 may convert the electrical signal into a mechanical stimulus (vibration, motion, etc.) or an electrical stimulus that the user may recognize through a sense of touch or motion. The haptic module ED79 may include a motor, a piezoelectric element, and / or an electrical stimulus.

[0152] The camera module ED80 can capture still images and moving images. The camera module ED80 may include a lens assembly having one or more lenses, an image sensor, an image signal processor, and / or a flash. The lens assembly included in the camera module ED80 may gather light emitted from an object to be photographed.

[0153] The power management module ED88 may manage power supplied to the electronic device ED01. The power management module ED88 may be implemented as a part of a power management integrated circuit (PMIC).

[0154] The battery ED89 can supply power to the components of the electronic device ED01. The battery ED89 can include a non-rechargeable primary battery, a rechargeable secondary battery, and / or a fuel cell.

[0155] The communication module ED90 can establish a direct (wired) communication channel and / or a wireless communication channel between the electronic device ED01 and another electronic device (electronic device ED02, electronic device ED04, server ED08, etc.), and support communication through the established communication channel. The communication module ED90 may include one or more communication processors, which operate independently of the processor ED20 (application processor, etc.) and support direct communication and / or wireless communication. The communication module ED90 may include a wireless communication module ED92 (cellular communication module, short-range wireless communication module, GNSS (Global Navigation Satellite System) communication module, etc.) and / or a wired communication module ED94 (local area network (LAN) communication module, power line communication module, etc.). The corresponding communication modules in these communication modules can communicate with other electronic devices through a first network ED98 (short-range communication network, such as Bluetooth, WiFi Direct or Infrared Data Association (IrDA)) or a second network ED99 (long-range communication network, such as a cellular network, the Internet or a computer network (LAN, WAN, etc.)). These various types of communication modules may be integrated into a single component (e.g., a single chip, etc.), or may be implemented as multiple separate components (multiple chips). The wireless communication module ED92 may identify and authenticate the electronic device ED01 in a communication network (e.g., a first network ED98 and / or a second network ED99) using subscriber information (e.g., an International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module ED96.

[0156] The antenna module ED97 can send signals and / or power to the outside (for example, another electronic device, etc.) or receive signals and / or power from the outside. The antenna may include a radiator formed by a conductive pattern formed on a substrate (PCB, etc.). The antenna module ED97 may include one or more antennas. When multiple antennas are included, an antenna suitable for a communication scheme used in a communication network (for example, a first network ED98 and / or a second network ED99) can be selected from the multiple antennas through the communication module ED90. Signals and / or power can be sent or received between the communication module ED90 and another electronic device through the selected antenna. Other components (RFIC, etc.) other than the antenna may be included as part of the antenna module ED97.

[0157] Some components are connected to each other and can exchange signals (commands, data, etc.) via a communication scheme (bus, general purpose input and output (GPIO), serial peripheral interface (SPI), mobile industry processor interface (MIPI), etc.), and can exchange signals (commands, data, etc.) between peripherals.

[0158] Commands or data can be sent or received between the electronic device ED01 and the external electronic device ED04 through the server ED08 connected to the second network ED99. The other electronic devices ED02 and ED04 can be devices of the same or different types as the electronic device ED01. All or some operations performed in the electronic device ED01 can be performed in one or more of the other electronic devices ED02, ED04 and ED08. For example, when the electronic device ED01 needs to perform a function or service, it can request one or more other electronic devices to perform part or all of the function or service instead of performing the function or service by itself. The one or more other electronic devices that receive the request can perform additional functions or services related to the request and send the execution results to the electronic device ED01. To this end, cloud computing, distributed computing and / or client-server computing technologies can be used.

[0159] Fig.12 It shows the settings in Fig.11 A block diagram of a camera module ED80 in an electronic device. Fig.12 The camera module ED80 may include a lens assembly 1170, a flash 1120, an image sensor 1000, an image stabilizer 1140, an auto focus (AF) controller 1130, a memory 1150 (buffer memory, etc.), an actuator 1180 and / or an image signal processor (ISP) 1160.

[0160] The lens assembly 1170 can gather light emitted from a target as an object of image capture. The lens assembly 1170 may include one or more optical lenses. The lens assembly 1170 may include a path switching member that changes the path of light to face the image sensor 1000. According to the arrangement of the path switching member and the arrangement form with the optical lens, the camera module ED80 may have a vertical form or a folded form. The camera module ED80 may include a plurality of lens assemblies 1170, and in this case, the camera module ED80 may be a dual camera, a 360° camera, or a spherical camera. Some of the plurality of lens assemblies 1170 may have the same lens properties (viewing angle, focal length, autofocus, F number, optical zoom, etc.) or other lens properties. The lens assembly 1170 may include a wide-angle lens or a telephoto lens.

[0161] The actuator 1180 may drive the lens assembly 1170. For example, at least some of the optical lenses and the path switching member constituting the lens assembly 1170 may be moved by the actuator 1180. The optical lens may be moved along the optical axis, and the distance between adjacent lenses may be adjusted by moving at least a portion of the optical lens included in the lens assembly 1170, thereby adjusting the optical zoom ratio.

[0162] The actuator 1180 may adjust the position of any one optical lens included in the lens assembly 1170 so that the image sensor 1000 is located at the focal length of the lens assembly 1170. The actuator 1180 may drive the lens assembly 1170 according to the AF driving signal transmitted from the AF controller 1130.

[0163] The flash 1120 may emit light for enhancing light emitted or reflected from a target. The flash 1120 may emit visible light or infrared light. The flash 1120 may include one or more light emitting diodes (RGB (red-green-blue) LEDs, white LEDs, infrared LEDs, ultraviolet LEDs, etc.) and / or xenon lamps. The image sensor 1000 may be Figure 1 The image sensor 200 described in . The image sensor 1000 may convert light emitted or reflected from a target and passing through the lens assembly 1170 into an electrical signal and obtain an image corresponding to the target.

[0164] The image sensor 1000 may be equipped with the above-mentioned metaprism array MP, and each pixel may include a plurality of light sensing units forming a plurality of channels, for example, a plurality of light sensing units arranged in a 2×2 matrix. Some of these pixels may be used as AF pixels, and the image sensor 1000 may generate an AF driving signal based on signals from a plurality of channels in the AF pixels.

[0165] In response to the movement of the camera module ED80 or the electronic device ED01 including the camera module ED80, the image stabilizer 1140 may move one or more lenses included in the lens assembly 1170 or the image sensor 1000 in a specific direction, or control the operating characteristics (readout timing adjustment, etc.) of the image sensor 1000 to compensate for the negative effects caused by the movement. The image stabilizer 1140 may detect the movement of the camera module ED80 or the electronic device ED01 using a gyro sensor or an acceleration sensor placed inside or outside the camera module ED80. The image stabilizer 1140 may be implemented optically.

[0166] The AF controller 1130 may generate an AF driving signal according to a signal value sensed from the AF pixel of the image sensor 1000. The AF controller 1130 may control the actuator 1180 according to the AF driving signal.

[0167] The memory 1150 may store some or all of the data of the image acquired by the image sensor 1000 for the next image processing operation. For example, if a plurality of images are acquired at a high speed, the acquired raw data (Bayer pattern data, high-resolution data, etc.) may be stored in the memory 1150, only a low-resolution image may be displayed, and then the raw data of the selected image may be sent to the image signal processor 1160. The memory 1150 may be integrated into the memory ED30 of the electronic device ED01, or may be configured as a separate memory that operates independently.

[0168] ISP 1160 may perform image processing on an image obtained by image sensor 1000 or image data stored in memory 1150. Image processing may include depth map generation, three-dimensional modeling, panorama generation, feature point extraction, image synthesis and / or image compensation (noise reduction, resolution adjustment, brightness adjustment, blur, sharpening, softening, etc.). Image signal processor 1160 may perform control (exposure time control, readout timing control, etc.) on components (image sensor 1000, etc.) included in camera module ED80. The image processed by image signal processor 1160 may be stored again in memory 1150 for further processing, or may be provided to external components (memory ED30, display device ED60, electronic device ED02, electronic device ED04, server ED08, etc.) of camera module ED80. Image signal processor 1160 may be integrated into processor ED20, or may be configured as a separate processor that operates independently of processor ED20. In the case where the image signal processor 1160 is composed of a processor separate from the processor ED20 , the image processed by the image signal processor 1160 may be displayed through the display device ED60 after additional image processing by the processor ED20 .

[0169] The AF controller 1130 may be integrated into the image signal processor 1160. The ISP 1160 processes a signal from an auto focus pixel of the image sensor 1000 to generate an AF signal, and the AF controller 1130 may convert the AF signal into a driving signal for driving the actuator 1180 and transmit the driving signal to the actuator 1180.

[0170] The image sensor 1000 according to the embodiment may be applied to various electronic devices.

[0171] The image sensor 1000 according to the embodiment can be applied to a mobile phone or a smart phone, a tablet computer or a smart tablet computer, a digital camera or a video recorder (referred to as a camera), a laptop computer, a television, a smart TV, etc. For example, a smart phone or a smart tablet computer may include a plurality of high-resolution cameras each equipped with a high-resolution image sensor. The high-resolution camera can be used to extract depth information from a target in an image, adjust the defocus of an image, or automatically identify a target in an image.

[0172] In addition, the image sensor 1000 can be applied to smart refrigerators, security cameras, robots, medical cameras, etc. For example, a smart refrigerator can use an image sensor to automatically identify food in the refrigerator and notify the user through a smart phone of the presence of specific food, the type of food stored or taken out, etc. Security cameras can provide ultra-high resolution images and can use high sensitivity to identify objects or people in images even in dark environments. Robots can be deployed at disaster or industrial sites that people cannot directly access to provide high-resolution images. Medical cameras can provide high-resolution images for diagnosis or surgery, and the field of view can be adjusted dynamically.

[0173] In addition, the image sensor 1000 can be applied to a vehicle. The vehicle may include a plurality of vehicle cameras arranged at various positions. Each vehicle camera may include an image sensor according to an embodiment. The vehicle may use a plurality of vehicle cameras to provide the driver with various information about the interior or surrounding environment of the vehicle, and may automatically identify objects or people in the image to provide information required for autonomous driving.

[0174] The image sensor including the lens array and the image acquisition device including the image sensor have been described with reference to the embodiments shown in the accompanying drawings.

[0175] While example embodiments have been particularly shown and described, it will be understood that various changes in form and details may be made therein without departing from the spirit and scope of the appended claims.

Claims

1. An image acquisition device, comprising: An image sensor comprising a microlens and a plurality of pixels arranged adjacent to each other and sharing the microlens; as well as A processor configured to process an input image acquired by the image sensor, wherein the processor is further configured to: Generating a plurality of disparity images by extracting and combining sub-images having the same disparity from the input images; generating an RGB demosaiced image by interpolating color values ​​of empty pixels of each of the plurality of disparity images using color values ​​of surrounding pixels; generating a plurality of upsampled images by upsampling each of the RGB demosaiced images to an input image size; and An output image is generated by weightedly summing a reference up-sampled image selected from the up-sampled images and the other up-sampled images based on similarities between the one reference up-sampled image and the other up-sampled images.

2. The image acquisition device according to claim 1, wherein: The processor is further configured to generate the RGB demosaiced image by using one of average value interpolation, linear interpolation, cubic interpolation, and directional bilinear interpolation.

3. The image acquisition device according to claim 1, wherein: The image sensor comprises: a quad-Bayer pattern array in which pixels arranged in a 2×2 matrix include corresponding color filters having the same color; or A quad Bayer pattern array in which pixels arranged in a 4×4 matrix include corresponding color filters having the same color.

4. The image acquisition device according to claim 3, wherein: The processor is further configured to generate a parallax A image, a parallax B image, a parallax C image, and a parallax D image as the plurality of parallax images based on the input image.

5. The image acquisition device according to claim 4, wherein: The processor comprises: A plurality of logics are configured to: simultaneously interpolate each of the parallax A image, the parallax B image, the parallax C image, and the parallax D image; or A single logic is configured to sequentially interpolate the parallax A image, the parallax B image, the parallax C image, and the parallax D image.

6. The image acquisition device according to claim 1, wherein: The processor is further configured to generate the upsampled image by using one of bilinear interpolation, bicubic interpolation, and a deep learning network.

7. The image acquisition device according to claim 1, wherein: The processor is further configured to determine a similarity between the reference up-sampled image and the other up-sampled images by using one of a sum of absolute differences (SAD), a sum of squared differences (SSD), a normalized cross correlation (NCC), and a zero-mean normalized cross correlation (ZNCC).

8. The image acquisition device according to claim 1, wherein: The processor is further configured to detect an aliasing region of the upsampled image.

9. The image acquisition device according to claim 8, wherein: The processor is further configured to: increase a weight in the aliasing region if a similarity between the reference up-sampled image and the other up-sampled images is lower than a preset value in the aliasing region.

10. The image acquisition device according to claim 8, wherein: The processor is further configured to: By using a directional filter, the edge result value is output; Outputting an edge absolute value by performing absolute value processing on the edge result value; and An edge map is generated by using input values ​​determined based on the edge absolute values ​​and a control signal.

11. The image acquisition device according to claim 10, wherein: The directional filters include a horizontal directional filter, a vertical directional filter, a 45 degree directional filter and a 135 degree directional filter.

12. The image acquisition device according to claim 11, wherein: The control signal includes one of the following: a first control signal for generating the edge map by using a maximum value among the following as the input value: an edge absolute value of the horizontal direction filter, an edge absolute value of the vertical direction filter, an edge absolute value of the 45 degree direction filter, and an edge absolute value of the 135 degree direction filter; a second control signal for generating the edge map by using an average value of the following items as the input value: an edge absolute value of the horizontal direction filter, an edge absolute value of the vertical direction filter, an edge absolute value of the 45 degree direction filter, and an edge absolute value of the 135 degree direction filter; and A third control signal is used to generate the edge map by using the maximum value of the following items as the input value: the weighted average of the edge absolute value of the horizontal direction filter and the edge absolute value of the vertical direction filter, and the weighted average of the edge absolute value of the 45 degree direction filter and the edge absolute value of the 135 degree direction filter.

13. The image acquisition device according to claim 10, wherein: The processor is further configured to: Determining that the input value is not an edge based on the input value being less than a first threshold; Based on the input value being greater than a second threshold, determining that the input value is an edge; and Based on determining that the input value is between the first threshold and the second threshold, linear interpolation is performed.

14. The image acquisition device according to claim 1, wherein: The processor is further configured to: perform white balance processing on the input image before generating the plurality of parallax images.

15. The image acquisition device according to claim 14, wherein: The processor is further configured to perform color shading compensation on the input image before the white balance processing.

16. The image acquisition device according to claim 15, wherein: The processor is further configured to: between the color shading compensation and the white balance processing, crop only a target area from the input image.

17. An image acquisition method used by an image acquisition device including an image sensor, the image sensor including a microlens and a plurality of pixels arranged adjacent to each other and sharing the microlens, the image acquisition method comprising: generating a plurality of disparity images by extracting and combining sub-images having the same disparity from input images acquired by the image sensor; demosaicing the image by interpolating color values ​​of empty pixels of each of the plurality of disparity images using color values ​​of surrounding pixels; generating a plurality of upsampled images by upsampling each of the interpolated disparity images to a size of the input image; as well as The image synthesis comprises generating an output image by: performing a weighted summation of an upsampled image selected from the upsampled images and the other upsampled images based on similarities between the upsampled image and the other upsampled images.

18. The image acquisition method according to claim 17, wherein: The image sensor comprises: a quad-Bayer pattern array in which pixels arranged in a 2×2 matrix include corresponding color filters having the same color; or A quad Bayer pattern array in which pixels arranged in a 4×4 matrix include corresponding color filters having the same color.

19. The image acquisition method according to claim 17, wherein: The image synthesis further includes: detecting an aliasing area of ​​the upsampled image.

20. The image acquisition method according to claim 19, wherein: In the image synthesis, when the similarity in the aliasing region is lower than a preset value, a weight is increased in the aliasing region.

Images