Apparatus for improving image resolution and method of operating the same

Abstract

Provided are an electronic device and a method thereof and a digital camera, including: an image sensor that generates pixel data based on light received through a lens that allows a captured image to be distorted by being compressed in a first direction; and an image signal processor that performs a re-stitching process on the pixel data to correct the distortion occurring in the first direction, and generates re-stitched pixel data.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to Korean Patent Application No. 10-2020-0136546, filed on October 21, 2020, with the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The embodiments of this disclosure described herein relate to electronic devices, and more specifically, to electronic devices and methods of operation thereof for improving image resolution of camera systems including lenses that allow distortion. Background Technology

[0004] Image sensors can be categorized into charge-coupled device (CCD) image sensors, complementary metal-oxide-semiconductor (CMOS) image sensors, and others. CMOS image sensors consist of pixels implemented using CMOS transistors, and convert light energy into electrical signals using photoelectric conversion elements included in each pixel. CMOS image sensors obtain information about captured or photographed images by using the electrical signals generated at each pixel.

[0005] Smartphones can be equipped with cameras that provide a wider field of view than classic views such as a cinematic widescreen. Images can be cropped to obtain images with a wider aspect ratio compared to the image sensor. In this case, the cropped image has a narrower field of view, and a portion of the original image is discarded during the image cropping process. For this reason, it may be more difficult to ensure a wider field of view as the user expects.

[0006] A wider field of view can be ensured by using lenses that allow for greater image distortion. This image distortion can be addressed to obtain an image with an aspect ratio that provides the wider field of view desired by the user. Summary of the Invention

[0007] Embodiments of this disclosure provide an electronic device and a method of operation thereof for improving image resolution in a camera system including lenses that allow distortion by using a re-stitching process of an image sensor with a multi-pixel structure.

[0008] According to an embodiment, an electronic device includes: an image sensor that generates pixel data based on light received through a lens, the lens allowing the captured image to be compressed in a first direction; and an image signal processor that performs a re-stitching process on the generated pixel data to correct the distortion occurring in the first direction and generates re-stitched pixel data.

[0009] According to an embodiment, an electronic device includes: an image sensor that generates pixel data based on light received through a lens, the lens allowing the captured image to be compressed in a first direction; and an image signal processor that performs re-stitching processing on the pixel data to correct the distortion occurring in the first direction and generates re-stitched pixel data. The pixel array includes: a first unit pixel comprising a first plurality of sub-pixels sharing a first floating diffusion region and a first color filter; a second unit pixel comprising a second plurality of sub-pixels sharing a second floating diffusion region and a second color filter; and a third unit pixel comprising a third plurality of sub-pixels sharing a third floating diffusion region and a third color filter.

[0010] According to an embodiment, a method for processing a signal output from an image sensor, the image sensor including a first unit pixel, a second unit pixel, and a third unit pixel, the first unit pixel including a first plurality of sub-pixels and a first color filter, the second unit pixel including a second plurality of sub-pixels and a second color filter, and the third unit pixel including a third plurality of sub-pixels and a third color filter, the method comprising: acquiring pixel values ​​from the plurality of sub-pixels based on light received through a lens, the lens allowing the captured image to be compressed in a first direction to reduce distortion; generating pixel data by performing correlated double sampling on the pixel values; and generating re-stitched pixel data by performing a re-stitching process on the pixel data to correct the distortion occurring in the first direction.

[0011] According to an embodiment, a digital camera includes: a lens that allows compression of incident light in a first direction; an image sensor configured to generate pixel data based on incident light received through the lens; and an image signal processor configured to generate re-stitched pixel data by decompressing the generated pixel data in the first direction.

[0012] The image sensor may include: a first unit pixel including a first plurality of sub-pixels sharing a first floating diffusion area; a second unit pixel including a second plurality of sub-pixels sharing a second floating diffusion area; and a third unit pixel including a third plurality of sub-pixels sharing a third floating diffusion area, wherein the image signal processor is configured to re-stitch only the first plurality of sub-pixels, the second plurality of sub-pixels, and the third plurality of sub-pixels that are adjacent to each other in a first direction. Attached Figure Description

[0013] The above and other embodiments of this disclosure will become apparent from the detailed description of the embodiments with reference to the accompanying drawings.

[0014] Figure 1 This is a block diagram illustrating the configuration of an image processing section according to an embodiment of the present disclosure.

[0015] Figure 2 It is shown Figure 1 A block diagram of the configuration of the pixel array.

[0016] Figure 3 It is shown that it is from the included Figure 2 A block diagram showing the re-stitching process associated with pixel data output from an image sensor with a pixel array.

[0017] Figure 4 Is to show execution Figure 3 A flowchart illustrating the method for reassembling the components.

[0018] Figure 5 It is shown Figure 1 A block diagram of another configuration of the pixel array. Figure 6 It is shown Figure 5 A circuit diagram of one of the unit pixels.

[0019] Figure 7 It is shown that it is from the included Figure 5 A block diagram showing the digital binning associated with the first pixel data output by the image sensor with a pixel array and the re-stitching process associated with the second pixel data.

[0020] Figure 8 Is to show execution Figure 7 A flowchart illustrating the method for digital binning and reassembly processing.

[0021] Figure 9 This is a conceptual diagram illustrating a process for improving image resolution through re-stitching according to an embodiment of the present disclosure.

[0022] Figure 10 This is a flowchart illustrating an operation method of an apparatus for improving image resolution according to an embodiment of the present disclosure.

[0023] Figure 11 This is a block diagram illustrating the configuration of an image processing block and an application processor according to another embodiment of the present disclosure.

[0024] Figure 12 This is a block diagram illustrating the configuration of an electronic device including a multi-camera module employing an image resolution enhancement method according to an embodiment of the present disclosure.

[0025] Figure 13 It is shown Figure 12 A block diagram showing the configuration of the camera module. Specific Implementation

[0026] The embodiments of this disclosure are described in detail below, so that those skilled in the art can easily implement the embodiments and other embodiments of this disclosure.

[0027] The components described in the detailed description with reference to the terms "unit," "module," "block," etc., and the functional blocks shown in the accompanying drawings, may be implemented using software, hardware, or a combination thereof. For example, software may include machine code, firmware, embedded code, application software, or a combination thereof. Hardware may include electrical circuits, electronic circuits, processors, computers, integrated circuits, integrated circuit cores, image sensors, pressure sensors, inertial sensors, microelectromechanical systems (MEMS), passive components, or a combination thereof.

[0028] Figure 1 The image processing block 100 according to an embodiment of the present disclosure is illustrated. The image processing block 100 may be implemented as part of various electronic devices such as smartphones, digital cameras, laptop computers, desktop computers, etc. The image processing block 100 may include a lens 110, an image sensor 120, and an image signal processor 130.

[0029] Light can be received from an object or scene 10 targeted by a lens 110 that can receive reflected light. Specifically, the lens 110 used in embodiments of this disclosure ensures a wide field of view, but may allow distortion of the image in a first direction in which the field of view is widened. For example, the lens 110 may be an anamorphic lens, which ensures a wide field of view in a particular direction, but allows distortion of the image in a particular direction such as where the field of view is widened. Hereinafter, it may be assumed that the direction in which distortion occurs due to the lens 110 is the same as the direction in which the field of view is widened due to the lens 110, but is not limited thereto.

[0030] Image sensor 120 can generate electrical signals based on light received through lens 110. For example, image sensor 120 can be implemented using a complementary metal-oxide-semiconductor (CMOS) image sensor or the like. For example, image sensor 120 can be a multi-pixel image sensor with a dual-pixel structure or a tetra-cell structure.

[0031] The image signal processor 130 can generate image data associated with light captured from scene 10 by appropriately processing the electrical signals generated by image sensor 120. In the following description, scene 10 may include background, landscape, etc., and targets that typically reflect or emit light to be captured by image sensor 120.

[0032] Image sensor 120 may include pixel array 121. Pixels in pixel array 121 convert light into electrical signals to generate pixel values. Image sensor 120 may perform correlated double sampling (CDS) on the pixel values ​​to generate pixel data PDAT. Specifically, because lens 110 of this disclosure allows distortion, pixel data PDAT may indicate an image that has been distorted. For example, in the case where lens 110 is an anamorphic lens that allows distortion in the direction of widening field of view, pixel data PDAT may indicate an image that has been distorted in the direction of widening field of view. In this disclosure, an image or data that has been distorted refers to an image or data generated based on light received through a lens that allows distortion.

[0033] Image signal processor 130 may include re-stitching processor 131. Re-stitching processor 131 may perform distortion correction on pixel data PDAT in the direction in which distortion occurs and may generate re-stitched pixel data RDAT. For example, re-stitching processor 131 may perform re-stitching processing on pixel data PDAT in the direction in which distortion occurs.

[0034] For example, the re-stitching processor 131 can perform re-stitching processing on pixel data PDAT of an image indicating distortion in the direction of widening field of view, and can generate re-stitched pixel data RDAT. In this case, the re-stitched pixel data RDAT can indicate an image whose resolution in the direction of distortion has been improved by correcting the distortion caused by lens 110.

[0035] For example, the re-stitching process may include stretching (such as upscaling the size of the pixel data PDAT in the direction where distortion occurs) and interpolation based on the value of the pixel data PDAT. Conversely, the re-stitching processor 131 may not perform the re-stitching process in the direction where no distortion occurs (such as in the direction where the field of view does not widen).

[0036] The image signal processor 130 can generate image data IDAT associated with scene 10 based on the re-stitched pixel data RDAT. To this end, the image signal processor 130 can perform the following processes on the re-stitched pixel data RDAT: color correction, automatic white balance, gamma correction, color saturation correction, bad pixel correction, and tone correction.

[0037] exist Figure 1The diagram shows a lens 110 and an image sensor 120. However, in another embodiment, the image processing block 100 may include multiple lenses and multiple image sensors. In this case, the multiple lenses may have different fields of view and may result in different distortions. The multiple image sensors may have different functions, different performance, and / or different characteristics, and the configuration or structure of the pixel arrays included in the multiple image sensors may differ. In addition, the image sensor 120 may also include analog-to-digital converters (ADCs) for performing correlated double sampling (CDS), counters, buffers, etc.

[0038] In addition, although Figure 1 The illustration shows a case where the re-stitching processor 131 is implemented in hardware within the image processing block 100, but this disclosure is not limited thereto. In another embodiment, Figure 1 The re-stitching processor 131 can be implemented in software on a main processor, such as an application processor independent of the image processing block 100. (See reference...) Figure 11 An embodiment of the reassembly processor 131 is described as being implemented in software on an application processor.

[0039] Figure 2 Show Figure 1 The pixel array 121 is configured as follows. The pixel array 200 may include multiple unit pixels (such as 201 to 204) arranged as a matrix with M columns and N rows, where M and N are integers.

[0040] Each of the plurality of unit pixels may include a color filter. For example, each of the plurality of unit pixels may include a red filter that allows red light of the red spectrum to pass through, a green filter that allows light of the green spectrum to pass through, or a blue filter that allows light of the blue spectrum to pass through. Figure 2 In this image, each of unit pixels 201 and 204 includes a green filter, unit pixel 202 includes a red filter, and unit pixel 203 includes a blue filter. In the following description, pixels including a green filter are designated as Gr pixels or Gb pixels, pixels including a red filter are designated as R pixels, and pixels including a blue filter are designated as B pixels.

[0041] Multiple unit pixels of pixel array 200 can be arranged in a Bayer pattern. For example, Gr pixels and R pixels can be alternately set in each row of odd-numbered rows (such as the first and third rows) of pixel array 200. For example, B pixels and Gb pixels can be alternately set in each row of even-numbered rows (such as the second and fourth rows) of pixel array 200.

[0042] Each of the plurality of unit pixels (such as 201 to 204) in pixel array 200 may include two sub-pixels (such as two of 201a to 204b). The two sub-pixels (such as 201a and 201b) constituting a unit pixel (such as 201) may share a microlens and may include the same color filter. That is, pixel array 200 may have a dual-pixel structure. Each sub-pixel may include a photoelectric conversion element such as a photodiode. Figure 2 In the diagram, thick solid lines indicate unit pixels, and thin solid lines indicate sub-pixels.

[0043] Two sub-pixels (such as 201a and 201b) constituting a unit pixel (such as 201) may share a floating diffusion region, or may each include a floating diffusion region. Each sub-pixel may output an analog signal, and the analog signal output from each sub-pixel may be converted into a digital signal by an analog-to-digital converter (ADC). Pixel data PDAT may be a set of signals output from each sub-pixel.

[0044] When the unit pixels of pixel array 200 are arranged in a matrix form with M columns and N rows, the size of pixel data PDAT can be represented by "M×N". In the following description, when the size of image data is represented by "M×N", the image data indicates an image represented by unit pixels arranged in a matrix form with M columns and N rows.

[0045] Figure 2 This illustration shows a case where each subpixel occupies the same size space (e.g., the space occupied by subpixel 201a is the same size as the space occupied by subpixel 201b). However, this disclosure is not limited thereto. For example, in another embodiment, subpixels may occupy different sizes of space, such as the space occupied by subpixel 201a being wider than the space occupied by subpixel 201b.

[0046] Figure 3 Showing and from including Figure 2 The pixel data PDAT output by the image sensor 120 with pixel array 200 is associated with the re-stitching process. Figure 3 The pixel data PDATa shown in the embodiment is 4×4 in size, and the re-stitched pixel data RDATa is 8×4 in size, but this disclosure is not limited thereto. Hereinafter, reference will be made to... Figures 1 to 3 Let's all provide a description.

[0047] Because each unit pixel of pixel array 200 comprises two sub-pixels, the image data output from each unit pixel of pixel array 200 may include pixel values ​​output from the two sub-pixels that constitute each unit pixel. Therefore, pixel data PDATa may include pixel values ​​output from each sub-pixel of pixel array 200.

[0048] The re-stitching processor 131 can perform re-stitching processing on the pixel data PDATa in the direction of distortion (such as the direction of field of view widening), and can generate re-stitched pixel data RDATa. In other words, the image indicated by the re-stitched pixel data RDATa can be an image in which the data size is enlarged in the direction of distortion, such as being proportionally enlarged in the direction of distortion, compared to the image indicated by the pixel data PDATa.

[0049] The pixel values ​​of the re-stitched pixel data RDATa can be determined based on the pixel values ​​of the pixel data PDATa (such as the pixel values ​​output from the individual sub-pixels constituting the pixel array 200). For example, the pixel values ​​of the first row 220 of the re-stitched pixel data RDATa can be determined by interpolation based on the pixel values ​​of the first row 210 of the pixel data PDATa. When the re-stitching process described above is performed, the re-stitched pixel data RDATa can indicate an image with improved resolution in the direction in which distortion occurs by correcting the distortion caused by the lens 110.

[0050] Figure 4 Show execution Figure 3 The method for reassembling and reassembling. In detail, Figure 4 Showing by the Figure 3 The first row 210 of the pixel data PDATa is re-stitched to generate Figure 3 The processing of the first row of 220 pixel values ​​of the re-stitched pixel data RDATa.

[0051] The first row 210 of the pixel data PDATa may include eight pixel values ​​Gr1 to R4 output from four unit pixels (e.g., eight sub-pixels). The first row 220 of the re-stitched pixel data RDATa may include eight pixel values ​​Gr1 R Up to R4 R The re-stitching processor 131 can determine the pixel values ​​of the first row 220 of the re-stitched pixel data RDATa based on the pixel values ​​of the first row 210 of the pixel data PDATa. For example, the re-stitching processor 131 can determine the pixel value Gr1 of the first row 220 of the re-stitched pixel data RDATa by performing interpolation based on the pixel values ​​Gr1 to R4 of the first row 210 of the pixel data PDATa. R Up to R4 R .

[0052] The Gr, Gb, R, and B pixel values ​​of the re-stitched pixel data RDATa can be determined based on the Gr, Gb, R, and B pixel values ​​of the pixel data PDATa. For example, the pixel value Gr1 of the re-stitched pixel data RDATa...R The pixel value Gr1 of the re-stitched pixel data PDATa can be the same. Similarly, the pixel value R2 of the re-stitched pixel data RDATa can also be the same. R Pixel value Gr3 R and pixel value R4 R These values ​​can be the same as pixel values ​​R2, Gr3, and R4 of pixel data PDATa, respectively.

[0053] For example, the pixel value Gr2 of the re-stitched pixel data RDATa can be determined based on the pixel values ​​Gr2 and Gr3 of the pixel data PDATa. R Among them, Gr2 has a greater weight than Gr3 because Gr2 R In the first direction, it is closer to Gr2 than to Gr3, but it is not limited to this.

[0054] Equation 1

[0055]

[0056] For example, when determining the pixel value Gr2 R In the above method, the pixel value R1 can be determined similarly based on the pixel value of the pixel data PDATa. R Pixel value R3 R and pixel value Gr4 R However, this disclosure is not limited thereto. For example, the pixel values ​​of the re-stitched pixel data RDATa can be determined by methods different from those described above.

[0057] Figure 5 Show Figure 1 Another configuration of pixel array 121. Pixel array 300 may include multiple unit pixels (such as 301 to 304) arranged as an N×M matrix, where M and N are integers.

[0058] Each of the multiple unit pixels can include a color filter. Figure 5 In the pixel array 300, each of unit pixels 301 and 304 includes a green filter, unit pixel 302 includes a red filter, and unit pixel 303 includes a blue filter. The plurality of unit pixels in the pixel array 300 can be arranged in a Bayer pattern.

[0059] Each of the plurality of unit pixels (such as 301 to 304) in the pixel array 300 may include four sub-pixels (such as four of 301a to 304d). The four sub-pixels (such as 301a to 301d) constituting a unit pixel (such as 301) may have the same color filter. That is, the pixel array 300 may be implemented in a four-unit structure. Each sub-pixel may include a photoelectric conversion element such as a photodiode. In addition, each sub-pixel may include a microlens, and the four sub-pixels (such as 301a to 301d) constituting a unit pixel (such as 301) may share a floating diffusion region.

[0060] Each sub-pixel can output an analog signal, and the analog signal output from each sub-pixel can be converted into a digital signal by an analog-to-digital converter. Pixel data PDAT can be a set of signals output from each sub-pixel.

[0061] Figure 6 Show Figure 5 A circuit diagram of one of the unit pixels. For example, unit pixel 301 may include photoelectric conversion elements PD1 to PD4, transmission transistors Tx1 to Tx4, reset transistor RST, dual conversion transistor DC, drive transistor Dx, and selection transistor SEL. Unit pixel 301 may include sub-pixels 301a to 301d.

[0062] In this embodiment, although sub-pixel 301a is closer to sub-pixel 301b in the first direction than to sub-pixel 301c in the second direction, this may be specifically applicable to ensuring an image with a wider field of view in the first direction, but the embodiment is not limited thereto. For example, in an alternative embodiment, sub-pixel 301a may be farther from sub-pixel 301b in the first direction than from sub-pixel 301c in the second direction, or it may be substantially equidistant from sub-pixel 301b and sub-pixel 301c.

[0063] Furthermore, although sub-pixel 301a is substantially horizontally aligned with sub-pixel 301b in a first direction and substantially vertically aligned with sub-pixel 301c in a second direction, the embodiments are not limited thereto. For example, in alternative embodiments, sub-pixel 301a may be horizontally aligned with sub-pixel 301b but vertically staggered with sub-pixel 301c, horizontally staggered with sub-pixel 301b but vertically aligned with sub-pixel 301c, or horizontally staggered with sub-pixel 301b and vertically staggered with sub-pixel 301c.

[0064] The first sub-pixel 301a may include a first photoelectric conversion element PD1 and a first transmission transistor Tx1, and each of the remaining sub-pixels 301b, 301c, and 301d may include elements similar to those of the first sub-pixel 301a. Each of sub-pixels 301a to 301d may share a reset transistor RST, a dual conversion transistor DC, a drive transistor Dx, and a selection transistor SEL. Additionally, as shown in the reference... Figure 5 As described, sub-pixels 301a to 301d can share the floating diffusion region FD.

[0065] The dual-conversion transistor DC can be driven by the dual-conversion signal VDC. When the dual-conversion transistor DC is off, the floating diffusion region FD can be connected to the first floating diffusion capacitor CFD1. In this case, the capacitance of the floating diffusion region FD corresponds to the capacitance of the first floating diffusion capacitor CFD1.

[0066] When the dual-conversion transistor DC is turned on, the floating diffusion region FD can be connected to the second floating diffusion capacitor CFD2 and the first floating diffusion capacitor CFD1. The extended floating diffusion region FD' indicates the floating diffusion region when the dual-conversion transistor DC is turned on. The second floating diffusion capacitor CFD2 can be provided to prevent saturation. For example, the floating diffusion capacitors CFD1 and CFD2 can be parasitic capacitors and / or capacitor elements.

[0067] Transmission transistors Tx1 to Tx4 can be driven by transmission signals VT1 to VT4 respectively, and can transfer the charge generated or integrated by photoelectric conversion elements PD1 to PD4 to the floating diffusion region FD or the extended floating diffusion region FD'. For example, the first terminals of transmission transistors Tx1 to Tx4 can be connected to photoelectric conversion elements PD1 to PD4 respectively, and the second terminals of transmission transistors Tx1 to Tx4 can be connected together to the floating diffusion region FD.

[0068] The floating diffusion region FD or the extended floating diffusion region FD' can accumulate charge corresponding to the amount of incident light. When the transmission transistors Tx1 to Tx4 are turned on by transmission signals VT1 to VT4 respectively, the floating diffusion region FD or the extended floating diffusion region FD' can accumulate the charge supplied from the photoelectric conversion elements PD1 to PD4. Because the floating diffusion region FD is connected to the gate terminal of the drive transistor Dx, which operates as a source follower amplifier, a voltage corresponding to the accumulated charge can be formed at the floating diffusion region FD.

[0069] The reset transistor RST can be driven by the reset signal VRST and can provide the power supply voltage VDD to the floating diffusion region FD or the extended floating diffusion region FD'. As a result, the charge accumulated in the floating diffusion region FD or the extended floating diffusion region FD' can be moved to the terminal for the power supply voltage such as VDD, and the voltage of the floating diffusion region FD or the extended floating diffusion region FD' can be reset.

[0070] The driving transistor Dx amplifies the voltage applied to its gate terminal's floating diffusion region FD or extended floating diffusion region FD' to a voltage such as VDD applied to its input terminal to generate the pixel signal PIX. The selection transistor SEL can be driven by the selection signal VSEL and can select the pixel to be read on a row-by-row basis. When the selection transistor SEL is turned on, the pixel signal PIX can be output through the column line CL.

[0071] Figure 7 Showing and from including Figure 5 The image sensor 120 outputs the first pixel data PDATb_1 of the pixel array 300, which is associated with digital binning and re-stitching processing associated with the second pixel data PDATb_2. The embodiment in... Figure 5 The first pixel data PDATb_1 and the second pixel data PDATb_2 are shown to be 4×4 in size, and the reassembled pixel data RDATb is 8×4 in size, but this disclosure is not limited thereto. Referring below... Figure 1 , Figure 5 and Figure 7 Provide a description.

[0072] Because each unit pixel of pixel array 300 comprises four sub-pixels, the image data output from each unit pixel of pixel array 300 may include pixel values ​​output from the four sub-pixels that constitute each unit pixel. Therefore, the first pixel data PDATb_1 may include pixel values ​​output from each sub-pixel of pixel array 300.

[0073] The second pixel data PDATb_2 can be generated as a result of digital binning the pixel values ​​of the first pixel data PDATb_1 in a distortion-free direction (such as the column direction). For example, the image signal processor 130 can generate the second pixel data PDATb_2 by performing digital binning on the pixel values ​​of the first pixel data PDATb_1 in a distortion-free direction. For example, the pixel values ​​of the first row 320 of the second pixel data PDATb_2 can be determined by performing digital binning on the pixel values ​​of the first row 310 of the first pixel data PDATb_1 in a distortion-free direction.

[0074] In another embodiment, the image sensor 120 can perform analog binning. Referring below... Figure 6 This describes the situation where simulated binning is performed on one of the unit pixels (such as 301) that constitute the pixel array 300.

[0075] The first transmission transistor Tx1 and the third transmission transistor Tx3 of a unit pixel can be turned on simultaneously, and the unit pixel can output a first pixel signal corresponding to the sum of the charges transferred from the first photoelectric conversion element PD1 and the third photoelectric conversion element PD3. Similarly, the second transmission transistor Tx2 and the fourth transmission transistor Tx4 of a unit pixel can be turned on simultaneously, and the unit pixel can output a second pixel signal corresponding to the sum of the charges transferred from the second photoelectric conversion element PD2 and the fourth photoelectric conversion element PD4. Each of the first pixel signal and the second pixel signal can be converted into a digital signal by an analog-to-digital converter.

[0076] Each of the remaining unit pixels can also output its own pixel signal corresponding to the sum of charges, and the output pixel signals can be converted into digital signals respectively. A set of such converted digital signals can be output as pixel data. Thus, the pixel array 300 can output pixel data, such as the second pixel data PDATb_2 obtained as a result of analog binning.

[0077] The re-stitching processor 131 can perform re-stitching processing on the second pixel data PDATb_2 in the direction where distortion occurs (such as in the direction of widening of the field of view), and can generate re-stitched pixel data RDATb. In other words, compared with the image indicated by the second pixel data PDATb_2, the image indicated by the re-stitched pixel data RDATb can be an image in which the data size is enlarged in the direction where distortion occurs (e.g., the data size is enlarged proportionally in the direction where distortion occurs, and the data size is kept equal in the direction where no distortion occurs).

[0078] The pixel values ​​of the re-stitched pixel data RDATb can be determined based on the pixel values ​​of the second pixel data PDATb_2 (such as pixel values ​​output from the individual sub-pixels constituting the pixel array 300). For example, the pixel values ​​of the first row 330 of the re-stitched pixel data RDATb can be determined by interpolation based on the pixel values ​​of the first row 320 of the second pixel data PDATb_2. When the above-described re-stitching process is performed, the re-stitched pixel data RDATb can indicate an image with improved resolution in the direction where distortion occurs by correcting for distortion caused by the lens 110.

[0079] Figure 8 Show execution Figure 7 The method for digital binning and reassembly processing. In detail, Figure 8 Showing by the Figure 7The first pixel data PDATb_1, consisting of 310 pixel values ​​in the first row, is generated by performing digital binning. Figure 7 The second pixel data PDATb_2 consists of 320 pixel values ​​in the first row, and is generated by re-stitching the 320 pixel values ​​in the first row of the second pixel data PDATb_2. Figure 7 The processing of the first row (330 pixels) of the re-stitched pixel data RDATb. Below, we will refer to... Figure 1 , Figure 5 and Figure 8 Provide a description.

[0080] The first row 310 of the first pixel data PDATb_1 may include 16 pixel values ​​Gr1 to R8 output from 4 unit pixels (such as 16 sub-pixels). The first row 320 of the second pixel data PDATb_2 may include 8 pixel values ​​Gr1 B Up to R4 B The first row 330 of the reassembled pixel data RDATb can include 8 pixel values ​​Gr1. R Up to R4 R .

[0081] The image signal processor 130 can determine the pixel values ​​of the first row 320 of the second pixel data PDATb_2 by performing digital binning on the pixel values ​​of the first row 310 of the first pixel data PDATb_1 in a direction where no distortion occurs (such as the column direction). For example, the pixel value Gr1 of the second pixel data PDATb_2 B It can be determined as the average of pixel values ​​Gr1 and Gr3 of the first pixel data PDATb_1, and the pixel value Gr2 of the second pixel data PDATb_2. B The average of pixel values ​​Gr2 and Gr4, which can be determined as the first pixel data PDATb_1.

[0082] Similarly, each of the remaining pixel values ​​of the second pixel data PDATb_2 can be determined by calculating the average of the pixel values ​​of the first pixel data PDATb_1. However, this disclosure is not limited thereto. For example, the image signal processor 130 can perform digital binning on the pixel values ​​of the first pixel data PDATb_1 using a method different from that described above.

[0083] The re-stitching processor 131 can determine the pixel values ​​of the first row 330 of the re-stitched pixel data RDATb based on the pixel values ​​of the first row 320 of the second pixel data PDATb_2. For example, the re-stitching processor 131 can determine the pixel values ​​of the first row 320 of the second pixel data PDATb_2 based on the pixel values ​​Gr1. B Up to R4 BPerform interpolation to determine the pixel value Gr1 of the first row of RDATb, which is the pixel data to be re-stitched. R Up to R4 R .

[0084] Based on the first row of the second pixel data PDATb_2, the pixel value Gr1 of 320. B Up to R4 B Methods and references for performing interpolation Figure 4 The methods described are the same, therefore, to avoid repetition, additional descriptions will be omitted. However, this disclosure is not limited thereto. For example, the pixel values ​​of the re-stitched pixel data RDATb can be determined by methods different from those described above.

[0085] Figure 8 This illustration shows the re-stitching processor 131 performing digital binning on the first pixel data PDATb_1 before the re-stitching process, but this disclosure is not limited thereto. In another embodiment, as shown in reference... Figure 6 and Figure 7 As described, pixel data can be output as the result of performing analog binning. Meanwhile, in another embodiment, both analog and digital binning may not be performed, and the re-stitching processor 131 can perform the re-stitching process using all pixel values ​​of the first pixel data PDATb_1.

[0086] Furthermore, this disclosure is not limited to Figure 2 dual-unit structure or Figure 5 To achieve this, a four-unit structure is used. Figure 1 The image sensor 120 is an example. In another embodiment, it can be implemented with a multi-pixel structure (where each unit pixel consists of multiple sub-pixels, such as four or more, each including the same color filter). Figure 1 Pixel array 121. (See reference...) Figure 2 and Figure 5 As described, each unit pixel may include multiple sub-pixels, and each sub-pixel includes a photoelectric conversion element such as a photodiode. Multiple sub-pixels constituting a unit pixel may share a single floating diffusion region or multiple floating diffusion regions.

[0087] Figure 9 The process of improving image resolution by re-stitching according to embodiments of the present disclosure is conceptually illustrated.

[0088] The lens 110 used in embodiments of this disclosure may be an anamorphic lens, which ensures a wide field of view in direction D1 and allows for image compression distortion in direction D1 (such as directions in which the field of view widens relative to scene 10). For example, a region with an aspect ratio of 2M:N may be a region that can be captured by the lens 110 used in embodiments of this disclosure, and a region “b” with an aspect ratio of M:N may be a region that can be captured by a lens that does not cause distortion.

[0089] The pixel array 121 can generate pixel data PDAT as an electrical signal based on the light received through the lens 110. Because image distortion occurs due to compression in direction D1 caused by the lens 110, the size of the pixel data PDAT can be "M×N". (See reference...) Figures 2 to 8 As described, the re-stitching processor 131 can perform re-stitching processing on pixel data PDAT to produce re-stitched pixel data RDAT. The image indicated by the re-stitched pixel data RDAT can be an image whose data size is enlarged from "M" to "2M" in direction D1 compared to the image indicated by the pixel data PDAT (such as the data size being scaled up in direction D1).

[0090] In other words, the size of the re-stitched pixel data RDAT can be “2M×N”, and the aspect ratio can be “2M:N”. The re-stitched pixel data RDAT can indicate an image with improved resolution (such as from “M” to “2M”) in the direction where distortion occurs by correcting for distortion caused by lens 110.

[0091] Conversely, to obtain an image with an aspect ratio of 2M:N using a classic lens that does not cause distortion, it may be necessary to capture region "b" and then crop the captured image to the size corresponding to region "c". Therefore, an image with a specific aspect ratio obtained by the image sensor 120 of this disclosure can show a considerably wider area compared to an image with a specific aspect ratio obtained by a general image sensor.

[0092] Figure 10 This is a flowchart illustrating an operation method of an apparatus for improving image resolution according to an embodiment of the present disclosure. Referring hereafter... Figure 1 and Figure 10 Provide a description.

[0093] In operation S110, pixel array 121 may generate pixel values ​​based on light received through lens 110. In operation S120, image sensor 120 may generate pixel data PDAT by performing correlated double sampling (CDS) on the pixel values ​​generated by pixel array 121. Specifically, because lens 110 used in embodiments of this disclosure allows distortion, pixel data PDAT may indicate an image in which distortion occurs. For example, in the case where lens 110 is an anamorphic lens that allows distortion in the direction of widening field of view, pixel data PDAT may indicate an image in which distortion occurs in the direction of widening field of view.

[0094] In operation S130, the re-stitching processor 131 can perform re-stitching processing on the pixel data PDAT in the direction where distortion occurs, and can generate re-stitched pixel data RDAT. In this case, the re-stitched pixel data RDAT can indicate an image whose resolution in the direction where distortion occurs has been improved by correcting the distortion.

[0095] Figure 11 This is a block diagram illustrating the configuration of an image processing block 100 and an application processor 140 according to another embodiment of this disclosure. Figure 1 different, Figure 11 An embodiment is shown in which the re-stitching processor 141 is not included in the image signal processor 130, and is implemented as software in the application processor 140, which is independent of the image processing block 100.

[0096] Pixel array 121 can generate pixel data PDAT based on light received through lens 110, and image signal processor 130 can generate image data IDAT associated with scene 10 based on pixel data PDAT. Re-stitching processor 141 can perform re-stitching processing on image data IDAT to generate re-stitched pixel data RDAT. Except that re-stitching processor 141 is included in application processor 140, the method for performing re-stitching processing on image data IDAT is similar to that described above. Figures 2 to 8 The method for re-stitching pixel data PDAT is the same as described, so additional descriptions will be omitted to avoid repetition.

[0097] Figure 12 The configuration of an electronic device including a multi-camera module employing an image resolution enhancement method according to an embodiment of the present disclosure is shown. Figure 13 Show Figure 12 The configuration of the camera module.

[0098] Reference Figure 12 The electronic device 1000 may include a camera module group 1100, an application processor 1200, a PMIC (power management integrated circuit) 1300, and an external memory 1400.

[0099] The camera module group 1100 may include multiple camera modules 1100a, 1100b and 1100c. Figure 12 The diagram illustrates an electronic device comprising three camera modules 1100a, 1100b, and 1100c, but this disclosure is not limited thereto. In an embodiment, the camera module group 1100 may be modified to include only two camera modules. Additionally, in an embodiment, the camera module group 1100 may be modified to include "n" camera modules, where n is a natural number of 4 or greater.

[0100] Below, we will refer to Figure 13 The detailed configuration of camera module 1100b is described in more detail below, but the following description can also be applied to the remaining camera modules 1100a and 1100c.

[0101] Reference Figure 13 The camera module 1100b may include a prism 1105, an optical path folding element (OPFE) 1110, an actuator 1130, an image sensing device 1140, and a storage device 1150.

[0102] The prism 1105 may include a reflective plane 1107 of reflective material and may alter the path of light “L” incident from the outside.

[0103] In this embodiment, prism 1105 can change the path of light "L" incident along the first direction "X" to a second direction "Y" perpendicular to the first direction "X". Furthermore, prism 1105 can change the path of light "L" incident in the first direction "X" to a second direction "Y" perpendicular to the first direction "X" by rotating the reflective plane 1107 of the reflective material in direction "A" around the central axis 1106 or by rotating the central axis 1106 in direction "B". In this case, OPFE 1110 can move along a third direction "Z" perpendicular to the first direction "X" and the second direction "Y".

[0104] In an embodiment, as shown in the figure, the maximum rotation angle of the prism 1105 in direction "A" can be equal to or less than 15 degrees in the positive A direction and greater than 15 degrees in the negative A direction, but this disclosure is not limited thereto.

[0105] In an embodiment, prism 1105 can move within approximately 20 degrees, between 10 and 20 degrees, or between 15 and 20 degrees in the positive or negative B direction; here, prism 1105 can move at the same angle in the positive or negative B direction, or can move at similar angles within approximately 1 degree.

[0106] In one embodiment, the prism 1105 can move the reflective plane 1107 of the reflective material in a third direction (such as the Z direction, which is parallel to the direction in which the central axis 1106 extends).

[0107] For example, OPFE 1110 may include an optical lens composed of "m" lenses (where m is a natural number). Here, the "m" lenses can be moved in the second direction "Y" to change the optical zoom ratio of camera module 1100b. For example, when the default optical zoom ratio of camera module 1100b is "Z", the optical zoom ratio of camera module 1100b can be changed to 3Z, 5Z, or a larger optical zoom ratio by moving the "m" optical lenses included in OPFE 1110. OPFE 1110 may also include an optical lens, such as an anamorphic lens, composed of "n" lenses (where n is a natural number preceding the aforementioned "m" lenses).

[0108] Actuator 1130 can move OPFE 1110 or optical lens to a specific position. For example, actuator 1130 can adjust the position of optical lens so that image sensor 1142 is positioned at the focal length of optical lens for accurate sensing.

[0109] Image sensing device 1140 may include image sensor 1142, control logic 1144, and memory 1146. Image sensor 1142 senses an image of a target using light "L" provided through an optical lens. Control logic 1144 controls the overall operation of camera module 1100b. For example, control logic 1144 may control the operation of camera module 1100b based on control signals provided through control signal line CSLb. Furthermore, control logic 1144 may include components such as those for performing the re-stitching process disclosed herein. Figure 1 The 131 re-assembly processor.

[0110] The memory 1146 may store information required for the operation of the camera module 1100b, such as calibration data 1147. Calibration data 1147 may include information required by the camera module 1100b to generate image data using externally supplied light "L". Calibration data 1147 may include, for example, information about the degree of rotation, information about the focal length, information about the optical axis, etc. In the case where the camera module 1100b is implemented as a multi-state camera in which the focal length varies depending on the position of the optical lens, calibration data 1147 may include the focal length value for each position (or state) of the optical lens and information about autofocus. Calibration data 1147 may include information about the direction in which the re-stitching process of this disclosure is performed.

[0111] The memory 1150 can store image data sensed by the image sensor 1142. The memory 1150 can be disposed outside the image sensing device 1140 and can be implemented in the form of a stack of the memory 1150 and the sensor chip constituting the image sensing device 1140. In an embodiment, the memory 1150 can be implemented with an electrically erasable programmable read-only memory (EEPROM), but this disclosure is not limited thereto.

[0112] Refer to together Figure 12 and Figure 13 In an embodiment, each of the plurality of camera modules 1100a, 1100b, and 1100c may include an actuator 1130. Thus, the same calibration data 1147 or different calibration data 1147 may be included in the plurality of camera modules 1100a, 1100b, and 1100c depending on the operation of the actuators 1130 in the plurality of camera modules 1100a, 1100b, and 1100c.

[0113] In an embodiment, one of the plurality of camera modules 1100a, 1100b and 1100c (such as 1100b) may be a camera module in the shape of a folded lens including the aforementioned prism 1105 and OPFE 1110, and the remaining camera modules (such as 1100a and 1100c) may be camera modules in the shape of a vertical shape excluding the aforementioned prism 1105 and OPFE 1110; however, this disclosure is not limited thereto.

[0114] In an embodiment, one of the plurality of camera modules 1100a, 1100b, and 1100c (such as 1100c) may be a vertically shaped depth camera, for example, that extracts depth information using infrared (IR). In this case, the application processor 1200 may merge image data provided from the depth camera with image data provided from any other camera module such as 1100a or 1100b, and may produce a three-dimensional (3D) depth image.

[0115] In an embodiment, at least two camera modules (such as 1100a and 1100b) of the plurality of camera modules 1100a, 1100b and 1100c may have different fields of view. In this case, at least two camera modules (such as 1100a and 1100b) of the plurality of camera modules 1100a, 1100b and 1100c may include different optical lenses, but are not limited thereto.

[0116] Additionally, in this embodiment, the fields of view of the multiple camera modules 1100a, 1100b, and 1100c may be different. In this case, the multiple camera modules 1100a, 1100b, and 1100c may include different optical lenses, but are not limited thereto.

[0117] In this embodiment, the multiple camera modules 1100a, 1100b, and 1100c may be configured to be physically separate from each other. That is, the multiple camera modules 1100a, 1100b, and 1100c may not use the sensing area of ​​a single image sensor 1142, but rather the multiple camera modules 1100a, 1100b, and 1100c may each include an independent image sensor 1142.

[0118] Return to Figure 12 The application processor 1200 may include an image processing device 1210, a memory controller 1220, and internal memory 1230. The application processor 1200 may be implemented separately from the plurality of camera modules 1100a, 1100b, and 1100c. For example, the application processor 1200 and the plurality of camera modules 1100a, 1100b, and 1100c may be implemented using separate semiconductor chips. In an embodiment, the application processor 1200 may include components such as those for performing the re-stitching processes of this disclosure. Figure 11 The 141 re-assembly processor.

[0119] The image processing apparatus 1210 may include a plurality of sub-image processors 1212a, 1212b and 1212c, an image generator 1214, and a camera module controller 1216.

[0120] The image processing device 1210 may include a plurality of sub-image processors 1212a, 1212b and 1212c, the number of which corresponds to the number of the plurality of camera modules 1100a, 1100b and 1100c.

[0121] Image data generated from camera modules 1100a, 1100b, and 1100c can be provided to corresponding sub-image processors 1212a, 1212b, and 1212c via separate image signal lines ISLa, ISLb, and ISLc, respectively. For example, image data generated from camera module 1100a can be provided to sub-image processor 1212a via image signal line ISLa, image data generated from camera module 1100b can be provided to sub-image processor 1212b via image signal line ISLb, and image data generated from camera module 1100c can be provided to sub-image processor 1212c via image signal line ISLc. For example, this image data transmission can be performed using a camera serial interface (CSI based on MIPI (Mobile Industrial Processor Interface)), but this disclosure is not limited thereto.

[0122] In this embodiment, a sub-image processor may be configured to correspond to multiple camera modules. For example, sub-image processors 1212a and 1212c may be implemented integratedly, rather than as shown in the example. Figure 12As shown, they are separated from each other; in this case, one of the multiple image data provided from camera module 1100a and camera module 1100c can be selected by a selection element such as a multiplexer, and the selected image data can be provided to the integrated sub-image processor.

[0123] Image data provided to sub-image processors 1212a, 1212b, and 1212c can be provided to image generator 1214. Image generator 1214 can generate an output image by using image data provided from sub-image processors 1212a, 1212b, and 1212c, respectively, based on image generation information (generation information) or a pattern signal.

[0124] More specifically, the image generator 1214 can generate an output image by merging at least a portion of image data generated from camera modules 1100a, 1100b, and 1100c, which have different fields of view, based on image generation information (generation information) or a mode signal. Furthermore, the image generator 1214 can generate an output image by selecting one of the image data generated from camera modules 1100a, 1100b, and 1100c, which have different fields of view, based on the image generation information (generation information) or the mode signal.

[0125] In an embodiment, image generation information (generation information) may include a zoom signal or a zoom factor. Additionally, in an embodiment, the mode signal may be, for example, a signal based on a mode selected by the user.

[0126] When the image generation information (generation information) is a zoom signal or zoom factor and camera modules 1100a, 1100b, and 1100c have different fields of view, image generator 1214 can perform different operations depending on the type of zoom signal. For example, when the zoom signal is a first signal, image generator 1214 can merge image data output from camera module 1100a and image data output from camera module 1100c, and can generate an output image by using the merged image signal and image data output from camera module 1100b that was not used in the merging operation. When the zoom signal is a second signal different from the first signal, without an image data merging operation, image generator 1214 can select one of the image data output from camera modules 1100a, 1100b, and 1100c respectively, and can output the selected image data as the output image. However, this disclosure is not limited to this, and the disclosed method of processing image data can be modified without limitation.

[0127] In an embodiment, the image generator 1214 can generate merged image data with increased dynamic range by receiving multiple image data with different exposure times from at least one of a plurality of sub-image processors 1212a, 1212b and 1212c and performing high dynamic range (HDR) processing on the multiple image data. In addition, in an embodiment, the image generator 1214 can generate image data with improved resolution by receiving distorted image data and then performing re-stitching processing on the distorted image data in the direction of distortion.

[0128] The camera module controller 1216 can provide control signals to camera modules 1100a, 1100b and 1100c respectively. The control signals generated from the camera module controller 1216 can be provided to the corresponding camera modules 1100a, 1100b and 1100c respectively through the separate control signal lines CSLa, CSLb and CSLc.

[0129] One of a plurality of camera modules 1100a, 1100b, and 1100c may be designated as the master camera (such as 1100b) based on image generation information (generation information) or a mode signal including a zoom signal, and the remaining camera modules (such as 1100a and 1100c) may be designated as slave cameras. The aforementioned designation information may be included in control signals, and the control signals including the designation information may be provided to the respective camera modules 1100a, 1100b, and 1100c via separate control signal lines CSLa, CSLb, and CSLc.

[0130] The camera module operating as both a master and slave camera can change based on the zoom factor or operating mode signal. For example, when the field of view of camera module 1100a is wider than that of camera module 1100b and the zoom factor indicates a low zoom ratio, camera module 1100b can operate as the master camera, and camera module 1100a can operate as the slave camera. Conversely, when the zoom factor indicates a high zoom ratio, camera module 1100a can operate as the master camera, and camera module 1100b can operate as the slave camera.

[0131] In an embodiment, the control signals provided from the camera module controller 1216 to each of the camera modules 1100a, 1100b, and 1100c may include a synchronization enable signal. For example, when camera module 1100b is used as the main camera and camera modules 1100a and 1100c are used as slave cameras, the camera module controller 1216 may send a synchronization enable signal to camera module 1100b. Camera module 1100b, provided with the synchronization enable signal, may generate a synchronization signal based on the provided synchronization enable signal and provide the generated synchronization signal to camera modules 1100a and 1100c via the synchronization signal line SSL. The synchronization signal can be used to synchronize camera module 1100b and camera modules 1100a and 1100c to send image data to application processor 1200.

[0132] In an embodiment, the control signals provided from the camera module controller 1216 to each of the camera modules 1100a, 1100b, and 1100c may include mode information based on a mode signal. Based on the mode information, the plurality of camera modules 1100a, 1100b, and 1100c may operate in a first operating mode and a second operating mode with respect to the sensing speed.

[0133] In the first operating mode, multiple camera modules 1100a, 1100b, and 1100c can generate image signals at a first speed (e.g., generate image signals at a first frame rate), encode image signals at a second speed higher than the first speed (e.g., encode image signals at a second frame rate higher than the first frame rate), and send the encoded image signals to the application processor 1200. In this case, the second speed may be 30 times or less than the first speed.

[0134] Application processor 1200 may store the received image signal (i.e., the encoded image signal) in internal memory 1230 provided therein or in external memory 1400 located outside application processor 1200. Application processor 1200 may then read the encoded image signal from internal memory 1230 or external memory 1400 and decode the encoded image signal, and may display image data generated based on the decoded image signal. For example, a corresponding one of the sub-image processors 1212a, 1212b, and 1212c of image processing apparatus 1210 may perform decoding on the encoded image signal and may also perform image processing on the decoded image signal.

[0135] In the second operating mode, multiple camera modules 1100a, 1100b, and 1100c can generate image signals at a third speed lower than the first speed and send the image signals to the application processor 1200. For example, the camera modules can generate image signals at a third frame rate lower than the first frame rate. The image signals provided to the application processor 1200 can be unencoded signals. The application processor 1200 can perform image processing on the received image signals, or it can store the image signals in internal memory 1230 or external memory 1400.

[0136] PMIC 1300 can supply power, for example, power supply voltage, to multiple camera modules 1100a, 1100b, and 1100c respectively. For example, under the control of application processor 1200, PMIC 1300 can supply first power to camera module 1100a via power signal line PSLa, supply second power to camera module 1100b via power signal line PSLb, and supply third power to camera module 1100c via power signal line PSLc.

[0137] The PMIC 1300 can generate power corresponding to each of the plurality of camera modules 1100a, 1100b, and 1100c in response to a power control signal PCON from the application processor 1200, and can adjust the power level. The power control signal PCON may include a power adjustment signal for each operating mode of the plurality of camera modules 1100a, 1100b, and 1100c. For example, the operating mode may include a low-power mode. In this case, the power control signal PCON may include information about the camera module operating in low-power mode and the set power level. The power levels supplied to the plurality of camera modules 1100a, 1100b, and 1100c may be the same or different from each other. Furthermore, the power level may be changed dynamically.

[0138] According to embodiments of this disclosure, resolution can be improved by correcting image distortion through a re-stitching process. Specifically, image resolution can be increased or decreased (e.g., for low-light conditions) by using an image sensor with a multi-pixel structure.

[0139] Although this disclosure has been described with reference to embodiments thereof, it will be apparent to those skilled in the art that various changes and modifications may be made to this disclosure without departing from the spirit and scope of the disclosure as set forth in the appended claims.

Claims

1. An electronic device comprising: An image sensor configured to generate pixel data based on light received through a lens, the lens allowing the captured image to be compressed in a first direction; as well as An image signal processor is configured to perform a re-stitching process on the generated pixel data to correct distortion occurring in the first direction, and to generate re-stitched pixel data. The image sensor includes multiple unit pixels, each unit pixel having multiple sub-pixels arranged in the first direction and corresponding floating diffusion regions shared only by the multiple sub-pixels. The image signal processor performs binning on the pixel values ​​output from the sub-pixels constituting each unit pixel in a second direction different from the first direction, and performs re-stitching on the binned pixel values.

2. The electronic device according to claim 1, wherein, The image sensor includes a pixel array, the pixel array comprising: The first unit pixel includes a first plurality of sub-pixels, a first color filter, and a first microlens; The second unit pixel includes a second plurality of sub-pixels, a second color filter, and a second microlens; and The third unit pixel includes a third plurality of sub-pixels, a third color filter, and a third microlens.

3. The electronic device according to claim 2, wherein, The re-stitching process includes increasing the resolution of the pixel data in the first direction.

4. The electronic device according to claim 3, wherein, The image signal processor performs the re-stitching process based on at least a portion of the pixel data that have the same type of pixel data.

5. The electronic device according to claim 1, wherein, The image sensor includes a pixel array, the pixel array comprising: The first unit pixel includes a first plurality of sub-pixels that share a first floating diffusion region and a first color filter; The second unit pixel includes a second plurality of sub-pixels sharing a second floating diffusion region and a second color filter; and The third unit pixel comprises a third plurality of sub-pixels that share a third floating diffusion region and a third color filter. Each of the plurality of sub-pixels includes a microlens.

6. The electronic device according to claim 5, wherein, The re-stitching process includes increasing the resolution of the pixel data in the first direction.

7. The electronic device according to claim 6, wherein, The image signal processor performs the re-stitching process based on at least a portion of the pixel data that have the same type of pixel data.

8. The electronic device according to claim 5, wherein, The image signal processor performs digital binning on at least a portion of the pixel values ​​output from the sub-pixels constituting each unit pixel, and The re-stitching process includes increasing the resolution of the pixel data in the first direction.

9. The electronic device according to claim 8, wherein, The image signal processor performs the re-stitching process based on at least a portion of the pixel data obtained as a result of the digital binning that has the same type of pixel data.

10. An electronic device comprising: An image sensor configured to generate pixel data based on light received through a lens, the lens allowing the captured image to be compressed in a first direction; as well as An image signal processor is configured to perform a re-stitching process on the generated pixel data to correct distortion occurring in the first direction, and to generate re-stitched pixel data. The image sensor includes multiple unit pixels, each unit pixel having multiple sub-pixels arranged in the first direction and a corresponding floating diffusion region shared only by the multiple sub-pixels. The image sensor includes a pixel array, which comprises: The first unit pixel includes a first plurality of sub-pixels that share a first floating diffusion region and a first color filter; The second unit pixel includes a second plurality of sub-pixels sharing a second floating diffusion region and a second color filter; and The third unit pixel includes a third plurality of sub-pixels that share a third floating diffusion region and a third color filter, and The image signal processor performs binning on the pixel values ​​output from the sub-pixels constituting each unit pixel in a second direction different from the first direction, and performs re-stitching on the binned pixel values.

11. The electronic device according to claim 10, wherein, The re-stitching process includes increasing the resolution of the pixel data in the first direction.

12. The electronic device according to claim 11, wherein, The image signal processor performs the re-stitching process based on at least a portion of the pixel data that have the same type of pixel data.

13. The electronic device according to claim 10, in, The image signal processor performs digital binning on at least a portion of the pixel values ​​output from the sub-pixels constituting each unit pixel, and The re-stitching process includes increasing the resolution of the pixel data in the first direction.

14. The electronic device according to claim 13, wherein, The image signal processor performs the re-stitching process based on at least a portion of the pixel data obtained as a result of the digital binning that has the same type of pixel data.

15. A method for processing a signal output from an image sensor, the image sensor comprising a plurality of unit pixels, each unit pixel having a plurality of sub-pixels arranged in a first direction and corresponding floating diffusion regions shared only by the plurality of sub-pixels, the plurality of unit pixels comprising a first unit pixel, a second unit pixel, and a third unit pixel, the first unit pixel comprising a first plurality of sub-pixels and a first color filter, the second unit pixel comprising a second plurality of sub-pixels and a second color filter, the third unit pixel comprising a third plurality of sub-pixels and a third color filter, the method comprising: Based on acquiring pixel values ​​from multiple sub-pixels through light received via a lens, the lens allows for compressed distortion of the captured image in a first direction; Pixel data is generated by performing correlated double sampling on the pixel values; Binning is performed on the pixel values ​​output from the sub-pixels constituting each unit pixel in a second direction different from the first direction, and Re-stitched pixel data is generated by performing a re-stitching process on the binned pixel values ​​to correct the distortion that occurs in the first direction.

16. The method according to claim 15, wherein, Generating the re-stitched pixel data includes increasing the resolution of the pixel data in the first direction.

17. The method according to claim 16, wherein, Generating the re-stitched pixel data further includes performing the re-stitching process based on at least a portion of the pixel data that have the same type of pixel data.

18. The method of claim 15, further comprising: Before generating the re-stitched pixel data, digital binning is performed on at least a portion of the pixel values ​​output from the sub-pixels that constitute each unit pixel.

19. The method according to claim 18, wherein, Generating the re-stitched pixel data includes increasing the resolution of the pixel data in the first direction.

20. The method according to claim 19, wherein, Generating the re-stitched pixel data further includes performing the re-stitching process based on at least a portion of the pixel data obtained as a result of the digital binning that has the same type of pixel data.

Images