Image sensor, electronic device, and method of operating an image sensor

Abstract

An image sensor is provided, comprising: a pixel array including a plurality of pixel groups, each of the plurality of pixel groups including a first pixel with a first conversion gain and a second pixel with a second conversion gain; a readout circuit configured to: for each of the plurality of pixel groups, receive a first pixel signal corresponding to the first pixel and a second pixel signal corresponding to the second pixel by a single readout, generate first image data based on the first pixel signals of the plurality of pixel groups, and generate second image data based on the second pixel signals of the plurality of pixel groups; and an image signal processor configured to generate output image data by merging the first image data and the second image data in units of pixel groups.

Description

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to the following applications: Korean Patent Application No. 10-2020-0067909 filed with the Korean Intellectual Property Office on June 4, 2020, and Korean Patent Application No. 10-2021-0042229 filed with the Korean Intellectual Property Office on March 31, 2021, the entire contents of each of which are incorporated herein by reference. Technical Field

[0003] At least some exemplary embodiments of the present invention relate to image sensors, and more particularly to image sensors and methods of operation thereof for generating image data with multiple conversion gains applied by a single readout. Background Technology

[0004] Image sensors can sense images of objects by using photoelectric conversion elements that react to the intensity of light reflected from the object and generate image data.

[0005] In recent years, dual conversion gain (DCG) technology has been applied to achieve high dynamic range (HDR) images from image sensors. In some example embodiments of DCG technology in the prior art, HDR images are obtained by performing multiple readouts with applied high conversion gain (HCG) and low conversion gain (LCG). Therefore, a reduction in frame rate is a problem. Summary of the Invention

[0006] At least some exemplary embodiments of the present invention provide an image sensor, an electronic device, and a method of operating the image sensor for generating image data with multiple conversion gains applied through a single readout.

[0007] According to at least some exemplary embodiments of the present invention, an image sensor includes: a pixel array including a plurality of pixel groups, each of the plurality of pixel groups including a first pixel with a first conversion gain and a second pixel with a second conversion gain; a readout circuit configured to: for each of the plurality of pixel groups, receive a first pixel signal corresponding to the first pixel and a second pixel signal corresponding to the second pixel by a single readout, generate first image data based on the first pixel signals of the plurality of pixel groups, and generate second image data based on the second pixel signals of the plurality of pixel groups; and an image signal processor configured to generate output image data by merging the first image data and the second image data on a pixel-group basis.

[0008] According to at least some exemplary embodiments of the present invention, an electronic device includes: an image sensor having a plurality of pixel groups arranged therein, each pixel group comprising a plurality of pixels, and the image sensor being configured to: generate a plurality of pixel signals corresponding to a plurality of conversion gains by a single readout for each pixel group, generate a plurality of image data corresponding to the plurality of pixel signals based on the plurality of pixel signals, and generate output image data by merging the plurality of image data; and a processor configured to perform image processing on the output image data.

[0009] According to at least some exemplary embodiments of the present invention, an operation method of an image sensor includes: outputting a first pixel signal corresponding to a first pixel corresponding to a first conversion gain from each of a plurality of pixel groups included in a pixel array by a single readout; outputting a second pixel signal corresponding to a second pixel corresponding to a second conversion gain from each of the plurality of pixel groups included in the pixel array by a single readout; generating first image data based on the first pixel signals of the plurality of pixel groups; generating second image data based on the second pixel signals of the plurality of pixel groups; and generating output image data by merging the first image data and the second image data on a pixel group basis. Attached Figure Description

[0010] The above and other features and advantages of the exemplary embodiments of the inventive concept will become more apparent from the detailed description of these embodiments with reference to the accompanying drawings. The drawings are intended to depict exemplary embodiments of the inventive concept and should not be construed as limiting the intended scope of the claims. Unless explicitly stated otherwise, the drawings should not be considered as being drawn to scale.

[0011] Figure 1 This is a diagram illustrating an image sensor and an electronic device including the image sensor according to an example embodiment of the inventive concept;

[0012] Figure 2 This is a diagram illustrating a pixel array according to an example embodiment of the inventive concept;

[0013] Figure 3 This is a diagram illustrating a pixel group of a pixel array according to an example embodiment of the inventive concept;

[0014] Figure 4 This is a diagram illustrating a readout operation during a binning operation, according to an exemplary embodiment of the present invention.

[0015] Figure 5 This is a diagram illustrating a method for generating synthetic image data according to an exemplary embodiment of the present invention;

[0016] Figures 6 to 8B This is a diagram illustrating a method for generating synthetic image data based on the pattern type of a pixel array;

[0017] Figure 9 and Figure 10 This is a diagram illustrating a method for generating multi-frame high dynamic range (HDR) image data according to an exemplary embodiment of the present invention;

[0018] Figure 11 This is a flowchart of an operation method of an image sensor according to an exemplary embodiment of the present invention;

[0019] Figure 12 This is a diagram illustrating an electronic device according to an example embodiment of the inventive concept;

[0020] Figure 13 This is a diagram illustrating a portion of an electronic device according to an example embodiment of the inventive concept; and

[0021] Figure 14 This is a diagram illustrating a specific configuration of a camera module according to an exemplary embodiment of the present invention. Detailed Implementation

[0022] As is common in the field of the inventive concept, embodiments are described and illustrated in the accompanying drawings in terms of functional blocks, units, and / or modules. Those skilled in the art will understand that these blocks, units, and / or modules are physically implemented by electronic (or optical) circuitry such as logic circuits, discrete components, microprocessors, hardwired circuitry, memory elements, wiring connections, etc., wherein said electronic (or optical) circuitry can be formed using semiconductor-based manufacturing techniques or other manufacturing techniques. Where blocks, units, and / or modules are implemented by microprocessors, etc., they can be programmed using software (e.g., microcode) to perform the various functions discussed herein, and may optionally be driven by firmware and / or software. Alternatively, each block, unit, and / or module may be implemented by dedicated hardware or as a combination of dedicated hardware performing some functions and processors performing other functions (e.g., one or more programmed microprocessors and associated circuitry). Furthermore, without departing from the scope of the inventive concept, each block, unit, and / or module of the embodiments may be physically divided into two or more interactive and discrete blocks, units, and / or modules. Furthermore, without departing from the scope of the inventive concept, the blocks, units, and / or modules of the embodiments can be physically combined into more complex blocks, units, and / or modules.

[0023] Figure 1 This is a diagram illustrating an image sensor and an electronic device including the image sensor, according to an example embodiment of the inventive concept.

[0024] Reference Figure 1 The electronic device 10 may include an image sensor 100 and a processor 200. The image sensor 100 converts light signals from an object incident through an optical lens LW into image data. The image sensor 100 can be installed in an electronic device with image or optical sensing capabilities. For example, the image sensor 100 can be installed in the electronic device 10, which may be, for example, a digital still camera, digital video camera, smartphone, wearable device, Internet of Things (IoT) device, tablet PC, personal digital assistant (PDA), portable multimedia player (PMP), navigation device, etc. The image sensor 100 can also be installed in the electronic device 10 as a component of a vehicle, furniture, manufacturing facility, door, various measuring equipment, etc. For example, the processor 200 may be an application processor and / or an image processor. Therefore, in this specification, the processor 200 may also be referred to as an image processor 200.

[0025] According to at least some exemplary embodiments of the present invention, the image processor 200 may be or include: hardware including logic circuitry; hardware / software combination executing software; or a combination thereof. For example, the image processor may more specifically include, but is not limited to, one or more of the following: a central processing unit (CPU), a processor core, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field-programmable gate array (FPGA), a programmable logic unit, a microprocessor, an application-specific integrated circuit (ASIC), etc. According to at least some exemplary embodiments of the present invention, the image processor 200 may be specifically constructed and / or programmed (e.g., via computer-executable program code) to perform and / or control some or all of the operations described herein performed by the image processor or elements of the image processor.

[0026] Reference Figure 1The image sensor 100 may include a pixel array 110, a readout circuit 120, and an image signal processor 130. In an example embodiment, the pixel array 110, the readout circuit 120, and the image signal processor 130 may be implemented together as a single semiconductor chip or semiconductor module. In an example embodiment, the pixel array 110 and the readout circuit 120 may be implemented together as a single semiconductor chip or semiconductor module, and the image signal processor 130 may be implemented as another semiconductor chip or semiconductor module. According to at least some example embodiments of the present invention, the image signal processor 130 may be or include: hardware including logic circuitry; a hardware / software combination executing software; or a combination thereof. For example, the image signal processor 130 may more specifically include, but is not limited to, one or more of the following: a central processing unit (CPU), a processor core, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field-programmable gate array (FPGA), a programmable logic unit, a microprocessor, an application-specific integrated circuit (ASIC), etc. According to at least some exemplary embodiments of the invention, the image signal processor 130 may be specifically constructed and / or programmed (e.g., via computer-executable program code) to perform and / or control some or all of the operations described herein performed by the signal processor or elements thereof.

[0027] The pixel array 110 can be specifically implemented as a photoelectric conversion element, such as a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS), or various other photoelectric conversion elements. The pixel array 110 may include multiple pixels PX for converting received optical signals (light) into electrical signals, and the multiple pixels PX may be arranged in a matrix. The pixel array 110 includes multiple row lines and multiple column lines connected to the multiple pixels PX.

[0028] Each of the multiple pixels (PX) includes an optical sensing element (or photoelectric conversion element). Examples of optical sensing elements may include photodiodes, phototransistors, photogates, pinned photodiodes, perovskite photodiodes and organic photodiodes, organic photoconductor films, etc., or various optical sensing elements may be used.

[0029] Multiple pixels (PX) can sense light through optical sensing elements and convert the sensed light into electrical signals. Each of the multiple pixels (PX) can sense light in a specific spectral region. For example, the multiple pixels may include a pixel for converting light in the red spectral region into an electrical signal (hereinafter referred to as a red pixel), a pixel for converting light in the green spectral region into an electrical signal (hereinafter referred to as a green pixel), and a pixel for converting light in the blue spectral region into an electrical signal (hereinafter referred to as a blue pixel). However, exemplary embodiments of the inventive concept are not limited thereto, and the multiple pixels (PX) may further include white pixels. As another example, the multiple pixels (PX) may include a combination of pixels of different colors (e.g., yellow pixels, cyan pixels, magenta pixels, etc.).

[0030] A color filter array for transmitting light in a specific spectral region can be set on multiple pixels PX, and the color to be sensed by the multiple pixels PX can be determined by the color filters on the multiple pixels PX. However, the exemplary embodiments of the present invention are not limited thereto, and in some exemplary embodiments of a specific optical sensing element, light of a specific wavelength band can be converted into an electrical signal based on the level of the electrical signal provided to the optical sensing element.

[0031] The charge generated by the photoelectric conversion element of each pixel in a multi-pixel PX can accumulate in a floating diffusion node, and the charge accumulated in the diffusion node can be read out by being converted into voltage. In some example embodiments, the rate at which the charge accumulated in the floating diffusion node is converted into voltage can be referred to as the conversion gain.

[0032] The conversion gain of each of the multiple pixels PX can be varied according to the capacitance of the floating diffusion node. Specifically, the conversion gain can decrease when the capacitance of the floating diffusion node increases, and the conversion gain can increase when the capacitance of the floating diffusion node decreases. The conversion gain of each of the multiple pixels PX can be changed by a conversion gain transistor (not shown) or a capacitor (not shown) connected to the floating diffusion node.

[0033] Multiple conversion gains (e.g., high conversion gain (HCG) and low conversion gain (LCG)) can be applied to multiple pixels PX. However, the inventive concept is not limited thereto, and the multiple conversion gains applied to multiple pixels PX may include three or more conversion gains. The value of HCG is higher than the value of LCG.

[0034] Figure 2 This is a diagram illustrating a pixel array according to an example embodiment of the inventive concept.

[0035] Reference Figure 2Pixel array 110 may include multiple pixel groups PG, each of which includes two or more pixels PX that are adjacent to each other. For example, pixel array 110 may include multiple pixel groups PG, each of which includes pixels PX arranged in a 2n×2n matrix (n is a positive integer). However, this disclosure is not limited to... Figure 2 The example shown. For example, pixel array 110 may include multiple pixel groups (PGs) comprising pixels (PXs) arranged in a 3N×3N matrix (N is a positive integer).

[0036] Multiple pixel groups PG are basic units of the readout method according to an exemplary embodiment of the present invention applied when the image sensor 100 operates in a first mode performing a combined operation, and the multiple pixel groups PG may correspond to multiple combined regions of image data generated based on readout signals. Pixel array 110 can output the pixel values ​​of pixels PX included in each of the multiple pixel groups PG through a single readout. During readout, the pixel values ​​of pixels PX included in a pixel group can be summed and output as at least one pixel signal.

[0037] In some example embodiments, pixel array 110 can output multiple pixel signals corresponding to multiple conversion gains of multiple pixel groups PG. In example embodiments, the multiple pixel groups PG can be divided into multiple subgroups corresponding to multiple conversion gains, and the pixel values ​​of pixels PX included in each of the multiple subgroups can be summed and output as multiple pixel signals corresponding to multiple conversion gains.

[0038] For example, when pixel group PG includes a first subgroup corresponding to high conversion gain HCG and a second subgroup corresponding to low conversion gain LCG, pixel array 110 can sum the pixel values ​​of pixels PX included in the first subgroup and output a first pixel signal, and sum the pixel values ​​of pixels PX included in the second subgroup and output a second pixel signal. The following will refer to... Figure 3 and Figure 4 This will be described in detail.

[0039] When the image sensor 100 operates in a second mode (e.g., a normal mode without combining), the pixel array 110 can read out multiple pixel signals of multiple pixels PX in rows.

[0040] The readout circuit 120 can receive pixel signals from the pixel array 110 and convert the pixel signals into digital data to generate image data (which may be referred to as an image). For ease of explanation, the image data generated by the readout circuit 120 will be referred to as the first image data IDT1 below.

[0041] For example, in the second mode where no combination operation is performed, the readout circuit 120 can generate first image data IDT1, including the pixel values ​​of multiple pixels PX, based on the pixel signal output from the pixel array 110.

[0042] As another example, in a first mode of performing the combined operation, the readout circuit 120 can receive multiple pixel signals corresponding to multiple conversion gains from multiple pixel groups PG, and generate multiple first image data IDT1 corresponding to multiple conversion gains based on the received multiple pixel signals. For example, the readout circuit 120 can generate first image data IDT1 corresponding to a first conversion gain (e.g., high conversion gain (HCG)) based on multiple first pixel signals output from a first subgroup of the multiple pixel groups PG. The readout circuit 120 can generate first image data IDT1 corresponding to a second conversion gain (e.g., low conversion gain (LCG)) based on multiple second pixel signals output from a second subgroup of the multiple pixel groups PG.

[0043] The image signal processor 130 can perform image processing on the first image data IDT1 output from the readout circuit 120. For example, the image signal processor 130 can perform image processing on the image data (e.g., the first image data IDT1) such as defect pixel correction, color correction, and image quality improvement.

[0044] According to an exemplary embodiment of the present invention, in a first mode of performing a combination operation, the image signal processor 130 can synthesize a plurality of first image data IDT1 corresponding to a plurality of conversion gains to generate output image data OIDT. In an exemplary embodiment, the image signal processor 130 can generate output image data OIDT by synthesizing the plurality of first image data IDT1 corresponding to a plurality of conversion gains in units of pixel groups PG. Reference will be made below. Figure 5 This will be described in detail.

[0045] Additionally, the image signal processor 130 can provide image-processed image data (e.g., output image data OIDT) to the image processor 200 (e.g., application processor, main processor of electronic device 10, graphics processor, etc.).

[0046] Figure 3 This is a diagram illustrating a pixel group of a pixel array according to an exemplary embodiment of the inventive concept. Specifically, Figure 3 This is a diagram showing pixel groups PG1, PG2, PG3 and PG4 of a pixel array 110 with an RGBW pattern.

[0047] Reference Figure 3A pixel array 110 with an RGBW pattern may include a first row ROW1 and a second row ROW2 in which red pixels R, white pixels W, green pixels (e.g., a first green pixel Gr) and white pixels W are arranged sequentially, and a third row ROW3 and a fourth row ROW4 in which green pixels (e.g., a second green pixel Gb), white pixels W, blue pixels B, and white pixels W are arranged sequentially. In the RGBW pattern, the white pixels W in the first row ROW1 to the fourth row ROW4 may be arranged in a diagonal direction.

[0048] Pixel array 110 may include multiple pixel groups PG1, PG2, PG3, and PG4, each pixel group comprising four adjacent pixels PX. For example, see reference... Figure 3 The pixel array 110 may include: a first pixel group PG1 (including two red pixels R and two white pixels W), a second pixel group PG2 (including two first green pixels Gr and two white pixels W), a third pixel group PG3 (including two second green pixels Gb and two white pixels W), and a fourth pixel group PG4 (including two blue pixels B and two white pixels W). That is, pixel groups PG1, PG2, PG3, and PG4 may include colored pixels of the same color and white pixels.

[0049] The pixels PX included in each of the multiple pixel groups PG1, PG2, PG3, and PG4 can be divided into multiple subgroups. In an example embodiment, the multiple pixels PX included in each of the multiple pixel groups PG1, PG2, PG3, and PG4 can be divided into multiple subgroups based on whether the multiple pixels PX are white pixels. The multiple subgroups can correspond to multiple conversion gains.

[0050] For example, refer to Figure 3 In the pixel PX of the first pixel group PG1, red pixels R (excluding white pixels W) can be grouped into a first subgroup, while white pixels W can be grouped into a second subgroup. In the pixel PX of the second pixel group PG2, first green pixels Gr can be grouped into a first subgroup, while white pixels W can be grouped into a second subgroup. In the pixel PX of the third pixel group PG3, second green pixels Gb can be grouped into a first subgroup, while white pixels W can be grouped into a second subgroup. In the pixel PX of the fourth pixel group PG4, blue pixels B can be grouped into a first subgroup, while white pixels W can be grouped into a second subgroup. The first subgroups of the multiple pixel groups PG1, PG2, PG3, and PG4 can correspond to a first conversion gain (e.g., low conversion gain (LCG)), and the second subgroups of the multiple pixel groups PG1, PG2, PG3, and PG4 can correspond to a second conversion gain (e.g., high conversion gain (HCG)). The number of conversion gains can be equal to or less than the number of subgroups.

[0051] However, the inventive concept is not limited thereto, and according to the example embodiment, multiple subgroups can be divided according to the position of the pixel PX of the group. For example, refer to Figure 3 Pixels located in the first diagonal direction can be divided into a first subgroup, and pixels located in the second diagonal direction, which is different from the first diagonal direction, can be divided into a second subgroup.

[0052] Pixel array 110 can output pixel signals of a first subgroup and pixel signals of a second subgroup for multiple pixel groups PG1, PG2, PG3 and PG4 through a single readout. For example, pixel array 110 can output pixel signals of a first subgroup including red pixels R and pixel signals of a second subgroup including white pixels W for the first pixel group PG1 through a single readout.

[0053] although Figure 3 The diagram shows pixel array 110 comprising a 4×4 pixel matrix, but the inventive concept is not limited thereto, and pixel array 110 may comprise an M×N pixel matrix (M and N are positive integers). Alternatively, pixel array 110 may have various patterns as well as RGBW patterns. For example, pixel array 110 may have an RGBY pattern in which yellow pixels Y are set instead of white pixels W.

[0054] Figure 4 This is a diagram illustrating a readout operation during a combination operation, based on an exemplary embodiment of the present invention. Specifically, Figure 4 It is used for explanation Figure 3 A diagram illustrating the readout operation of pixel array 110 with respect to pixel groups PG1, PG2, PG3 and PG4.

[0055] Reference Figure 4 When the image sensor 100 operates in a first mode performing combined operations, multiple pixel groups PG1, PG2, PG3, and PG4 can output pixel signals as subgroups. In an example embodiment, during readout, the pixel values ​​of pixels PX included in the same subgroup can be summed and output as a single pixel signal. For example, refer to... Figure 4 During the readout process, the pixel values ​​of the red pixels R included in the first subgroup of the first pixel group PG1 can be summed and output as the first pixel signal, and the pixel values ​​of the white pixels W included in the second subgroup of the first pixel group PG1 can be summed and output as the second pixel signal.

[0056] In some example embodiments, during readout, a conversion gain corresponding to the subgroup can be applied to the pixels PX included in the subgroup. For example, refer to... Figure 4When the first subgroup of the first pixel group PG1 corresponds to a first conversion gain (e.g., low conversion gain), a first pixel signal with low conversion gain can be output from the red pixel R included in the first subgroup. When the second subgroup of the first pixel group PG1 corresponds to a second conversion gain (e.g., high conversion gain), a second pixel signal with high conversion gain can be output from the white pixel W included in the second subgroup.

[0057] As described above, when the image sensor 100 performs a combined operation, it can output multiple pixel signals corresponding to multiple conversion gains from a single pixel group through a single readout.

[0058] The readout circuit 120 can generate multiple first image data IDT1_CG1 and IDT1_CG2 corresponding to multiple conversion gains based on multiple pixel signals from multiple pixel groups PG1, PG2, PG3, and PG4. In an example embodiment, the readout circuit 120 can receive first pixel signals corresponding to first conversion gains from a first subgroup of the multiple pixel groups PG1, PG2, PG3, and PG4, and generate first image data IDT1_CG1 corresponding to the first conversion gain based on the received first pixel signals.

[0059] For example, refer to Figure 4 The readout circuit 120 can receive a first pixel signal from the red pixel R included in the first subgroup of the first pixel group PG1, and calculate the first pixel value R1 corresponding to the first pixel group PG1 based on the received first pixel signal. The readout circuit 120 can receive a first pixel signal from the first green pixel Gr included in the first subgroup of the second pixel group PG2, and calculate the first pixel value Gr1 corresponding to the second pixel group PG2 based on the received first pixel signal. The readout circuit 120 can receive a first pixel signal from the second green pixel Gb included in the first subgroup of the third pixel group PG3, and calculate the first pixel value Gb1 corresponding to the third pixel group PG3 based on the received first pixel signal. The readout circuit 120 can receive a first pixel signal from the blue pixel B included in the first subgroup of the fourth pixel group PG4, and calculate the first pixel value B1 corresponding to the fourth pixel group PG4 based on the received first pixel signal. The readout circuit 120 can generate first image data IDT1_CG1 corresponding to the first conversion gain based on the first pixel values ​​R1, Gr1, Gb1 and B1 calculated corresponding to the plurality of pixel groups PG1, PG2, PG3 and PG4.

[0060] The readout circuit 120 can receive a second pixel signal corresponding to the second conversion gain from the second subgroup of multiple pixel groups PG1, PG2, PG3 and PG4, and generate first image data IDT1_CG2 corresponding to the second conversion gain based on the received second pixel signal.

[0061] For example, refer to Figure 4 The readout circuit 120 can receive a second pixel signal from the white pixels W included in the second subgroup of the first pixel group PG1, and calculate a second pixel value W1 corresponding to the first pixel group PG1 based on the received second pixel signal. The readout circuit 120 can receive a second pixel signal from the white pixels W included in the second subgroup of the second pixel group PG2, and calculate a second pixel value W2 corresponding to the second pixel group PG2 based on the received second pixel signal. The readout circuit 120 can receive a second pixel signal from the white pixels W included in the second subgroup of the third pixel group PG3, and calculate a second pixel value W3 corresponding to the third pixel group PG3 based on the received second pixel signal. The readout circuit 120 can receive a second pixel signal from the white pixels W included in the second subgroup of the fourth pixel group PG4, and calculate a second pixel value W4 corresponding to the fourth pixel group PG4 based on the received second pixel signal. Subsequently, the readout circuit 120 can generate first image data IDT1_CG2 corresponding to the second conversion gain based on the calculated second pixel values ​​W1, W2, W3 and W4 corresponding to the multiple pixel groups PG1, PG2, PG3 and PG4.

[0062] Figure 5 This is an illustration illustrating a method for generating synthetic image data according to an exemplary embodiment of the present invention. Specifically, Figure 5 This is a diagram illustrating a method for generating output image data OIDT, which is synthetic image data obtained by combining multiple first image data IDT1_CG1 and IDT1_CG2.

[0063] Reference Figure 5 The readout circuit 120 can send multiple first image data IDT1_CG1 and IDT1_CG2 corresponding to multiple conversion gains to the image signal processor 130. The image signal processor 130 can generate output image data ODIT by synthesizing the multiple first image data IDT1_CG1 and IDT1_CG2 in units of pixel groups.

[0064] For example, the image signal processor 130 can calculate the third pixel value R2 by merging the first pixel values ​​R1 and R2 of multiple first image data IDT1_CG1 and IDT1_CG2 corresponding to the first pixel group PG1. The image signal processor 130 can calculate the third pixel value Gr2 by merging the first pixel values ​​Gr1 and W2 of multiple first image data IDT1_CG1 and IDT1_CG2 corresponding to the second pixel group PG2. The image signal processor 130 can calculate the third pixel value Gb2 by merging the first pixel values ​​Gb1 and W3 of multiple first image data IDT1_CG1 and IDT1_CG2 corresponding to the third pixel group PG3. The image signal processor 130 can calculate the third pixel value B2 by merging the first pixel values ​​B1 and W4 of multiple first image data IDT1_CG1 and IDT1_CG2 corresponding to the fourth pixel group PG4. The image signal processor 130 can generate output image data OIDT based on the calculated third pixel values ​​R2, Gr2, Gb2, and B2 corresponding to multiple pixel groups PG1, PG2, PG3, and PG4.

[0065] As described above, the image sensor 100 of the exemplary embodiment of the present invention can generate multiple image data corresponding to multiple conversion gains through a single readout, so as to maintain a high frame rate and generate images with a rich color range.

[0066] Despite the above references Figures 3 to 5 The invention describes pixel array 110 as having an RGBW pattern, but the inventive concept is not limited thereto. For example, the inventive concept can also be applied to example embodiments in which pixel array 110 has a pattern different from the RGBW pattern. Reference will be made below. Figures 6 to 8B The pixel array 110 of the example embodiment of the present invention is described in the case of having different patterns.

[0067] Figures 6 to 8B This is an illustration used to illustrate a method for generating synthetic image data based on the pattern type of a pixel array. The following will describe... Figures 6 to 8B The part with is omitted. Figures 3 to 5 The description of the same part in the text.

[0068] first, Figure 6 This is a diagram illustrating a method for generating composite image data when pixel array 110a has a TETRA pattern. (Refer to...) Figure 6The pixel array 110a may have four patterns, including a red pixel group PG1 including red pixels R arranged in a 2×2 matrix, a first green pixel group PG2 including a first green pixel Gr arranged in a 2×2 matrix, a second green pixel group PG3 including a second green pixel Gb arranged in a 2×2 matrix, and a blue pixel group PG4 including blue pixels B arranged in a 2×2 matrix, which are repeatedly arranged.

[0069] The pixels PX included in each pixel group PG1, PG2, PG3, and PG4 of pixel array 110a can be divided into multiple subgroups. In the example embodiment, the multiple subgroups can be divided according to the position of the pixels PX in the pixel group. For example, referring to... Figure 7 Within a pixel group, pixels located along the first diagonal direction (e.g., pixels located on the upper left and lower right sides of the pixel group) can be grouped into a first subgroup, and pixels located along the second diagonal direction (e.g., pixels located on the upper right and lower left sides of the pixel group) can be grouped into a second subgroup.

[0070] In pixel array 110a, a first conversion gain can be applied to pixels PX included in a first subgroup, and a second conversion gain can be applied to pixels PX included in a second subgroup. Pixel array 110a can output pixel signals of the first subgroup and pixel signals of the second subgroup for multiple pixel groups PG1, PG2, PG3 and PG4 through a single readout.

[0071] For example, pixel array 110a can output, through a single readout, the first pixel signal of the first subgroup including the red pixel R located on the first diagonal direction, and the second pixel signal of the second subgroup including the red pixel R located on the second diagonal direction, for the first pixel group PG1. Pixel array 110a can output the pixel signals of other pixel groups PG2, PG3, and PG4 using the above method.

[0072] The readout circuit 120 can generate first image data IDT1_CG1 corresponding to the first conversion gain based on the first pixel signals of multiple pixel groups PG1, PG2, PG3, and PG4. For example, the readout circuit 120 can generate first image data IDT1_CG1 corresponding to the first conversion gain based on the first pixel values ​​R1, Gr1, Gb1, and B1 generated based on the first pixel signals of multiple pixel groups PG1, PG2, PG3, and PG4.

[0073] Furthermore, the readout circuit 120 can generate first image data IDT1_CG2 corresponding to the second conversion gain based on the second pixel signals of multiple pixel groups PG1, PG2, PG3, and PG4. For example, the readout circuit 120 can generate first image data IDT1_CG2 corresponding to the second conversion gain based on the second pixel values ​​R2, Gr2, Gb2, and B2 generated based on the second pixel signals of multiple pixel groups PG1, PG2, PG3, and PG4.

[0074] The image signal processor 130 can generate output image data ODIT by synthesizing the first image data IDT1_CG1 and IDT1_CG2 in units of pixel groups. For example, the image signal processor 130 can generate output image data OIDT including third pixel values ​​R3, Gr3, Gb3, and B3 by synthesizing the first image data IDT1_CG1 and IDT1_CG2 in units of multiple pixel groups PG1, PG2, PG3, and PG4.

[0075] Figure 7 This is a diagram illustrating a method for generating composite image data when pixel array 110b has a NONA pattern. (See reference...) Figure 6 The pixel array 110a may have a NONA pattern, which includes a red pixel group PG1 including red pixels R arranged in a 3×3 matrix, a first green pixel group PG2 including a first green pixel Gr arranged in a 3×3 matrix, a second green pixel group PG3 including a second green pixel Gb arranged in a 3×3 matrix, and a blue pixel group PG4 including blue pixels B arranged in a 3×3 matrix, which are repeatedly arranged.

[0076] The pixels PX in each of the pixel groups PG1, PG2, PG3, and PG4 of the pixel array 110b can be divided into multiple subgroups. In the example embodiment, the multiple subgroups can be divided according to the position of the pixels PX in the pixel group.

[0077] For example, refer to Figure 7 In a pixel group, pixels in the first column can be divided into a first subgroup, pixels in the second column into a second subgroup, and pixels in the third column into a third subgroup. However, the inventive concept is not limited to this, and for example, pixels in the first row can be divided into a first subgroup, pixels in the second row into a second subgroup, and pixels in the third row into a third subgroup.

[0078] In pixel array 110b, a first conversion gain can be applied to pixels PX included in a first subgroup, a second conversion gain can be applied to pixels PX included in a second subgroup, and a third conversion gain can be applied to pixels PX included in a third subgroup. The first to third conversion gains can be the same or different from each other. Pixel array 110b can output the pixel signals of the first, second, and third subgroups for multiple pixel groups PG1, PG2, PG3, and PG4 through a single readout.

[0079] For example, pixel array 110b can output a first pixel signal including a first subgroup of red pixels R located in the first column, a second pixel signal including a second subgroup of red pixels R located in the second column, and a third pixel signal including a third subgroup of red pixels R located in the third column. Pixel array 110b can output pixel signals of other pixel groups PG2, PG3, and PG4 using the above method.

[0080] The readout circuit 120 can generate first image data IDT1_CG1 corresponding to the first conversion gain based on the first pixel signals of multiple pixel groups PG1, PG2, PG3, and PG4. For example, the readout circuit 120 can generate first image data IDT1_CG1 corresponding to the first conversion gain based on the first pixel values ​​R1, Gr1, Gb1, and B1 generated based on the first pixel signals of multiple pixel groups PG1, PG2, PG3, and PG4.

[0081] Furthermore, the readout circuit 120 can generate first image data IDT1_CG2 corresponding to the second conversion gain based on the second pixel signals of multiple pixel groups PG1, PG2, PG3, and PG4. For example, the readout circuit 120 can generate first image data IDT1_CG2 corresponding to the second conversion gain based on the second pixel values ​​R2, Gr2, Gb2, and B2 generated based on the second pixel signals of multiple pixel groups PG1, PG2, PG3, and PG4.

[0082] Furthermore, the readout circuit 120 can generate first image data IDT1_CG3 corresponding to the third conversion gain based on the third pixel signals of multiple pixel groups PG1, PG2, PG3, and PG4. For example, the readout circuit 120 can generate first image data IDT1_CG3 corresponding to the third conversion gain based on the third pixel values ​​R3, Gr3, Gb3, and B3 generated based on the third pixel signals of multiple pixel groups PG1, PG2, PG3, and PG4.

[0083] The image signal processor 130 can generate output image data ODIT by synthesizing the first image data IDT1_CG1, IDT1_CG2, and IDT1_CG3 in units of pixel groups. In an example embodiment, the image signal processor 130 can generate output image data OIDT including fourth pixel values ​​R4, Gr4, Gb4, and B4 by synthesizing the first image data IDT1_CG1, IDT1_CG2, and IDT1_CG3 in units of multiple pixel groups PG1, PG2, PG3, and PG4.

[0084] For example, the image signal processor 1130 can calculate a fourth pixel value R4 corresponding to the first pixel group PG1 based on the pixel values ​​R1, R2, and R3 of the first image data IDT1_CG1, IDT1_CG2, and IDT1_CG3 corresponding to the first pixel group PG1. In this way, the image signal processor 130 can calculate fourth pixel values ​​Gr4, Gb4, and B4 corresponding to other pixel groups PG2, PG3, and PG4. Thereafter, the image signal processor 130 can generate output image data OIDT based on the fourth pixel values ​​R4, Gr4, Gb4, and B4.

[0085] Figure 8A and Figure 8B This is a diagram illustrating a method for generating composite image data when pixel array 110c has a HEXADECA pattern. (Refer to...) Figure 6 The pixel array 110a may have a HEXADECA pattern, which includes a red pixel group PG1 including red pixels R arranged in a 4×4 matrix, a first green pixel group PG2 including a first green pixel Gr arranged in a 4×4 matrix, a second green pixel group PG3 including a second green pixel Gb arranged in a 4×4 matrix, and a blue pixel group PG4 including blue pixels B arranged in a 4×4 matrix, which are repeatedly arranged.

[0086] The pixels PX included in each of the multiple pixel groups PG1, PG2, PG3, and PG4 of the pixel array 110c can be divided into multiple subgroups. In the example embodiment, the multiple subgroups can be divided according to the position of the pixels PX in the pixel group.

[0087] For example, refer to Figure 8AEach of the multiple pixel groups PG1, PG2, PG3, and PG4 can be divided into four subgroups. Specifically, among the pixels included in the multiple pixel groups PG1, PG2, PG3, and PG4, the top-left four pixels can be grouped into a first subgroup, the top-right four pixels into a second subgroup, the bottom-left four pixels into a third subgroup, and the bottom-right four pixels into a fourth subgroup. However, the inventive concept is not limited to this, and for example, in a pixel group, pixels located in the first row (first column) can be grouped into a first subgroup, pixels located in the second row (second column) can be grouped into a second subgroup, pixels located in the third row (third column) can be grouped into a third subgroup, and pixels located in the fourth row (fourth column) can be grouped into a fourth subgroup.

[0088] In pixel array 110c, a first conversion gain can be applied to pixels PX included in a first subgroup, a second conversion gain can be applied to pixels PX included in a second subgroup, a third conversion gain can be applied to pixels PX included in a third subgroup, and a fourth conversion gain can be applied to pixels PX included in a fourth subgroup. The first to fourth conversion gains can be the same or different from each other. Pixel array 110c can output the pixel signals of the first, second, third, and fourth subgroups through a single readout for multiple pixel groups PG1, PG2, PG3, and PG4.

[0089] For example, refer to Figure 8A The pixel array 110c can output the first pixel signal of the first subgroup PG1, including the upper left red pixel R; the second pixel signal of the second subgroup, including the upper right red pixel R; the third pixel signal of the third subgroup, including the lower left red pixel R; and the fourth pixel signal of the fourth subgroup, including the lower right red pixel R. The pixel array 110c can output the pixel signals of other pixel groups PG2, PG3, and PG4 using the same method.

[0090] The readout circuit 120 can generate first image data IDT1_CG1 corresponding to the first conversion gain based on the first pixel signals of multiple pixel groups PG1, PG2, PG3, and PG4. For example, the readout circuit 120 can generate first image data IDT1_CG1 corresponding to the first conversion gain based on the first pixel values ​​R1, Gr1, Gb1, and B1 generated from the first pixel signals of the first subgroups of multiple pixel groups PG1, PG2, PG3, and PG4.

[0091] Furthermore, the readout circuit 120 can generate second image data IDT1_CG2 corresponding to the second conversion gain based on the second pixel signals of multiple pixel groups PG1, PG2, PG3, and PG4. For example, the readout circuit 120 can generate first image data IDT1_CG2 corresponding to the second conversion gain based on the second pixel values ​​R2, Gr2, Gb2, and B2 generated from the second pixel signals of the second subgroups of multiple pixel groups PG1, PG2, PG3, and PG4.

[0092] The readout circuit 120 can generate first image data IDT1_CG3 corresponding to a third conversion gain based on the third pixel signals of multiple pixel groups PG1, PG2, PG3, and PG4. For example, the readout circuit 120 can generate first image data IDT1_CG3 corresponding to a third conversion gain based on the third pixel values ​​R3, Gr3, Gb3, and B3 generated from the third pixel signals of the third subgroups of multiple pixel groups PG1, PG2, PG3, and PG4.

[0093] The readout circuit 120 can generate first image data IDT1_CG4 corresponding to the fourth conversion gain based on the fourth pixel signals of multiple pixel groups PG1, PG2, PG3, and PG4. For example, the readout circuit 120 can generate first image data IDT1_CG4 corresponding to the fourth conversion gain based on the fourth pixel values ​​R4, Gr4, Gb4, and B4 generated from the fourth pixel signals of the fourth subgroups of multiple pixel groups PG1, PG2, PG3, and PG4.

[0094] The image signal processor 130 can generate output image data OD by synthesizing the first image data IDT1_CG1, IDT1_CG2, IDT1_CG3, and IDT1_CG4 in units of pixel groups. In an example embodiment, the image signal processor 130 can generate output image data ODIDT including fifth pixel values ​​R5, Gr5, Gb5, and B5 by synthesizing the first image data IDT1_CG1, IDT1_CG2, IDT1_CG3, and IDT1_CG4 in units of multiple pixel groups PG1, PG2, PG3, and PG4.

[0095] For example, the image signal processor 130 can calculate the fifth pixel value R5 corresponding to the first pixel group PG1 based on the pixel values ​​R1, R2, R3, and R4 of the first image data IDT1_CG1, IDT1_CG2, IDT1_CG3, and IDT1_CG4 corresponding to the first pixel group PG1. In this way, the image signal processor 130 can calculate the fifth pixel values ​​Gr5, Gb5, and B5 corresponding to other pixel groups PG2, PG3, and PG4. Thereafter, the image signal processor 130 can generate output image data OIDT based on the fifth pixel values ​​Gr5, Gb5, and B5.

[0096] For example, refer to Figure 8A Each of the multiple pixel groups PG1, PG2, PG3, and PG4 can be divided into two subgroups. Specifically, among the pixels included in each of the multiple pixel groups PG1, PG2, PG3, and PG4, the eight upper pixels can be grouped into a first subgroup, and the eight lower pixels can be grouped into a second subgroup. However, the inventive concept is not limited to this, and for example, in a pixel group, the left pixels can be grouped into a first subgroup, and the right pixels can be grouped into a second subgroup.

[0097] In pixel array 110c, a first conversion gain can be applied to pixels PX included in a first subgroup, and a second conversion gain can be applied to pixels PX included in a second subgroup. The first conversion gain and the second conversion gain can be the same or different from each other. Pixel array 110 can output pixel signals of the first subgroup and pixel signals of the second subgroup for multiple pixel groups PG1, PG2, PG3 and PG4 through a single readout.

[0098] For example, refer to Figure 8B Pixel array 110a can output the first pixel signal of the first subgroup, including the upper red pixel R, and the second pixel signal of the second subgroup, including the lower red pixel R, for the first pixel group PG1 by a single readout. Pixel array 110c can output the pixel signals of other pixel groups PG2, PG3, and PG4 using the above method.

[0099] The readout circuit 120 can generate first image data IDT1_CG1 corresponding to the first conversion gain based on the first pixel signals of multiple pixel groups PG1, PG2, PG3 and PG4. For example, the readout circuit 120 can generate first image data TDT1_CG1 corresponding to the first conversion gain based on the first pixel values ​​R1, Gr1, Gb1 and B1 generated from the first pixel signals of the first subgroups of multiple pixel groups PG1, PG2, PG3 and PG4.

[0100] Furthermore, the readout circuit 120 can generate second image data IDT1_CG2 corresponding to the second conversion gain based on the second pixel signals of multiple pixel groups PG1, PG2, PG3, and PG4. For example, the readout circuit 120 can generate first image data IDT1_CG2 corresponding to the second conversion gain based on the second pixel values ​​R2, Gr2, Gb2, and B2 generated from the second pixel signals of the second subgroups of multiple pixel groups PG1, PG2, PG3, and PG4.

[0101] The image signal processor 130 can generate output image data OIDT by synthesizing the first image data IDT1_CG1 and IDT1_CG2 in units of pixel groups. In an example embodiment, the image signal processor 130 can generate output image data OIDT including third pixel values ​​R3, Gr3, Gb3, and B3 by synthesizing the first image data IDT1_CG1 and IDT1_CG2 in units of multiple pixel groups PG1, PG2, PG3, and PG4.

[0102] For example, refer to Figure 8B The image signal processor 130 can calculate a third pixel value R3 corresponding to the first pixel group PG1 based on pixel values ​​R1 and R2 of the first image data IDT1_CG1 and IDT1_CG2 corresponding to the first pixel group PG1. In this way, the image signal processor 130 can calculate third pixel values ​​Gr3, Gb3, and B3 corresponding to other pixel groups PG2, PG3, and PG4. Furthermore, the image signal processor 130 can generate output image data OIDT based on the third pixel values ​​R3, Gr3, Gb3, and B3.

[0103] Figure 9 and Figure 10 This diagram illustrates a method for generating multi-frame high dynamic range (HDR) image data according to an exemplary embodiment of the present invention. Although for ease of illustration, it is assumed that the pixel array 110 has an RGBW pattern, the inventive concept is not limited thereto, and the following description can be substantially applied to exemplary embodiments in which the pixel array 110 has an RGBY pattern, a TETRA pattern, a NONA pattern, or a HEXADECA pattern. The following will describe… Figure 9 and Figure 10 The part with is omitted. Figure 3 The description of the parts that are the same as those in Figure 8.

[0104] Reference Figure 9 , Figure 1 Electronic device 10 can be referenced above Figure 3 The method described in Figure 8 generates multiple synthetic image data for each of multiple frame segments (e.g., Figure 5The electronic device 10 can generate multi-frame HDR images by synthesizing multiple generated composite image data.

[0105] In the example embodiment, the image sensor 100 can be referenced above. Figure 3 The method described in Figure 8 generates first image data IDT1_CG1 corresponding to the first gain and first image data IDT1_CG2 corresponding to the second conversion gain in the first frame portion tFRAME1 corresponding to the first exposure. The image signal processor 130 of the image sensor 100 can be referenced above. Figure 3 The method described in Figure 8 merges the first image data IDT1_CG1 and IDT1_CG2 corresponding to the first frame portion tFRAME1 to generate the first output image data OIDT1.

[0106] Image sensor 100 can be referenced above Figure 3 The method described in Figure 8 generates second image data IDT2_CG1 corresponding to the first gain and second image data IDT2_CG2 corresponding to the second conversion gain in the second frame portion tFRAME2 corresponding to the second exposure. The second exposure may have a different exposure period than the first exposure. The image signal processor 130 of the image sensor 100 can be referenced above. Figure 3 The method described in Figure 8 merges the second image data IDT2_CG1 and IDT2_CG2 corresponding to the second frame portion tFRAME2 to generate the second output image data OIDT2.

[0107] Reference Figure 10 The image signal processor 130 can provide the first output image data OIDT1 and the second output image data OIDT2 to the processor 200. The processor 200 can generate composite image data IDT_HDR by merging the first output image data OIDT1 and the second output image data OIDT2 in units of pixel groups.

[0108] For example, the processor 200 can calculate the first pixel value R5 of the synthesized image data IDT_HDR based on the pixel values ​​R2 and R4 of the first output image data OIDT1 and the second output image data OIDT2 corresponding to the first pixel group. For example, the processor 200 can calculate other pixel values ​​Gr5, Gb5, and B5 of the synthesized image data IDT_HDR based on the pixel values ​​of the first output image data OIDT1 and the second output image data OIDT2 corresponding to other pixel groups.

[0109] As described above, the electronic device 10 of the exemplary embodiment of the present invention can generate multi-frame HDR image data by using multiple output image data corresponding to each of the multiple frame portions in the above method, thereby significantly increasing the color range.

[0110] Despite the above references Figure 9 and Figure 10 The invention describes how an electronic device 10 generates multi-frame HDR image data by using two output image data corresponding to two frame portions. However, the invention concept is not limited thereto, and multi-frame HDR image data can be generated using three output image data corresponding to three or more frame portions.

[0111] Figure 11 This is a flowchart illustrating an operation method of an image sensor according to an exemplary embodiment of the present invention. Specifically, Figure 11 yes Figure 1 A flowchart of the operation method of the image sensor 100.

[0112] Reference Figure 11 The image sensor 100 can output a first pixel signal corresponding to a first pixel with a first conversion gain applied from each of a plurality of pixel groups included in the pixel array by a single readout (S110). Specifically, the image sensor 100 can output the first pixel signal by summing the first pixel values ​​of the first pixel in each of the plurality of pixel groups.

[0113] Next, the image sensor 100 can output a second pixel signal corresponding to a second pixel with a second conversion gain applied from each of the plurality of pixel groups included in the pixel array via a single readout (S120). Specifically, the image sensor 100 can output the second pixel signal by summing the second pixel values ​​of the second pixels in each of the plurality of pixel groups. S110 and S120 can be performed simultaneously via a single readout.

[0114] Image sensor 100 can generate first image data based on first pixel signals of multiple pixel groups (S130). Image sensor 100 can generate second image data based on second pixel signals of multiple pixel groups (S140). Thereafter, image sensor 100 can generate output image data by merging the first image data and the second image data in units of pixel groups (S150).

[0115] Figure 12 This is a diagram illustrating an electronic device according to an example embodiment of the inventive concept.

[0116] Reference Figure 12Electronic device 1000 may include image sensor 1100, application processor 1200, display 1300, memory 1400, storage device 1500, user interface 1600 and wireless transceiver 1700. Figure 12 The image sensor 1100 can correspond to Figure 1 The image sensor 100, and Figure 14 Application processor 1200 can correspond to Figure 1 Processor 200. (The last part, "omitted," appears to be a fragment and doesn't translate directly.) Figure 1 The description of the same part in the text.

[0117] The application processor 1200 can control the overall operation of the electronic device 1000 and can be provided in the form of a system-on-a-chip (SoC) for driving applications, operating systems, etc.

[0118] Memory 1400 may store programs and / or data to be processed or executed by application processor 1200. Storage device 1500 may be specifically implemented as a non-volatile storage device such as NAND flash memory or resistive memory, and may be provided, for example, in the form of a memory card (MMC, eMMC, SD, or micro SD). Storage device 1500 may store data related to the execution algorithms of image processing operations used to control application processor 1200 and / or programs, and when image processing operations are performed, data and / or programs may be loaded into memory 1400.

[0119] User interface 1600 can be specifically implemented as various types of devices capable of receiving user input, such as a keyboard, curtain key panel, touch panel, fingerprint sensor, or microphone. User interface 1600 can receive user input and provide signals corresponding to the received user input to application processor 1200. Wireless transceiver 1700 may include modem 1710, transceiver 1720, and antenna 1730.

[0120] Figure 13 This is a diagram illustrating a portion of an electronic device according to an example embodiment of an inventive concept. Figure 14 This is a diagram illustrating a specific configuration of a camera module according to an exemplary embodiment of the present invention. Specifically, Figure 13 It is shown as Figure 12 A diagram of electronic device 2000, which is a part of electronic device 1000, and Figure 14 It shows Figure 13 A diagram showing the specific configuration of the 2100b camera module.

[0121] Reference Figure 13The electronic device 2000 may include a multi-camera module 2100, an application processor 2200, and a memory 2300. The memory 2300 can perform operations related to... Figure 12 The memory 1400 has the same function as the memory 2300, so the description of the parts in the memory 2300 that are the same as those in the memory 1400 will be omitted.

[0122] Electronic device 2000 can capture and / or store images of objects using a CMOS image sensor, and can be specifically implemented as a mobile phone, tablet computer, or portable electronic device. Portable electronic devices can include laptop computers, mobile phones, smartphones, tablet PCs, wearable devices, etc.

[0123] The multi-camera module 2100 may include a first camera module 2100a, a second camera module 2100b, and a third camera module 2100c. The multi-camera module 2100 can perform operations related to... Figure 1 It has the same function as the image sensor 100. Although Figure 13 The multi-camera module 2100 is shown to include three camera modules 2100a to 2100c, but the inventive concept is not limited thereto, and the multi-camera module 2100 may include a variety of numbers of camera modules.

[0124] The following will refer to Figure 14 The configuration of camera module 2100b is described in more detail, but the following description is also applicable to other camera modules 2100a and 2100b according to the example embodiments.

[0125] Reference Figure 14 The camera module 2100b may include a prism 2105, an optical path folding element (OPFE) 2110, an actuator 2130, an image sensing device 2140, and a memory 2150.

[0126] The prism 2105 can change the path of light L incident from the outside, which includes a reflective surface 2107 of a light-reflecting material.

[0127] In an example embodiment, prism 2105 can change the path of light incident in the first direction X to a second direction Y perpendicular to the first direction X. Alternatively, prism 2105 can change the path of light L incident in the first direction X to a second direction Y perpendicular to the first direction X by rotating the reflective surface 2107 of the light-reflecting material about the central axis 2106 in direction A or about the central axis 2106 in direction B. In some example embodiments, OPFE 2110 can be moved in a third direction Z perpendicular to the first direction X and the second direction Y.

[0128] OPFE 2110 may include, for example, m groups of optical lenses (where m is a natural number). The m groups of lenses can be moved in the second direction Y to change the optical zoom ratio of camera module 2100b.

[0129] Actuator 2130 can move OPFE 2110 or optical lens (hereinafter referred to as optical lens) to a position. For example, for accurate sensing, actuator 2130 can adjust the position of optical lens so that image sensor 2142 is located at the focal length of optical lens.

[0130] Image sensing device 2140 may include image sensor 2142, control logic device 2144, and memory 2146. Image sensor 2142 can sense an image of the object to be sensed by using light L provided through an optical lens. Figure 14 The image sensor 2142 is functionally compatible with... Figure 1 The image sensor 100 is similar to the second camera module 2100b, so its redundant description will be omitted. The control logic device 2144 can control the overall operation of the second camera module 2100b.

[0131] The memory 2146 may store information necessary for operating the second camera module 2100b, such as calibration data 2147. Calibration data 2147 may include information necessary for the second camera module 2100b to generate image data using externally supplied light L. Calibration data 2147 may include information such as rotation, focal length, and optical axis. When the second camera module 2100b is in the form of a multi-state camera where the focal length varies depending on the position of the optical lens, calibration data 2147 may include the focal length value for each position (or state) of the optical lens, as well as information related to autofocus.

[0132] The memory 2150 can store image data sensed by the image sensor 2142. The memory 2150 can be located external to the image sensing device 2140 and can be stacked together with the sensor chip of the image sensing device 2140. In an example embodiment, the memory 2150 can be specifically implemented as an electrically erasable programmable read-only memory (EEPROM), but the example embodiment is not limited thereto.

[0133] Reference Figure 13 and Figure 14 In an example embodiment, each of the plurality of camera modules 2100a, 2100b, and 2100c may include an actuator 2130. Therefore, depending on the operation of the actuators 1130 in the plurality of camera modules 2100a, 2100b, and 2100c, the plurality of camera modules 2100a, 2100b, and 2100c may include the same or different calibration data 2147.

[0134] In the example embodiment, one of the camera modules 2100a, 2100b and 2100c (e.g., the second camera module 2100b) may be a folding lens type camera module including prism 2105 and OPFE 2110, and the other camera modules (e.g., camera modules 2100a and 2100b) may be vertical camera modules excluding prism 2105 and OPFE 2110, but the example embodiment is not limited thereto.

[0135] In example embodiments, one of the camera modules 2100a, 2100b, and 2100c (e.g., the third camera module 2100c) may be a vertical depth camera, for example, that extracts depth information using infrared (IR). In some example embodiments, the application processor 2200 may generate a three-dimensional (3D) depth image by merging image data provided from the depth camera and image data provided from another camera module (e.g., the first camera module 2100a or the second camera module 2100b).

[0136] In example embodiments, at least two camera modules of the plurality of camera modules 2100a, 2100b, and 2100c (e.g., first camera module 2100a and second camera module 2100b) may have different fields of view (FOVs) (different angles of view). In some example embodiments, for example, at least two camera modules of the plurality of camera modules 2100a, 2100b, and 2100c (e.g., camera modules 2100a and 2100b) may include different optical lenses, but the example embodiments are not limited thereto. For example, the FOV of the first camera module 2100a of the plurality of camera modules 2100a, 2100b, and 2100c may be smaller than the FOV of the second camera module 2100b and the third camera module 2100c. However, the example embodiments are not limited thereto, and the multiple camera modules 2100 may further include camera modules having a larger FOV than the FOV of the initially used camera modules 2100a, 2100b, and 2100c.

[0137] In some example embodiments, the viewing angles of the multiple camera modules 2100a, 2100b, and 2100c may be different from each other. In some example embodiments, the optical lenses included in the multiple camera modules 2100a, 2100b, and 2100c may be different from each other, but the example embodiments are not limited thereto.

[0138] In some example embodiments, the multiple camera modules 2100a, 2100b, and 2100c can be arranged to be physically separated from each other. That is, the sensing area of ​​an image sensor 2142 is not divided and used by the multiple camera modules 2100a, 2100b, and 2100c, but the image sensor 1142 can be independently disposed in each of the multiple camera modules 2100a, 2100b, and 2100c.

[0139] Application processor 2200 may include multiple subprocessors 2210a, 2210b, and 2210c, camera module controller 2230, memory controller 2400, and internal memory 2500. Application processor 2200 may be separate from the multiple camera modules 2100a, 2100b, and 2100c. For example, application processor 2200 and the multiple camera modules 2100a, 2100b, and 2100c may be separate semiconductor chips.

[0140] Image data generated by camera module 2100a, camera module 2100b, and camera module 2100c can be provided to their respective subprocessors 2210a, 2210b, and 2210c via separate image signal lines ISLa, ISLb, and ISLc. Such image data can be transmitted, for example, using a Camera Serial Interface (CSI) based on a Mobile Industrial Processor Interface (MIPI), but the example embodiment is not limited thereto.

[0141] In an example embodiment, a subprocessor may be arranged to correspond to multiple camera modules. For example, unlike that shown in the figures, the first subprocessor 2210a and the third subprocessor 2210b may not be separate from each other, but may be integrated together as a single subprocessor, and image data provided from camera modules 2100a and 2100c may be selected by a selector (e.g., a multiplexer) and provided to the subprocessor.

[0142] Subprocessors 2210a, 2210b and 2210c can perform image processing on the received image data and output the processed image data to image generator 2220.

[0143] The camera module controller 2230 can provide control signals to camera modules 2100a, 2100b, and 2100c. The control signals generated by the camera module controller 2230 can be provided to camera modules 2100a, 2100b, and 2100c through separate control signal lines CSLa, CSLb, and CSLc.

[0144] Exemplary embodiments of the inventive concept have been described herein, and it will be apparent that they can be modified in various ways. These changes should not be considered as departing from the intended spirit and scope of the exemplary embodiments of the inventive concept, and it will be apparent to those skilled in the art that all such modifications are intended to be included within the scope of the appended claims.

Claims

1. An image sensor, comprising: A pixel array comprising multiple pixel groups, each of the multiple pixel groups comprising pixels arranged in a 2n×2n matrix or a 3n×3n matrix, where n is a positive integer, and the pixels comprising a first pixel with a first conversion gain applied and a second pixel with a second conversion gain applied; The readout circuit is configured to: for each of the plurality of pixel groups, receive a first pixel signal corresponding to the first pixel as a pixel signal and a second pixel signal corresponding to the second pixel as a pixel signal by a single readout; generate first image data based on the first pixel signal of the plurality of pixel groups; and generate second image data based on the second pixel signal of the plurality of pixel groups. as well as An image signal processor is configured to generate output image data by merging the first image data and the second image data in units of pixel groups.

2. The image sensor according to claim 1, wherein, For each of the plurality of pixel groups, the first pixel signal is read by summing the first pixel value of the first pixel, and the second pixel signal is read by summing the second pixel value of the second pixel.

3. The image sensor according to claim 1, wherein, Each of the first conversion gain and the second conversion gain is either a high conversion gain or a low conversion gain.

4. The image sensor according to claim 1, wherein, Each of the plurality of pixel groups includes: There are 2n first pixels and 2n second pixels, where n is a positive integer.

5. The image sensor according to claim 4, in, The first pixel of each of the plurality of pixel groups includes: Red pixel, blue pixel, first green pixel or second green pixel, and Wherein, the second pixel of each of the plurality of pixel groups includes: White pixels or yellow pixels.

6. The image sensor according to claim 4, wherein, The first and second pixels of each of the plurality of pixel groups include: Red pixel, blue pixel, first green pixel or second green pixel.

7. The image sensor according to claim 1, wherein, Each of the plurality of pixel groups includes: The third pixel is subject to a third conversion gain.

8. The image sensor according to claim 7, wherein, The first pixel, second pixel, and third pixel of each of the plurality of pixel groups include: Red pixel, blue pixel, first green pixel or second green pixel.

9. The image sensor according to claim 7, wherein, The readout circuit is further configured to: For each of the plurality of pixel groups, a third pixel signal corresponding to the third pixel to which the third conversion gain is applied is received by a single readout, and The third image data is generated based on the third pixel signal of the plurality of pixel groups.

10. The image sensor according to claim 9, wherein, The image signal processor is further configured to generate the output image data by merging the first image data, the second image data, and the third image data in units of pixel groups.

11. The image sensor according to claim 1, wherein, Each of the plurality of pixel groups includes: The third pixel with a third conversion gain and the fourth pixel with a fourth conversion gain.

12. The image sensor according to claim 11, wherein, The readout circuit is further configured to: For each of the plurality of pixel groups, the third pixel signal corresponding to the third pixel and the fourth pixel signal corresponding to the fourth pixel are received individually through readout. Third image data is generated based on the third pixel signal of the plurality of pixel groups, and fourth image data is generated based on the fourth pixel signal of the plurality of pixel groups.

13. The image sensor according to claim 12, wherein, The image signal processor is further configured to generate the output image data by merging the first image data, the second image data, the third image data, and the fourth image data in units of pixel groups.

14. An electronic device, comprising: An image sensor having multiple pixel groups arranged therein, each pixel group comprising multiple pixels arranged in a 2n×2n matrix or a 3n×3n matrix, where n is a positive integer, wherein the image sensor is configured as follows: For each of the plurality of pixel groups, multiple pixel signals corresponding to multiple conversion gains are generated by reading out a single pixel signal. The pixel signal corresponding to one of the multiple conversion gains is generated as a single pixel signal. Based on the plurality of pixel signals, multiple image data corresponding to the plurality of pixel signals are generated, and Output image data is generated by merging the multiple image data sets; and The processor is configured to perform image processing on the output image data.

15. The electronic device according to claim 14, wherein, Each of the plurality of pixel groups includes: The pixels corresponding to the plurality of conversion gains, and The image sensor is further configured to generate a plurality of pixel signals corresponding to the plurality of conversion gains by summing the pixel values ​​of the pixels corresponding to one of the plurality of conversion gains for each of the plurality of pixel groups.

16. The electronic device according to claim 14, wherein, Each of the plurality of pixel groups includes: The first RGB pixel corresponding to the first conversion gain; and The white pixel corresponding to the second conversion gain, and The image sensor is further configured to generate a first pixel signal corresponding to the first RGB pixel and a second pixel signal corresponding to the white pixel for each of the plurality of pixel groups.

17. The electronic device according to claim 14, wherein, Each of the plurality of pixel groups includes: The first RGB pixel corresponding to the first conversion gain; and The second RGB pixel corresponding to the second conversion gain, and The image sensor is further configured to generate a first pixel signal corresponding to the first RGB pixel and a second pixel signal corresponding to the second RGB pixel for each of the plurality of pixel groups.

18. The electronic device according to claim 14, wherein, The image sensor is further configured to generate first output image data corresponding to a portion of the first frame and second output image data corresponding to a portion of the second frame, and The processor is also configured to generate multi-frame image data by merging the first output image data and the second output image data.

19. A method for operating an image sensor, the method comprising: Each pixel group in the pixel array is read out individually and output as a pixel signal corresponding to the first pixel corresponding to the first conversion gain. Each pixel group in the pixel array includes pixels arranged in a 2n×2n matrix or a 3n×3n matrix, where n is a positive integer. From each of the plurality of pixel groups included in the pixel array, a second pixel signal corresponding to the second pixel corresponding to the second conversion gain is output as a pixel signal through the single readout; First image data is generated based on the first pixel signal of the plurality of pixel groups; Second image data is generated based on the second pixel signals of the plurality of pixel groups; as well as Output image data is generated by merging the first image data and the second image data in units of pixel groups.

20. The operating method according to claim 19, wherein, Outputting the first pixel signal includes: summing the first pixel values ​​of the first pixel to output the first pixel signal, and Outputting the second pixel signal includes: outputting the second pixel signal by summing the second pixel values ​​of the second pixel.

Images