Image sensor and operation method thereof, and image device

Abstract

To provide an image sensor having improved performance, an operation method thereof, and an image device.SOLUTION: An image sensor of the present invention includes: a pixel array including a plurality of pixels; a row driver configured to control the plurality of pixels; and an analog-to-digital converter configured to digitize a result sensed by the pixel array to generate a first image. The pixel array includes: first pixel groups, each of the first pixel groups including first white pixels and first color pixels among the plurality of pixels; and second pixel groups, each of the second pixel groups including second white pixels and second color pixels among the plurality of pixels. First pixel data of the first image are generated based on the first white pixels and the first color pixels, and second pixel data of the first image are generated based on the second color pixels.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to semiconductor devices, and more particularly to an image sensor and a method for operating the same, and an imaging device. [Background technology]

[0002] Image sensors included in smartphones, tablet PCs, digital cameras, etc. convert light waves reflected from external objects into electrical signals to acquire image information about the external objects. Various image signal processing operations are performed on the electrical signals acquired from the image sensor to convert them into image information that can be actually recognized by humans or to improve image quality.

[0003] In recent years, image sensors with multi-color filter arrays have been widely adopted, as well as pixels with shared floating diffusion regions. In particular, the application of white pixels can improve sensing sensitivity. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 8,139,130 [Patent Document 2] U.S. Patent No. 8,330,839 [Patent Document 3] U.S. Patent No. 7,855,740 [Patent Document 4] U.S. Patent No. 7,889,921 [Patent Document 5] U.S. Patent No. 7,876,956 [Patent Document 6] U.S. Patent No. 8,452,082 [Patent Document 7] US Patent Application Publication No. 2020 / 0195872 [Patent Document 8] US Patent Application Publication No. 2016 / 0057367 [Patent Document 9] U.S. Patent Application Publication No. 2021 / 0029309 [Patent Document 10] European Patent Application Publication No. 3751840 [Patent Document 11] Chinese Patent Publication No. 111741277 [Patent Document 12] Korean Patent Registration No. 10-1637671 [Patent Document 13] Korean Patent No. 10-1332689 [Patent Document 14] Korean Patent Registration No. 10-1923957 [Non-patent literature]

[0005] [Non-Patent Document 1] Eric Dubois, Frequency-Domain Methods for Demosaicking of Bayer-Sampled Color Images, IEEE SIGNAL PROCESSING LETTERS, VOL. 12, NO. 12, DECEMBER 2005 [Non-patent document 2] Kaiming He et al., Guided Image Filtering, 2010 [Non-patent document 3] Chulhee Park et al., G-Channel Restoration for RWB CFA with Double-Exposed W Channel, Sensors 2017, 17, 293; doi:10.3390 / s17020293 Summary of the Invention [Problem to be solved by the invention]

[0006] SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned conventional techniques, and an object of the present invention is to provide an image sensor having improved performance, an operating method thereof, and an imaging device. [Means for solving the problem]

[0007] In order to achieve the above object, an image sensor according to one aspect of the present invention comprises a pixel array including a plurality of pixels, a row driver configured to control the plurality of pixels, and an analog-to-digital converter configured to digitize the results sensed by the pixel array to generate a first image, wherein the pixel array includes a first pixel group each including a first white pixel and a first color pixel from the plurality of pixels, and a second pixel group each including a second white pixel and a second color pixel from the plurality of pixels, and wherein first pixel data from the first image is generated based on the first white pixel and the first color pixel, and second pixel data from the first image is generated based on the second color pixel.

[0008] In order to achieve the above object, according to one aspect of the present invention, a method for operating an image sensor including a first pixel group and a second pixel group includes: a first readout step, performing a first summation operation for the first pixel group including a first white pixel and a first color pixel, sampling and outputting a first signal based on the first white pixel and the first color pixel; a second readout step, performing a second summation operation for the second pixel group including a second white pixel and a second color pixel, sampling and outputting a second signal based on the second color pixel but not based on the second white pixel; converting the first signal and the second signal into digital signals to generate a low image; extracting a white image based on the low image; performing a crosstalk compensation operation based on the white image to generate a crosstalk-compensated white image; and performing a subtraction operation using the crosstalk-compensated white image on the low image to generate a Bayer image.

[0009] In order to achieve the above object, one aspect of the present invention provides an image device comprising: an image sensor including a pixel set, the image sensor configured to generate a raw image by performing a first summation operation for each of first through third pixel groups of the pixel set to generate first pixel data based on all pixels that share a floating diffusion region, and a second summation operation for a fourth pixel group of the pixel set to generate second pixel data based on only a portion of the pixels that share a floating diffusion region, and an image signal processor configured to perform a signal processing operation on image data received from the image sensor, the pixel set including first through fourth pixel groups, the first pixel group including a first white pixel and a first green pixel, the second pixel group including a second white pixel and a red pixel, the third pixel group including a third white pixel and a blue pixel, and the fourth pixel group including a fourth white pixel and a second green pixel. [Effects of the Invention]

[0010] According to the present invention, the image sensor can generate a non-Bayer pattern image and convert the non-Bayer pattern image to a Bayer pattern image in a single readout operation, thereby improving the overall performance of the image sensor. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram illustrating an example of an image device according to one embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram illustrating an example of an image signal processor of FIG. 1. [Figure 3] FIG. 2 is a block diagram illustrating an example of the image sensor of FIG. 1. [Figure 4A] 4 is a circuit diagram showing an example of a pixel group PG in FIG. 3. FIG. [Figure 4B]FIG. 4B is a diagram illustrating a pixel set including the pixel group of FIG. 4A. [Figure 4C] FIG. 4B is a diagram illustrating a pixel set including the pixel group of FIG. 4A. [Figure 5] FIG. 2 is a timing diagram illustrating an example of signal waveforms applied to a pixel set. [Figure 6] 4 is a diagram showing an example of a raw image output from the ADC of FIG. 3. [Figure 7] 2 is a flowchart showing the operation of the image sensor of FIG. 1. [Figure 8] 8 is a flowchart showing step S110 of FIG. 7 in more detail. [Figure 9] FIG. 4 is a block diagram illustrating the white extraction module of FIG. 3 in more detail. [Figure 10] FIG. 10 is a block diagram illustrating the guide filtering module of FIG. 9 in more detail. [Figure 11A] FIG. 10 is a diagram illustrating a method for converting a raw image into a Bayer image. [Figure 11B] FIG. 10 is a diagram illustrating a method for converting a raw image into a Bayer image. [Figure 11C] FIG. 10 is a diagram illustrating a method for converting a raw image into a Bayer image. [Figure 12A] 1A-1D are block diagrams illustrating various examples of image devices according to an embodiment of the present invention. [Figure 12B] 1A-1D are block diagrams illustrating various examples of image devices according to an embodiment of the present invention. [Figure 12C] 1A-1D are block diagrams illustrating various examples of image devices according to an embodiment of the present invention. [Figure 13] 1 is a block diagram showing an example of the configuration of an electronic device including a multi-camera module according to an embodiment of the present invention. [Figure 14] FIG. 14 is a block diagram showing an example of the configuration of the camera module of FIG. 13. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, specific examples of embodiments of the present invention will be described in detail with reference to the drawings.

[0013] As is common in the art, the embodiments are described and illustrated in terms of described functions or blocks that perform functions. These blocks, referred to herein as units, modules, etc., or by names such as converters, processors, controllers, etc., are physically realized by analog or digital circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, wired circuits, etc., and are driven by firmware and software. The circuits are implemented on a substrate support such as one or more semiconductor chips or a printed circuit board. The circuits included in the blocks are realized by dedicated hardware or processors (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware that performs some functions of the block and a processor that performs other functions of the block. Each block of the embodiments is physically divided into two or more interacting individual blocks. Similarly, the blocks of the embodiments are physically combined into more complex blocks.

[0014] 1 is a block diagram illustrating an example of an image device according to an embodiment of the present invention. Referring to FIG. 1, the image device 100 includes an image sensor 110 and an image signal processor 120. The image device 100 may be included in a variety of computing systems, such as a desktop computer, a laptop computer, a tablet computer, a smartphone, a wearable device, a black box, a digital camera, etc.

[0015] The image sensor 110 outputs image data based on light waves incident from the outside. For example, the outside may refer to the outside of the image sensor 110, the outside of the image device 100, or the outside of a computing system or other device that includes the image sensor 110 or the image device 100. For example, the image sensor 110 includes a plurality of pixels. Each of the plurality of pixels is configured to output an electrical signal corresponding to light incident from the outside. The image sensor 110 outputs image data based on the electrical signal. In one embodiment, the image data output from the image sensor 110 includes brightness information, color information, etc. for a particular frame.

[0016] For example, the image sensor 110 is realized by a CMOS (Complementary Metal Oxide Semiconductor) image sensor, etc. For example, the image sensor 110 is a multi-pixel image sensor having a dual pixel structure or a tetracell structure.

[0017] Four pixels among the plurality of pixels of the image sensor 110 share one floating diffusion region. This will be described in more detail in FIG. 4A. Hereinafter, for convenience of explanation, an operation of sampling and outputting signal voltage levels based on four pixels sharing one floating diffusion region will be referred to as a first summing operation (e.g., FD4sum), an operation of sampling and outputting signal voltage levels based on two pixels sharing one floating diffusion region will be referred to as a second summing operation (e.g., FD2sum), and performing the first summing operation on a first portion of the plurality of pixels and the second summing operation on a second portion of the plurality of pixels will be referred to as a third summing operation (e.g., FD4sum+FD2sum).

[0018] For ease of explanation, hereinafter, the basic unit of the first type array pattern includes four pixels, including a green pixel Gr, a red pixel R, a green pixel Gb, and a blue pixel B, which are arranged sequentially clockwise from the upper left corner. The first type array pattern is referred to as a Bayer pattern. The basic unit of the second type array pattern includes one pixel, including a white pixel W. The second type array pattern is referred to as a white pattern.

[0019] An image sensor that uses a general RGBW pattern performs two readout operations: the image sensor performs a second summing operation in the first readout operation to generate a Bayer pattern image, and then performs a second summing operation in the second readout operation to generate a white pattern image.

[0020] Meanwhile, the image sensor according to this embodiment generates a raw image RAW by performing one read operation. The image sensor 110 generates a non-Bayer pattern raw image RAW by performing a third summation operation. The image sensor 110 extracts a Bayer pattern image (hereinafter referred to as a "Bayer image") from the raw image RAW. This allows the image sensor 110 to improve the frame rate and reduce analog power consumption.

[0021] The image sensor 110 includes a white extraction module 111, a crosstalk compensation module 112, and a Bayer extraction module 113. The structures and operation methods of the white extraction module 111, the crosstalk compensation module 112, and the Bayer extraction module 113 according to this embodiment will be described in more detail with reference to the following drawings.

[0022] The image signal processor 120 receives image data from the image sensor 110 and performs various signal processing operations on the received image data. In one embodiment, the image signal processor 120 performs various signal processing operations on the received image data, such as noise reduction, white balance, gamma correction, color correction, color conversion, etc. The processed image data is transferred to an external device (e.g., a display device) or stored in a separate storage device.

[0023] Figure 2 is a block diagram showing an example of the image signal processor of Figure 1. Referring to Figures 1 and 2, the image signal processor 120 includes a noise reduction module 121, a demosaic module 122, a color correction module 123, a gamma correction module 124, and a color conversion module 125.

[0024] The noise reduction module 121 is configured to remove noise from the Bayer image received from the image sensor 110. For example, the noise reduction module 121 is configured to remove fixed-pattern noise or temporal random noise according to a color filter array (CFA) of the image sensor 110.

[0025] The demosaic module 122 is configured to convert the output of the noise reduction module 121 into full-color data. For example, the output of the noise reduction module 121 has a data format (e.g., Bayer format, tetra format, etc.) according to the CFA pattern of the image sensor 110. The demosaic module 122 is configured to convert the data format according to the CFA pattern of the image sensor 110 into an RGB format.

[0026] The color correction module 123 is configured to correct the hue of the high quality image converted to RGB format. The gamma correction module 124 is configured to correct the gamma value for the output from the color correction module 123.

[0027] The color conversion module 125 is configured to convert the output from the gamma correction module 124 into a specific format. For example, the output from the gamma correction module 124 has an RGB format. The color conversion module 125 converts the RGB format into a YUV format.

[0028] The image generated by the image signal processor 120 is provided to an external display device or storage device. In one embodiment, the configuration of the image signal processor 120 shown in FIG. 2 is merely an example, and the scope of the present invention is not limited thereto. For example, the image signal processor 120 may further include additional components configured to perform signal processing operations other than those described above.

[0029] Figure 3 is a block diagram showing an example of the image sensor of Figure 1. Referring to Figures 1 and 3, the image sensor 110 includes a white extraction module 111, a crosstalk compensation module 112, a Bayer extraction module 113, a pixel array 114, a row driver 115, an analog to digital converter (ADC) 116, an output circuit 117, and a control logic circuit 118.

[0030] The pixel array 114 includes a plurality of pixels PX. Each of the plurality of pixels is configured to output an electrical signal, i.e., an analog signal, proportional to the magnitude of the incident light based on light waves incident from outside the pixel. In one embodiment, each of the plurality of pixels is coupled to a different color filter (e.g., R, G, B, W, etc.) to receive light waves of different wavelengths.

[0031] The pixels constituting the pixel array 114 include multiple pixel groups PG. Each pixel group PG includes two or more pixels. The pixels constituting a pixel group PG share one floating diffusion node. However, the present invention is not limited to this, and the pixels constituting a pixel group PG may share multiple floating diffusion nodes. For example, a pixel group PG may include nine pixels PX arranged in three rows and three columns, or four pixels PX arranged in two rows and two columns, but the number of pixels constituting a pixel group PG is not limited to this.

[0032] The pixel group PG includes color pixels and white pixels for outputting color-related information. For example, the pixel group PG may include a red pixel R that converts light waves in the red spectral region into electrical signals, or a green pixel (Gr, Gb) that converts light waves in the green spectral region into electrical signals, or a blue pixel B that converts light waves in the blue spectral region into electrical signals, or a white pixel W that converts light waves in spectral regions corresponding to the red pixel R, the green pixel (Gr, Gb), and the blue pixel B into electrical signals. To this end, a plurality of color filters are formed on the pixel group PG to form a multi-color filter array (Multi-CFA). The color filter array may be formed based on at least one of various patterns, such as a Bayer pattern or a Tetra pattern.

[0033] In one embodiment, a plurality of color filters are formed on pixel groups PG constituting the pixel array 114 to form a multi-color filter array (Multi-CFA). The height at which the color filters are formed may vary depending on the order of processes. For example, a color filter formed relatively early may have a relatively low height formed from the substrate. On the other hand, a color filter formed relatively late may have a relatively high height formed from the substrate. A color filter formed relatively high may affect a color filter formed relatively low, which may cause crosstalk between pixels.

[0034] The row driver 115 is configured to control a plurality of pixels included in the pixel array 114. For example, the row driver 115 generates various control signals (e.g., shutter signals, transfer signals, reset signals, select signals, etc.) for controlling the plurality of pixels. In one embodiment, the row driver 115 controls the plurality of pixels on a row-by-row basis, although the scope of the present invention is not limited in this respect.

[0035] The ADC 116 converts the analog signals generated from each of the plurality of pixels into digital signals and outputs the converted digital signals as data. In one embodiment, the ADC 116 generates data based on correlated double sampling (CDS). Although not shown, the image sensor 110 further includes a storage circuit or memory configured to store the data output from the ADC 116 or a ramp signal generator configured to generate a ramp signal used for the operation of the ADC 116. The ADC 116 outputs a non-Bayer pattern raw image RAW to the white extraction module 111.

[0036] The white extraction module 111 generates a white image WI based on the raw image RAW. For example, the white extraction module 111 receives the raw image RAW from the ADC 116. The white extraction module 111 performs a white extraction operation to generate a white image WI of a white pattern. The white extraction module 111 outputs the white image WI and the raw image RAW to the crosstalk compensation module 112. A more detailed description of the white extraction operation is provided in FIG. 8.

[0037] The crosstalk compensation module 112 performs a crosstalk compensation operation based on the raw image RAW and the white image WI. For example, the crosstalk compensation module 112 receives the raw image RAW and the white image WI from the white extraction module 111. The crosstalk compensation module 112 performs a crosstalk compensation operation to generate a crosstalk-compensated white image WI_XT. The crosstalk compensation module 112 outputs the crosstalk-compensated white image WI_XT and the raw image RAW to the Bayer extraction module 113. A more detailed description of the crosstalk compensation operation will be provided with reference to FIG. 7.

[0038] The Bayer extraction module 113 generates a Bayer image BI based on the raw image RAW and the white image WI. For example, the Bayer extraction module 113 receives the raw image RAW and the crosstalk-compensated white image WI_XT from the crosstalk compensation module 112. The Bayer extraction module 113 performs a subtraction operation on the raw image RAW and the crosstalk-compensated white image WI_XT to extract a Bayer image. A detailed description of this will be provided in FIG. 11C.

[0039] In one embodiment, the white extraction module 111 and the crosstalk compensation module 112 each transmit the received raw image RAW, although the scope of the present invention is not limited in this respect. As shown in FIG. 3, the ADC 116 can directly output the raw image RAW to the crosstalk compensation module 112 and the Bayer extraction module 113.

[0040] The output circuit 117 transmits the Bayer image BI output from the Bayer extraction module 113 to an external device (e.g., a display or a storage device). The control logic circuit 118 is configured to control various components within the image sensor 110 under the control of an external control device (e.g., a controller or application processor of the image sensor device).

[0041] Figure 4A is a circuit diagram illustrating an example of a pixel group PG of Figure 3. For example, the pixel group of Figure 4A is included in pixel array 114. In one embodiment, pixel group PG includes first to fourth pixels (PX1 to PX4).

[0042] The pixel group PG includes photoelectric conversion elements (PDa to PDd), transfer transistors (Ta to Td), dual conversion transistors DT, reset transistors RT, source follower transistors SF, and selection transistors SE. The first pixel PX1 includes the first photoelectric conversion element PDa and the first transfer transistor Ta, the second pixel PX2 includes the second photoelectric conversion element PDb and the second transfer transistor Tb, the third pixel PX3 includes the third photoelectric conversion element PDc and the third transfer transistor Tc, and the fourth pixel PX4 includes the fourth photoelectric conversion element PDd and the fourth transfer transistor Td. Each of the first to fourth pixels (PX1 to PX4) shares the dual conversion transistor DT, reset transistor RT, source follower transistor SF, selection transistor SE, and floating diffusion regions (FDa, FDb).

[0043] Each of the transfer transistors (Ta to Td) transfers charges generated by each of the photoelectric conversion elements (PDa to PDd) to the first floating diffusion region FDa. For example, during the period in which the transfer transistor Ta is turned on by the transfer signal TGa received from the row driver 115, charges provided from the photoelectric conversion element PDa are accumulated in the first floating diffusion region FDa. The remaining transfer transistors (TGb to TGd) operate in the same manner, so that charges provided from the photoelectric conversion elements (PDb to PDd) are also accumulated in the first floating diffusion region FDa. One end of the transfer transistors (Ta to Td) is connected to each of the photoelectric conversion elements (PDa to PDd), and the other end is commonly connected to the first floating diffusion region FDa.

[0044] The first floating diffusion region FDa accumulates the charges converted by at least one of the photoelectric conversion elements (PDa-PDd). For example, the capacitance of the first floating diffusion region FDa is shown as a first capacitance CFDa. The first floating diffusion region FDa is connected to the gate terminal of a source follower transistor SF, which operates as a source follower amplifier. As a result, a voltage potential corresponding to the charges accumulated in the first floating diffusion region FDa is formed.

[0045] The reset transistor RT is turned on by a reset signal RG to provide a reset voltage (e.g., a power supply voltage VDD) to the first floating diffusion region FDa, so that the charge stored in the first floating diffusion region FDa is transferred to the power supply voltage VDD terminal, and the voltage of the first floating diffusion region FDa is reset.

[0046] The source follower transistor SF amplifies the change in the electrical potential of the first floating diffusion region FDa and generates a corresponding voltage (i.e., output signal OUT). The selection transistor SE is driven by a selection signal SEL to select pixels to be read row by row. When the selection transistor SE is turned on, the output signal OUT is output via the column line CL.

[0047] Meanwhile, under normal circumstances, the first floating diffusion region FDa does not easily saturate, so there is no need to increase the capacitance of the first floating diffusion region FDa (i.e., CFDa). However, under high-illumination circumstances, the first floating diffusion region FDa easily saturates. Therefore, to prevent saturation, the dual-conversion transistor DT is turned on, for example, by the dual-conversion signal DCG, so that the first floating diffusion region FDa and the second floating diffusion region FDb are electrically connected, and the capacitance of the floating diffusion regions (FDa, FDb) is increased to the sum of the first capacitance CFDa and the second capacitance CFDb.

[0048] In one embodiment, when the image processing device operates in normal mode, the row driver 115 uses the output signals OUT output from each pixel constituting the pixel group PG individually, i.e., controls the transfer signals TGa to TGad so that the transfer transistors Ta to Td are turned on at different times, thereby outputting the output signals OUT corresponding to the charges converted by each photoelectric conversion element via the column lines CL at different times.

[0049] In one embodiment, when the image processing device operates in a binning mode, the charges converted by the pixels (PX1 to PX4) constituting the pixel group PG are simultaneously used to generate one output signal OUT. For example, the transfer transistors (Ta to Td) are simultaneously or temporarily turned on to store the charges converted by the pixels (PX1 to PX4) in the first floating diffusion region FDa, and output an output signal OUT corresponding to the sum of the charges converted by the photoelectric conversion elements (PDa to PDd) via the column line CL.

[0050] FIG. 4B is a diagram illustrating a pixel set PS including the pixel group of FIG. 4A. Referring to FIGS. 3, 4A, and 4B, the pixel set PS includes first to fourth pixel groups (PG1 to PG4). For example, the first pixel group PG1 is located in the first row and first column, the second pixel group PG2 is located in the first row and second column, the third pixel group PG3 is located in the second row and first column, and the fourth pixel group PG4 is located in the second row and second column. The pixel set PS includes 16 pixels arranged in a 4x4 array. The scope of the present invention is not limited thereto, and the number of pixels included in the pixel set PS may vary. Each pixel has a digital value or pixel value (i.e., code level) as the output of the image sensor 110.

[0051] The pixel set PS has a non-Bayer pattern. For convenience of explanation, the pixel set PS is a third type of array pattern. For example, in the third type of array pattern, the first row and the first column are positions of white color corresponding to the white pixel W1, the first row and the second column are positions of green color corresponding to the green pixel Gr1, the first row and the third column are positions of white color corresponding to the white pixel W3, the first row and the fourth column are positions of red color corresponding to the red pixel R1, the second row and the first column are positions of green color corresponding to the green pixel Gr2, the second row and the second column are positions of white color corresponding to the white pixel W2, the second row and the third column are positions of red color corresponding to the red pixel R2, and the second row and the fourth column are positions of white color corresponding to the white pixel W4. The third row and first column are the positions of white color corresponding to white pixel W5, the third row and second column are the positions of blue color corresponding to blue color pixel B1, the third row and third column are the positions of white color corresponding to white pixel W7, the third row and fourth column are the positions of green color corresponding to green color pixel Gb1, the fourth row and first column are the positions of blue color corresponding to blue color pixel B2, the fourth row and second column are the positions of white color corresponding to white pixel W6, the fourth row and third column are the positions of green color corresponding to green color pixel Gb2, and the fourth row and fourth column are the positions of white color corresponding to white pixel W8.

[0052] The pixel set PS is divided into multiple pixel groups (PG1 to PG4). The first pixel group PG1 includes green color pixels (Gr1, Gr2) and white pixels (W1, W2), the second pixel group PG2 includes red color pixels (W3, W4) and white pixels (W3, W4), the third pixel group PG3 includes blue color pixels (B1, B2) and white pixels (W5, W6), and the fourth pixel group PG4 includes green color pixels (Gb1, Gb2) and white pixels (W7, W8).

[0053] 4C is a diagram illustrating a pixel set including the pixel group of FIG. 4A. Hereinafter, for convenience of explanation and simplicity of the drawings, detailed descriptions or reference numerals for components that are the same as or similar to the components described above will be omitted. The components omitted below may be implemented by each of the overall embodiments described in the detailed description of the present invention or a combination thereof.

[0054] Referring to Figures 3, 4A, and 4C, the pixels (W1, W2, Gr1, Gr2) included in the first pixel group PG1 of the pixel set PS share the first floating diffusion region FD1, the pixels (W3, W4, R1, R2) included in the second pixel group PG2 share the second floating diffusion region FD2, the pixels (W5, W6, B1, B2) included in the third pixel group PG3 share the third floating diffusion region FD3, and the pixels (W7, W8, Gb1, Gb2) included in the fourth pixel group PG4 share the fourth floating diffusion region FD4.

[0055] In one embodiment, the first pixel group PG1 includes first to fourth transfer transistors T1 to T4 and first to fourth photoelectric conversion elements PD1 to PD4. For example, the first white pixel W1 corresponds to the first transfer transistor T1 and the first photoelectric conversion element PD1, the first green pixel Gr1 corresponds to the second transfer transistor T2 and the second photoelectric conversion element PD2, the second green pixel Gr2 corresponds to the third transfer transistor T3 and the third photoelectric conversion element PD3, and the second white pixel W2 corresponds to the fourth transfer transistor T4 and the fourth photoelectric conversion element PD4.

[0056] The second pixel group PG2 includes fifth to eighth transfer transistors (T5 to T8) and fifth to eighth photoelectric conversion elements (PD5 to PD8). For example, the third white pixel W3 corresponds to the fifth transfer transistor T5 and the fifth photoelectric conversion element PD5, the first red pixel R1 corresponds to the sixth transfer transistor T6 and the sixth photoelectric conversion element PD6, the second red pixel R2 corresponds to the seventh transfer transistor T7 and the seventh photoelectric conversion element PD7, and the fourth white pixel W4 corresponds to the eighth transfer transistor T8 and the eighth photoelectric conversion element PD8.

[0057] The third pixel group PG3 includes ninth to twelfth transfer transistors (T9 to T12) and ninth to twelfth photoelectric conversion elements (PD9 to PD12). For example, the fifth white pixel W5 corresponds to the ninth transfer transistor T9 and the ninth photoelectric conversion element PD9, the first blue pixel B1 corresponds to the tenth transfer transistor T10 and the tenth photoelectric conversion element PD10, the second blue pixel B2 corresponds to the eleventh transfer transistor T11 and the eleventh photoelectric conversion element PD11, and the sixth white pixel W6 corresponds to the twelfth transfer transistor T12 and the twelfth photoelectric conversion element PD12.

[0058] The fourth pixel group PG4 includes thirteenth to sixteenth transfer transistors (T13 to T16) and thirteenth to sixteenth photoelectric conversion elements (PD13 to PD16). For example, the seventh white pixel W7 corresponds to the thirteenth transfer transistor T13 and the thirteenth photoelectric conversion element PD13, the first green pixel Gb1 corresponds to the fourteenth transfer transistor T14 and the fourteenth photoelectric conversion element PD14, the second green pixel Gb2 corresponds to the fifteenth transfer transistor T15 and the fifteenth photoelectric conversion element PD15, and the eighth white pixel W8 corresponds to the sixteenth transfer transistor T16 and the sixteenth photoelectric conversion element PD16.

[0059] In one embodiment, the first transfer signal TG1 is applied to the gate terminal of the first transfer transistor T1, the second transfer signal TG2 is applied to the gate terminal of the second transfer transistor T2, the third transfer signal TG3 is applied to the gate terminal of the third transfer transistor T3, and the fourth transfer signal TG4 is applied to the gate terminal of the fourth transfer transistor T4. The remaining transfer signals (TG5 to TG16) are similar to the above and will not be described in detail.

[0060] 5 is a timing diagram showing an example of waveforms of signals applied to pixel groups. Referring to FIGS. 4A, 4C, and 5, the image sensor 110 according to this embodiment performs a third summation operation to generate a raw image RAW. For example, the image sensor 110 performs a first summation operation on the first, second, and third pixel groups PG1 to PG3, and a second summation operation on the fourth pixel group PG4.

[0061] A first reset signal RG1 is applied to each of the reset transistors of the first and second pixel groups (PG1, PG2), a second reset signal RG2 is applied to each of the reset transistors of the third and fourth pixel groups (PG3, PG4), a first select signal SEL1 is applied to each of the select transistors of the first and second pixel groups (PG1, PG2), and a second select signal SEL2 is applied to each of the select transistors of the third and fourth pixel groups (PG3, PG4).

[0062] During the first readout period, the first selection signal SEL1 is activated, the first reset signal RG1 is activated and then deactivated again, and then at least one of the first to eighth transfer signals (TG1 to TG8) is activated and then deactivated again. During the first readout period, the second selection signal SEL2 is deactivated, the second reset signal RG2 is deactivated, and the ninth to sixteenth transfer signals (TG9 to TG16) are deactivated.

[0063] After the first select signal SEL1 is activated and the first reset signal RG1 is activated and then deactivated again, and before at least one of the first to eighth transfer signals (TG1 to TG8) is activated, the readout circuit of the first pixel group PG1 samples and outputs the reset voltage level of the first floating diffusion region FD1, and the readout circuit of the second pixel group PG2 samples and outputs the reset voltage level of the second floating diffusion region FD2 (RST SMP1).After at least one of the first to eighth transfer signals (TG1 to TG8) is activated and then deactivated again, the readout circuit of the first pixel group PG1 samples and outputs the signal voltage level of the first floating diffusion region FD1, and the readout circuit of the second pixel group PG2 samples and outputs the signal voltage level of the second floating diffusion region FD2 (SIG SMP1).

[0064] During the second readout period, the second select signal SEL2 is activated, the second reset signal RG2 is activated and then deactivated again, and then at least one of the ninth to twelfth, fourteenth, and fifteenth transfer signals (TG9 to TG12, TG14, and TG15) is activated and then deactivated again. During the second readout period, the first select signal SEL1 is deactivated, the first reset signal RG1 is deactivated, the first to eighth transfer signals (TG1 to TG8) are deactivated, and the thirteenth and sixteenth transfer signals (TG13 and TG16) are deactivated.

[0065] After the second select signal SEL2 is activated and the second reset signal RG2 is activated and then deactivated again, and before at least one of the ninth to twelfth, fourteenth, and fifteenth transfer signals (TG9 to TG12, TG14, and TG15) is activated, the readout circuit of the third pixel group PG3 samples and outputs the reset voltage level of the third floating diffusion region FD3, and the readout circuit of the fourth pixel group PG4 samples and outputs the reset voltage level of the fourth floating diffusion region FD4 (RST SMP2).After at least one of the ninth to twelfth, fourteenth, and fifteenth transfer signals (TG9 to TG12, TG14, and TG15) is activated and then deactivated again, the readout circuit of the third pixel group PG3 samples and outputs the signal voltage level of the third floating diffusion region FD3, and the readout circuit of the fourth pixel group PG4 samples and outputs the signal voltage level of the fourth floating diffusion region FD4 (SIG SMP2).

[0066] The first to eighth transfer signals (TG1 to TG8) in the first readout period are activated and then deactivated again, and the ninth to twelfth transfer signals (TG9 to TG12) in the second readout period are activated and then deactivated again. Therefore, the first to third floating diffusion regions (FD1 to FD3) accumulate and store electrons accumulated by all of the corresponding photoelectric conversion elements. For example, the first floating diffusion region FD1 accumulates and stores electrons accumulated by the first to fourth photoelectric conversion elements (PD1 to PD4). The signal sampling value of the first pixel group PG1 is generated based on the first and second white pixels (W1, W2) and the first and second green color pixels (Gr1, Gr2).

[0067] During the second readout period, the thirteenth and sixteenth transfer signals (TG13, TG16) remain inactive, and the fourteenth and fifteenth transfer signals (TG14, TG15) are activated and then deactivated again. Therefore, the fourth floating diffusion region FD4 does not accumulate and store electrons accumulated by the thirteenth and sixteenth photoelectric conversion elements (PD13, PD16). The fourth floating diffusion region FD4 accumulates and stores only electrons accumulated by the fourteenth and fifteenth photoelectric conversion elements (PD14, PD15). The signal sample value of the fourth pixel group PG4 is generated based only on the first and second green pixels (Gb1, Gb2) and not on the seventh and eighth white pixels (W7, W8). In other words, the signal sample value of the fourth pixel group PG4 is generated based only on the first and second green pixels (Gb1, Gb2), not on the seventh and eighth white pixels (W7, W8).

[0068] As described above, the first to twelfth transfer signals (TG1 to TG12) are activated and then deactivated during the readout period, so the first to third pixel groups (PG1 to PG3) sample and output signal voltage levels based on the four pixels included in the pixel groups. Meanwhile, the thirteenth and sixteenth transfer signals (TG13, TG16) are deactivated during the readout period, and the fourteenth and fifteenth transfer signals (TG14, TG15) are activated and then deactivated, so the fourth pixel group PG4 samples and outputs signal voltage levels based on the two pixels (Gb1, Gb2) included in the pixel group.

[0069] In the image sensor 110 according to this embodiment, a first portion of the plurality of pixels performs a first summing operation, and a second portion of the plurality of pixels performs a second summing operation. That is, the image sensor 110 performs a third summing operation. Therefore, the image sensor according to this embodiment can improve the frame rate and reduce analog power consumption.

[0070] Figure 6 is a diagram showing an example of a raw image output from the ADC of Figure 3. Referring to Figures 3 and 6, pixel array 114 includes first to fourth pixel sets (PS1 to PS4). The scope of the present invention is not limited thereto, and the number of pixel sets and the number of pixels included in the pixel array may vary.

[0071] In one embodiment, the first pixel set PS1 includes the first through second white pixels (W1 through W8), the first and second green pixels (Gr1 and Gr2), the first and second red pixels (R1 and R2), the first and second blue pixels (B1 and B2), and the first and second green pixels (Gb1 and Gb2). The second pixel set PS2 includes the ninth through sixteenth white pixels (W9 through W16), the third and fourth green pixels (Gr3 and Gr4), the third and fourth red pixels (R3 and R4), the third and fourth blue pixels (B3 and B4), and the third and fourth green pixels (Gb3 and Gb4). The third pixel set PS3 includes the 17th to 24th white pixels (W17 to W24), the 5th and 6th green pixels (Gr5, Gr6), the 5th and 6th red pixels (R5, R6), the 5th and 6th blue pixels (B5, B6), and the 5th and 6th green pixels (Gb5, Gb6). The fourth pixel set PS4 includes the 25th to 32nd white pixels (W25 to W32), the 7th and 8th green pixels (Gr7, Gr8), the 7th and 8th red pixels (R7, R8), the 7th and 8th blue pixels (B7, B8), and the 7th and 8th green pixels (Gb7, Gb8). Each of the pixel sets (PS1 to PS4) is an arrangement pattern of the third type. For convenience of explanation, detailed descriptions of the above pixel sets are omitted.

[0072] The ADC 116 outputs a raw image RAW. For example, the raw image RAW includes 16 pixels PX arranged in a 4x4 matrix. The scope of the present invention is not limited thereto, and the number of pixels included in the raw image RAW may vary. Each pixel has a digital value or pixel value (i.e., code level) as the output of the image sensor 110.

[0073] The raw image RAW is divided into a plurality of basic units (U1 to U4). Each of the plurality of basic units (U1 to U4) includes four pixels arranged in a 2x2 matrix, and each pixel data has a digital value. The raw image RAW has a fourth type of array pattern. The basic unit of the fourth type of array pattern includes four pixels arranged in a 2x2 matrix, including a white-green color pixel W+Gr, a white-red color pixel W+R, a green color pixel Gb1, and a white-blue color pixel W+B, which are arranged sequentially clockwise from the upper left corner.

[0074] For example, in the first basic unit U1, the first row and first column are the positions of the white-green color pixel W+Gr_1, the first row and second column are the positions of the white-red color pixel W+R_1, the second row and first column are the positions of the white-blue color pixel W+B_1, and the second row and second column are the positions of the green color pixel Gb1. In a similar manner, the remaining basic units (U2 to U4) are arranged with positions for each color. Detailed description of this will be omitted.

[0075] In one embodiment, the first pixel set PS1 corresponds to the first basic unit U1, the second pixel set PS2 corresponds to the second basic unit U2, the third pixel set PS3 corresponds to the third basic unit U3, and the fourth pixel set PS4 corresponds to the fourth basic unit U4.

[0076] In the first basic unit U1, the white-green color pixel W+Gr_1 corresponds to the first pixel group PG1 of the first pixel set PS1, the white-red color pixel W+R_1 corresponds to the second pixel group PG2 of the first pixel set PS1, the white-blue color pixel W+B_1 corresponds to the third pixel group PG3 of the first pixel set PS1, and the green color pixel Gb1 corresponds to the fourth pixel group PG4 of the first pixel set PS1.

[0077] More specifically, the digital value of the white-green pixel W+Gr_1 is determined based on the first and second white pixels (W1, W2) and the first and second green pixels (Gr1, Gr2). The digital value of the white-red pixel W+R_1 is determined based on the third and fourth white pixels (W3, W4) and the first and second red pixels (R1, R2). The digital value of the white-blue pixel W+B_1 is determined based on the fifth and sixth white pixels (W5, W6) and the first and second blue pixels (B1, B2). The digital value of the green pixel Gb1 is determined based on the first and second green pixels (Gb1, Gb2). The digital values ​​of the remaining pixels of the raw image RAW can be determined in a similar manner, so a detailed description thereof will be omitted.

[0078] Fig. 7 is a flowchart showing the operation of the image sensor in Fig. 1. Referring to Figs. 1, 3, and 7, the image sensor 110 generates a Bayer pattern image based on a non-Bayer pattern image. That is, the image sensor 110 converts the non-Bayer pattern image into a Bayer pattern image.

[0079] In step S110, the image sensor 110 performs a white extraction operation based on the raw image RAW. For example, the white extraction module 111 receives the raw image RAW from the ADC 116. The white extraction module 111 extracts a white image WI from the raw image RAW. A more detailed description of the white extraction operation is provided in FIG. 8.

[0080] In step S120, the image sensor 110 performs a crosstalk compensation operation based on the white image WI. For example, the crosstalk compensation module 112 receives the white image WI and the raw image RAW from the white extraction module 111. The crosstalk compensation module 112 performs a crosstalk compensation operation on the white image WI based on the calibration data to generate a crosstalk-compensated white image WI_XT. The crosstalk compensation module 112 outputs the crosstalk-compensated white image WI_XT and the raw image RAW to the Bayer extraction module 113.

[0081] Crosstalk refers to signals generated by interference between pixels. Crosstalk includes optical crosstalk caused by microlenses and electrical crosstalk caused by electromagnetic interference in silicon. For example, the crosstalk compensation module 112 performs crosstalk compensation according to differences in the heights of the color filters of the pixels that make up the pixel array 114.

[0082] In one embodiment, optical crosstalk is affected by surrounding pixels. In particular, crosstalk is more affected by pixels horizontally or vertically adjacent to a pixel. For example, referring to FIG. 6, the pixels adjacent in the first and second directions to the green color pixels (Gr:Gr1-Gr8), red color pixels (R:R1-R8), blue color pixels (B:B1-B8), and green color pixels (Gb:Gb1-Gb8) are white pixels. Therefore, the green, red, blue, and green color pixels (Gr, R, B, Gb) are only affected by crosstalk relative to the white pixel W.

[0083] Meanwhile, the colors of the pixels adjacent to the white pixels (W: W1 to W32) in the first and second directions differ depending on the position of each white pixel W. Therefore, the crosstalk for each white pixel W differs depending on the pixels adjacent to it in the first and second directions.

[0084] In one embodiment, the image sensor 110 generates and stores calibration data necessary to perform the crosstalk compensation operation based on pixels adjacent in the first and second directions to each white pixel W. For example, the calibration data is a value measured in advance, compressed, and stored in an external memory.

[0085] The crosstalk compensation module 112 loads the calibration data from the external memory and performs a crosstalk compensation operation on the white image WI based on the calibration data. For example, the crosstalk compensation module 112 multiplies the code level of the white image WI by the corresponding calibration value in the calibration data to generate a crosstalk-compensated white image WI_XT.

[0086] In step S130, the image sensor 110 generates a Bayer image BI based on the white image WI and the raw image RAW. For example, the Bayer extraction module 113 receives the crosstalk-compensated white image WI_XT and the raw image RAW from the crosstalk compensation module 112.

[0087] In one embodiment, the Bayer extraction module 113 performs a subtraction operation on the raw image RAW and the crosstalk-compensated white image WI_XT to generate the Bayer image BI, i.e., the Bayer extraction module 113 subtracts the crosstalk-compensated white image WI_XT from the raw image RAW to extract the Bayer image BI.

[0088] As described above, the image sensor 110 according to this embodiment generates a raw image RAW through a single read operation and converts the raw image RAW into a Bayer image BI.

[0089] FIG. 8 is a flowchart illustrating step S110 of FIG. 7 in more detail. FIG. 9 is a block diagram illustrating the white extraction module of FIG. 3 in more detail. Referring to FIGS. 3, 7, 8, and 9, the image sensor 110 performs a white extraction operation based on a raw image RAW. In one embodiment, the white extraction module 111 includes a white balance module 130, a luma extraction module 140, a guide filtering module 150, and a white generation module 160. The configurations and operation methods of the white extraction modules of FIGS. 8 and 9 are merely examples, and the scope of the present invention is not limited thereto.

[0090] Color is expressed by hue, saturation, and value. The saturation of an achromatic object is "0 (zero)." The green (G), red (R), and blue (B) values ​​of the hue of an achromatic object are the same. Achromatic objects have various values.

[0091] In step S111, the image sensor 110 performs a white balance operation. For example, the white balance module 130 receives a raw image RAW from the ADC 116. The white balance module 130 performs a white balance operation to generate a white balance raw image RAW_WB. The white balance module 130 outputs the white balance raw image RAW_WB to the luma extraction module 140.

[0092] For example, the white balance operation adjusts the gain of green G, red R, and blue B hues for light waves reflected from an object and captured by an image sensor. The white balance module 130 adjusts the white balance gain for the raw image RAW and performs the white balance operation based on the adjusted white balance gain.

[0093] In step S112, the image sensor 110 performs a luma extraction operation. For example, the luma extraction module 140 receives a white-balanced raw image RAW_WB. The luma extraction module 140 performs the luma extraction operation to generate a panchromatic image (PI). For example, the luma extraction module 140 performs the luma extraction operation based on the white-balanced raw image RAW_WB to generate a panchromatic image PI (e.g., an achromatic image) with a saturation of 0 (zero) and the same values ​​for green G, red R, and blue B. The luma extraction module 140 outputs the panchromatic image PI to the guided filtering module 150.

[0094] In step S113, the image sensor 110 performs a guided filtering operation. For example, the guided filtering module 150 receives a raw image RAW and a panchromatic image PI. The guided filtering module 150 performs a guided filtering operation based on the raw image RAW and the panchromatic image PI to generate a white-green full image W+G_FI and a green full image G_FI. The guided filtering module 150 outputs the white-green full image W+G_FI and the green full image G_FI to the white generation module 160.

[0095] In one embodiment, the white-green full image W+G_FI has a fifth type of array pattern. The basic unit of the fifth type of array pattern includes one pixel, which is a white-green pixel W+G. The green full image G_FI has a sixth type of array pattern. The basic unit of the sixth type of array pattern includes one pixel, which is a green pixel G.

[0096] In operation S114, the image sensor 110 performs a white image generation operation. For example, the white generation module 160 receives the white-green full image W+G_FI and the green full image G_FI. The image sensor 110 performs a subtraction operation on the white-green full image W+G_FI and the green full image G_FI to generate a white image W_I.

[0097] Figure 10 is a block diagram showing in more detail the guided filtering module of Figure 9. Figures 11A to 11C are diagrams for explaining a method of converting a raw image into a Bayer image. Referring to Figures 3, 9, and 10, the guided filtering module 150 includes a sampling module 151 and a full generation module 152.

[0098] In one embodiment, the sampling module 151 receives a raw image RAW. The sampling module 151 generates a white-green sampled image W+G_SI and a green sampled image G_SI based on the raw image RAW. For example, the white-green sampled image W+G_SI has a fifth type of arrangement pattern, and the green sampled image G_SI has a sixth type of arrangement pattern.

[0099] 11A, detailed description of the above-mentioned raw image RAW is omitted. For example, the white-green sampled image W+G_SI includes four pixels PX arranged in a 2x2 matrix. The size of the white-green sampled image W+G_SI is 1 / 4 of the size of the raw image RAW. The white-green sampled image W+G_SI includes only the white-green color pixels (W+Gr_1 to W+Gr_4) of the raw image RAW.

[0100] For example, the green sampled image G_SI includes four pixels PX arranged in a 2x2 matrix. The size of the green sampled image G_SI is 1 / 4 of the size of the raw image RAW. The green sampled image G_SI includes only the green color pixels (Gb1 to Gb4) of the raw image RAW.

[0101] In one embodiment, the full generation module 152 receives a panchromatic image PI, a white-green sampled image W+G_SI, and a green sampled image G_SI. The full generation module 152 generates a white-green full image W+G_FI based on the panchromatic image PI and the white-green sampled image W+G_SI. The full generation module 152 generates a green full image G_FI based on the panchromatic image PI and the green sampled image G_SI. The full generation module 152 outputs the white-green full image W+G_FI and the green full image G_FI.

[0102] In one embodiment, the white-green full image W+G_FI includes 16 pixels (W+G_11 to W+G_44) arranged in a 4x4 matrix. The size of the white-green full image W+G_FI is the same as the size of the raw image RAW. For example, the code level of a first portion (W+G_11, W+G_12, W+G_21, W+G_22) of the white-green full image W+G_FI is the same as the code level of the first pixel W+Gr_1 of the white-green sampled image W+G_SI. The code level of a second portion (W+G_13, W+G_14, W+G_23, W+G_24) of the white-green full image W+G_FI is the same as the code level of the second pixel W+Gr_2 of the white-green sampled image W+G_SI. The code level of the third portion (W+G_31, W+G_32, W+G_41, W+G_42) of the white-green full image W+G_FI is the same as the code level of the third pixel W+Gr_3 of the white-green sampled image W+G_SI. The code level of the fourth portion (W+G_33, W+G_34, W+G_43, W+G_44) of the white-green full image W+G_FI is the same as the code level of the fourth pixel W+Gr_4 of the white-green sampled image W+G_SI.

[0103] The green full image G_FI includes 16 pixels (G11 to G44) arranged in a 4x4 matrix. The size of the green full image G_FI is the same as the size of the raw image RAW. For example, the code level of the first portion (G11, G12, G21, G22) of the green full image G_FI is the same as the code level of the first pixel Gb1 of the green sampled image G_SI. The code level of the second portion (G13, G14, G23, G24) of the green full image G_FI is the same as the code level of the second pixel Gb2 of the green sampled image G_SI. The code level of the third portion (G31, G32, G41, G42) of the green full image G_FI is the same as the code level of the third pixel Gb3 of the green sampled image G_SI. The code level of the fourth portion (G33, G34, G43, G44) of the green full image G_FI is the same as the code level of the fourth pixel Gb4 of the green sampled image G_SI.

[0104] 11B, the white generation module 160 generates a white image WI based on the white-green full image W+G_FI and the green full image G_FI. For example, the white generation module 160 generates the white image WI by subtracting the green full image G_FI from the white-green full image W+G_FI. In other words, the code level of pixel W11 of the white image WI is calculated by subtracting the code level of pixel G11 of the green full image G_FI from the code level of the corresponding pixel W+G_11 of the white-green full image W+G_FI. The white image WI includes 16 pixels (W11 to W44) arranged in a 4x4 matrix. The size of the white image WI is the same as the size of the raw image RAW.

[0105] 11C, the Bayer extraction module 113 generates a Bayer image BI based on the raw image RAW and the crosstalk-compensated white image WI_XT. For example, the Bayer image BI includes 16 pixels (Gr1 to Gr4, R1 to R4, B1 to B4, Gb1 to Gb4) arranged in a 4x4 matrix. The size of the Bayer image BI is the same as the size of the raw image RAW.

[0106] In one embodiment, the Bayer extraction module 113 subtracts the crosstalk-compensated white image WI_XT from the raw image RAW to generate the Bayer image BI. Each of the raw image RAW, the crosstalk-compensated white image WI_XT, and the Bayer image BI is divided into four basic units.

[0107] The Bayer extraction module 113 performs a subtraction operation on the first raw image basic unit RAW_U1 and the first crosstalk-compensated white image basic unit WI_XT_U1 to generate the first Bayer image basic unit BI_U1, performs a subtraction operation on the second raw image basic unit RAW_U2 and the second crosstalk-compensated white image basic unit WI_XT_U2 to generate the second Bayer image basic unit BI_U2, performs a subtraction operation on the third raw image basic unit RAW_U3 and the third crosstalk-compensated white image basic unit WI_XT_U3 to generate the third Bayer image basic unit BI_U3, and performs a subtraction operation on the fourth raw image basic unit RAW_U4 and the fourth crosstalk-compensated white image basic unit WI_XT_U4 to generate the fourth Bayer image basic unit BI_U4.

[0108] For example, in the first basic unit, the Bayer extraction module 113 calculates the code level of the green pixel Gr1 located in the first row and first column of the Bayer image B1 by subtracting the white pixel W11 located in the first row and first column of the crosstalk-compensated white image WI_XT from the white-green pixel W+Gr_1 located in the first row and first column of the raw image RAW. The Bayer extraction module 113 calculates the code level of the red pixel R1 located in the first row and second column of the Bayer image B1 by subtracting the white pixel W12 located in the first row and second column of the crosstalk-compensated white image WI_XT from the white-red pixel W+R_1 located in the first row and second column of the raw image RAW. The Bayer extraction module 113 calculates the code level of the blue pixel B1 located in the second row and first column of the Bayer image B1 by subtracting the white pixel W21 located in the second row and first column of the crosstalk-compensated white image WI_XT from the white-blue pixel W+B_1 located in the second row and first column of the raw image RAW.

[0109] The Bayer extraction module 113 does not perform a subtraction operation on the second row and second column. That is, the code level of the green color pixel Gb1 located in the second row and second column of the Bayer image BI is the same as the code level of the green color pixel Gb1 located in the second row and second column of the raw image RAW. The remaining basic units (G2 to G4) are similar to this, so detailed description will be omitted.

[0110] For ease of explanation, the first portion of the raw image RAW is divided into pixels related to white pixels (W+Gr_1 to W+Gr_4, W+R_1 to W+R_4, W+B_1 to W+B_4), and the second portion of the raw image is divided into pixels unrelated to white pixels (Gb1 to Gb4). That is, the first portion of the raw image RAW corresponds to the first to third pixel groups (PG1 to PG3) that performed the first summation operation on the pixel set PS, and the second portion of the raw image RAW corresponds to the fourth pixel group PG4 that performed the second summation operation on the pixel set PS.

[0111] A first portion of the crosstalk-compensated white image WI_XT corresponds to a first portion of the raw image RAW, and a second portion of the crosstalk-compensated white image WI_XT corresponds to a second portion of the raw image RAW. A first portion of the Bayer image BI corresponds to the first portion of the raw image RAW, and a second portion of the Bayer image BI corresponds to the second portion of the raw image RAW. In Figure 11C, the hatched (or shaded) pixels correspond to the second portion, and the other pixels correspond to the first portion.

[0112] As described above, the Bayer extraction module 113 performs a subtraction operation on the first portion of the raw image RAW and the first portion of the crosstalk-compensated white image WI_XT to generate the first portion of the Bayer image BI, and generates the second portion of the Bayer image BI based on the second portion of the raw image RAW.

[0113] 12A to 12C are block diagrams illustrating various examples of an image device according to an embodiment of the present invention. Referring to FIG. 12A, an image device 200a includes an image sensor 210a and an image signal processor 220a. Unlike the image sensor 110 described above, the image sensor 210a is configured to output a raw image. That is, the image sensor 210a does not include a white extraction module, a crosstalk compensation module, or a Bayer extraction module.

[0114] Unlike the image signal processor 120 described above, the image signal processor 220a is configured to receive a raw image RAW. That is, the image signal processor 220a of Fig. 12A includes a white extraction module 221a, a crosstalk compensation module 222a, and a Bayer extraction module 223a. Based on the scheme described with reference to Figs. 1 to 11, the white extraction module 221a performs a white extraction operation, the crosstalk compensation module 222a performs a crosstalk compensation operation, and the Bayer extraction module 223a generates a Bayer image.

[0115] 12B, an image device 200b includes an image sensor 210b and an image signal processor 220b. Unlike the image sensor 110 described above, the image sensor 210a is configured to output a raw image RAW and a white image WI. That is, the image sensor 210a includes a white extraction module 211b, but does not include a crosstalk compensation module or a Bayer extraction module. The white extraction module 221b performs a white extraction operation based on the method described with reference to FIGS. 1 to 11.

[0116] Unlike the image signal processor 120 described above, the image signal processor 220b is configured to receive a raw image RAW and a white image WI. That is, the image signal processor 220b of Fig. 12B includes a crosstalk compensation module 222b and a Bayer extraction module 223b. The crosstalk compensation module 222b performs a crosstalk compensation operation, and the Bayer extraction module 223b generates a Bayer image based on the scheme described with reference to Figs. 1 to 11.

[0117] 12C, an image device 200c includes an image sensor 210c and an image signal processor 220c. Unlike the image sensor 110 described above, the image sensor 210c is configured to output a raw image RAW and a crosstalk-compensated white image WI_XT. That is, the image sensor 210c includes a white extraction module 211c and a crosstalk compensation module 212c, but does not include a Bayer extraction module. The white extraction module 211c performs a white extraction operation, and the crosstalk compensation module 212c performs a crosstalk compensation operation based on the scheme described with reference to FIGS. 1 to 11.

[0118] Unlike the image signal processor 120 described above, the image signal processor 220c is configured to receive the raw image RAW and the crosstalk-compensated white image WI_XT. That is, the image signal processor 220c of Fig. 12C includes a Bayer extraction module 223c. The Bayer extraction module 223c generates a Bayer image BI based on the scheme described with reference to Figs. 1 to 11.

[0119] Fig. 13 is a block diagram showing an example of the configuration of an electronic device including a multi-camera module according to an embodiment of the present invention, and Fig. 14 is a block diagram showing an example of the configuration of the camera module of Fig. 13.

[0120] Referring to FIG. 13, an electronic device 1000 includes a camera module group 1100, an application processor 1200, a PM (power management) IC 1300, and an external memory 1400.

[0121] The camera module group 1100 includes multiple camera modules (1100a, 1100b, 1100c). While the drawings show an embodiment with three camera modules (1100a, 1100b, 1100c), the embodiment is not limited to this. In one embodiment, the camera module group 1100 is modified to include only two camera modules. In another embodiment, the camera module group 1100 is modified to include n camera modules (n is a natural number greater than or equal to 4).

[0122] The detailed configuration of camera module 1100b will be described in more detail below with reference to FIG. 14, but the following description also applies to other camera modules (1100a, 1100c) depending on the embodiment.

[0123] Referring to FIG. 14, a camera module 1100b includes a prism 1105, an optical path folding element (hereinafter referred to as “OPFE”) 1110, an actuator 1130, an image sensing device 1140, and a storage device 1150.

[0124] The prism 1105 includes a reflecting surface 1107 made of a light-reflecting material, and changes the path of light L incident from the outside.

[0125] In one embodiment, the prism 1105 changes the path of light L incident in a first direction X to a second direction Y perpendicular to the first direction X. The prism 1105 rotates a reflective surface 1107 made of a light-reflecting material in a direction A around a central axis 1106 or rotates the central axis 1106 in a direction B to change the path of light L incident in the first direction X to the perpendicular second direction Y. At this time, the OPFE 1110 also moves in a third direction Z perpendicular to the first direction X and the second direction Y.

[0126] In one embodiment, as shown, the maximum rotation angle of prism 1105 in the A direction is less than or equal to 15 degrees in the positive (+) A direction and greater than 15 degrees in the negative (-) A direction, although the embodiment is not limited thereto.

[0127] In one embodiment, prism 1105 moves in the plus (+) or minus (-) B direction by 20 degrees in or out, between 10 degrees and 20 degrees, or between 15 degrees and 20 degrees, where the angle of movement is the same angle in the plus (+) or minus (-) B direction, or to a similar angle within a range of 1 degree in or out.

[0128] In one embodiment, the prism 1105 moves the reflective surface 1107 of the light-reflecting material in a third direction (eg, Z direction) parallel to the extension direction of the central axis 1106 .

[0129] The OPFE 1110 includes, for example, m (where m is a natural number) groups of optical lenses. The m lenses move in the second direction Y to change the optical zoom ratio of the camera module 1100b. For example, if the basic optical zoom ratio of the camera module 1100b is Z, then by moving the m optical lenses included in the OPFE 1110, the optical zoom ratio of the camera module 1100b is changed to 3Z, 5Z, or an optical zoom ratio of 5Z or more. The OPFE 1110 further includes n (where n is a natural number) groups of optical lenses (for example, anamorphic lenses) in front of the above-mentioned m lenses.

[0130] The actuator 1130 moves the OPFE 1110 or the optical lens (hereinafter referred to as the optical lens) to a specific position. For example, the actuator 1130 adjusts the position of the optical lens so that the image sensor 1142 is located at the focal length of the optical lens for accurate sensing.

[0131] The image sensing device 1140 includes an image sensor 1142, control logic 1144, and memory 1146. The image sensor 1142 senses an image of a sensing target using light L provided through an optical lens. The control logic 1144 controls the overall operation of the camera module 1100b. For example, the control logic 1144 controls the operation of the camera module 1100b according to a control signal provided via a control signal line CSLb. Furthermore, the image sensor 1142 and the control logic 1144 are configured to perform an operation of converting a non-Bayer image into a Bayer image according to FIGS. 1 to 12C.

[0132] The memory 1146 stores information necessary for the operation of the camera module 1100b, such as calibration data 1147. The calibration data 1147 includes information necessary for the camera module 1100b to generate image data using externally provided light L. The calibration data 1147 includes, for example, information regarding the degree of rotation, the focal length, and the optical axis. If the camera module 1100b is implemented in the form of a multi-state camera in which the focal length changes depending on the position of the optical lens, the calibration data 1147 includes focal length values ​​for each position (or state) of the optical lens and information regarding autofocusing. The calibration data 1147 includes data necessary for the white balance operation, crosstalk compensation operation, and the like, described with reference to FIGS. 1 to 11C.

[0133] The storage device 1150 stores image data sensed through the image sensor 1142. The storage device 1150 is disposed outside the image sensing device 1140 and is implemented in a stacked form on the sensor chip that constitutes the image sensing device 1140. In one embodiment, the storage device 1150 is implemented as an EEPROM (Electrically Erasable Programmable Read-Only Memory), but the embodiment is not limited thereto.

[0134] 13 and 14 together, in one embodiment, each of the plurality of camera modules (1100a, 1100b, 1100c) includes an actuator 1130. Accordingly, each of the plurality of camera modules (1100a, 1100b, 1100c) includes the same or different calibration data 1147 depending on the operation of the actuator 1130 included therein.

[0135] In one embodiment, one of the multiple camera modules (1100a, 1100b, 1100c) (e.g., 1100b) is a folded lens type camera module including the above-mentioned prism 1105 and OPFE 1110, and the remaining camera modules (e.g., 1100a, 1100b) are vertical type camera modules that do not include the prism 1105 and OPFE 1110, but the embodiment is not limited to this.

[0136] In one embodiment, one of the camera modules 1100a, 1100b, and 1100c (e.g., 1100c) is a vertical depth camera that extracts depth information using, for example, infrared rays (IR). In this case, the application processor 1200 merges image data provided by the vertical depth camera with image data provided by another camera module (e.g., 1100a or 1100b) to generate a 3D depth image.

[0137] In one embodiment, at least two camera modules (e.g., 1100a, 1100b) of the plurality of camera modules (1100a, 1100b, 1100c) have different fields of view (fields of view), for example, but not limited to, by using different optical lenses for at least two camera modules (e.g., 1100a, 1100b) of the plurality of camera modules (1100a, 1100b, 1100c).

[0138] In one embodiment, the viewing angles of the camera modules (1100a, 1100b, 1100c) are different from each other, and in this case, the optical lenses included in the camera modules (1100a, 1100b, 1100c) are also different from each other, but this is not limiting.

[0139] In one embodiment, each of the multiple camera modules (1100a, 1100b, 1100c) is physically separated from the other. That is, instead of the multiple camera modules (1100a, 1100b, 1100c) sharing the same sensing area of ​​a single image sensor 1142, an independent image sensor 1142 is disposed within each of the multiple camera modules (1100a, 1100b, 1100c).

[0140] 13, the application processor 1200 includes an image processing unit 1210, a memory controller 1220, and an internal memory 1230. The application processor 1200 is implemented separately from the multiple camera modules (1100a, 1100b, 1100c). For example, the application processor 1200 and the multiple camera modules (1100a, 1100b, 1100c) are implemented separately on different semiconductor chips. In one embodiment, the application processor 1200 converts a raw image into a Bayer image, as described with reference to FIGS. 1 to 11C.

[0141] The image processing device 1210 includes a number of sub-image processors (1212a, 1212b, 1212c), an image generator 1214, and a camera module controller 1216.

[0142] The image processing device 1210 includes sub-image processors (1212a, 1212b, 1212c) the number of which corresponds to the number of camera modules (1100a, 1100b, 1100c).

[0143] Image data generated from each camera module (1100a, 1100b, 1100c) is provided to the corresponding sub-image processor (1212a, 1212b, 1212c) via separate image signal lines (ISLa, ISLb, ISLc). For example, image data generated from camera module 1100a is provided to sub-image processor 1212a via image signal line ISLa, image data generated from camera module 1100b is provided to sub-image processor 1212b via image signal line ISLb, and image data generated from camera module 1100c is provided to sub-image processor 1212c via image signal line ISLc. Such image data transfer is performed using, for example, a camera serial interface (CSI) based on MIPI (Mobile Industry Processor Interface), but the embodiment is not limited thereto.

[0144] On the other hand, in one embodiment, one sub-image processor is arranged to correspond to multiple camera modules. For example, sub-image processor 1212a and sub-image processor 1212c are not implemented separately from each other as shown in the figure, but are integrated into one sub-image processor, and image data provided from camera module 1100a and camera module 1100c is selected via a selection element (e.g., a multiplexer) and then provided to the integrated sub-image processor.

[0145] The image data provided to each sub-image processor (1212a, 1212b, 1212c) is provided to an image generator 1214. The image generator 1214 generates an output image using the image data provided from each sub-image processor (1212a, 1212b, 1212c) in accordance with image generation information or a mode signal.

[0146] Specifically, the image generator 1214 generates an output image by merging at least some of the image data generated from the camera modules 1100a, 1100b, and 1100c having different viewing angles according to the image generation information or the mode signal. The image generator 1214 may also generate an output image by selecting one of the image data generated from the camera modules 1100a, 1100b, and 1100c having different viewing angles according to the image generation information or the mode signal.

[0147] In one embodiment, the image generation information includes a zoom signal or zoom factor, and in one embodiment, the mode signal is based on a mode selected by, for example, a user.

[0148] If the image generation information is a zoom signal (zoom factor) and each camera module (1100a, 1100b, 1100c) has a different field of view (viewing angle), the image generator 1214 operates differently depending on the type of zoom signal. For example, if the zoom signal is a first signal, the image generator 1214 merges the image data output from camera module 1100a with the image data output from camera module 1100c, and generates an output image using the merged image signal and the image data output from camera module 1100b that was not used in the merging. If the zoom signal is a second signal different from the first signal, the image generator 1214 does not merge the image data, but instead selects one of the image data output from each camera module (1100a, 1100b, 1100c) to generate an output image. However, embodiments are not limited to this, and the method of processing image data may be variously modified as needed.

[0149] In one embodiment, the image generator 1214 receives a plurality of image data with different exposure times from at least one of the plurality of sub-image processors (1212a, 1212b, 1212c) and performs high dynamic range (HDR) processing on the plurality of image data to generate merged image data with an increased dynamic range. In one embodiment, the image generator 1214 performs a compensation operation on the second image to generate an image with reduced hue difference from the final image generated in the first mode.

[0150] The camera module controller 1216 provides control signals to each of the camera modules (1100a, 1100b, 1100c). The control signals generated by the camera module controller 1216 are provided to the corresponding camera modules (1100a, 1100b, 1100c) via separate control signal lines (CSLa, CSLb, CSLc).

[0151] Any one of the multiple camera modules (1100a, 1100b, 1100c) can be designated as a master camera (e.g., 1100b) in response to image generation information or a mode signal including a zoom signal, and the remaining camera modules (e.g., 1100a, 1100c) can be designated as slave cameras. Such information is included in a control signal and provided to the corresponding camera modules (1100a, 1100b, 1100c) via separate control signal lines (CSLa, CSLb, CSLc), respectively.

[0152] The camera module operating as the master or slave is changed depending on the zoom factor or the operation mode signal. For example, when the viewing angle of camera module 1100a is wider than that of camera module 1100b and the zoom factor indicates a low zoom magnification, camera module 1100b operates as the master and camera module 1100a operates as the slave. Conversely, when the zoom factor indicates a high zoom magnification, camera module 1100a operates as the master and camera module 1100b operates as the slave.

[0153] In one embodiment, the control signals provided from the camera module controller 1216 to each of the camera modules (1100a, 1100b, 1100c) include a sink enable signal. For example, if the camera module 1100b is the master camera and the camera modules (1100a, 1100c) are slave cameras, the camera module controller 1216 sends a sink enable signal to the camera module 1100b. Upon receiving the sink enable signal, the camera module 1100b generates a sink signal based on the received sink enable signal and provides the generated sink signal to the camera modules (1100a, 1100c) via the sink signal line SSL. The camera modules 1100b and the camera modules (1100a, 1100c) send image data to the application processor 1200 in synchronization with the sink signals.

[0154] In one embodiment, the control signals provided from the camera module controller 1216 to the camera modules (1100a, 1100b, 1100c) include mode information corresponding to the mode signal, and based on the mode information, the camera modules (1100a, 1100b, 1100c) operate in a first operation mode and a second operation mode associated with sensing speeds.

[0155] In a first operating mode, the multiple camera modules (1100a, 1100b, 1100c) generate image signals at a first rate (e.g., generate image signals at a first frame rate), encode the image signals at a second rate higher than the first rate (e.g., encode image signals at a second frame rate higher than the first frame rate), and send the encoded image signals to the application processor 1200. In this case, the second rate is 30 times faster than the first rate or less.

[0156] The application processor 1200 stores the received image signal, i.e., the encoded image signal, in an internal memory 1230 or an external memory 1400 of the application processor 1200, and then reads and decodes the encoded image signal from the internal memory 1230 or the external memory 1400. Then, the application processor 1200 displays image data generated based on the decoded image signal. For example, the decoding is performed by a corresponding sub-image processor among the plurality of sub-image processors (1212a, 1212b, 1212c) of the image processing device 1210. The application processor 1200 performs image processing on the decoded image signal.

[0157] In the second operating mode, the camera modules (1100a, 1100b, 1100c) generate image signals at a third rate lower than the first rate (e.g., generate image signals at a third frame rate lower than the first frame rate) and send the image signals to the application processor 1200. The image signals provided to the application processor 1200 are unencoded signals. The application processor 1200 performs image processing on the received image signals or stores the image signals in the internal memory 1230 or the external memory 1400.

[0158] The PMIC 1300 supplies power, e.g., a power supply voltage, to each of the multiple camera modules (1100a, 1100b, and 1100c). For example, under the control of the application processor 1200, the PMIC 1300 supplies a first power to the camera module 1100a via a power signal line PSLa, a second power to the camera module 1100b via a power signal line PSLb, and a third power to the camera module 1100c via a power signal line PSLc.

[0159] The PMIC 1300 generates power for each of the camera modules (1100a, 1100b, 1100c) and adjusts the power level in response to a power control signal PCON from the application processor 1200. The power control signal PCON includes a power adjustment signal for each operation mode of the camera modules (1100a, 1100b, 1100c). For example, the operation mode may include a low-power mode, and the power control signal PCON includes information about the camera module operating in the low-power mode and the power level to be set. The power levels provided to each of the camera modules (1100a, 1100b, 1100c) may be the same or different. The power levels may be dynamically changed.

[0160] Although the embodiments of the present invention have been described in detail above with reference to the drawings, the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the technical concept of the present invention. [Explanation of symbols]

[0161] 100, 200a, 200b, 200c Image Device 110, 210a, 210b, 210c, 1142 Image sensor 111, 221a, 221b, 221c White Extraction Module 112, 222a, 222b, 222c Crosstalk Compensation Modules 113, 223a, 223b, 223c Bayer Extraction Module 114 pixel array 115 line driver 116 Analog-to-Digital Converter (ADC) 117 Output circuit 118 Control Logic Circuit 120, 220a, 220b, 220c Image signal processor 121 Noise Reduction Module 122 Demosaic Module 123 Color Correction Module 124 Gamma Correction Module 125 Color Conversion Module 130 White Balance Module 140 Luma Extraction Module 150 Guide Filtering Module 151 Sampling Module 152 Full Generation Module 160 White Generation Module 1000 electronic devices 1100 Camera Module Group 1100a~1100c Camera Module 1105 Prism 1106 Center axis 1107 Reflective surface 1110 Optical path bending element (OPFE) 1130 Actuator 1140 Image sensing device 1144 Control Logic 1146 memory 1147 Calibration Data 1150 Storage Device 1200 Application Processor 1210 Image Processing Device 1212a~1212c Sub-image processor 1214 Image Generator 1216 Camera Module Controller 1220 memory controller 1230 internal memory 1300 PM (power management) IC 1400 external memory B1~B8 1st~8th blue color pixels BI Bayer Image BI_U1~BI_U4 Bayer image basic units 1~4 CFDa, CFDb Secondary Capacitance CL column line CSLa~CSLc control signal lines DCG dual conversion signal DT Dual Conversion Transistor FD1~FD4 1st to 4th floating diffusion regions FDa, FDb First and second floating diffusion regions G_FI Green Full Image G_SI Green Sampling Image G, Gb, Gr Green color pixel ( Gb1~Gb8 1st to 8th green color pixels Gr1~Gr8 1st to 8th green color pixels ISLa~ISLc Image signal lines OUT Output signal PCON Power Control Signal PG Pixel Group PG1~PG4 1st~4th pixel group PD1 to PD16 1st to 16th photoelectric conversion elements PDa to PDd 1st to 4th photoelectric conversion elements PI panchromatic image PS pixel set PS1~PS4 1st~4th pixel set PSLa~PSLc Power signal line PX pixels PX1~PX4 1st to 4th pixels R Red color pixel R1~R8 1st~8th red color pixels RAW raw image RAW_U1~RAW_U4 1st~4th raw image basic units RAW_WB White balance low image RG Reset signal RG1, RG2 First and second reset signals RT Reset transistor SE select transistor SF source follower transistor SEL selection signal SEL1, SEL2 First and second selection signals SSL Sync Signal Line T1 to T16 1st to 16th transfer transistors Ta to Td First to fourth transfer transistors TG1~TG16 1st to 16th transfer signals TGa~TGd 1st~4th transfer signals U1~U4 1st~4th basic units VDD power supply voltage W White Pixel W1~W24 1st~24th white pixels WI White Image WI_XT Crosstalk Compensated White Image WI_XT_U1~WI_XT_U4 1st~4th crosstalk compensation white image basic units W+B, W+B_1~W+B_4 White-blue color pixels W+G, W+Gr, W+Gr_1~W+Gr_4 White-green color pixels W+R, W+R_1~W+R_4 White-red color pixels W+G_FI White Green Full Image W+G_SI White Green Sampling Image

Claims

1. a pixel array including a plurality of pixels; a row driver configured to control the plurality of pixels; an analog-to-digital converter configured to digitize the sensed results by the pixel array to generate a first image; The pixel array a first pixel group including a first white pixel and a first color pixel among the plurality of pixels; a second pixel group including a second white pixel and a second color pixel among the plurality of pixels, First pixel data of the first image is generated based on the first white pixel and the first color pixel; second pixel data of the first image is generated based on the second color pixels; The image sensor further comprises at least one processor configured to receive the first image and perform a white extraction operation based on the first image to generate a second image of a white pattern.

2. The image sensor of claim 1 , wherein the at least one processor is further configured to perform a crosstalk compensation operation based on the second image to generate a third image.

3. 3. The image sensor of claim 2, wherein the at least one processor is configured to load pre-stored calibration data and perform a crosstalk compensation operation to multiply each code level of the second image by a corresponding calibration data value from the calibration data to generate the third image.

4. 3. The image sensor of claim 2, wherein the at least one processor is further configured to subtract the third image from the first image to generate a Bayer pattern image.

5. The at least one processor performing a subtraction operation on the first pixel data of the first image by the third image; 5. The image sensor of claim 4, wherein the image sensor is configured not to perform a subtraction operation on the second pixel data of the first image using the third image.

6. 5. The image sensor of claim 4, wherein the second pixel data of the first image is the same as corresponding pixel data of the Bayer pattern image.

7. The at least one processor performing a white balance operation based on the first image to generate and output a fourth image; performing a luma extraction operation based on the fourth image to generate and output a fifth image having an achromatic color; performing a guide filtering operation based on the fifth image and the first image to output a sixth image of white and green patterns and a seventh image of green patterns; 2. The image sensor of claim 1, further comprising: a subtraction operation performed on the sixth image by the seventh image to generate the second image.

8. The image sensor of claim 7 , wherein the at least one processor is further configured to adjust a gain for the first image and perform a white balance operation based on the adjusted gain.

9. the first pixel group includes a green color pixel group having a first portion of the first white pixels and a first green color pixel of the first color pixels; the second color pixel corresponds to a second green color pixel; The at least one processor generating a first sampled image including white-green pixel data among the first pixel data generated based on the green color pixel group, and generating a second sampled image including the second pixel data; 8. The image sensor of claim 7, further configured to generate a sixth image corresponding to the size of the first image based on the first sampled image, and to generate a seventh image corresponding to the size of the first image based on the second sampled image.

10. 1. A method of operating an image sensor including a first group of pixels and a second group of pixels, comprising: In the first readout step, performing a first summation operation on the first pixel group including a first white pixel and a first color pixel, sampling and outputting a first signal based on the first white pixel and the first color pixel; performing a second summation operation for sampling and outputting a second signal based on the second color pixel for the second pixel group including a second white pixel and a second color pixel in the first readout process; converting the first signal and the second signal into a digital signal to generate a raw image; extracting a white image based on the raw image; performing a crosstalk compensation operation based on the white image to generate a crosstalk-compensated white image; and performing a subtraction operation on the raw image by the crosstalk-compensated white image to generate a Bayer image.

11. The step of extracting a white image based on the raw image includes: performing a white balance operation based on the raw image to generate a white-balanced raw image; generating a panchromatic image by performing a luma extraction operation based on the white balanced raw image; generating a white-green full image and a green full image by performing a guided filtering operation based on the panchromatic image and the raw image; 11. The method of claim 10, further comprising: performing a subtraction operation on the white-green full image by the green full image to generate the white image.

12. generating a white-green full image and a green full image by performing a guided filtering operation based on the panchromatic image and the raw image, extracting a white-green sampled image based on the raw image; extracting a green sampled image based on the raw image; generating the white-green full image based on the panchromatic image and the white-green sampled image; and generating the full green image based on the panchromatic image and the green sampled image.

13. The step of performing a first summation operation on the first pixel group including the first white pixel and the first color pixel to sample and output a first signal based on the first white pixel and the first color pixel includes: activating a first selection signal coupled to a selection transistor included in the first pixel group; deactivating a first reset signal coupled to a reset transistor included in the first pixel group; activating and deactivating a first transfer signal connected to the first white pixel; 13. The method of claim 12, further comprising: activating and deactivating second transfer signals coupled to the first color pixels.

14. The step of performing a second summation operation for sampling and outputting a second signal based on the second color pixel for the second pixel group including the second white pixel and the second color pixel includes: activating a first selection signal coupled to a selection transistor included in the second pixel group; deactivating a first reset signal coupled to a reset transistor included in the second pixel group; deactivating a third transfer signal connected to the second white pixel; 13. The method of claim 12, further comprising: activating and deactivating a fourth transfer signal coupled to the second color pixels.

15. an image sensor including a pixel set, configured to generate a raw image by performing a first summation operation for each of first through third pixel groups of the pixel set to generate first pixel data based on all pixels that share a floating diffusion region, and performing a second summation operation for a fourth pixel group of the pixel set to generate second pixel data based on only a portion of pixels that share a floating diffusion region; an image signal processor configured to perform signal processing operations on image data received from the image sensor; the pixel set includes first to fourth pixel groups; the first pixel group includes a first white pixel and a first green pixel; the second pixel group includes a second white pixel and a red color pixel; the third pixel group includes a third white pixel and a blue color pixel; The fourth pixel group includes a fourth white pixel and a second green pixel.

Images