Demosaicing device and demosaicing method of image sensor
By employing a pixel array with switchable and transparent pixels in image sensors, the demosaicing process is simplified, allowing for high-resolution and high-quality image reconstruction by capturing both full-color and color images with reduced signal loss.
Patent Information
- Application Number
- CN202410502464.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-01-12
- Filing Date
- 2024-04-25
- Publication Date
- 2025-07-15
AI Technical Summary
The quality and resolution of the images after the existing image sensor is reconstructed in demosaic processing are insufficient, and the existing methods cannot be effectively improved.
Using a pixel array including switchable pixels and transparent pixels, a high-resolution color image is reconstructed by applying bias voltage to the electrochromic layer of the switchable pixels to switch between the transparent state and the colored state, combined with weighted averaging operations, signal deletion and interpolation, etc.
This enables full-color and color images to be captured using the same pixel array, simplifies demosaic processing, reduces signal loss, and reconstructs high-resolution and high-quality images.
Smart Images
Figure CN120321516A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an image sensor, and more particularly to a demosaicing device and a demosaicing method for an image sensor. Background Art
[0002] After an image sensor with a pixel array captures an image, a demosaicing method can be performed on the captured image to obtain a reconstructed image. Since the quality and resolution of the reconstructed image largely depend on the demosaicing processing method, it is essential to improve the demosaicing method and the corresponding image sensor. Summary of the Invention
[0003] According to some embodiments of the present disclosure, a demosaicing method for an image sensor includes the following steps. Use a pixel array including a plurality of switchable pixels and a plurality of transparent pixels to capture a panchromatic image, where the switchable pixels and the transparent pixels are in a transparent state. Use the pixel array to capture a color image, where the switchable pixels are in a colored state and the transparent pixels are in a transparent state. Perform a weighted average operation and signal subtraction on the color image and the panchromatic image to generate a plurality of first-resolution color images, where the resolution of the first-resolution color images is the same as that of the panchromatic image and the color image. Fuse the first-resolution color images with the panchromatic image to generate a plurality of demosaiced color images.
[0004] In some embodiments, each of the switchable pixels includes an electrochromic layer, where when each of the switchable pixels is in a transparent state, the electrochromic layer is transparent. When each of the switchable pixels is in a colored state, the electrochromic layer is colored.
[0005] In some embodiments, when each of the switchable pixels is in a transparent state, the electrochromic layer is not biased. When each of the switchable pixels is in a colored state, the electrochromic layer is biased.
[0006] In some embodiments, performing a weighted average operation and signal subtraction to generate a first-resolution color image includes: generating a subtracted color image from the color image and the panchromatic image, where the resolution of the subtracted color image is lower than that of the first-resolution color image.
[0007] In some embodiments, generating the subtracted color image includes: performing a weighted average operation on the color image and the panchromatic image to generate an average color image and an average panchromatic image, and performing signal subtraction by subtracting the signal of the average panchromatic image from the signal of the average color image to generate the subtracted color image.
[0008] In some embodiments, generating a reduced-color image includes: performing signal reduction to generate a reduced image by subtracting the signal of a panchromatic image from the signal of a color image, and performing a weighted average operation on the reduced image to generate a reduced-color image.
[0009] In some embodiments, the method further includes: performing interpolation on the reduced-color image to generate a plurality of second-resolution color images, where the resolution of the second-resolution color images is the same as the resolution of the reduced-color image.
[0010] In some embodiments, the method further includes: sampling the second-resolution color images to generate first-resolution color images.
[0011] In some embodiments, a pixel group of a pixel array includes two of the switchable pixels and two of the transparent pixels, and the two switchable pixels and the two transparent pixels are arranged in a 2-row × 2-column array.
[0012] In some embodiments, performing a weighted average operation on the panchromatic image includes averaging a plurality of signals of the switchable pixels in the transparent state and a plurality of signals of the transparent pixels in the transparent state to generate an average signal of the pixel group.
[0013] In some embodiments, performing a weighted average operation on the color image includes: averaging a plurality of signals of the switchable pixels in the colored state to generate an average signal of the pixel group.
[0014] In some embodiments, performing signal reduction on the color image and the panchromatic image includes: subtracting the signal of one of the switchable pixels in the transparent state from the signal of one of the switchable pixels in the colored state to generate a reduced signal of the one of the switchable pixels.
[0015] In some embodiments, performing signal reduction on the color image and the panchromatic image includes: subtracting the signal of one of the transparent pixels of the panchromatic image from the signal of one of the transparent pixels of the color image to generate a reduced signal of zero.
[0016] In some embodiments, the pixel array includes two first pixel groups, a second pixel group, and a third pixel group, where each of the first pixel group, the second pixel group, and the third pixel group includes two of the switchable pixels and two of the transparent pixels, and the first pixel group, the second pixel group, and the third pixel group are arranged in a 2-row × 2-column array.
[0017] In some embodiments, the switchable pixels in the colored state of the first pixel group are green pixels, the switchable pixels in the colored state of the second pixel group are red pixels, and the switchable pixels in the colored state of the third pixel group are blue pixels.
[0018] In some embodiments, the transparent pixels and the switchable pixels are arranged alternately.
[0019] According to some embodiments of the present disclosure, a demosaicing device of an image sensor includes a pixel array that captures a panchromatic image and a color image. The pixel array includes switchable pixels and transparent pixels adjacent to the switchable pixels. The switchable pixels include a first photosensitive element and an electrochromic layer above the first photosensitive element, and the transparent pixels include a second photosensitive element. When the electrochromic layer is not biased, the switchable pixels and the transparent pixels are in a transparent state. When the electrochromic layer is biased, the switchable pixels are in a colored state, and the transparent pixels are in a transparent state. The demosaicing device further includes an arithmetic circuit that performs a weighted average operation and signal reduction on the color image and the panchromatic image to generate a plurality of first-resolution color images. The demosaicing device further includes a correction circuit that fuses the first-resolution color images with the panchromatic image to generate a plurality of demosaiced color images.
[0020] In some embodiments, the first photosensitive element and the electrochromic layer are independently controlled by different transistors.
[0021] In some embodiments, the switchable pixels further include a bottom electrode located between the first photosensitive element and the electrochromic layer, an electrolyte layer located on the electrochromic layer, an ion storage layer located on the electrolyte layer, a top electrode located on the ion storage layer, and a microlens array located on the top electrode.
[0022] In some embodiments, the electrolyte layer, the ion storage layer, the top electrode, and the microlens array extend above the second photosensitive element.
[0023] According to the above embodiments, the demosaicing device of the image sensor includes switchable pixels and transparent pixels. By applying or not applying a bias voltage to the electrochromic layer in the switchable pixels, the switchable pixels can be switched between a transparent state and a colored state. Therefore, the same pixel array can be used to capture a panchromatic image and a color image, thereby reconstructing a high-resolution image and simplifying the demosaicing process of the image sensor to reduce signal loss. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Aspects of the present disclosure are best understood from the following detailed description when read with the accompanying drawings. It should be noted that, in accordance with standard practice in the industry, various features are not drawn to scale. In fact, for the sake of clear discussion, the dimensions of various features may be arbitrarily increased or decreased.
[0025] Figure 1A A cross-sectional view of a demosaicing device of an image sensor is illustrated according to some embodiments of the present disclosure, where the demosaicing device has an unbiased electrochromic layer.
[0026] Figure 1B A cross-sectional view of a demosaicing device for an image sensor is illustrated according to some embodiments of the present disclosure, where the demosaicing device has a biased electrochromic layer.
[0027] Figure 2A A schematic diagram of a pixel array of a demosaicing device for capturing a panchromatic image is illustrated according to some embodiments of the present disclosure.
[0028] Figure 2B A schematic diagram of a pixel array of a demosaicing device for capturing a color image is illustrated according to some embodiments of the present disclosure.
[0029] Figure 3A A flowchart of a demosaicing method is illustrated according to some embodiments of the present disclosure.
[0030] Figure 3B Illustrating the use of Figure 3A in the demosaicing method to perform demosaicing processing on an image.
[0031] Figure 4A A flowchart of a demosaicing method is illustrated according to some embodiments of the present disclosure.
[0032] Figure 4B Illustrating the use of Figure 4A in the demosaicing method to perform demosaicing processing on an image.
[0033] Among them, the reference numerals are explained as follows:
[0034] 10: Demosaicing device
[0035] 100: Pixel array
[0036] 100C: Transparent pixel
[0037] 100S: Switchable pixel
[0038] 110: Pixel group
[0039] 110G: First pixel group
[0040] 110R: Second pixel group
[0041] 110B: Third pixel group
[0042] 130: Arithmetic circuit
[0043] 132: Processing circuit
[0044] 134: Calibration circuit
[0045] 140: Arithmetic circuit
[0046] 142: Processing circuit
[0047] 144: Calibration Circuit
[0048] 200: Substrate
[0049] 210: Circuit Layer
[0050] 212, 214: Transistors
[0051] 220: Passivation Layer
[0052] 222, 224: Photosensitive Elements
[0053] 230: Electrochromic Layer
[0054] 240: Bottom Electrode
[0055] 250: Electrolyte Layer
[0056] 260: Ion Storage Layer
[0057] 270: Top Electrode
[0058] 280: Contact Through-Hole
[0059] 290: Microlens Array
[0060] 300a: Panoramic Image
[0061] 300b: Color Image
[0062] 310a: Average Panoramic Image
[0063] 310b: Average Color Image
[0064] 320: Reduced Color Image
[0065] 330: Second Resolution Color Image
[0066] 340: First Resolution Color Image
[0067] 350: Demosaiced Color Image
[0068] 400a: Panoramic Image
[0069] 400b: Color Image
[0070] 410: Reduced Image
[0071] 420: Reduced Color Image
[0072] 430: Second Resolution Color Image
[0073] 440: First Resolution Color Image
[0074] 450: Demosaiced Color Image
[0075] S300: Method
[0076] S310, S320, S330, S340, S350, S360, S370, S380, S390: Steps
[0077] S400: Method
[0078] S410, S420, S430, S440, S450, S460, S470, S480: Steps
[0079] V: Bias voltage Detailed implementation manners
[0080] To implement different features of the mentioned subject matter, the following disclosure provides many different embodiments or examples. The following describes specific examples of components, configurations, etc. to simplify the present disclosure. Of course, these are merely examples and not restrictive. For example, in the following description, forming the first feature above or on top of the second feature may include embodiments where the first feature and the second feature are formed in direct contact, and may also include embodiments where additional features are formed between the first feature and the second feature so that the first feature and the second feature may not be in direct contact. Additionally, the present disclosure may repeat reference numerals and / or letters in various examples. This repetition is for the purpose of simplicity and clarity and does not in itself indicate a relationship between the various embodiments and / or configurations discussed.
[0081] In addition, spatial relative terms such as "under", "below", "lower", "above", "upper", etc. may be used herein to facilitate the description of the relationship of one element or feature to another element or feature as shown in the figures. Except for the orientation shown in the figures, the spatial relative terms are intended to include different orientations of the device in use or operation. The device may be oriented in other ways (rotated 90 degrees or in other directions), and the spatial relative description symbols used herein may be interpreted accordingly.
[0082] The present disclosure provides a demosaicing device and a demosaicing method for an image sensor. The demosaicing device includes switchable pixels and transparent pixels arranged in a pixel array. Since the switchable pixels can be switched between a clear state and a color state according to an electrochromic layer in the switchable pixels, a panchromatic image and a chromic image of an object can be captured using the same pixel array of the demosaicing device. The panchromatic image and the chromic image can be processed together through a simplified operation of the demosaicing method, thereby providing a high-resolution reconstructed image and reducing signal loss during demosaicing processing.
[0083] According to some embodiments of the present disclosure, Figure 1A and Figure 1B FIG. shows a cross-sectional view of a demosaicing device 10 of an image sensor. The demosaicing device 10 includes a plurality of switchable pixels 100S and a plurality of transparent pixels 100C, wherein each transparent pixel 100C is arranged adjacent to at least one of the switchable pixels 100S. The pixel array (e.g., Figure 2A and Figure 2B the pixel array 100 in) formed by the switchable pixels 100S and the transparent pixels 100C in the demosaicing device 10 can be used to capture a panchromatic image and a chromic image, which will be further discussed in detail below.
[0084] Specifically, the switchable pixel 100S includes a substrate 200, a circuit layer 210 above the substrate 200, a photosensitive element 222 in a passivation layer 220 above the circuit layer 210, and an electrochromic layer 230 above the photosensitive element 222. A transistor 212 in the circuit layer 210 controls the photosensitive element 222 to convert incident light into an electronic signal. A transistor 214 in the circuit layer 210 controls the electrochromic layer 230 to switch between a transparent appearance and a colored appearance. In some embodiments, different transistors 212 and 214 can be used to independently control the photosensitive element 222 and the electrochromic layer 230, so that the photosensitive element 222 can generate an electronic signal regardless of whether the electrochromic layer 230 is transparent or colored. For example, the transistor 212 for the photosensitive element 222 can include a set of sub-transistors, and the transistor 214 for the electrochromic layer 230 can be a single transistor other than the sub-transistors.
[0085] When the electrochromic layer 230 is transparent, incident light having a wide wavelength range can penetrate the electrochromic layer 230 and reach the photosensitive element 222. When the electrochromic layer 230 switches to a colored appearance, the electrochromic layer 230 first filters the incident light before the photosensitive element 222 receives the incident light. In some embodiments, the switchable pixel 100S further includes a bottom electrode 240 between the photosensitive element 222 and the electrochromic layer 230, an electrolyte layer 250 disposed on and in contact with the electrochromic layer 230, an ion storage layer 260 disposed on the electrolyte layer 250, and a top electrode 270 disposed on the ion storage layer 260 to switch between a transparent appearance and a colored appearance.
[0086] The bottom electrode 240 can be electrically connected to the transistor 214 through a contact via 280 in the passivation layer 220, such that the bottom electrode 240 and the top electrode 270 can apply a bias to the electrochromic layer 230 under the control of the transistor 214. When a bias is applied to the electrochromic layer 230, a redox reaction occurs in the electrochromic layer 230, causing a change in the valence amount of the electrochromic layer 230. Therefore, after the electrochromic layer 230 is biased, the transparent electrochromic layer 230 can be switched to a colored electrochromic layer 230. During the bias, the electrolyte layer 250 and the ion storage layer 260 can serve as the electron path and electron storage of the electrochromic layer 230 to achieve the redox reaction. In some embodiments, the bottom electrode 240 and the top electrode 270 can be formed of a transparent conductive material to reduce the loss before the incident light reaches the photosensitive element 222.
[0087] When the electrochromic layer 230 is Figure 1A not biased as shown, the electrochromic layer 230 is transparent and allows the incident light to penetrate the electrochromic layer 230 with little absorption by the electrochromic layer 230. Therefore, the incident light reaching the photosensitive element 222 has a wide wavelength range, providing the switchable pixel 100S in the transparent state. Alternatively, when the electrochromic layer 230 is Figure 1B biased as shown, the electrochromic layer 230 switches to a colored appearance (as shown by the dots in Figure 1B ) and allows the incident light within a specific wavelength range to penetrate. The incident light is filtered by the electrochromic layer 230 before reaching the photosensitive element 222, providing the switchable pixel 100S in the colored state. In other words, the biased electrochromic layer 230 serves as a color filter for the switchable pixel 100S, such that Figure 1B the spectrum of the switchable pixel 100S in
[0088] The transparent pixel 100C adjacent to the switchable pixel 100S has a structure similar to that of the switchable pixel 100S, but the transparent pixel 100C does not have the electrochromic layer 230. As Figure 1A and Figure 1B shown, the transparent pixel 100C includes a substrate 200, a circuit layer 210 above the substrate 200, and a photosensitive element 224 in the passivation layer 220. The photosensitive element 224 in the transparent pixel 100C does not overlap with the electrochromic layer 230. Therefore, even when the electrochromic layer 230 in the switchable pixel 100S is under bias voltage, the light received by the transparent pixel 100C remains unfiltered. Therefore, Figure 1A the transparent pixel 100C in Figure 1B and the transparent pixel 100C in
[0089] are both in a transparent state.
[0090] In some embodiments, the electrolyte layer 250, the ion storage layer 260, and the top electrode 270 in the switchable pixel 100S can extend above the photosensitive element 224 in the transparent pixel 100C to maintain the structural balance of the demosaicing device 10. For example, the transparent pixel 100C can also include an electrolyte layer 250, an ion storage layer 260, and a top electrode 270 above the photosensitive element 224, where these layers have flush top surfaces in the switchable pixel 100S and the transparent pixel 100C respectively. In such embodiments, the switchable pixel 100S and the transparent pixel 100C can further include a microlens array 290 on the top electrode 270, where the microlens array 290 extends above the photosensitive element 222 and the photosensitive element 224 to increase the incident light entering the switchable pixel 100S and the transparent pixel 100C.
[0090] According to some embodiments of the present disclosure, Figure 2A and Figure 2B respectively show schematic diagrams of a pixel array 100 of a demosaicing device of an image sensor. The pixel array 100 includes Figure 1A and Figure 1B the switchable pixel 100S and the transparent pixel 100C in the demosaicing device 10 shown, where the switchable pixel 100S and the transparent pixel 100C are alternately arranged in a two-dimensional array. For example, the pixel array 100 can be composed of eight switchable pixels 100S and eight transparent pixels 100C, and the switchable pixel 100S and the transparent pixel 100C are alternately arranged in a 4-row × 4-column two-dimensional array. The pixel array 100 can be the smallest repeating unit in the demosaicing device, such that the demosaicing device can capture a mosaic image (not shown) through the pixel array 100.
[0091] Figure 2A the pixel array 100 in Figure 1AThe demosaicing device 10 therein, where the electrochromic layer 230 in the switchable pixel 100S is not under bias voltage and remains transparent. In other words, Figure 2A Both the switchable pixel 100S and the transparent pixel 100C in the pixel array 100 therein are in a transparent state. Since the incident light reaching the photosensitive elements of the switchable pixel 100S and the transparent pixel 100C is not filtered by the unbiased electrochromic layer 230, Figure 2A the pixel array 100 therein can be used to capture a full-color image and provide the luminance signal of the object.
[0092] Alternatively, Figure 2B the pixel array 100 therein can correspond to Figure 1B the demosaicing device 10 therein, where the electrochromic layer 230 in the switchable pixel 100S is under bias voltage and switches to a colored appearance. In other words, Figure 2B the switchable pixel 100S in the pixel array 100 therein is in a colored state, while the transparent pixel 100C is in a transparent state. Since the incident light reaching the photosensitive element of the switchable pixel 100S is filtered by the biased electrochromic layer 230, Figure 2B the pixel array 100 therein can be used to capture a color image and provide the chromaticity signal of the object.
[0093] It should be noted that the same pixel array 100 can be used to capture the full-color image and the color image of the object. Since the switchable pixel 100S can be easily switched between the transparent state and the colored state by applying a bias voltage to the electrochromic layer, the demosaicing device including the pixel array 100 can quickly capture the full-color image and the color image of the same object. Two object images with different signals can be demosaiced together to reconstruct an image with high resolution and high quality.
[0094] In some embodiments, the switchable pixels 100S and the transparent pixels 100C of the pixel array 100 can be grouped into multiple pixel groups 110. The pixel group 110 is the smallest repeating unit for some processing operations in the demosaicing method of the pixel array 100, such as Figure 3A the demosaicing method S300 in Figure 4A and the demosaicing method S400 in
[0095] In some embodiments where pixel group 110 includes two switchable pixels 100S and two transparent pixels 100C, pixel array 100 may be composed of four pixel groups 110 having multiple spectra when the switchable pixels 100S are in the colored state. As Figure 2B shown, pixel array 100 may include two first pixel groups 110G, one second pixel group 110R, and one third pixel group 110B, where each of the first pixel group 110G, the second pixel group 110R, and the third pixel group 110B includes two switchable pixels 100S and two transparent pixels 100C. When the electrochromic layer of the switchable pixel 100S is biased, the switchable pixels 100S of the first pixel group 110G, the second pixel group 110R, and the third pixel group 110B have different spectra. The first pixel group 110G, the second pixel group 110R, and the third pixel group 110B are arranged in a 2-row × 2-column array, and any two adjacent pixel groups 110 have different spectra when the switchable pixels 100S are in the colored state.
[0096] For example, pixel array 100 may include a first pixel group 110G located at the upper left corner and the lower right corner of pixel array 100, a second pixel group 110R located at the upper right corner of pixel array 100, and a third pixel group 110B located at the lower left corner of pixel array 100. The switchable pixels 100S in the colored state in the first pixel group 110G are green pixels that allow green light to pass through. The switchable pixels 100S in the colored state in the second pixel group 110R are red pixels that allow red light to pass through. The switchable pixels 100S in the colored state in the third pixel group 110B are blue pixels that allow blue light to pass through. In such an embodiment, since the human eye is more sensitive to green light, the number of the first pixel groups 110G in a pixel array 100 may be more than the number of the second pixel group 110R or the third pixel group 110B to capture an image with higher chromatic accuracy. It should be noted that the combination of green pixels / red pixels / blue pixels is one embodiment of pixel array 100, and the switchable pixels 100S of pixel array 100 in some other embodiments may include but are not limited to cyan, magenta, yellow, or other spectra.
[0097] As described above, the demosaicing device 10 may use the pixel array 100 to capture a panchromatic image and a color image. Since the panchromatic image and the color image of the same object are captured using the same pixel array 100, the panchromatic image and the color image can be used together in the demosaicing method of the image sensor to simplify the operation of reconstructing the image. In addition, the pixels of the panchromatic image may match the pixels of the color image, so the demosaicing method using the panchromatic image and the color image can reconstruct an image with high resolution and high quality.
[0098] According to some embodiments of the present disclosure,Figure 3A The flowchart of the demosaicing method S300 is shown, Figure 3B The use of Figure 3A in the method S300 for demosaicing the image is shown. According to some other embodiments of the present disclosure, Figure 4A The flowchart of the demosaicing method S400 is shown, Figure 4B The use of Figure 4A in the method S400 for demosaicing the image is shown. The method S300 and the method S400 can be used, for example, to reconstruct Figure 1A and Figure 1B the images captured by the demosaicing device 10 therein. However, those skilled in the art should understand that the method S300 and the method S400 can also be used to reconstruct the images captured by a demosaicing device with switchable pixels within the scope of the present disclosure. Additionally, additional operations can be provided before, during, or after the method S300 and the method S400, and in other embodiments, some of the operations can be replaced, deleted, or moved.
[0099] The method S300 described below in conjunction with Figure 3A and Figure 3B begins with steps S310 and S320, and uses the pixel array 100 of the demosaicing device to capture the panchromatic image 300a and the color image 300b of an object, such as Figure 2A and Figure 2B the pixel array 100 therein. Specifically, in step S310, the switchable pixels and the transparent pixels of the pixel array can be set to the transparent state to capture the panchromatic image 300a. In step S320, a bias voltage is applied to the switchable pixels to switch them to the colored state, while the transparent pixels remain in the transparent state to capture the color image 300b. In some embodiments, the order of steps S310 and S320 can be interchanged, and this is not intended to limit the present disclosure.
[0100] In other words, the same pixel array is used to separately capture the panchromatic image 300a and the color image 300b, but the switchable pixels of the pixel array are in different states when capturing the two images. Therefore, the panchromatic image 300a and the color image 300b have the same resolution. Since both the panchromatic image 300a and the color image 300b include all the pixels of the pixel array, the panchromatic image 300a and the color image 300b can have a high resolution.
[0101] Next, method S300 proceeds to steps S330 to S370, where a weighted average calculation, signal subtraction, interpolation, and upsampling are performed on the panchromatic image 300a and the color image 300b using the arithmetic circuit 130 and the processing circuit 132 of the demosaicing device to generate a plurality of first-resolution color images 340. After steps S330 to S370, the resolution of the first-resolution color images 340 is the same as the resolution of the panchromatic image 300a and the color image 300b.
[0102] In step S330, a weighted average calculation is performed on the panchromatic image 300a using the arithmetic circuit 130 to generate an average panchromatic image 310a. Specifically, the weighted average calculation is based on pixel groups of the pixel array such that the average signal of multiple pixels in a pixel group represents the signal of the pixel group. In an embodiment where a pixel group includes four pixels, the signals of two switchable pixels in a pixel group that are in the transparent state and the signals of two transparent pixels in the transparent state are averaged to generate the average signal of the pixel group. As a result, the panchromatic image 300a represented by a 4-row × 4-column pixel array generates an average panchromatic image 310a represented by a 2-row × 2-column pixel group array. Therefore, the resolution of the average panchromatic image 310a is lower than the resolution of the panchromatic image 300a.
[0103] In step S340, a weighted average calculation is then performed on the color image 300b using the arithmetic circuit 130 to generate an average color image 310b. The weighted average calculation performed on the color image 300b is different from the weighted average calculation performed on the panchromatic image 300a. For the color image 300b, the signals of two switchable pixels in a pixel group that are in the colored state are averaged to generate the average signal of the pixel group. The signals of two transparent pixels in the transparent state may not be used for the weighted average calculation. As a result, the color image 300b represented by a 4-row × 4-column pixel array generates an average color image 310b represented by a 2-row × 2-column pixel group array. Therefore, the resolution of the average color image 310b is lower than the resolution of the color image 300b but the same as the resolution of the average panchromatic image 310a.
[0104] In step S350, the arithmetic circuit 130 performs signal subtraction on the average panchromatic image 310a and the average color image 310b to generate a subtracted color image 320. Specifically, the signal of the average panchromatic image 310a is subtracted from the signal of the average color image 310b to generate the signal of the subtracted color image 320. The signal subtraction is based on pixel groups of the pixel array. In other words, the average signal of a pixel group of the average color image 310b is subtracted from the average signal of the corresponding pixel group of the average panchromatic image 310a to generate the subtracted average signal of a pixel group of the subtracted color image 320. Therefore, the resolution of the subtracted color image 320 is the same as the resolution of the average color image 310b.
[0105] In step S360, the processing circuit 132 then performs interpolation on the subtracted color image 320 to generate a plurality of second-resolution color images 330. Specifically, the pixel array of the subtracted color image 320 includes switchable pixels having different spectra. Interpolation is performed on the subtracted color image 320 to generate a plurality of second-resolution color images 330, where the number of the second-resolution color images 330 corresponds to the number of spectra included in the subtracted color image 320. The subtracted average signal of a pixel group included in the subtracted color image 320 represents the signal of all pixel groups of one second-resolution color image 330.
[0106] For example, the subtracted color image 320 may include two first pixel groups having a first spectrum, one second pixel group having a second spectrum, and one third pixel group having a third spectrum. After performing interpolation, three second-resolution color images 330 are generated corresponding to the first spectrum, the second spectrum, and the third spectrum. The plurality of pixel groups of one of the second-resolution color images 330 has the subtracted average signal obtained by averaging the two first pixel groups. The plurality of pixel groups of the other two of the second-resolution color images 330 respectively has the subtracted average signal of the second pixel group and the subtracted average signal of the third pixel group. Therefore, the resolution of the second-resolution color images 330 is the same as the resolution of the subtracted color image 320.
[0107] In step S370, the processing circuit 132 is then used to sample the second-resolution color image 330 to generate the first-resolution color image 340. Each second-resolution color image 330 is sampled into one of the first-resolution color images 340, so the number of first-resolution color images 340 is the same as the number of second-resolution color images 330. For example, the pixel array of the panchromatic image 300a has sixteen pixels arranged in 4 rows × 4 columns, while the pixel group array of the second-resolution color image 330 has four pixel groups arranged in 2 rows × 2 columns. The sampling process divides the four pixel groups of the second-resolution color image 330 into sixteen pixels to generate the first-resolution color image 340. Therefore, the resolution of the first-resolution color image 340 is higher than that of the second-resolution color image 330, but the same as the resolution of the panchromatic image 300a and the color image 300b.
[0108] Next, the method S300 proceeds to step S380, where the correction circuit 134 of the demosaicing device is used to fuse the first-resolution color image 340 and the panchromatic image 300a to generate a plurality of demosaiced color images 350. Specifically, the signal of one of the first-resolution color images 340 is fused with the signal of the panchromatic image 300a to generate one of the demosaiced color images 350, such that the demosaiced color image 350 includes both the object luminance and object chrominance signals. The plurality of first-resolution color images 340 and the panchromatic image 300a are fused independently, so the number of demosaiced color images 350 is the same as the number of first-resolution color images 340. Since the same number of pixels is maintained during step S370, the resolution of the demosaiced color images 350 is the same as the resolution of the first-resolution color images 340 (i.e., the resolution of the panchromatic image 300a and the color image 300b).
[0109] The method S300 can further proceed to step S390, where the object image is reconstructed using the demosaiced color images 350. Since the demosaiced color images 350 are generated using the panchromatic image 300a and the color image 300b captured by all the pixels of the pixel array, the reconstructed image can have high resolution and high quality. Additionally, since the demosaiced color images 350 are generated using the panchromatic image 300a and the color image 300b captured by the same pixel array, the operation of the method S300 can be simplified and signal loss during processing can be reduced.
[0110] The following combines Figure 4A and Figure 4B to describe the method S400. The method S400 starts with steps S410 and S420, using the pixel array 100 of the demosaicing device to capture the panchromatic image 400a and the color image 400b of the object, such as Figure 2A and Figure 2BThe pixel array 100 in it. The operations of step S410 and step S420 are similar to those of step S310 and step S320, so they are not described in detail here.
[0111] Next, method S400 proceeds to step S430 to step S460, and uses the arithmetic circuit 140 and the processing circuit 142 of the demosaicing device to perform weighted average operations, signal deletion, interpolation, and sampling on the panchromatic image 400a and the color image 400b to generate a plurality of first-resolution color images 440. After step S430 to step S460, the resolution of the first-resolution color image 440 is the same as that of the panchromatic image 400a and the color image 400b.
[0112] In step S430, the arithmetic circuit 140 is used to perform signal deletion on the panchromatic image 400a and the color image 400b to generate a deleted image 410. Specifically, the signal of the panchromatic image 400a is deleted from the signal of the color image 400b to generate the signal of the deleted image 410. The execution of signal deletion is based on the pixels of the pixel array. For transparent pixels, the signal of a corresponding transparent pixel in the panchromatic image 400a is deleted from the signal of a transparent pixel in the color image 400b to generate the deleted signal of a transparent pixel in the deleted image 410. Since the transparent pixels are in a transparent state when the panchromatic image 400a and the color image 400b are captured, the deleted signal of the transparent pixel can be zero.
[0113] For switchable pixels, the signal of a corresponding switchable pixel in the transparent state in the panchromatic image 400a is deleted from the signal of a switchable pixel in the colored state in the color image 400b to generate the deleted signal of a switchable pixel in the deleted image 410. Therefore, the resolution of the deleted image 410 is the same as that of the panchromatic image 400a and the color image 400b.
[0114] In step S440, then the arithmetic circuit 140 is used to perform a weighted average operation on the deleted image 410 to generate a deleted color image 420. Specifically, the weighted average operation is executed based on the pixel groups of the pixel array, such that the average signal of multiple pixels in a pixel group represents the signal of a pixel group. In an embodiment where a pixel group includes four pixels, the deleted signals of two switchable pixels are averaged to generate the average deleted signal of the pixel group. The signals of two transparent pixels may not be used for the weighted average operation. As a result, the deleted image 410 represented by a 4-row × 4-column pixel array generates a deleted color image 420 represented by a 2-row × 2-column pixel group array. Therefore, the resolution of the deleted color image 420 is lower than that of the deleted image 410.
[0115] In step S450, the processing circuit 142 is then used to perform interpolation on the pruned color image 420 to generate a plurality of second-resolution color images 430. The operation of step S450 is similar to that of step S360, and thus will not be described in detail herein. After step S450, the resolution of the second-resolution color images 430 is the same as that of the pruned color image 420.
[0116] In step S460, the processing circuit 142 is then used to sample the second-resolution color images 430 to generate a first-resolution color image 440. The operation of step S460 is similar to that of step S370, and thus will not be described in detail herein. After step S460, the resolution of the first-resolution color image 440 is higher than that of the second-resolution color images 430, but the same as that of the panchromatic image 400a and the color image 400b.
[0117] Next, method S400 proceeds to step S470, where the correction circuit 144 of the demosaicing device is used to fuse the first-resolution color image 440 with the panchromatic image 400a to generate a plurality of demosaiced color images 450. Method S400 can further proceed to step S480 to reconstruct the object image using the demosaiced color images 450. The operations of step S470 and step S480 are similar to those of step S380 and step S390, and thus will not be described in detail herein. The resolution of the demosaiced color images is the same as that of the panchromatic image and the color image, resulting in a reconstructed image with high resolution and high quality. Additionally, the simplified operation of method S400 can reduce signal loss during processing.
[0118] According to the above embodiments, the demosaicing device of the image sensor of the present disclosure includes switchable pixels and transparent pixels arranged in a pixel array. The switchable pixels can be switched between a transparent state and a colored state by applying or not applying a bias voltage to the electrochromic layer in the switchable pixels, while the transparent pixels without an electrochromic layer remain in the transparent state. Therefore, in the demosaicing method, the same pixel array can be used to capture the panchromatic image and the color image of an object to reconstruct a high-resolution image. Additionally, the demosaicing method of the image sensor of the present disclosure has a simplified operation, and thus can reduce signal loss during demosaicing processing.
[0119] The features of some embodiments are outlined above so that those skilled in the art can better understand the concepts of the present disclosure. Those skilled in the art should understand that they can easily use the present disclosure as a basis for designing or modifying other processes and structures to achieve the same purposes and / or achieve the same advantages as the embodiments described herein. Those skilled in the art should also understand that such equivalent structures do not depart from the spirit and scope of the present disclosure, and various changes, substitutions, and alterations can be made without departing from the spirit and scope of the present disclosure.
Claims
1. A method for demosaicking an image sensor, characterized in that, Comprising: Taking a panchromatic image using a pixel array including a plurality of switchable pixels and a plurality of transparent pixels, wherein the plurality of switchable pixels and the plurality of transparent pixels are in a transparent state; Taking a color image using the pixel array, wherein the plurality of switchable pixels are in a colored state and the plurality of transparent pixels are in the transparent state; Performing a weighted average operation and a signal deletion on the color image and the panchromatic image to generate a plurality of first-resolution color images, wherein a resolution of the plurality of first-resolution color images is the same as a resolution of the panchromatic image and the color image; and Fusing the plurality of first-resolution color images with the panchromatic image to generate a plurality of demosaiced color images.
2. The demosaicing method according to claim 1, wherein each of the plurality of switchable pixels includes an electrochromic layer, wherein when each of the plurality of switchable pixels is in the transparent state, the electrochromic layer is transparent and not biased, wherein when each of the plurality of switchable pixels is in the colored state, the electrochromic layer is colored and biased.
3. The demosaicing method according to claim 1, wherein performing the weighted average operation and the signal deletion to generate the plurality of first-resolution color images includes: Performing the weighted average operation on the color image and the panchromatic image to generate an average color image and an average panchromatic image; And Performing the signal deletion by deleting a signal of the average panchromatic image from a signal of the average color image to generate a deleted color image, wherein a resolution of the deleted color image is lower than the resolution of the plurality of first-resolution color images.
4. The demosaicing method according to claim 1, wherein performing the weighted average operation and the signal deletion to generate the plurality of first-resolution color images includes: Performing the signal deletion by deleting a signal of the panchromatic image from a signal of the color image to generate a deleted image; and Performing the weighted average operation on the deleted image to generate a deleted color image, wherein a resolution of the deleted color image is lower than the resolution of the plurality of first-resolution color images.
5. The demosaicing method according to claim 1, wherein performing the weighted average operation and the signal deletion to generate the plurality of first-resolution color images includes: Generating a deleted color image from the color image and the panchromatic image, wherein a resolution of the deleted color image is lower than the resolution of the plurality of first-resolution color images; Performing an interpolation on the deleted color image to generate a plurality of second-resolution color images, wherein a resolution of the plurality of second-resolution color images is the same as the resolution of the deleted color image; and Performing a sampling on the plurality of second-resolution color images to generate the plurality of first-resolution color images.
6. The demosaicing method according to claim 1, wherein a pixel group of the pixel array includes two of the plurality of switchable pixels and two of the plurality of transparent pixels, and the two of the plurality of switchable pixels and the two of the plurality of transparent pixels are alternately arranged in an array of 2 rows × 2 columns.
7. The demosaicing method according to claim 6, wherein performing the weighted average operation on the panchromatic image includes: averaging a plurality of signals of the plurality of switchable pixels in the transparent state and a plurality of signals of the plurality of transparent pixels in the transparent state to generate a first average signal of the pixel group; wherein performing the weighted average operation on the color image includes: averaging a plurality of signals of the plurality of switchable pixels in the colored state to generate a second average signal of the pixel group; and wherein performing the signal subtraction on the color image and the panchromatic image includes: subtracting a signal of one of the plurality of switchable pixels in the transparent state from a signal of one of the plurality of switchable pixels in the colored state to generate a first subtracted signal of the one of the plurality of switchable pixels, or subtracting a signal of one of the plurality of transparent pixels in the panchromatic image from a signal of one of the plurality of transparent pixels in the color image to generate a second subtracted signal of zero.
8. The demosaicing method according to claim 1, wherein the pixel array includes two first pixel groups, a second pixel group, and a third pixel group, wherein each of the two first pixel groups, the second pixel group, and the third pixel group includes two of the plurality of switchable pixels and two of the plurality of transparent pixels, wherein the two first pixel groups, the second pixel group, and the third pixel group are arranged in an array of 2 rows × 2 columns, and wherein the plurality of switchable pixels in the colored state of the two first pixel groups are green pixels, the plurality of switchable pixels in the colored state of the second pixel group are red pixels, and the plurality of switchable pixels in the colored state of the third pixel group are blue pixels.
9. A demosaicing device for an image sensor, characterized in that, Comprising: A pixel array for capturing a panchromatic image and a color image, including: A switchable pixel, including a first photosensitive element and an electrochromic layer above the first photosensitive element; And A transparent pixel, adjacent to the switchable pixel and including a second photosensitive element, wherein when the electrochromic layer is not biased, the switchable pixel and the transparent pixel are in a transparent state, wherein when the electrochromic layer is biased, the switchable pixel is in a colored state and the transparent pixel is in the transparent state; An arithmetic circuit, performing a weighted average operation and a signal subtraction on the color image and the panchromatic image to generate a plurality of first-resolution color images; and A correction circuit, fusing the plurality of first-resolution color images with the panchromatic image to generate a plurality of demosaiced color images.
10. The demosaicing device according to claim 9, wherein the first photosensitive element and the electrochromic layer are independently controlled by different transistors, and wherein the switchable pixel further comprises: a bottom electrode located between the first photosensitive element and the electrochromic layer; an electrolyte layer located on the electrochromic layer; an ion storage layer located on the electrolyte layer; a top electrode located on the ion storage layer; and a microlens array located on the top electrode, wherein the electrolyte layer, the ion storage layer, the top electrode, and the microlens array extend above the second photosensitive element.