Electronic device
The imaging unit in electronic devices maintains resolution and enhances information capture by employing pixel groups with diverse filters and polarizing elements, addressing the resolution loss from special-purpose pixels.
Patent Information
- Application Number
- JP2021575655
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-02-03
- Filing Date
- 2020-12-23
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-12-23
AI Technical Summary
The integration of special-purpose pixels in imaging units of electronic devices, such as smartphones and PCs, leads to a decrease in image resolution due to the reduction in the number of normal pixels.
An imaging unit with a plurality of pixel groups, including first and second pixel groups with specific filter arrangements and polarizing elements, allows for increased information capture without reducing resolution through color and polarization corrections.
The solution maintains image resolution while enhancing the types of information obtained, including phase difference detection and flare correction, by utilizing adjacent pixels with different filters and polarizing elements.
Smart Images

Figure 0007593953000001 
Figure 0007593953000002 
Figure 0007593953000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to electronic devices. [Background technology]
[0002] Recent electronic devices such as smartphones, mobile phones, and PCs (Personal Computers) are equipped with cameras to facilitate video calls and video capture. Meanwhile, in the imaging section that captures an image, in addition to normal pixels that output imaging information, special purpose pixels such as polarization pixels and pixels with complementary color filters may be arranged. The polarization pixels are used, for example, to correct flare, and the pixels with complementary color filters are used for color correction.
[0003] However, when a large number of special purpose pixels are arranged, the number of normal pixels decreases, which may result in a decrease in the resolution of the image captured by the imaging section. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2019-106634 A [Patent Document 2] JP 2012-168339 A Summary of the Invention [Problem to be solved by the invention]
[0005] One aspect of the present disclosure provides an electronic device that can increase the types of information obtained by an imaging section while suppressing a decrease in the resolution of a captured image. [Means for solving the problem]
[0006] In order to solve the above problems, the present disclosure provides an image pickup apparatus including an image pickup unit having a plurality of pixel groups each formed of two adjacent pixels, At least one first pixel group of the plurality of pixel groups a first pixel that photoelectrically converts a portion of the incident light collected via the first lens; a second pixel different from the first pixel that photoelectrically converts a part of the incident light condensed through the first lens; having At least one second pixel group different from the first pixel group among the plurality of pixel groups, a third pixel that performs photoelectric conversion on the incident light collected via the second lens; a fourth pixel different from the third pixel that photoelectrically converts incident light collected via a third lens different from the second lens; An electronic device is provided having the following:
[0007] the imaging unit is configured with a plurality of pixel regions in which the pixel group is arranged in a 2×2 matrix; The plurality of pixel regions include a first pixel region in which four of the first pixel groups are arranged; a second pixel region in which three of the first pixel groups and one of the second pixel groups are arranged; may have the following structure:
[0008] In the first pixel region, any one of a red filter, a green filter, and a blue filter may be disposed in correspondence with the first pixel group that receives red light, green light, and blue light.
[0009] in the second pixel region, at least two of a red filter, a green filter, and a blue filter are arranged corresponding to the first pixel group that receives at least two colors of red light, green light, and blue light; At least one of the two pixels in the second pixel group may have one of a cyan filter, a magenta filter, and a yellow filter.
[0010] At least one of the two pixels in the second pixel group may be a pixel having a blue wavelength range.
[0011] The image sensor may further include a signal processing unit that performs color correction on an output signal output by at least one of the pixels of the first pixel group, based on an output signal of at least one of the two pixels of the second pixel group.
[0012] At least one pixel in the second group of pixels may include a polarising element.
[0013] The third pixel and the fourth pixel may have the polarizing element, and the polarizing element of the third pixel and the polarizing element of the fourth pixel may have different polarization orientations.
[0014] The image sensor may further include a correction section that corrects output signals of pixels in the first pixel group using polarization information based on output signals of pixels having the polarization elements.
[0015] the incident light is incident on the first pixel and the second pixel through a display unit, The correction section may remove a polarized component of at least one of reflected light and diffracted light generated when passing through the display section and incident on the first pixel and the second pixel and imaged.
[0016] The correction unit may correct the digital pixel data by performing a subtraction process on the digital pixel data that has been photoelectrically converted and digitized at the first pixel and the second pixel, by applying a correction amount based on polarization information data that has been digitized from a polarization component that has been photoelectrically converted at a pixel having the polarizing element.
[0017] a drive unit that reads out electric charges from each pixel of the plurality of pixel groups a plurality of times during one imaging frame; an analog-to-digital conversion unit that performs analog-to-digital conversion in parallel on each of a plurality of pixel signals based on a plurality of readouts of electric charges; may further comprise:
[0018] The drive section may read out a common black level corresponding to the third pixel and the fourth pixel.
[0019] The plurality of pixels each formed of two adjacent pixels may be in a square shape.
[0020] A phase difference may be detectable based on output signals of two pixels in the first pixel group.
[0021] The signal processing unit may perform white balance processing after performing color correction on the output signal.
[0022] The image display device may further include an interpolation section that interpolates an output signal of a pixel having the polarizing element from outputs of pixels surrounding the pixel.
[0023] The first to third lenses may be on-chip lenses that focus incident light onto photoelectric conversion units of the corresponding pixels.
[0024] Further comprising a display unit, The incident light may be incident on the plurality of pixel groups via the display section. [Brief description of the drawings]
[0025] [Figure 1] 1 is a schematic cross-sectional view of an electronic device according to a first embodiment. [Diagram 2] 2(a) is a schematic external view of the electronic device in FIG. 1, and (b) is a cross-sectional view taken along line AA in (a). [Diagram 3] FIG. 2 is a schematic plan view for explaining a pixel arrangement in an imaging section. [Figure 4] FIG. 2 is a schematic plan view showing the relationship between a pixel array and an on-chip lens array in an imaging unit. [Diagram 5] FIG. 4 is a schematic plan view illustrating an arrangement of pixels in a first pixel region. [Figure 6A] FIG. 4 is a schematic plan view illustrating an arrangement of pixels in a second pixel region. [Figure 6B] FIG. 6B is a schematic plan view illustrating an arrangement of pixels in a second pixel region different from that in FIG. 6A. [Figure 6C] FIG. 7 is a schematic plan view for explaining an arrangement of pixels in a second pixel region different from those in FIGS. 6A and 6B. [Figure 7A] FIG. 13 is a diagram showing a pixel array of a second pixel region with respect to an R array. [Figure 7B] 7B is a diagram showing a pixel arrangement of a second pixel region different from that of FIG. 7A with respect to the R arrangement. [Figure 7C] 8A and 8B show a pixel arrangement in a second pixel region, which is different from the R arrangement in FIGS. 7A and 7B. [Figure 8] FIG. 6 is a diagram showing the structure of the cross section AA of FIG. 5. [Figure 9] FIG. 6B is a diagram showing the structure of the cross section AA of FIG. 6A. [Figure 10] FIG. 1 is a diagram showing an example of a system configuration of an electronic device. [Figure 11] FIG. 4 is a diagram showing an example of a data area stored in a memory unit. [Figure 12] FIG. 13 is a diagram showing an example of charge readout driving. [Figure 13] 1 is a diagram showing the relative sensitivities of red, green, and blue pixels. [Figure 14] FIG. 4 is a diagram showing the relative sensitivities of cyan, yellow, and magenta pixels. [Figure 15] FIG. 11 is a schematic plan view for explaining a pixel arrangement in an imaging section according to a second embodiment. [Figure 16] FIG. 11 is a schematic plan view showing the relationship between a pixel array and an on-chip lens array in an imaging unit according to a second embodiment. [Figure 17A] FIG. 4 is a schematic plan view illustrating an arrangement of pixels in a second pixel region. [Figure 17B] FIG. 17B is a schematic plan view for explaining an arrangement of pixels having a different polarizing element from that in FIG. 17A. [Figure 17C] FIG. 17B is a schematic plan view for explaining an arrangement of pixels having different polarizing elements from those in FIGS. [Figure 17D] FIG. 13 is a schematic plan view for explaining the arrangement of polarizing elements in the B arrangement. [Figure 17E] FIG. 17E is a schematic plan view for explaining an arrangement of pixels having a different polarizing element from that in FIG. 17D. [Figure 17F] FIG. 17D is a schematic plan view for explaining an arrangement of pixels having different polarizing elements from those in FIGS. 17E and D. [Figure 18]FIG. 17B is a diagram showing the AA cross-sectional structure of FIG. 17A. [Figure 19] FIG. 4 is a perspective view showing an example of a detailed structure of each polarizing element. [Figure 20] 1A and 1B are diagrams illustrating a state in which flare occurs when an object is photographed by an electronic device. [Figure 21] 21 is a diagram showing signal components contained in the captured image of FIG. 20. [Figure 22] 10A and 10B are diagrams conceptually explaining correction processing. [Figure 23] FIG. 11 is another diagram conceptually illustrating the correction process. [Figure 24] FIG. 2 is a block diagram showing the internal configuration of the electronic device 1. [Diagram 25] 4 is a flowchart showing the procedure of a photographing process performed by the electronic device. [Figure 26] FIG. 13 is a plan view of the electronic device applied to a capsule endoscope. [Figure 27] FIG. 11 is a rear view of the electronic device when applied to a digital single-lens reflex camera. [Figure 28] FIG. 11 is a plan view showing an example in which the electronic device is applied to a head mounted display. [Figure 29] A diagram showing the current HMD. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0026] Hereinafter, an embodiment of an electronic device will be described with reference to the drawings. The following description will focus on the main components of the electronic device, but the electronic device may include components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described.
[0027] (First embodiment) FIG. 1 is a schematic cross-sectional view of an electronic device 1 according to a first embodiment. The electronic device 1 in FIG. 1 is any electronic device having both a display function and an image capturing function, such as a smartphone, a mobile phone, a tablet, or a PC. The electronic device 1 in FIG. 1 includes a camera module (image capturing unit) disposed on the opposite side to the display surface of the display unit 2. In this manner, the electronic device 1 in FIG. 1 includes a camera module 3 disposed on the rear side of the display surface of the display unit 2. Therefore, the camera module 3 captures images through the display unit 2.
[0028] FIG. 2(a) is a schematic external view of the electronic device 1 in FIG. 1, and FIG. 2(b) is a cross-sectional view taken along line AA in FIG. 2(a). In the example of FIG. 2(a), the display screen 1a extends to nearly the outer size of the electronic device 1, and the width of the bezel 1b around the display screen 1a is set to a few mm or less. Usually, a front camera is often mounted on the bezel 1b, but in FIG. 2(a), as shown by the dashed line, a camera module 3 functioning as a front camera is disposed on the rear side of the display screen 1a at approximately the center. In this way, by providing the front camera on the rear side of the display screen 1a, it becomes unnecessary to dispose the front camera on the bezel 1b, and the width of the bezel 1b can be narrowed. 2(a), the camera module 3 is disposed on the rear side of the display screen 1a at approximately the center thereof, but in this embodiment, the camera module 3 may be disposed on any rear side of the display screen 1a, for example, on the rear side near the periphery of the display screen 1a. In this manner, the camera module 3 in this embodiment is disposed at any position on the rear side that overlaps with the display screen 1a.
[0029] As shown in FIG. 1, the display unit 2 is a structure in which a display panel 4, a circular polarizer 5, a touch panel 6, and a cover glass 7 are laminated in this order. The display panel 4 may be, for example, an OLED (Organic Light Emitting Device) unit, a liquid crystal display unit, a MicroLED, or a display unit 2 based on other display principles. The display panel 4, such as an OLED unit, is composed of multiple layers. The display panel 4 is often provided with a member having low transmittance, such as a color filter layer. As will be described later, a through hole may be formed in the member having low transmittance in the display panel 4 in accordance with the arrangement location of the camera module 3. If the subject light passing through the through hole is made incident on the camera module 3, the image quality of the image captured by the camera module 3 can be improved.
[0030] The circular polarizer 5 is provided to reduce glare and improve the visibility of the display screen 1a even in a bright environment. A touch sensor is incorporated in the touch panel 6. There are various types of touch sensors, such as a capacitance type and a resistive film type, and any type may be used. The touch panel 6 and the display panel 4 may be integrated together. The cover glass 7 is provided to protect the display panel 4 and the like.
[0031] The camera module 3 has an imaging unit 8 and an optical system 9. The optical system 9 is disposed on the light incident surface side of the imaging unit 8, i.e., on the side closer to the display unit 2, and collects light that has passed through the display unit 2 onto the imaging unit 8. The optical system 9 is usually composed of multiple lenses.
[0032] The imaging unit 8 has a plurality of photoelectric conversion units. The photoelectric conversion units photoelectrically convert light incident via the display unit 2. The photoelectric conversion units may be CMOS (Complementary Metal Oxide Semiconductor) sensors or CCD (Charge Coupled Device) sensors. The photoelectric conversion units may be photodiodes or organic photoelectric conversion films.
[0033] 3 to 6C, examples of the pixel array and on-chip lens array in the imaging unit 8 will be described. An on-chip lens is a lens that is provided on the surface of each pixel on the light incident side and focuses incident light onto the photoelectric conversion unit of the corresponding pixel.
[0034] FIG. 3 is a schematic plan view for explaining the pixel arrangement in the imaging unit 8. FIG. 4 is a schematic plan view showing the relationship between the pixel arrangement in the imaging unit 8 and the on-chip lens arrangement. FIG. 5 is a schematic plan view for explaining the arrangement of a pair of pixels 80, 82 in the first pixel region 8a. FIG. 6A is a schematic plan view for explaining the arrangement of a pair of pixels 80a, 82a in the second pixel region 8b. FIG. 6B is a schematic plan view for explaining the arrangement of the pixels 80a, 82a in the second pixel region 8c. FIG. 6C is a schematic plan view for explaining the arrangement of the pixels 80a, 82a in the second pixel region 8d.
[0035] 3, the imaging unit 8 has a plurality of pixel groups each composed of two adjacent pixels (80, 82) and (80a, 82a) that form a pair. The pixels 80, 82, 80a, and 82a are rectangular, and the two adjacent pixels (80, 82) and (80a, 82a) are square.
[0036] The symbol R indicates a pixel that receives red light, the symbol G indicates a pixel that receives green light, the symbol B indicates a pixel that receives blue light, the symbol C indicates a pixel that receives cyan light, the symbol Y indicates a pixel that receives yellow light, and the symbol M indicates a pixel that receives magenta light. The same is true in the other drawings.
[0037] The imaging section 8 has a first pixel region 8a and second pixel regions 8b, 8c, and 8d. In Fig. 3, one group each of the second pixel regions 8b, 8c, and 8d is illustrated. That is, the remaining 13 groups are the first pixel region 8a.
[0038] In the first pixel region 8a, pixels are arranged in a form in which one pixel in a normal Bayer array is replaced with two pixels 80, 82 arranged in a row. That is, the pixels are arranged in a form in which R, G, and B in the Bayer array are each replaced with two pixels 80, 82.
[0039] On the other hand, in the second pixel regions 8b, 8c, and 8d, the pixels are arranged such that R and G in the Bayer array are replaced by two pixels 80 and 82, respectively, and B in the Bayer array is replaced by two pixels 80a and 82a. For example, the combination of two pixels 80a and 82a is a combination of B and C in the second pixel region 8b, a combination of B and Y in the second pixel region 8c, and a combination of B and M in the second pixel region 8d.
[0040] 4 to 6C, one circular on-chip lens 22 is provided for every two pixels 80, 82. This allows each of the pixels 80, 82 in the pixel groups 8a, 8b, 8c, and 8d to detect an image plane phase difference. Furthermore, by adding the outputs of the pixels 80, 82, the pixels 80, 82 function in the same manner as a normal imaging pixel. In other words, by adding the outputs of the pixels 80, 82, imaging information can be obtained.
[0041] On the other hand, as shown in FIG. 4 to FIG. 6C, two pixels 80a, 82a are provided with elliptical on-chip lenses 22a. As shown in FIG. 6A, in the second pixel region 8b, the pixel 82a is different from the pixel B in the first pixel region 8a in that it is a pixel that receives cyan light. As a result, the two pixels 80a, 82a can receive blue light and cyan light independently. Similarly, as shown in FIG. 6B, in the second pixel region 8c, the pixel 82a receives yellow light. As a result, the two pixels 80a, 82a can receive blue light and yellow light independently. Similarly, as shown in FIG. 6C, in the second pixel region 8d, the pixel 82a receives magenta light. As a result, the two pixels 80a, 82a can receive blue light and magenta light independently.
[0042] In the first pixel region 8a, the pixels of the B array can acquire only blue color information, whereas the pixels of the B array in the second pixel region 8b can acquire cyan color information in addition to blue color information. Similarly, the pixels of the B array in the second pixel region 8c can acquire yellow color information in addition to blue color information. Similarly, the pixels of the B array in the second pixel region 8d can acquire magenta color information in addition to blue color information.
[0043] The color information of cyan, yellow, and magenta acquired by the pixels 80a and 82a of the second pixel regions 8b, 8c, and 8d can be used for color correction. In other words, the pixels 80a and 82a of the second pixel regions 8b, 8c, and 8d are special-purpose pixels arranged for color correction. Here, the special-purpose pixels according to this embodiment refer to pixels used for correction processes such as color correction and polarization correction. These special-purpose pixels can also be used for purposes other than normal imaging.
[0044] The on-chip lenses 22a of the pixels 80a and 82a in the second pixel regions 8b, 8c, and 8d are elliptical, and the amount of light received is half that of the total amount of light received by the pixels 80 and 82 that receive light of the same color. The light receiving distribution and light amount, i.e., sensitivity, can be corrected by signal processing.
[0045] On the other hand, the pixels 80a and 82a can obtain two different types of color information, which is effectively used for color correction. In this way, the second pixel regions 8b, 8c, and 8d can increase the types of information that can be obtained without reducing the resolution. The color correction process will be described in detail later.
[0046] In this embodiment, the pixel of the B array of the Bayer array is composed of two pixels 80a and 82a, but is not limited to this. For example, as shown in Figures 7A to 7C, the pixel of the R array of the Bayer array may be composed of two pixels 80a and 82a.
[0047] 7A is a diagram showing the pixel arrangement of the second pixel region 8e. In the second pixel region 8e, the pixel 82a in the R array of the Bayer array is different from the pixel arrangement in the first pixel region 8a in that it is a pixel that receives cyan light. This allows the two pixels 80a and 82a to receive red light and cyan light independently.
[0048] 7B is a diagram showing the pixel arrangement of the second pixel region 8f. In the second pixel region 8f, the pixel 82a in the R array of the Bayer array is different from the pixel arrangement in the first pixel region 8a in that it is a pixel that receives yellow light. This allows the two pixels 80a and 82a to receive red light and yellow light independently.
[0049] 7C is a diagram showing the pixel arrangement of the second pixel region 8g. In the second pixel region 8g, the pixel 82a in the R array of the Bayer array is different from the pixel arrangement in the first pixel region 8a in that it is a pixel that receives magenta light. This allows the two pixels 80a and 82a to receive red light and magenta light, respectively, independently. In this embodiment, the pixel array is configured as a Bayer array, but is not limited to this. For example, it may be an interline array, a checkerboard array, a stripe array, or other array. In other words, the ratio of the number of pixels 80a, 82a to the number of pixels 80, 82, the type of light receiving color, and the arrangement location are arbitrary.
[0050] Fig. 8 is a diagram showing the structure of the AA cross section of Fig. 5. As shown in Fig. 8, a plurality of photoelectric conversion units 800a are arranged in the substrate 11. A plurality of wiring layers 12 are arranged on the first surface 11a side of the substrate 11. An interlayer insulating film 13 is arranged around the plurality of wiring layers 12. Contacts (not shown) are provided to connect these wiring layers 12 to each other and between the wiring layers 12 and the photoelectric conversion units 800a, but are omitted in Fig. 8.
[0051] On the second surface 11b side of the substrate 11, a light-shielding layer 15 is disposed near the boundary of the pixels via a planarization layer 14, and a base insulating layer 16 is disposed around the light-shielding layer 15. A planarization layer 20 is disposed on the base insulating layer 16. A color filter layer 21 is disposed on the planarization layer 20. The color filter layer 21 has filter layers of three colors, RGB. In this embodiment, the color filter layer 21 of the pixels 80 and 82 has filter layers of three colors, RGB, but is not limited to this. For example, it may have filter layers of cyan, magenta, and yellow, which are complementary colors. Alternatively, it may have a filter layer that transmits colors other than visible light such as infrared light, a filter layer having multispectral characteristics, or a filter layer of subtractive colors such as white. By transmitting light other than visible light such as infrared light, sensing information such as depth information can be detected. An on-chip lens 22 is disposed on the color filter layer 21.
[0052] FIG. 9 is a diagram showing the structure of the AA cross section of FIG. 6A. In the cross-sectional structure of FIG. 8, one circular on-chip lens 22 is arranged in the pixels 80 and 82, but in FIG. 9, an on-chip lens 22a is arranged for each of the pixels 80a and 82a. The color filter layer 21 of one pixel 80a is, for example, a blue filter. The other pixel 82a is, for example, a cyan filter. In the second pixel regions 8c and d, the other pixel 82a is, for example, a yellow filter or a magenta filter. In the second pixel regions 8e, f, and g, the color filter layer 21 of one pixel 80a is, for example, a red filter. The position of the filter of one pixel 80a and the position of the filter of the other pixel 82a may be reversed. Here, the blue filter is a transmission filter that transmits blue light, the red filter is a transmission filter that transmits red light, and the green filter is a transmission filter that transmits green light. Similarly, the cyan, magenta and yellow filters are transmission filters that transmit cyan light, magenta light and yellow light, respectively.
[0053] As can be seen from these, the pixels 80, 82 and the pixels 80a, 82a have different shapes of the on-chip lenses 22, 22a and combinations of the color filter layers 21, but have the same structure below the planarization layer 20. Therefore, data can be read out from the pixels 80, 82 and data can be read out from the pixels 80a, 82a in the same manner. This makes it possible to increase the types of information that can be obtained from the output signals of the pixels 80a, 82a, as will be described later in detail, and to prevent a decrease in the frame rate.
[0054] Here, a system configuration example of the electronic device 1 and a data reading method will be described with reference to Figs. 10, 11, and 12. Fig. 10 is a diagram showing a system configuration example of the electronic device 1. The electronic device 1 according to the first embodiment includes an imaging unit 8, a vertical driving unit 130, analog-to-digital conversion (hereinafter referred to as "AD conversion") units 140 and 150, column processing units 160 and 170, a memory unit 180, a system control unit 19, a signal processing unit 510, and an interface unit 520.
[0055] In the imaging unit 8, for the matrix-like pixel arrangement, pixel drive lines are wired in the row direction for each pixel row, and for each pixel column, for example, two vertical signal lines 310, 32 are wired in the column direction. The pixel drive lines transmit drive signals for driving the pixels 80, 82, 80a, 82a when reading out signals. One end of the pixel drive line is connected to an output terminal of the vertical drive unit 130 corresponding to each row.
[0056] The vertical driving unit 130 is composed of a shift register, an address decoder, etc., and drives each pixel 80, 82, 80a, 82a of the imaging unit 8 simultaneously for all pixels or row by row. That is, the vertical driving unit 130, together with the system control unit 190 that controls the vertical driving unit 130, constitutes a driving unit that drives each pixel 80, 82, 80a, 82a of the imaging unit 8. This vertical driving unit 130 is generally configured to have two scanning systems, a read scanning system and a sweep scanning system. The read scanning system selects and scans each pixel 80, 82, 80a, 82a in sequence row by row. The signals read out from each pixel 80, 82, 80a, 82a are analog signals. The sweep scanning system performs sweep scanning on the read rows that are read out by the read scanning system, prior to the read scanning by the shutter speed.
[0057] The sweep-out scanning by this sweep-out scanning system sweeps out unnecessary charges from the photoelectric conversion units of the pixels 80, 82, 80a, 82a in the readout row, thereby resetting the photoelectric conversion units. Then, by sweeping out (resetting) the unnecessary charges by this sweep-out scanning system, a so-called electronic shutter operation is performed. Here, the electronic shutter operation refers to the operation of discarding the photoelectric charges in the photoelectric conversion units and starting a new exposure (starting the accumulation of photoelectric charges).
[0058] The signal read by the read operation by the read scanning system corresponds to the amount of light received after the immediately preceding read operation or electronic shutter operation. The period from the read timing of the immediately preceding read operation or the sweep timing of the electronic shutter operation to the read timing of the current read operation is the exposure period of the photocharges in the unit pixel.
[0059] The pixel signals output from the pixels 80, 82, 80a, 82a in the pixel row selected by the vertical drive unit 130 are input to the AD conversion units 140, 150 through two systems of vertical signal lines 310, 320. Here, one system of vertical signal lines 310 is made up of a signal line group (first signal line group) that transmits the pixel signals output from the pixels 80, 82, 80a, 82a in the selected row in a first direction (one side in the pixel column direction / upward in the figure) for each pixel column. The other system of vertical signal lines 320 is made up of a signal line group (second signal line group) that transmits the pixel signals output from the pixels 80, 82, 80a, 82a in the selected row in a second direction (the other side in the pixel column direction / downward in the figure) opposite to the first direction.
[0060] The AD conversion units 140, 150 each include a set of AD converters 141, 151 (AD converter group) provided for each pixel column, are provided on either side of the imaging unit 8 in the pixel column direction, and perform AD conversion on pixel signals transmitted by two vertical signal lines 310, 320. That is, the AD conversion unit 140 includes a set of AD converters 141 that perform AD conversion on pixel signals transmitted in a first direction by the vertical signal line 31 for each pixel column. The AD conversion unit 150 includes a set of AD converters 151 that perform AD conversion on pixel signals transmitted in a second direction by the vertical signal line 320 for each pixel column.
[0061] That is, one end of the vertical signal line 310 is connected to an AD converter 141 of one system. Then, pixel signals output from each of the pixels 80, 82, 80a, 82a are transmitted in a first direction (upward in the figure) by the vertical signal line 310 and input to the AD converter 141. Also, one end of the vertical signal line 320 is connected to an AD converter 151 of the other system. Then, pixel signals output from each of the pixels 80, 82, 80a, 82a are transmitted in a second direction (downward in the figure) by the vertical signal line 320 and input to the AD converter 151.
[0062] The pixel data (digital data) after AD conversion in the AD conversion units 140 and 150 is supplied to the memory unit 180 via the column processing units 160 and 170. The memory unit 180 temporarily stores the pixel data that has passed through the column processing unit 160 and the pixel data that has passed through the column processing unit 170. The memory unit 180 also performs a process of adding the pixel data that has passed through the column processing unit 160 and the pixel data that has passed through the column processing unit 170 together.
[0063] Furthermore, when acquiring black level signals of each pixel 80, 82, 80a, 82a, a black level serving as a reference point may be commonly read out for each pair of adjacent pixels (80, 82), (80a, 82a). This allows the black level to be commonly read out, making it possible to increase the readout speed, i.e., the frame rate. In other words, after the black level serving as the reference point is commonly read out, it is possible to perform driving to read out the normal signal level individually.
[0064] 11 is a diagram showing an example of data areas stored in the memory unit 180. For example, pixel data read from each of the pixels 80, 82, and 80a is associated with pixel coordinates and stored in the first area 180a, and pixel data read from each of the pixels 82a is associated with pixel coordinates and stored in the second area 180b. As a result, the pixel data stored in the first area 180a is stored as R, G, and B image data in a Bayer array, and the pixel data stored in the second area 180b is stored as image data for correction processing.
[0065] The system control unit 190 is composed of a timing generator that generates various timing signals, and performs drive control of the vertical drive unit 130, the AD conversion units 140 and 150, and the column processing units 160 and 170, etc. based on the various timings generated by the timing generator.
[0066] The pixel data read out from the memory unit 180 undergoes predetermined signal processing in the signal processing unit 510, and is then output to the display panel 4 via the interface 520. In the signal processing unit 510, for example, processing is performed to calculate the sum or average of the pixel data in one imaging frame. The signal processing unit 510 will be described in detail later.
[0067] Fig. 12 is a diagram showing an example of a drive for reading out electric charges twice. Fig. 12 shows a schematic diagram of a shutter operation, a read operation, an accumulation state of electric charges, and an addition process when reading out electric charges twice from the photoelectric conversion unit 800a (Figs. 8 and 9).
[0068] In the electronic device 1 according to this embodiment, the vertical drive unit 130 performs drive to read out electric charges from the photoelectric conversion unit 800a, for example, twice during one imaging frame, under the control of the system control unit 190. The readout speed is made faster than the case of one charge readout, and the charges are read out twice, stored in the memory unit 180, and an addition process is performed, so that an amount of electric charge equivalent to the number of times the readouts can be read out from the photoelectric conversion unit 800a.
[0069] In the electronic device 1 according to the present embodiment, a configuration (two-parallel configuration) is adopted in which the AD conversion units 140 and 150 are provided in two parallel systems for two pixel signals based on two charge readouts. By providing two parallel systems of AD conversion units for two pixel signals read out in time series from each of the pixels 80, 82, 80a, and 82a, the two pixel signals read out in time series can be AD converted in parallel by the two AD conversion units 140 and 150. In other words, by providing two parallel systems of the AD conversion units 140 and 150, the second charge readout and the AD conversion of the pixel signals based thereon can be performed in parallel (in parallel) during the AD conversion of the image signal based on the first charge readout. This makes it possible to read out image data from the photoelectric conversion unit 800a at a higher speed.
[0070] Here, an example of color correction processing of the signal processing unit 510 will be described in detail with reference to Figs. 13 and 14. Fig. 13 is a diagram showing the relative sensitivity of R: red, G: green, and B: blue pixels (Fig. 3). The vertical axis indicates the relative sensitivity, and the horizontal axis indicates the wavelength. Similarly, Fig. 14 is a diagram showing the relative sensitivity of C: cyan, Y: yellow, and M: magenta pixels (Fig. 3). The vertical axis indicates the relative sensitivity, and the horizontal axis indicates the wavelength. As described above, the R (red) pixel has a red filter, the B (blue) pixel has a blue filter, the G (green) pixel has a green filter, the C (cyan) pixel has a cyan color filter, the Y (yellow) pixel has a yellow filter, and the M (magenta) pixel has a magenta color filter.
[0071] First, a correction process will be described in which the output signal CS1 of the C (cyan) pixel is used to generate corrected output signals BS3 and BS4 of the B (blue) pixel. As described above, the output signal RS1 of the R (red) pixel, the output signal GS1 of the G (green) pixel, and the output signal GB1 of the B (blue) pixel are stored in the first area (180a) of the memory unit 180. On the other hand, the output signal CS1 of the C (cyan) pixel, the output signal YS1 of the Y (yellow) pixel, and the output signal MS1 of the M (magenta) pixel are stored in the second area (180b) of the memory unit 180.
[0072] Comparing the wavelength characteristics of the C (cyan) pixel, B (blue) pixel, and G (green) pixel as shown in Figures 13 and 14, the output signal CS1 of the C (cyan) pixel can be approximated by adding the output signal BS1 of the B (blue) pixel and the output signal GS1 of the G (green) pixel.
[0073] Therefore, in the second pixel region 8b (FIG. 3), the signal processing unit 510 calculates the output signal BS2 of the B (blue) pixel using, for example, equation (1). BS2 = k1 × CS1 - k2 × GS1 (1) Here, k1 and k2 are coefficients for adjusting the signal strength.
[0074] Then, the signal processing unit 510 calculates the corrected output signal BS3 of the B (blue) pixel, for example, using equation (2). BS3=BS2+k3×BS1 = k1 × CS1 - k2 × GS1 + k3 × BS1 (2) Here, k3 is a coefficient for adjusting the signal strength.
[0075] Similarly, in the second pixel region 8e (FIG. 7A), the signal processing unit 510 calculates the output signal BS4 of the B (blue) pixel using, for example, equation (3). BS4=k1×CS1―k2×GS1+k4×BS1 (3) Here, k4 is a coefficient for adjusting the signal strength. In this manner, the signal processing unit 510 can obtain the corrected output signals BS3 and BS4 of the B (blue) pixels using the output signal CS1 of the C (cyan) pixel and the output signal GS1 of the G (green) pixel.
[0076] Next, a correction process will be described in which the output signal YS1 of the Y (yellow) pixel is used to generate corrected output signals RS3 and RS4 of the R (red) pixel. Comparing the wavelength characteristics of the Y (yellow) pixel, R (red) pixel, and G (green) pixel as shown in Figures 13 and 14, the output signal YS1 of the Y (yellow) pixel can be approximated by adding the output signal RS1 of the R (red) pixel and the output signal GS1 of the G (green) pixel.
[0077] Therefore, in the second pixel region 8c (FIG. 3), the signal processing unit 510 calculates the output signal RS2 of the R (red) pixel using, for example, equation (4). RS2 = k5 × YS1 - k6 × GS1 (4) Here, k5 and k6 are coefficients for adjusting the signal strength.
[0078] Then, the signal processing unit 510 calculates the corrected output signal RS3 of the R (red) pixel, for example, using equation (5). RS3 = k7 × RS1 + RS2 = k5 × YS1 - k6 × GS1 + k7 × RS1 (5) Here, k7 is a coefficient for adjusting the signal strength.
[0079] Similarly, in the second pixel region 8f (FIG. 7B), the signal processing unit 510 calculates the output signal RS4 of the R (red) pixel using, for example, equation (6). RS4=k5×YS1―k6×GS1+k8×RS1 (6) Here, k8 is a coefficient for adjusting the signal strength. In this manner, the signal processing unit 510 can obtain corrected output signals RS3 and RS4 of the R (red) pixel using the output signal YS1 of the Y (yellow) pixel and the output signal GS1 of the G (green) pixel.
[0080] Next, a correction process will be described in which the output signal MS1 of the M (magenta) pixel is used to generate corrected output signals BS6 and BS7 of the B (blue) pixel. Comparing the wavelength characteristics of the M (magenta) pixel, B (blue) pixel, and R (red) pixel as shown in Figures 13 and 14, the output signal MS1 of the M (magenta) pixel can be approximated by adding the output signal BS1 of the B (blue) pixel and the output signal RS1 of the R (red) pixel.
[0081] Therefore, in the second pixel region 8d (FIG. 3), the signal processing unit 510 calculates the output signal BS5 of the B (blue) pixel using, for example, equation (7). BS5=k9×MS1-k10×RS1 (7) Here, k9 and k10 are coefficients for adjusting the signal strength.
[0082] Then, the signal processing unit 510 calculates the corrected output signal BS6 of the B (blue) pixel using, for example, equation (8). BS6=BS5+k11×BS1 = k9 × MS1 - k10 × RS1 + k11 × BS1 (8) Here, k11 is a coefficient for adjusting the signal strength.
[0083] Similarly, in the second pixel region 8g (FIG. 7C), the signal processing unit 510 calculates the output signal BS7 of the B (blue) pixel using, for example, equation (9). BS7=k9×MS1―k10×RS1+k12×BS1 (9) Here, k12 is a coefficient for adjusting the signal strength. In this manner, the signal processing unit 510 can obtain the corrected output signals BS6 and BS7 of the B (blue) pixels using the output signal MS1 of the M (magenta) pixel and the output signal RS1 of the R (red) pixel.
[0084] Next, a correction process will be described in which the output signal MS1 of the M (magenta) pixel is used to generate corrected output signals RS6 and RS7 of the R (red) pixel. In the second pixel region 8d (FIG. 3), the signal processing unit 510 calculates the output signal RS5 of the R (red) pixel using, for example, equation (10). RS5 = k13 × MS1 - k14 × BS1 (10) Here, k13 and k14 are coefficients for adjusting the signal strength.
[0085] Then, the signal processing unit 510 calculates the corrected output signal RS6 of the R (red) pixel, for example, using equation (11). RS6=RS5+k15×RS1 = k13 × MS1 - k14 × BS1 + k16 × RS1 (11) Here, k16 is a coefficient for adjusting the signal strength.
[0086] Similarly, in the second pixel region 8g (FIG. 7C), the signal processing unit 510 calculates the output signal BS7 of the R (red) pixel using, for example, equation (12). RS7=k13×MS1―k14×BS1+k17×RS1 (12) Here, k17 is a coefficient for adjusting the signal strength. In this manner, the signal processing unit 510 can obtain the output signals RS6 and RS7 of the R (red) pixel corrected using the output signal MS1 of the M (magenta) pixel and the output signal BS1 of the B (blue) pixel.
[0087] Furthermore, the signal processing unit 510 performs various processes such as white balance adjustment, gamma correction, and edge enhancement to output a color image. In this way, since white balance adjustment is performed after color correction based on the output signals of the pixels 80a and 82a, it is possible to obtain a captured image with more natural color tones.
[0088] As described above, according to the present embodiment, the imaging unit 8 has a plurality of pixel groups each composed of two adjacent pixels, and the first pixel groups 80, 82 each having one on-chip lens 22 and the second pixel groups 80a, 82a each having an on-chip lens 22a are arranged. As a result, the first pixel groups 80, 82 are capable of phase difference detection and function as normal imaging pixels, and the second pixel groups 80a, 82a are capable of functioning as special-purpose pixels capable of acquiring independent imaging information. In addition, the pixel area of each of the pixel groups 80a, 82a capable of functioning as special-purpose pixels is half that of the pixel groups 80, 82 capable of functioning as normal imaging pixels, and it is possible to avoid obstructing the arrangement of the first pixel groups 80, 82 capable of normal imaging.
[0089] In the second pixel regions 8b to 8k, which are pixel regions in which three first pixel groups 80, 82 and one second pixel group 80a, 82a are arranged, at least two of a red filter, a green filter, and a blue filter are arranged corresponding to the first pixel groups 80, 82 that receive at least two colors of red light, green light, and blue light, and at least one of the two pixels 80a, 82a of the second pixel group is arranged with a cyan color filter, a magenta color filter, or a yellow filter. This makes it possible to color correct an output signal corresponding to any of R (red) pixels, G (green) pixels, and B (blue) pixels using an output signal corresponding to any of C (cyan) pixels, M (magenta) pixels, and Y (yellow) pixels. In particular, by color correcting an output signal corresponding to any of R (red) pixels, G (green) pixels, and B (blue) pixels using an output signal corresponding to any of C (cyan) pixels and M (magenta) pixels, it is possible to increase blue information without reducing resolution. In this way, the types of information obtained by the imaging unit 8 can be increased while suppressing a decrease in the resolution of the captured image.
[0090] Second embodiment The electronic device 1 according to the second embodiment differs from the electronic device 1 according to the first embodiment in that two pixels 80b, 82b in the second pixel region are configured as pixels having polarizing elements. The differences from the electronic device 1 according to the first embodiment will be described below.
[0091] Here, examples of pixel arrangement and on-chip lens arrangement in the imaging unit 8 according to the second embodiment will be described with reference to Figs. 15 to 17C. Fig. 15 is a schematic plan view for explaining the pixel arrangement in the imaging unit 8 according to the second embodiment. Fig. 16 is a schematic plan view showing the relationship between the pixel arrangement and the on-chip lens arrangement in the imaging unit 8 according to the second embodiment. Fig. 17A is a schematic plan view for explaining the arrangement of the pixels 80b, 82b in the second pixel region 8h. Fig. 17B is a schematic plan view for explaining the arrangement of the pixels 80b, 82b in the second pixel region 8i. Fig. 17C is a schematic plan view for explaining the arrangement of the pixels 80b, 82b in the second pixel region 8j.
[0092] As shown in FIG. 15, the imaging section 8 according to the second embodiment has a first pixel region 8a and second pixel regions 8h, 8i, and 8j. In the second pixel regions 8h, 8i, and 8j, pixels are arranged in such a manner that the G pixels 80 and 82 in the Bayer array are replaced with two special-purpose pixels 80b and 82b, respectively. In this embodiment, the pixels are arranged in such a manner that the G pixels 80 and 82 in the Bayer array are replaced with the special-purpose pixels 80b and 82b, but this is not limited thereto. For example, as described later, the B pixels 80 and 82 in the Bayer array may be replaced with the special-purpose pixels 80b and 82b.
[0093] As shown in FIG. 16 to FIG. 17C, as in the first embodiment, one circular on-chip lens 22 is provided for every two pixels 80b, 82b. Meanwhile, a polarizing element S is arranged in the two pixels 80b, 82b. FIGS. 17A to 17C are plan views that typically show combinations of polarizing elements S arranged in the pixels 80b, 82b. FIG. 17A shows a combination of a 45-degree polarizing element and a 0-degree polarizing element. FIG. 17B shows a combination of a 45-degree polarizing element and a 135-degree polarizing element. FIG. 17C shows a combination of a 45-degree polarizing element and a 90-degree polarizing element. In this way, for example, combinations of polarizing elements at 0 degrees, 45 degrees, 90 degrees, 135 degrees, etc. are possible in the two pixels 80b, 82b. Also, as shown in FIGS. 17D to 17F, the pixels are arranged in a form in which the B pixels 80, 82 in the Bayer array are replaced with two pixels 80b, 82b, respectively. In this way, the pixels may be arranged in a form in which the B and R pixels 80 and 82 of the Bayer array are replaced with two pixels 80b and 82b, respectively, without being limited to the G pixels 80 and 82 of the Bayer array. When the G pixels 80 and 82 of the Bayer array are replaced with the special-purpose pixels 80b and 82b, it is also possible to obtain R, G, and B information only from the pixel output in the second pixel regions 8h, 8i, and 8j. On the other hand, when the B pixels 80 and 82 of the Bayer array are replaced with the special-purpose pixels 80b and 82b, they can be used for phase detection without impairing the output of the G pixels, which have higher phase detection accuracy. In this way, the pixels 80b and 82b of the second pixel regions 8h, 8i, and 8j can extract polarized light components.
[0094] Fig. 18 is a diagram showing the AA cross-sectional structure of Fig. 17A. As shown in Fig. 18, a plurality of polarizing elements 9b are disposed at intervals on an underlying insulating layer 16. Each polarizing element 9b in Fig. 18 is a wire-grid polarizing element with a line-and-space structure disposed in a part of an insulating layer 17.
[0095] FIG. 19 is a perspective view showing an example of a detailed structure of each polarizing element 9b. As shown in FIG. 19, each of the multiple polarizing elements 9b has multiple line portions 9d of a convex shape extending in one direction and space portions 9e between each line portion 9d. There are multiple types of polarizing elements 9b, each of which has a different extending direction of the line portion 9d. More specifically, there are three or more types of polarizing elements 9b, and for example, the angle between the arrangement direction of the photoelectric conversion unit 800a and the extension direction of the line portion 9d may be three types of 0 degrees, 60 degrees, and 120 degrees. Alternatively, the angle between the arrangement direction of the photoelectric conversion unit 800a and the extension direction of the line portion 9d may be four types of 0 degrees, 45 degrees, 90 degrees, and 135 degrees, or may be any other angle. Alternatively, the multiple polarizing elements 9b may polarize light in only one direction. The material of the multiple polarizing elements 9b may be a metal material such as aluminum or tungsten, or an organic photoelectric conversion film.
[0096] In this manner, each polarizing element 9b has a structure in which a plurality of line portions 9d extending in one direction are arranged at intervals in a direction intersecting the one direction. There are a plurality of types of polarizing elements 9b in which the directions in which the line portions 9d extend are different from one another.
[0097] The line portion 9d has a laminated structure in which a light reflecting layer 9f, an insulating layer 9g, and a light absorbing layer 9h are laminated. The light reflecting layer 9f is made of a metal material such as aluminum. The insulating layer 9g is made of SiO2. The light absorbing layer 9h is made of a metal material such as tungsten.
[0098] Next, a characteristic operation of the electronic device 1 according to this embodiment will be described. Fig. 20 is a diagram showing a typical state in which flare occurs when an object is photographed by the electronic device 1 of Fig. 1. Flare occurs when a part of light incident on the display unit 2 of the electronic device 1 is repeatedly reflected by any member in the display unit 2, and then incident on the imaging unit 8 and is captured in the captured image. When flare occurs in the captured image, a difference in brightness or a change in color occurs as shown in Fig. 20, degrading the image quality.
[0099] Fig. 21 is a diagram showing signal components contained in the captured image of Fig. 20. As shown in Fig. 21, the captured image contains a subject signal and a flare component.
[0100] 22 and 23 are diagrams conceptually explaining the correction process according to this embodiment. As shown in FIG. 15, the imaging unit 8 according to this embodiment has a plurality of polarizing pixels 80b, 82b and a plurality of non-polarizing pixels 80, 82. The pixel information photoelectrically converted by the plurality of non-polarizing pixels 80, 82 shown in FIG. 15 includes an object signal and a flare component as shown in FIG. 21. In contrast, the polarization information photoelectrically converted by the plurality of polarizing pixels 80b, 82b shown in FIG. 15 is information of the flare component. Therefore, by subtracting the polarization information photoelectrically converted by the plurality of polarizing pixels 80b, 82b from the pixel information photoelectrically converted by the plurality of non-polarizing pixels 80, 82, the flare component is removed and an object signal is obtained as shown in FIG. 23. When an image based on this object signal is displayed on the display unit 2, an object image from which the flare that existed in FIG. 21 has been removed is displayed as shown in FIG. 23.
[0101] External light incident on the display unit 2 may be diffracted by wiring patterns or the like within the display unit 2, and the diffracted light may be incident on the imaging unit 8. In this way, at least one of flare and diffracted light may be captured in the captured image.
[0102] Fig. 24 is a block diagram showing the internal configuration of the electronic device 1 according to this embodiment. The electronic device 1 in Fig. 8 includes an optical system 9, an imaging unit 8, a memory unit 180, a clamp unit 32, a color output unit 33, a polarized output unit 34, a flare extraction unit 35, a flare correction signal generation unit 36, a defect correction unit 37, a linear matrix unit 38, a gamma correction unit 39, a luminance chroma signal generation unit 40, a focus adjustment unit 41, an exposure adjustment unit 42, a noise reduction unit 43, an edge enhancement unit 44, and an output unit 45. The vertical drive unit 130, the analog-digital conversion units 140 and 150, the column processing units 160 and 170, and the system control unit 19 shown in Fig. 10 are omitted in Fig. 24 for the sake of simplicity.
[0103] The optical system 9 includes one or more lenses 9a and an IR (Infrared Ray) cut filter 9b. The IR cut filter 9b may be omitted. As described above, the imaging unit 8 includes a plurality of non-polarized pixels 80 and 82 and a plurality of polarized pixels 80b and 82b.
[0104] The output values of the multiple polarizing pixels 80b, 82b and the output values of the multiple non-polarizing pixels 80, 82 are converted in analog-to-digital conversion units 140, 150 (not shown), and polarization information data obtained by digitizing the output values of the multiple polarizing pixels 80b, 82b is stored in the second area 180b (Figure 11), and digital pixel data obtained by digitizing the output values of the multiple non-polarizing pixels 80, 82 is stored in the first area 180a (Figure 11).
[0105] The clamping unit 32 performs a process for defining a black level, and subtracts black level data from each of the digital pixel data stored in the first area 180a (FIG. 11) of the memory unit 180 and the polarization information data stored in the second area 180b (FIG. 11). The output data of the clamping unit 32 is branched, and RGB digital pixel data is output from the color output unit 33, and polarization information data is output from the polarization output unit 34. The flare extraction unit 35 extracts at least one of a flare component and a diffracted light component from the polarization information data. In this specification, at least one of the flare component and the diffracted light component extracted by the flare extraction unit 35 may be referred to as a correction amount.
[0106] The flare correction signal generation unit 36 corrects the digital pixel data output from the color output unit 33 by subtracting the correction amount extracted by the flare extraction unit 35 from the digital pixel data. The output data of the flare correction signal generation unit 36 is digital pixel data from which at least one of the flare component and the diffracted light component has been removed. In this way, the flare correction signal generation unit 36 functions as a correction unit that corrects the captured image photoelectrically converted by the plurality of non-polarized pixels 80, 82 based on the polarization information.
[0107] The signal level of the digital pixel data at the pixel positions of polarization pixels 80b and 82b is lowered by the amount that the data has passed through polarization element 9b. For this reason, defect correction unit 37 regards polarization pixels 80b and 82b as defective and performs a predetermined defect correction process. In this case, the defect correction process may be an interpolation process using digital pixel data from surrounding pixel positions.
[0108] The linear matrix section 38 performs a matrix operation on color information such as RGB, to reproduce colors more accurately. The linear matrix section 38 is also called a color matrix section.
[0109] Gamma correction unit 39 performs gamma correction so as to enable display with excellent visibility in accordance with the display characteristics of display unit 2. For example, gamma correction unit 39 performs conversion from 10 bits to 8 bits while changing the gradient.
[0110] The luminance chrominance signal generating section 40 generates a luminance chrominance signal to be displayed on the display section 2 based on the output data of the gamma correction section 39 .
[0111] The focus adjustment unit 41 performs autofocus processing based on the luminance chroma signal after the defect correction processing. The exposure adjustment unit 42 performs exposure adjustment based on the luminance chroma signal after the defect correction processing. When performing exposure adjustment, the exposure adjustment may be performed by setting an upper limit clip so that the pixel values of each non-polarized pixel 82 do not saturate. Furthermore, if the pixel values of each non-polarized pixel 82 are saturated even after exposure adjustment, the pixel values of the saturated non-polarized pixels 82 may be estimated based on the pixel values of the polarized pixels 81 surrounding the non-polarized pixel 82.
[0112] The noise reduction unit 43 performs a process of reducing noise contained in the luminance chroma signal. The edge enhancement unit 44 performs a process of enhancing the edge of the subject image based on the luminance chroma signal. The noise reduction process by the noise reduction unit 43 and the edge enhancement process by the edge enhancement unit 44 may be performed only when a predetermined condition is satisfied. The predetermined condition is, for example, when the correction amount of the flare component or the diffracted light component extracted by the flare extraction unit 35 exceeds a predetermined threshold value. The more the flare component or the diffracted light component contained in the captured image, the more noise there will be in the image when the flare component or the diffracted light component is removed, and the more the edges will be blurred. For this reason, the frequency of the noise reduction process or the edge enhancement process can be reduced by performing the noise reduction process or the edge enhancement process only when the correction amount exceeds a threshold value.
[0113] At least some of the signal processing of the defect correction unit 37, the linear matrix unit 38, the gamma correction unit 39, the luminance chroma signal generation unit 40, the focus adjustment unit 41, the exposure adjustment unit 42, the noise reduction unit 43, and the edge enhancement unit 44 in FIG. 24 may be executed by a logic circuit in an image sensor having an image pickup unit 8, or may be executed by a signal processing circuit in an electronic device 1 equipped with an image sensor. Alternatively, at least some of the signal processing of FIG. 24 may be executed by a server on a cloud that transmits and receives information to and from the electronic device 1 via a network. As shown in the block diagram of FIG. 24, the electronic device 1 according to this embodiment performs various signal processing on digital pixel data from which at least one of the flare component and the diffracted light component has been removed in the flare correction signal generation unit 36. In particular, some of the signal processing such as the exposure processing, the focus adjustment processing, and the white balance adjustment processing cannot obtain good signal processing results even if the signal processing is performed in a state in which the flare component and the diffracted light component are included.
[0114] Fig. 25 is a flow chart showing the procedure of the photographing process performed by the electronic device 1 according to this embodiment. First, the camera module 3 is started (step S1). This causes a power supply voltage to be supplied to the imaging unit 8, and the imaging unit 8 starts capturing an image of the incident light. More specifically, the multiple non-polarized pixels 80, 82 perform photoelectric conversion of the incident light, and the multiple polarized pixels 80b, 82b acquire polarization information of the incident light (step S2). The analog-to-digital conversion units 140, 150 (Fig. 10) output polarization information data obtained by digitizing the output values of the multiple polarized pixels 81 and digital pixel data obtained by digitizing the output values of the multiple non-polarized pixels 82, and store them in the memory unit 180 (step S3).
[0115] Next, the flare extraction unit 35 determines whether or not flare or diffraction occurs based on the polarization information data stored in the memory unit 180 (step S4). Here, for example, if the polarization information data exceeds a predetermined threshold, it is determined that flare or diffraction occurs. If it is determined that flare or diffraction occurs, the flare extraction unit 35 extracts a correction amount for the flare component or the diffracted light component based on the polarization information data (step S5). The flare correction signal generation unit 36 subtracts the correction amount from the digital pixel data stored in the memory unit 180 to generate digital pixel data from which the flare component or the diffracted light component has been removed (step S6).
[0116] Next, various signal processing is performed on the digital pixel data corrected in step S6 or on the digital pixel data determined in step S4 to be free of flare or diffraction (step S7). More specifically, in step S7, as shown in Fig. 8, processes such as defect correction processing, linear matrix processing, gamma correction processing, luminance chroma signal generation processing, exposure processing, focus adjustment processing, white balance adjustment processing, noise reduction processing, edge enhancement processing, etc. are performed. Note that the types and execution order of signal processing are arbitrary, and signal processing of some of the blocks shown in Fig. 24 may be omitted, or signal processing other than that of the blocks shown in Fig. 24 may be performed.
[0117] The digital pixel data that has been subjected to the signal processing in step S7 is output from the output unit 45 and may be stored in a memory (not shown) or may be displayed on the display unit 2 as a live image (step S8).
[0118] As described above, in the second pixel regions 8h to 8k, which are pixel regions in which the above-mentioned three first pixel groups and one second pixel group are arranged, red filters, green filters, and blue filters are arranged corresponding to the first pixel groups that receive red light, green light, and blue light, and pixels 80b and 82b having a polarizing element are arranged in at least one of the two pixels of the second pixel group. The output of the pixels 80b and 82b having a polarizing element can be corrected as a normal pixel by interpolating using digital pixel data of the surrounding pixel positions. This makes it possible to increase the polarization information without reducing the resolution.
[0119] As described above, in the second embodiment, the camera module 3 is disposed on the side opposite to the display surface of the display unit 2, and polarization information of light passing through the display unit 2 is acquired by the multiple polarization pixels 80b, 82b. A portion of the light passing through the display unit 2 is repeatedly reflected within the display unit 2 and then incident on the multiple non-polarization pixels 80, 82 in the camera module 3. According to the present embodiment, by acquiring the above-described polarization information, it is possible to generate a captured image in a state in which flare components and diffracted light components contained in the light that is repeatedly reflected within the display unit 2 and then incident on the multiple non-polarization pixels 80, 82 have been simply and reliably removed.
[0120] (Third embodiment) There are various possible specific candidates for the electronic device 1 having the configurations described in the first and second embodiments. For example, Fig. 26 is a plan view of the electronic device 1 of the first and second embodiments applied to a capsule endoscope 50. The capsule endoscope 50 in Fig. 26 includes, in a housing 51 having, for example, hemispherical end faces and a cylindrical center, a camera (ultra-miniature camera) 52 for capturing images inside a body cavity, a memory 53 for recording image data captured by the camera 52, and a wireless transmitter 55 for transmitting the recorded image data to the outside via an antenna 54 after the capsule endoscope 50 is discharged from the subject's body.
[0121] Also provided within housing 51 are a CPU (Central Processing Unit) 56 and a coil (magnetic force-current conversion coil) 57. CPU 56 controls the shooting by camera 52 and the data storage operation in memory 53, and also controls the data transmission from memory 53 to a data receiving device (not shown) outside housing 51 by wireless transmitter 55. Coil 57 supplies power to camera 52, memory 53, wireless transmitter 55, antenna 54, and light source 52b described below.
[0122] Furthermore, the housing 51 is provided with a magnetic (reed) switch 58 for detecting when the capsule endoscope 50 is set in the data receiving device. When the reed switch 58 detects that the capsule endoscope 50 is set in the data receiving device and data transmission becomes possible, the CPU 56 supplies power from the coil 57 to the wireless transmitter 55.
[0123] The camera 52 has, for example, an imaging element 52a including an objective optical system 9 for capturing an image inside the body cavity, and a plurality of light sources 52b for illuminating the inside of the body cavity. Specifically, the camera 52 is configured with, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor equipped with an LED (Light Emitting Diode) or a CCD (Charge Coupled Device) as the light source 52b.
[0124] The display unit 2 in the electronic device 1 of the first and second embodiments is a concept including a light emitting body such as the light source 52b in Fig. 26. The capsule endoscope 50 in Fig. 26 has, for example, two light sources 52b, but these light sources 52b can be configured with a display panel 4 having a plurality of light source units or an LED module having a plurality of LEDs. In this case, by arranging the imaging unit 8 of the camera 52 below the display panel 4 or the LED module, restrictions on the layout arrangement of the camera 52 are reduced, and a smaller capsule endoscope 50 can be realized.
[0125] 27 is a rear view of the electronic device 1 of the first to second embodiments applied to a digital single-lens reflex camera 60. The digital single-lens reflex camera 60 and compact cameras are provided with a display unit 2 that displays a preview screen on the rear surface opposite the lens. A camera module 3 may be disposed on the opposite side of the display surface of the display unit 2 so that a facial image of the photographer can be displayed on the display screen 1a of the display unit 2. In the electronic device 1 according to the first to fourth embodiments, the camera module 3 can be disposed in an area overlapping with the display unit 2, so that it is not necessary to provide the camera module 3 in the frame portion of the display unit 2, and the size of the display unit 2 can be made as large as possible.
[0126] Fig. 28 is a plan view showing an example in which the electronic device 1 of the first and second embodiments is applied to a head mounted display (hereinafter, HMD) 61. The HMD 61 in Fig. 28 is used for VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), SR (Substituional Reality), or the like. Current HMDs, as shown in Fig. 29, are equipped with a camera 62 on the outer surface, and while a person wearing the HMD can view images of the surroundings, there is a problem in that people around cannot see the eyes or facial expressions of the person wearing the HMD.
[0127] 28, the display surface of the display unit 2 is provided on the outer surface of the HMD 61, and the camera module 3 is provided on the opposite side to the display surface of the display unit 2. This allows the facial expression of the wearer captured by the camera module 3 to be displayed on the display surface of the display unit 2, allowing people around the wearer to grasp the facial expression and eye movements of the wearer in real time.
[0128] 28, since the camera module 3 is provided on the back side of the display unit 2, there are no restrictions on where to install the camera module 3, which increases the freedom of design of the HMD 61. In addition, since the camera can be placed in an optimal position, it is possible to prevent problems such as the wearer's line of sight not being aligned with the display surface.
[0129] In this way, in the third embodiment, the electronic device 1 according to the first and second embodiments can be used for various purposes, thereby increasing the utility value.
[0130] The present technology can be configured as follows. (1) An imaging unit having a plurality of pixel groups each composed of two adjacent pixels; At least one first pixel group of the plurality of pixel groups a first pixel that photoelectrically converts a portion of the incident light collected via the first lens; a second pixel different from the first pixel that photoelectrically converts a part of the incident light condensed through the first lens; having At least one second pixel group different from the first pixel group among the plurality of pixel groups, a third pixel that performs photoelectric conversion on the incident light collected via the second lens; a fourth pixel different from the third pixel that photoelectrically converts incident light collected via a third lens different from the second lens; An electronic device having
[0131] (2) the imaging unit is configured with a plurality of pixel regions in which the pixel group is arranged in a 2×2 matrix; The plurality of pixel regions include a first pixel region in which four of the first pixel groups are arranged; a second pixel region in which three of the first pixel groups and one of the second pixel groups are arranged; The electronic device according to (1) ,
[0132] (3) The electronic device described in (2), wherein in the first pixel region, a red filter, a green filter, or a blue filter is arranged corresponding to the first pixel group that receives red light, green light, and blue light.
[0133] (4) in the second pixel region, at least two of a red filter, a green filter, and a blue filter are arranged corresponding to the first pixel group that receives at least two colors of red light, green light, and blue light; The electronic device according to (3), wherein at least one of the two pixels of the second pixel group has one of a cyan filter, a magenta filter, and a yellow filter.
[0134] (5) The electronic device described in (4), wherein at least one of the two pixels in the second pixel group is a pixel having a blue wavelength range.
[0135] (6) The electronic device described in (4), further comprising a signal processing unit that performs color correction of an output signal output by at least one of the pixels of the first pixel group based on an output signal of at least one of the two pixels of the second pixel group.
[0136] (7) The electronic device described in (2), wherein at least one pixel of the second pixel group has a polarizing element.
[0137] (8) The electronic device described in (7), wherein the third pixel and the fourth pixel have the polarizing element, and the polarizing element of the third pixel and the polarizing element of the fourth pixel have different polarization orientations.
[0138] (9) The electronic device according to (7), further comprising a correction unit that corrects output signals of pixels in the first pixel group using polarization information based on output signals of pixels having the polarizing element.
[0139] (10) The incident light is incident on the first pixel and the second pixel through a display unit, The electronic device described in (9), wherein the correction unit removes polarized components of at least one of reflected light and diffracted light generated when passing through the display unit and incident on the first pixel and the second pixel and imaged.
[0140] (11) The electronic device described in (10), wherein the correction unit corrects the digital pixel data by performing a subtraction process on the digital pixel data that has been photoelectrically converted and digitized by the first pixel and the second pixel, by subtracting a correction amount based on polarization information data that has been digitized from the polarization component that has been photoelectrically converted by the pixel having the polarization element.
[0141] (12) a drive unit that reads out electric charges from each pixel of the plurality of pixel groups a plurality of times during one imaging frame; an analog-to-digital conversion unit that performs analog-to-digital conversion in parallel on each of a plurality of pixel signals based on a plurality of readouts of electric charges; The electronic device according to any one of (1) to (11), further comprising:
[0142] (13) The electronic device according to (12), wherein the drive section reads out a common black level corresponding to the third pixel and the fourth pixel.
[0143] (14) The electronic device according to any one of (1) to (13), wherein the plurality of pixels each formed of two adjacent pixels are square shaped.
[0144] (15) The electronic device according to any one of (1) to (14), wherein a phase difference can be detected based on output signals of two pixels of the first pixel group.
[0145] (16) The electronic device according to (6), wherein the signal processing unit performs white balance processing after performing color correction of the output signal.
[0146] (17) An interpolation unit that interpolates an output signal of a pixel having the polarizing element from outputs of pixels surrounding the pixel, The electronic device according to (7) further comprises:
[0147] (18) The electronic device according to any one of (1) to (17), wherein the first to third lenses are on-chip lenses that focus incident light onto a photoelectric conversion unit of a corresponding pixel.
[0148] (19) Further comprising a display unit, The electronic device according to any one of (1) to (18), wherein the incident light is incident on the plurality of pixel groups via the display unit.
[0149] The aspects of the present disclosure are not limited to the above-mentioned individual embodiments, but include various modifications that may be conceived by a person skilled in the art, and the effects of the present disclosure are not limited to the above-mentioned contents. In other words, various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and intent of the present disclosure derived from the contents defined in the claims and their equivalents. [Explanation of symbols]
[0150] 1: electronic device, 2: display unit, 8: imaging unit, 8a: first pixel region, 8b to 8k: second pixel region, 22: on-chip lens, 22a: on-chip lens, 36: flare correction signal generation unit, 80: pixel, 80a: pixel, 82: pixel, 82a: pixel, 130: vertical drive unit, 140, 150: analog-to-digital conversion unit, 510: signal processing unit, 800a: photoelectric conversion unit.
Claims
1. An imaging unit having a plurality of pixel groups each composed of two adjacent pixels, At least one first pixel group of the plurality of pixel groups a first pixel that photoelectrically converts a portion of the incident light collected via the first lens; a second pixel different from the first pixel that photoelectrically converts a part of the incident light condensed through the first lens; having At least one second pixel group different from the first pixel group among the plurality of pixel groups, a third pixel that performs photoelectric conversion on the incident light collected via the second lens; a fourth pixel different from the third pixel that photoelectrically converts incident light collected via a third lens different from the second lens; having the imaging unit is configured with a plurality of pixel regions in which the pixel group is arranged in a 2×2 matrix; The plurality of pixel regions include a first pixel region in which four of the first pixel groups are arranged; a second pixel region in which three of the first pixel groups and one of the second pixel groups are arranged; An electronic device having
2. 2 . The electronic device according to claim 1 , wherein in the first pixel region, any one of a red filter, a green filter, and a blue filter is arranged corresponding to the first pixel group that receives red light, green light, and blue light.
3. in the second pixel region, at least two of a red filter, a green filter, and a blue filter are arranged in correspondence with the first pixel group that receives at least two colors of red light, green light, and blue light; The electronic device of claim 2 , wherein at least one of the two pixels in the second pixel group has one of a cyan filter, a magenta filter, and a yellow filter.
4. The electronic device according to claim 3 , wherein at least one of the two pixels in the second pixel group is a pixel having a blue wavelength region.
5. The electronic device according to claim 3 , further comprising a signal processing unit that performs color correction of an output signal output by at least one of the pixels of the first pixel group based on an output signal of at least one of the two pixels of the second pixel group.
6. The electronic device of claim 1 , wherein at least one pixel in the second group of pixels comprises a polarizing element.
7. The electronic device according to claim 6 , wherein the third pixel and the fourth pixel each have the polarizing element, and the polarizing element of the third pixel and the polarizing element of the fourth pixel have different polarization orientations.
8. The electronic device according to claim 6 , further comprising a correction section that corrects output signals of pixels in the first pixel group using polarization information based on output signals of pixels having the polarizing elements.
9. the incident light is incident on the first pixel and the second pixel via a display unit, The electronic device according to claim 8 , wherein the correction section removes a polarized component of at least one of reflected light and diffracted light generated when passing through the display section and incident on the first pixel and the second pixel and captured.
10. 10. The electronic device according to claim 9, wherein the correction unit corrects the digital pixel data by performing a subtraction process on the digital pixel data that has been photoelectrically converted and digitized by the first pixel and the second pixel, by a correction amount based on polarization information data that has been digitized from a polarization component that has been photoelectrically converted by a pixel having the polarization element.
11. a drive unit that reads out electric charges from each pixel of the plurality of pixel groups a plurality of times during one imaging frame; an analog-to-digital conversion unit that performs analog-to-digital conversion in parallel on each of a plurality of pixel signals based on a plurality of readouts of electric charges; The electronic device of claim 1 , further comprising:
12. The electronic device according to claim 11 , wherein the drive section reads out a common black level corresponding to the third pixel and the fourth pixel.
13. The electronic device according to claim 1 , wherein the plurality of pixels each formed of two adjacent pixels are in a square shape.
14. The electronic device according to claim 1 , wherein a phase difference can be detected based on output signals of two pixels in the first pixel group.
15. The electronic device according to claim 5 , wherein the signal processing unit performs white balance processing after performing color correction on the output signal.
16. an interpolation unit that interpolates an output signal of a pixel having the polarizing element using digital pixel data of pixel positions surrounding the pixel; The electronic device according to claim 6 , further comprising:
17. The electronic device according to claim 1 , wherein the first to third lenses are on-chip lenses that focus incident light onto photoelectric conversion units of corresponding pixels.
18. Further comprising a display unit, The electronic device according to claim 1 , wherein the incident light is incident on the plurality of pixel groups via the display unit.
Citation Information
Patent Citations
Image processing device
JP2012075059A
Image pickup apparatus
JP2012168339A
Imaging element and imaging device
JP2019106634A
Solid-state image pickup element and electronic device
WO2016098640A1
