Imaging device and electronic device

Abstract

This imaging element is provided with a light receiving part (11) which has: a plurality of photoelectric conversion elements (110) that are disposed in a lattice arrangement; and a plurality of lenses (30) that are disposed so as to correspond, on a one-on-one basis, to respective element sets each comprising two or more of the photoelectric conversion elements arranged adjacent to each other. In the light receiving part, of a plurality of pixel sets each comprising an element set and a lens provided to the element set among the plurality of lenses, at least two adjacent pixel sets have mutually different pupil correction amounts.

Description

[Technical field]

[0001] The present invention relates to an imaging device and an electronic device. [Background technology]

[0002] A configuration is known in which light from a main lens is irradiated onto a pixel array in which a plurality of pixels each equipped with a microlens is arranged. In this configuration, the entrance pupil diameter for the pixel changes according to the image height of each pixel relative to the optical axis position of the main lens, and the amount of light irradiated onto the pixel changes. For this reason, a pupil correction technique is known in which the position of the microlens or the like for each pixel is shifted according to the image height to suppress the change in the entrance pupil diameter.

[0003] Also, an image plane phase difference AF method is known in which autofocus (AF) processing and parallax detection are performed based on the phase difference due to pixel signals of a plurality of pixels arranged in the row direction (or column direction) in the pixel array described above. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-188633 A [Patent Document 2] JP 2018-014476 A Summary of the Invention [Problem to be solved by the invention]

[0005] In existing technology, the amount of pupil correction for each pixel included in one pixel array is fixed for each pixel. However, when the pixel array is mounted on a general camera, if the main lens is changed by lens replacement or zoom operation, the entrance pupil diameter of the main lens changes. In this way, when the entrance pupil diameter of the main lens changes, pupil correction is not performed appropriately, making it difficult to obtain the phase difference of the image signal with high accuracy.

[0006] An object of the present disclosure is to provide an imaging element and electronic device that are capable of acquiring the phase difference of an image signal with high accuracy over a wider range of entrance pupil diameters. [Means for solving the problem]

[0007] The imaging element according to the present disclosure includes a plurality of photoelectric conversion elements arranged in a lattice pattern, and adjacent ones of the plurality of photoelectric conversion elements. of 2 and a plurality of lenses provided in a one-to-one correspondence with each of the sets of elements made of photoelectric conversion elements, and the light receiving unit has a pupil correction amount different from each other for at least two of a plurality of pixel sets each including the set of elements and a lens provided in the set of elements among the plurality of lenses. and each of the plurality of pixel blocks includes n×n (n is 2 or 4) photoelectric conversion elements that receive light of the same wavelength band among the plurality of photoelectric conversion elements and are arranged in a lattice pattern, the plurality of pixel blocks are arranged according to a pattern in which pixel blocks having the same wavelength band received by the arranged photoelectric conversion elements are not adjacent to each other, and at least one of the plurality of pixel blocks includes two of the pixel sets having different pupil correction amounts, and when n is 4, at least one of the plurality of pixel blocks includes four of the pixel sets having different pupil correction amounts, or two of the pixel sets having directions in which adjacent photoelectric conversion elements among the plurality of photoelectric conversion elements included in the light receiving unit are aligned on the array that are different from each other. . [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing a configuration of an example of an electronic device commonly applicable to the embodiments. [Diagram 2] FIG. 2 is a block diagram showing an example of a basic configuration of an image sensor that can be commonly applied to each embodiment. [Diagram 3] FIG. 1 is a diagram showing an example of a commonly used Bayer array. [Figure 4] FIG. 2 is a diagram illustrating an example of a pixel configuration applicable to the first embodiment. [Diagram 5] FIG. 13 is a diagram illustrating an example in which one OCL is provided for two pixels, which is applicable to each embodiment. [Figure 6] FIG. 1 is a diagram illustrating a schematic cross section of a pixel set applicable to each embodiment. [Figure 7] 1A and 1B are diagrams for explaining a first example of a method according to an existing technology for realizing an image plane phase difference AF method. [Figure 8A] 11A and 11B are diagrams for explaining a second example of a method according to an existing technology for realizing an image plane phase difference AF method. [Figure 8B]11A and 11B are diagrams for explaining a second example of a method according to an existing technology for realizing an image plane phase difference AF method. [Figure 9] FIG. 2 is a diagram showing an example of a pixel configuration according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating a pixel array section. [Figure 11A] FIG. 2 shows a schematic cross-section of a pixel set with strong pupil correction according to a first embodiment; [Figure 11B] FIG. 2 shows a schematic cross-section of a pixel set with weak pupil correction according to a first embodiment; [Figure 12] 5A to 5C are diagrams for explaining the effect of performing pupil correction using a plurality of pupil correction amounts in one pixel block according to the first embodiment. [Figure 13] 11A and 11B are diagrams illustrating examples of regions in a pixel array portion, the regions having different directions relative to the center of the image height. [Figure 14A] 6A to 6C are diagrams illustrating an example of pupil correction in an area C according to the first embodiment. [Figure 14B] 4A to 4C are diagrams illustrating an example of pupil correction in an area L according to the first embodiment. [Figure 14C] 4A to 4C are diagrams illustrating an example of pupil correction in a region R according to the first embodiment. [Figure 14D] 4A to 4C are diagrams illustrating an example of pupil correction in an area CB according to the first embodiment. [Figure 14E] 5A to 5C are diagrams illustrating an example of pupil correction in a region CT according to the first embodiment. [Figure 14F] 4A to 4C are diagrams illustrating an example of pupil correction in a region LT according to the first embodiment. [Figure 14G] 4A to 4C are diagrams illustrating an example of pupil correction in an area RB according to the first embodiment. [Figure 15] 4A to 4C are diagrams illustrating a first method for reading out signals from each pixel in each pixel block according to the first embodiment. [Figure 16] 6A and 6B are diagrams for explaining a second method for reading out signals from each pixel in each pixel block according to the first embodiment. [Figure 17] FIG. 2 is a cross-sectional view showing a first example of a pixel configuration for suppressing color mixing between pixel sets, which can be applied to the first embodiment. [Figure 18] FIG. 11 is a cross-sectional view showing a second example of a pixel configuration for suppressing color mixing between pixel sets, which can be applied to the first embodiment. [Figure 19] 4A to 4C are diagrams illustrating examples of light blocking bodies arranged on boundaries of pixel blocks that can be applied to the first embodiment. [Figure 20] 11A to 11C are diagrams illustrating examples of light blocking bodies arranged on the boundaries of each pixel set, which are applicable to the first embodiment. [Figure 21A] FIG. 13 is a diagram illustrating an example of pupil correction in a region C according to a modified example of the first embodiment. [Figure 21B] FIG. 11 is a diagram illustrating an example of pupil correction in a region CT according to a modified example of the first embodiment. [Figure 21C] FIG. 13 is a diagram illustrating an example of pupil correction in an area CB according to a modified example of the first embodiment. [Figure 21D] FIG. 11 is a diagram illustrating an example of pupil correction in an area L according to a modified example of the first embodiment. [Figure 21E] FIG. 11 is a diagram illustrating an example of pupil correction in a region R according to a modified example of the first embodiment. [Figure 21F] FIG. 13 is a diagram showing an example of pupil correction in a region LT according to a modified example of the first embodiment. [Figure 21G] FIG. 13 is a diagram illustrating an example of pupil correction in a region RB according to a modified example of the first embodiment. [Figure 22] FIG. 13 is a diagram illustrating an example of a pixel configuration applicable to the second embodiment. [Figure 23A] 13 is a diagram illustrating an example of pupil correction in an area C according to a second embodiment. FIG. [Figure 23B] 13 is a diagram illustrating an example of pupil correction in an area L according to a second embodiment. FIG. [Figure 23C] 13 is a diagram illustrating an example of pupil correction in a region R according to a second embodiment. FIG. [Figure 23D]13A to 13C are diagrams illustrating an example of pupil correction in a region CT according to a second embodiment. [Figure 23E] 13A to 13C are diagrams illustrating an example of pupil correction in an area CB according to the second embodiment. [Figure 23F] 13A to 13C are diagrams illustrating an example of pupil correction in a region LT according to a second embodiment. [Figure 23G] 13 is a diagram showing an example of pupil correction in an area RB according to a second embodiment. FIG. [Figure 24A] FIG. 13 is a diagram illustrating an example of pupil correction in a region C according to a modified example of the second embodiment. [Figure 24B] FIG. 13 is a diagram illustrating an example of pupil correction in an area L according to a modified example of the second embodiment. [Figure 24C] FIG. 13 is a diagram illustrating an example of pupil correction in a region R according to a modified example of the second embodiment. [Figure 24D] FIG. 13 is a diagram illustrating an example of pupil correction in the region CT according to a modified example of the second embodiment. [Figure 24E] FIG. 13 is a diagram illustrating an example of pupil correction in an area CB according to a modified example of the second embodiment. [Figure 24F] FIG. 13 is a diagram showing an example of pupil correction in a region LT according to a modified example of the second embodiment. [Figure 24G] FIG. 13 is a diagram illustrating an example of pupil correction in a region RB according to a modified example of the second embodiment. [Diagram 25] 1A to 1C are diagrams showing examples of use of the image pickup elements according to the first embodiment and its modified example, and the second embodiment and its modified example. [Figure 26] 1 is a block diagram showing an example of a schematic configuration of a patient's in-vivo information acquisition system using a capsule endoscope to which the technology according to the present disclosure can be applied. [Figure 27] 1 is a diagram showing an example of a schematic configuration of an endoscopic surgery system to which the technology disclosed herein can be applied. [Figure 28] 2 is a block diagram showing an example of a functional configuration of a camera head and a CCU. FIG. [Figure 29]1 is a block diagram showing a schematic configuration example of a vehicle control system that is an example of a mobile object control system to which the technology according to the present disclosure can be applied. [Diagram 30] FIG. 4 is a diagram showing an example of an installation position of an imaging unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0010] (Configuration applicable to all embodiments) Fig. 1 is a block diagram showing an example of a configuration of an electronic device commonly applicable to each embodiment. In Fig. 1, the electronic device 1 includes an optical system 2, a control unit 3, an image sensor 4, an image processing unit 5, a memory 6, a storage unit 7, a display unit 8, an interface (I / F) unit 9, and an input device 10.

[0011] Here, a digital still camera, a digital video camera, a mobile phone or a smartphone with an imaging function, etc. can be used as the electronic device 1. In addition, a surveillance camera, an in-vehicle camera, a medical camera, etc. can also be used as the electronic device 1.

[0012] The imaging element 4 includes a plurality of photoelectric conversion elements arranged, for example, in a grid pattern. The photoelectric conversion elements convert received light into electric charges through photoelectric conversion. The imaging element 4 includes a drive circuit that drives the plurality of photoelectric conversion elements, and a signal processing circuit that reads out electric charges from each of the plurality of photoelectric conversion elements and generates image data based on the read out electric charges.

[0013] The optical system 2 includes a main lens formed by a combination of one or more lenses and a mechanism for driving the main lens, and forms an image of image light (incident light) from a subject on the light receiving surface of the image sensor 4 via the main lens. The optical system 2 also includes an autofocus mechanism that adjusts the focus according to a control signal, and a zoom mechanism that changes the zoom ratio according to a control signal. The electronic device 1 may also have the optical system 2 detachable so that it can be replaced with another optical system 2.

[0014] The image processing unit 5 executes a predetermined image processing on the image data output from the imaging element 4. For example, the image processing unit 5 is connected to a memory 6 such as a frame memory, and writes the image data output from the imaging element 4 into the memory 6. The image processing unit 5 executes a predetermined image processing on the image data written into the memory 6, and writes the image-processed image data back into the memory 6.

[0015] The storage unit 7 is a non-volatile memory such as a flash memory or a hard disk drive, and stores the image data output from the image processing unit 5 in a non-volatile manner. The display unit 8 includes a display device such as an LCD (Liquid Crystal Display) and a drive circuit for driving the display device, and can display an image based on the image data output from the image processing unit 5. The I / F unit 9 is an interface for transmitting the image data output from the image processing unit 5 to the outside. As the I / F unit 9, for example, a USB (Universal Serial Bus) can be applied. However, the I / F unit 9 is not limited to this, and may be an interface that can be connected to a network by wired communication or wireless communication.

[0016] The input device 10 includes an operator for receiving user input, etc. If the electronic device 1 is, for example, a digital still camera, a digital video camera, a mobile phone with an imaging function, or a smartphone, the input device 10 may include a shutter button for instructing the imaging element 4 to capture an image, or an operator for realizing the function of the shutter button.

[0017] The control unit 3 includes a processor such as a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory), and controls the overall operation of the electronic device 1 using the RAM as a work memory in accordance with a program pre-stored in the ROM. For example, the control unit 3 can control the operation of the electronic device 1 in response to a user input received by the input device 10. The control unit 3 can also control an autofocus mechanism in the optical system 2 based on the image processing result of the image processing unit 5.

[0018] Fig. 2 is a block diagram showing an example of a basic configuration of an image sensor 4 that can be commonly applied to each embodiment. In Fig. 2, the image sensor 4 includes a pixel array section 11, a vertical scanning section 12, an AD (Analog to Digital) conversion section 13, pixel signal lines 16, vertical signal lines 17, an output section 18, a control section 19, and a signal processing section 20.

[0019] The pixel array section 11 includes a plurality of pixels 110 each having a photoelectric conversion element that performs photoelectric conversion on received light. A photodiode can be used as the photoelectric conversion element. In the pixel array section 11, the plurality of pixels 110 are arranged in a two-dimensional lattice in the horizontal direction (row direction) and the vertical direction (column direction). In the pixel array section 11, the row direction arrangement of the pixels 110 is called a line. One frame of image (image data) is formed by pixel signals read out from a predetermined number of lines in this pixel array section 11. For example, when one frame of image is formed with 3000 pixels x 2000 lines, the pixel array section 11 includes at least 2000 lines each including at least 3000 pixels 110.

[0020] In addition, in the pixel array section 11, pixel signal lines 16 are connected to the rows and columns of the pixels 110 for each row, and vertical signal lines 17 are connected to the columns.

[0021] The ends of the pixel signal lines 16 that are not connected to the pixel array unit 11 are connected to the vertical scanning unit 12. The vertical scanning unit 12 transmits control signals such as drive pulses used when reading out pixel signals from the pixels 110 to the pixel array unit 11 via the pixel signal lines 16 under the control of a control unit 19, which will be described later. The ends of the vertical signal lines 17 that are not connected to the pixel array unit 11 are connected to an AD conversion unit 13. The pixel signals read out from the pixels are transmitted to the AD conversion unit 13 via the vertical signal lines 17.

[0022] The control of reading out pixel signals from pixels will be briefly described. Reading out pixel signals from pixels is performed by transferring charges accumulated in a photoelectric conversion element by exposure to a floating diffusion layer (FD; Floating Diffusion) and converting the transferred charges in the floating diffusion layer into a voltage. The voltage into which the charges are converted in the floating diffusion layer is output to a vertical signal line 17 via an amplifier.

[0023] More specifically, in the pixel 110, during exposure, the photoelectric conversion element and the floating diffusion layer are turned off (open) to accumulate charges generated in response to the incident light by photoelectric conversion in the photoelectric conversion element. After the exposure is completed, the floating diffusion layer and the vertical signal line 17 are connected in response to a selection signal supplied via the pixel signal line 16. Furthermore, the floating diffusion layer is connected to a supply line of a power supply voltage VDD or a black level voltage for a short period of time in response to a reset pulse supplied via the pixel signal line 16 to reset the floating diffusion layer. A voltage (referred to as voltage P) of a reset level of the floating diffusion layer is output to the vertical signal line 17. Thereafter, a transfer pulse supplied via the pixel signal line 16 turns the photoelectric conversion element and the floating diffusion layer into an on (closed) state, and the charges accumulated in the photoelectric conversion element are transferred to the floating diffusion layer. A voltage (referred to as voltage Q) corresponding to the amount of charge in the floating diffusion layer is output to the vertical signal line 17.

[0024] The AD conversion unit 13 includes an AD converter 1300 provided for each vertical signal line 17, a reference signal generation unit 14, and a horizontal scanning unit 15. The AD converter 1300 is a column AD converter that performs AD conversion processing for each column of the pixel array unit 11. The AD converter 1300 performs AD conversion processing on pixel signals supplied from the pixels 110 via the vertical signal lines 17, and generates two digital values ​​(values ​​corresponding to voltage P and voltage Q, respectively) for correlated double sampling (CDS) processing that reduces noise.

[0025] The AD converter 1300 supplies the two generated digital values ​​to the signal processing unit 20. The signal processing unit 20 performs CDS processing based on the two digital values ​​supplied from the AD converter 1300, and generates a pixel signal (pixel data) based on a digital signal. The pixel data generated by the signal processing unit 20 is output to the outside of the imaging element 4.

[0026] The image data output from the signal processing unit 20 is supplied to, for example, the image processing unit 5 and sequentially stored in, for example, a frame buffer memory 6. When one frame's worth of pixel data is stored in the frame buffer, the stored pixel data is read out from the frame buffer as one frame's image data.

[0027] The reference signal generating unit 14 generates a ramp signal RAMP used by each AD converter 1300 to convert a pixel signal into two digital values, based on an ADC control signal input from the control unit 19. The ramp signal RAMP is a signal whose level (voltage value) decreases with time at a constant slope, or a signal whose level decreases in a step-like manner. The reference signal generating unit 14 supplies the generated ramp signal RAMP to each AD converter 1300. The reference signal generating unit 14 is configured using, for example, a DA (Digital to Analog) conversion circuit or the like.

[0028] Under the control of the control unit 19, the horizontal scanning unit 15 performs selective scanning to select each AD converter 1300 in a predetermined order, thereby causing each AD converter 1300 to sequentially output each digital value temporarily held therein to the signal processing unit 20. The horizontal scanning unit 15 is configured using, for example, a shift register, an address decoder, etc.

[0029] The control unit 19 controls the driving of the vertical scanning unit 12, the AD conversion unit 13, the reference signal generation unit 14, the horizontal scanning unit 15, etc. The control unit 19 generates various driving signals that serve as references for the operations of the vertical scanning unit 12, the AD conversion unit 13, the reference signal generation unit 14, and the horizontal scanning unit 15. The control unit 19 generates control signals for the vertical scanning unit 12 to supply to each pixel 110 via the pixel signal lines 16, based on a vertical synchronization signal or an external trigger signal supplied from the outside (e.g., the control unit 3) and a horizontal synchronization signal. The control unit 19 supplies the generated control signals to the vertical scanning unit 12.

[0030] Based on a control signal supplied from the control unit 19, the vertical scanning unit 12 supplies various signals including drive pulses to the pixel signal lines 16 of a selected pixel row of the pixel array unit 11 for each pixel 110 on a line basis, and causes each pixel 110 to output a pixel signal to a vertical signal line 17. The vertical scanning unit 12 is configured using, for example, a shift register, an address decoder, etc.

[0031] The imaging element 4 configured in this manner is a column AD type CMOS (Complementary Metal Oxide Semiconductor) image sensor in which the AD converters 1300 are arranged on a column basis.

[0032] (Outline of color filter arrangement) Each pixel 110 may be provided with a filter that selectively transmits light of a predetermined wavelength band. When the transmitted wavelength band is in the visible light region, the filter is called a color filter. In the following, it is assumed that a color filter of each wavelength band of red (R), green (G), and blue (B) constituting the three primary colors is provided for each pixel 110. However, it is also possible to provide a color filter of each color that is complementary to each pixel 110, or a filter that selectively transmits light of a wavelength band in the infrared region, or a filter that transmits light of all wavelength bands in the visible light region. In the following, unless otherwise specified, these various filters will be described as being represented by color filters.

[0033] Fig. 3 is a diagram showing an example of a commonly used Bayer array. In Fig. 3, the Bayer array includes two pixels 110G in which a G color filter is arranged, one pixel 110R in which an R color filter is arranged, and a pixel 110B in which a B color filter is arranged. In the Bayer array, these four pixels are arranged in a 2 pixel x 2 pixel grid so that two pixels 110G are not adjacent to each other. In other words, the Bayer array is an array in which pixels 110 in which color filters that transmit light of the same wavelength band are arranged are not adjacent to each other.

[0034] In the following, unless otherwise specified, the "pixel 110R on which an R color filter is arranged" will be referred to as the "R pixel 110R" or simply as the "pixel 110R." The same applies to the pixel 110G on which a G color filter is arranged and the pixel 110B on which a B color filter is arranged. In addition, when the color filters are not of particular concern, the pixels 110R, 110G, and 110B will be described as being representative of the pixel 110.

[0035] Fig. 4 is a diagram showing an example of a pixel configuration applicable to the first embodiment described later. In the pixel configuration of Fig. 4, a pixel block is a unit in which 2 pixels x 2 pixels of the same color are arranged in a grid pattern, and each pixel block is composed of four R pixels 110R, four G pixels 110G, and four B pixels 110B, and is arranged in a pixel array conforming to the Bayer array. Hereinafter, unless otherwise specified, such a pixel array is called a 4-division Bayer type RGB array.

[0036] More specifically, in the four-division Bayer RGB array, pixel blocks made up of R pixels 110R, G pixels 110G, and B pixels 110B are arranged in a 2×2 grid such that the ratio of the numbers of pixels 110R, pixels 110G, and pixels 110B is 1:2:1, and pixel blocks made up of pixels of the same color are not adjacent to each other. In the example of Fig. 4, pixel blocks made up of G pixels 110G are arranged to the left and below the pixel blocks made up of B pixels 110B, and pixel blocks made up of R pixels 110R are arranged diagonally from the pixel blocks made up of B pixels 110B.

[0037] (Outline of OCL arrangement applicable to all embodiments) Each pixel 110 arranged in the pixel array unit 11 is provided with an OCL (On Chip Lens). In each embodiment, one OCL is provided in common for a plurality of adjacent pixels 110. Fig. 5 is a diagram that illustrates an example in which one OCL is provided for two pixels 110, which is applicable to each embodiment.

[0038] A more detailed description will be given by taking as an example four G pixels 110G1, 110G2, 110G3, and 110G4 in a 4-division Bayer RGB array shown in Fig. 5. One OCL 30 is provided for a pair of two pixels 110G1 and 110G2 adjacent to each other in the horizontal direction in Fig. 5. Similarly, one OCL 30 is provided for a pair of two pixels 110G3 and 110G4 adjacent to each other in the horizontal direction. Similarly, for the R pixel 110R and the B pixel 110B, one OCL 30 is provided for each pair of two pixels 110R adjacent to each other in the horizontal direction in Fig. 5 and each pair of two pixels 110B adjacent to each other in the horizontal direction.

[0039] In the following, a set of adjacent pixels 110 that share one OCL 30 and the OCL 30 are collectively referred to as a "pixel set" as appropriate. In the example of Fig. 5, a pixel set is configured including two pixels 110 that are adjacent in the horizontal direction of the pixel array unit 11. In the following, in each diagram showing the arrangement of the pixels 110 on a two-dimensional lattice such as Fig. 5, the left-right direction of the diagram is the horizontal direction and the up-down direction of the diagram is the vertical direction.

[0040] FIG. 6 is a diagram illustrating a schematic cross section of a pixel set applicable to each embodiment. In the example of FIG. 6, the pixel set includes two pixels 110 arranged adjacent to each other. In FIG. 6, each of the two pixels 110 included in the pixel set is roughly structured such that a color filter 112 is provided on an incident surface of a photoelectric conversion unit 111 that generates electric charges in response to incident light. The two color filters 112 included in one pixel set transmit light of the same wavelength band. An OCL 30 is provided on the color filter 112 of each of the two pixels 110 in common to the two pixels 110.

[0041] (About the image plane phase detection AF method) Next, the image plane phase difference AF method will be briefly described. In the image plane phase difference AF method, autofocus control and parallax detection are performed based on the phase difference of pixel signals extracted from pixels 110 at different positions. In the example of Fig. 5, for example, for a pixel set including two pixels 110G1 and 110G2 that share one OCL 30, the phase difference of each pixel signal by each of the pixels 110G1 and 110G2 is detected, and information for performing, for example, autofocus control is obtained based on the detected phase difference.

[0042] 7 is a diagram for explaining a first example of a method according to an existing technique for realizing an image plane phase difference AF method. In this first example, one pixel 110w is configured to include two photoelectric conversion units 111wa and 111wb arranged adjacent to each other. One color filter 112a and one OCL 31 are provided in common for the photoelectric conversion units 111wa and 111wb. A light shielding body 113 is provided between the color filter 112a and another color filter 112b adjacent to the color filter 112a and having a different wavelength band to that of the color filter 112a, thereby suppressing light leakage between the color filter 112a and the color filter 112b.

[0043] In such a configuration, for example, by detecting the phase difference of each pixel signal by each of the photoelectric conversion units 111wa and 111wb, image plane phase difference AF and parallax detection can be realized. That is, in the configuration of FIG. 7, the pixel 110w including the two adjacently arranged photoelectric conversion units 111wa and 111wb is a pixel for phase difference detection. Hereinafter, the "pixel for phase difference detection" will be referred to as a "phase difference pixel" as appropriate.

[0044] 7, the color filter 112a (light shield 113) and the OCL 31 are arranged to be shifted by a predetermined distance toward the right in FIG. 7 with respect to the incident surface of each of the photoelectric conversion units 111wa and 111wb included in the pixel 110w. This allows pupil correction to be performed on the light 40 that is incident on the incident surface from an oblique direction. By performing pupil correction, the range 41 in which the light 40 is incident on the incident surface can be made appropriate, and the accuracy of the image plane phase difference AF and the parallax detection can be improved.

[0045] The amount of pupil correction, i.e., the amount of shift of the color filter 112a (light shield 113) and OCL 31 relative to the incident surface, is set according to the image height relative to the optical axis position of the main lens in the pixel array in which the pixel 110w is arranged. For example, the higher the image height at the position where the pixel 110w is arranged, the larger the amount of pupil correction is. The amount of pupil correction of the pixel 110w arranged at the position (image height center) corresponding to the optical axis position of the main lens in the pixel array is set to zero, for example.

[0046] According to the configuration shown in Fig. 7, the amount of pupil correction is fixed for each image height. Therefore, in order to achieve accuracy, it is necessary to use a main lens with an EPD (Exit Pupil Distance) that corresponds to that amount of pupil correction. Therefore, when the main lens is replaced with another main lens with a different EPD due to lens replacement or when the EPD changes due to zooming, it becomes difficult to achieve high accuracy.

[0047] 8A and 8B are diagrams for explaining a second example of a method according to an existing technology for realizing an image plane phase difference AF method. In this second example, in a pixel array, a phase difference pixel is provided separately from a pixel for forming an image. In the example of FIG. 8A, two pixels, pixels 110a and 110b, are shown as examples of phase difference pixels.

[0048] In Fig. 8A, pixels 110a and 110b each have one photoelectric conversion unit 111a and 111b. In pixel 110a, the right half of the incident surface of photoelectric conversion unit 111a in Fig. 8A is shielded from light by using a light shield 50, and the left half is open. On the other hand, in pixel 110b, the left half of the incident surface of photoelectric conversion unit 111b in Fig. 8A is shielded from light by using a light shield 50, and the right half is open.

[0049] These pixels 110a and 110b are arranged close to each other (for example, adjacent to each other) along the shift direction of the opening position of the light shielding body 50 relative to the photoelectric conversion units 111a and 111b. By detecting the phase difference between the pixel signals of the pixels 110a and 110b, image plane phase difference AF and parallax detection can be realized. That is, in this second example, it can be considered that the two pixels 110a and 110b constitute one phase difference pixel.

[0050] Here, the example of Fig. 8A shows pixels 110a and 110b arranged on the left end side of the pixel array in Fig. 8A. The pixels 110a and 110b are arranged such that the OCLs 31 are shifted to the right in Fig. 8A with respect to the entrance surfaces of the photoelectric conversion units 111a and 111b, respectively, and pupil correction is performed according to the position (image height) on the pixel array.

[0051] In this second example, another phase difference pixel having a pupil correction amount different from that of the pixels 110a and 110b is provided at a position on the pixel array that is approximately equal in image height to the pixels 110a and 110b.

[0052] Fig. 8B is a diagram showing an example of another phase difference pixel according to the second example. In Fig. 8B, two pixels 110a' and 110b' constituting the phase difference pixel are an example in which the amount of pupil correction is increased so as to be compatible with a main lens of a shorter EPD compared to the pixels 110a and 110b shown in Fig. 8A.

[0053] In pixel 110a' in Fig. 8B, the opening defined by light shield 50 of photoelectric conversion unit 111a' is narrower toward the left end side than pixel 110a shown in Fig. 8A. On the other hand, in pixel 110b', the opening defined by light shield 50 of photoelectric conversion unit 111b' is wider toward the right than pixel 110b shown in Fig. 8A. That is, the configuration of Fig. 8B enables pupil correction for light 40a' and 40b' incident at a larger angle than light 40a and 40b for which pupil correction is possible for pixels 110a and 110b in Fig. 8A.

[0054] In the second example, by arranging a plurality of phase difference pixels with different pupil correction amounts in the pixel array, it is possible to widen the range of the EPD in which autofocus processing by image plane phase difference AF is possible. However, in the second example, since the phase difference pixels are not used as pixels for forming an image, arranging many phase difference pixels with different pupil correction amounts reduces the number of pixels for forming an image, resulting in degradation of image quality.

[0055] [First embodiment] Next, a first embodiment will be described. Fig. 9 is a diagram showing an example of a pixel configuration according to the first embodiment. Fig. 9 shows an example of a pixel configuration in an area L on the left end side of the image height center in Fig. 10 in a pixel array section 11 shown in Fig. 10.

[0056] In Fig. 9, the pixels 110R, 110G, and 110B are arranged in a four-division Bayer array as described with reference to Fig. 4, and each pixel block is made up of four pixels 110R, four pixels 110G, and four pixels 110B. As described with reference to Fig. 5, among the pixels 110R, 110G, and 110B, one OCL 30 is commonly arranged for two pixels of the same color that are adjacent in the horizontal direction, forming a pixel set. That is, each pixel block includes two pixel sets, a pixel set of two pixels in the upper row, and a pixel set of two pixels in the lower row.

[0057] Here, in the first embodiment, the pupil correction amounts of the two pixel sets included in each pixel block are made different from each other. More specifically, in the pixel set in the upper row of each pixel block in Fig. 9, stronger pupil correction is performed with a larger pupil correction amount toward the image height center in the right direction in Fig. 9 (hereinafter referred to as "strong pupil correction"). Also, in the pixel set in the lower row of each pixel block in Fig. 9, weaker pupil correction is performed with a smaller pupil correction amount compared to the pixel set in the upper row of the pixel block toward the image height center in the right direction in Fig. 9 (hereinafter referred to as "weak pupil correction").

[0058] In this way, in the first embodiment, in a pixel block in which pixels 110 of the same color are arranged in a 2 pixel x 2 pixel array, the amount of pupil correction is made different for each of the two pixel sets included in the pixel block, making it possible to perform appropriate pupil correction for two types of main lenses with different EPDs.

[0059] The pupil correction according to the first embodiment will be described more specifically with reference to FIGS. 11A and 11B.

[0060] Fig. 11A is a diagram showing a schematic cross section of a pixel set for performing strong pupil correction according to the first embodiment. Fig. 11A corresponds to a cross section of the pixel set in the upper row of each pixel block shown in Fig. 9 described above. For example, for a pixel block including four pixels 110G, a pixel set including two pixels 110G1 and 110G2 in the upper row of the pixel block will be described with reference to Fig. 5 as an example.

[0061] 11A, color filters 112G1 and 112G2 provided in pixels 110G1 and 110G2, respectively, are arranged to be shifted toward the image height center (rightward in FIG. 11A) with respect to the incident surfaces of photoelectric conversion units 111G1 and 111G2 of pixels 110G1 and 110G2, respectively. Furthermore, OCL 30 provided commonly to pixels 110G1 and 110G2 is arranged to be shifted toward the image height center with respect to color filters 112G1 and 112G2.

[0062] In this manner, the photoelectric conversion units 111G1 and 111G2, the color filters 112G1 and 112G2, and the OCL 30 are arranged with their positions shifted in the same direction, so that in each of the pixels 110G1 and 110G2, pupil correction is performed on the light 40c incident on the incident surfaces of the photoelectric conversion units 111G1 and 111G2 at a predetermined incident angle α.

[0063] This pupil correction is performed based on the amount of pupil correction corresponding to the amount of shift of each of the color filters 112G1 and 112G2 and the OCL 30 provided in common to the pixels 110G1 and 110G2 with respect to each of the photoelectric conversion units 111G1 and 111G2. This pupil correction can make appropriate the range 41c where the light 40c is incident on the incident surfaces of the photoelectric conversion units 111G1 and 111G2 at the incident angle α.

[0064] Fig. 11B is a diagram showing a schematic cross section of a pixel set for performing weak pupil correction according to the first embodiment. Fig. 11B corresponds to a cross section of a pixel set in the lower row of each pixel block shown in Fig. 9 described above. For example, for a pixel block including four pixels 110G, a pixel set consisting of two pixels 110G3 and 110G4 in the lower row of the pixel block will be described with reference to Fig. 5 as an example.

[0065] In Fig. 11B, the color filters 112G3 and 112G4 provided in the pixels 110G3 and 110G4, respectively, are shifted toward the image height center (rightward in Fig. 11B) with respect to the incident surfaces of the photoelectric conversion units 111G3 and 111G4 of the pixels 110G3 and 110G4, respectively, by a smaller shift amount than in Fig. 11A. Furthermore, the OCL 30 provided in common to the pixels 110G3 and 110G4 is shifted toward the image height center with respect to the color filters 112G3 and 112G4 by a smaller shift amount than in Fig. 11A.

[0066] In this way, the photoelectric conversion units 111G3 and 111G4, the color filters 112G3 and 112G4, and the OCL 30 are arranged in the same direction, with their positions shifted by a smaller shift amount than in the case of Fig. 11A. This allows pupil correction to be performed on the light 40d incident on the incident surfaces of the photoelectric conversion units 111G3 and 111G4 at a predetermined incident angle β smaller than the incident angle α in each of the pixels 110G3 and 110G4.

[0067] This pupil correction is performed based on the pupil correction amount according to the shift amount of each of the color filters 112G3 and 112G4 and the OCL 30 provided in common to the pixels 110G3 and 110G4 with respect to each of the photoelectric conversion units 111G3 and 111G4. This pupil correction can make the range 41d where the light 40d is incident on the incident surface of the photoelectric conversion units 111G3 and 111G4 appropriate.

[0068] 11A and 11B, for example, a metal film (metal mask) for limiting incident light can be further provided. Taking Fig. 11A as an example, a metal mask having an opening whose position is shifted with respect to each of the photoelectric conversion units 111R, 111G1, and 111G2 according to the pupil correction amount is provided between each of the photoelectric conversion units 111R, 111G1, and 111G2 and each of the color filters 112R, 112G1, and 112G2.

[0069] Fig. 12 is a diagram for explaining the effect of performing pupil correction using a plurality of pupil correction amounts in one pixel block according to the first embodiment. Fig. 12 is a diagram showing a schematic diagram of the accuracy of autofocus (AF) and parallax detection with respect to the EPD of the main lens. In Fig. 12, the horizontal axis shows the main lens EPD, and the vertical axis shows the accuracy of AF and parallax detection (AF·parallax detection accuracy).

[0070] 12, characteristic line 51 shows a schematic example of the AF / parallax detection accuracy when pupil correction is performed with an amount of pupil correction corresponding to the EPD of a specific main lens. The peak position of characteristic line 51 corresponds to the EPD of the main lens, and the AF / parallax detection accuracy attenuates with a spread in the direction of the EPD relative to the EPD of the main lens.

[0071] 12, characteristic line 50a shows an example where strong pupil correction is performed, and characteristic line 50b shows an example where weak pupil correction is performed. In this example, strong pupil correction is set to have a peak at a short EPD relative to the EPD due to a specific main lens shown by characteristic line 51, as shown by characteristic line 50a. On the other hand, weak pupil correction is set to have a peak at a long EPD relative to the EPD due to a specific main lens shown by characteristic line 51, as shown by characteristic line 50b.

[0072] Here, it is assumed that the required AF / parallax detection accuracy (required accuracy) is set with a margin relative to the peaks of the characteristic lines 50a, 50b, and 51. In the example of Fig. 12, the required accuracy is set to an accuracy lower than the intersection position of the characteristic lines 50a and 50b.

[0073] In this case, the range of EPDs in which the required accuracy can be obtained when pupil correction is performed with an amount of pupil correction corresponding to the EPD of a specific main lens is set to range A based on characteristic line 51. In this case, for example, when zooming is performed over a wide range from wide angle to telephoto, it becomes difficult to perform autofocus with the required accuracy over the relatively wide ranges on both the wide angle side and the telephoto side.

[0074] In contrast, when strong pupil correction is combined with weak pupil correction, as shown by characteristic lines 50a and 50b, there is an overlap between the range of EPD in which the required accuracy can be obtained by strong pupil correction and the range of EPD in which the required accuracy can be obtained by weak pupil correction. As a result, when strong pupil correction is combined with weak pupil correction, the required accuracy can be obtained in EPD range B, which is wider than range A. Therefore, even when zooming is performed over a wide range from wide angle to telephoto as described above, autofocus can be performed with the required accuracy on both the wide angle side and the telephoto side.

[0075] (Specific example of pupil correction according to the first embodiment) In the above, as shown in Fig. 10, pupil correction according to the first embodiment has been described by taking as an example the region L on the left end side in Fig. 10 with respect to the image height center of the pixel array unit 11. In reality, pupil correction is performed in each direction with respect to the image height center of the pixel array unit 11 according to the image height and the direction toward the image height center.

[0076] Fig. 13 is a diagram showing examples of regions in the pixel array unit 11 that have different directions relative to the image height center. In Fig. 13, region C is a region that corresponds to the image height center. Regions L and R are regions at the horizontal end of the image height center. Regions CT and CB are regions at the vertical end of the image height center. Regions LT and RB are regions at the upper left and lower right ends (corners) of Fig. 13 relative to the image height center.

[0077] 14A to 14G, examples of the directions of pupil correction in the above-mentioned regions C, L, R, CT, CB, LT, and RB according to the first embodiment will be described. Note that in FIGS. 14A to 14G, "None" on the right side indicates that no pupil correction is performed in the corresponding row. Also, "Strong" on the right side indicates that strong pupil correction is performed in the corresponding row, and "Weak" indicates that weak pupil correction is performed in the corresponding row.

[0078] 14A is a diagram showing an example of pupil correction in region C according to the first embodiment. In region C, as shown at the right end of each row in FIG. 14A, pupil correction is not performed on each pixel set in each pixel block.

[0079] Fig. 14B is a diagram showing an example of pupil correction in region L according to the first embodiment. Fig. 14B is a diagram similar to Fig. 9 described above, and in region L, strong pupil correction is performed in the pixel set in the upper row of each pixel block, and weak pupil correction is performed in the pixel set in the lower row, toward the right side (image height center) in Fig. 14B.

[0080] 14C is a diagram showing an example of pupil correction in region R according to the first embodiment. In region R, strong pupil correction is performed on the pixel set in the upper row of each pixel block toward the left side (center of image height) in FIG. 14C, and weak pupil correction is performed on the pixel set in the lower row.

[0081] Fig. 14D is a diagram showing an example of pupil correction in area CB according to the first embodiment. In area CB, pupil correction is performed with the same amount of pupil correction on the pixel sets in the upper and lower rows of each pixel block toward the upper side (image height center) in Fig. 14D.

[0082] Fig. 14E is a diagram showing an example of pupil correction in the region CT according to the first embodiment. In the region CT, pupil correction is performed with the same amount of pupil correction on the pixel sets in the upper and lower rows of each pixel block toward the lower side (image height center) in Fig. 14E.

[0083] Fig. 14F is a diagram showing an example of pupil correction in region LT according to the first embodiment. In region LT, strong pupil correction is performed on the upper pixel set of each pixel block and weak pupil correction is performed on the lower pixel set in Fig. 14F diagonally downward to the right (center of image height). In the example of Fig. 14F, pupil correction is performed in region LT in a direction that combines the direction of pupil correction in region L in Fig. 14B (right side) and the direction of pupil correction in region CT in Fig. 14E (downward).

[0084] Fig. 14G is a diagram showing an example of pupil correction in region RB according to the first embodiment. In region RB, strong pupil correction is performed on the pixel set in the upper row of each pixel block, and weak pupil correction is performed on the pixel set in the lower row, toward the upper left side (center of image height) in Fig. 14G. In the example of Fig. 14G, pupil correction is performed in region RB in a direction that combines the direction of pupil correction in region R in Fig. 14C (left side) and the direction of pupil correction in region CB in Fig. 14D (upper side).

[0085] In addition to the above-mentioned pupil correction according to each direction with respect to the image height center, the amount of pupil correction can be changed according to the image height.

[0086] In the above description, each pixel block includes a pixel set that performs strong pupil correction and a pixel set that performs weak pupil correction in all areas other than the areas having a predetermined width in the horizontal direction, including areas C, CT, and CB, which are included in the pixel array unit 11, but this is not limited to this example. For example, in all areas other than the areas having a predetermined width in the horizontal direction, including areas C, C, CT, and CB, which are included in the pixel array unit 11, at least one pixel block may include a pixel set that performs strong pupil correction and a pixel set that performs weak pupil correction.

[0087] (Reading Method for Each Pixel Block According to the First Embodiment) Next, a method for reading out signals from each pixel 110 (photoelectric conversion unit 111) in each pixel block according to the first embodiment will be described. In the first embodiment, in each pixel block, a first readout method for reading out signals individually from each pixel 110 and a second readout method for adding up and collectively reading out the signals of each pixel 110 can be executed.

[0088] Fig. 15 is a diagram for explaining a first method for reading out signals from each pixel 110 in each pixel block according to the first embodiment. Fig. 15 and Fig. 16, which will be described later, will be explained by taking as an example a pixel block including the pixels 110G1, 110G2, 110G3, and 110G4 in Fig. 5. Here, the pixel block is configured such that the pixels 110G1, 110G2, 110G3, and 110G4 share one floating diffusion layer.

[0089] In the first readout method, charges are sequentially read out from the photoelectric conversion elements (photoelectric conversion units 111) in the pixels 110G1, 110G2, 110G3, and 110G4 in accordance with the above-mentioned readout control.

[0090] That is, for example, under the control of the control unit 19, the vertical scanning unit 12 resets the floating diffusion layer in the pixel block, and then reads out charges from the photoelectric conversion unit 111 in the pixel 110G1 and transfers the read out charges to the floating diffusion layer. In the floating diffusion layer, the transferred charges are converted into a voltage according to the amount of charge, and are output to the vertical signal line 17 as a pixel signal read out from the pixel 110G1.

[0091] Next, the vertical scanning unit 12 resets the floating diffusion layer in the pixel block, and then reads out the charge from the photoelectric conversion unit 111 in the pixel 110G2 and transfers the read out charge to the floating diffusion layer. In the floating diffusion layer, the transferred charge is converted into a voltage according to the amount of charge, and is output to the vertical signal line 17 as a pixel signal read out from the pixel 110G2.

[0092] The vertical scanning unit 12 similarly reads out pixel signals from pixels 110G3 and 110G4, resetting the floating diffusion layers of each pixel, and then reads out charges from the photoelectric conversion unit 111 and transfers the read out charges to the floating diffusion layers.

[0093] The pixel signals read out from the pixels 110G1 to 110G4 are respectively supplied to, for example, the image processing unit 5. The image processing unit 5 detects a phase difference in the horizontal direction based on, for example, two pixels constituting a pixel set, for example, the pixel signals of the pixels 110G1 and 110G2, or the pixel signals of the pixels 110G3 and 110G4, out of the supplied pixel signals. The image processing unit 5 passes information indicating the detected phase difference to the control unit 3.

[0094] The control unit 3 generates a control signal for performing, for example, image plane phase difference AF based on the information indicating the phase difference passed from the image processing unit 5. The control unit 3 controls the optical system 2 based on this control signal, and causes the optical system 2 to perform an AF operation. Without being limited to this, the control unit 3 can also obtain parallax information based on the information indicating the phase difference passed from the image processing unit 5.

[0095] 16 is a diagram for explaining a second method for reading out signals from the pixels 110 in each pixel block according to the first embodiment. In the second readout method, charges accumulated in the photoelectric conversion units 111 in the pixels 110G1, 110G2, 110G3, and 110G4 are added together within the pixel block and read out.

[0096] That is, for example, under the control of the control unit 19, the vertical scanning unit 12 resets the floating diffusion layer in the pixel block, and then reads out charges from each photoelectric conversion unit 111 in the pixels 110G1 to 110G4 at a predetermined timing, and transfers the read out charges to the floating diffusion layer. In the floating diffusion layer, the charges transferred from each photoelectric conversion unit 111 are added up in the adder unit 120. In this case, the adder unit 120 corresponds to a floating diffusion layer common to the pixels 110G1 to 110G4. In the floating diffusion layer, the charges transferred and added up from each photoelectric conversion unit 111 are converted into a voltage according to the amount of charge, and output to the vertical signal line 17 as a total pixel signal of each of the pixels 110G1 to 110G4.

[0097] The total pixel signal of the pixels 110G1 to 110G4 is supplied to, for example, the image processing unit 5. The image processing unit 5 performs a predetermined image processing on the supplied pixel signal, and stores it in the memory 6 as, for example, image data in units of one frame. For example, the control unit 3 causes the image data that has been image-processed by the image processing unit 5 and stored in the memory 6 to be stored in the storage unit 7, and also causes the display unit 8 to display the image data. The control unit 3 can also transmit the image data to the outside via the I / F unit 9.

[0098] A brief description will be given of examples of using the first and second readout methods. When the electronic device 1 shown in FIG. 1 is a digital still camera, the shutter button is generally pressed halfway to instruct an autofocus operation, and the shutter button is pressed fully to instruct exposure. In response to the halfway press of the shutter button as the input device 10, the control unit 3 executes the first readout method described above and executes autofocus based on the phase difference. In response to the subsequent full press of the shutter button, the control unit 3 executes the second readout method described above and acquires a pixel signal based on the sum of the charges of the four pixels included in the pixel block.

[0099] By controlling in this way, for example, the electronic device 1 as a digital still camera can execute autofocus control based on the phase difference in response to a series of operations of the shutter button, and can acquire image data based on pixel signals in units of a plurality of pixels 110. At this time, autofocus based on the phase difference can be executed using all the pixels 110 included in the pixel array unit 11, so that more accurate autofocus control is possible. In addition, the acquired image data is configured based on pixel signals obtained by collectively processing the four pixels 110 included in the pixel block, so that a brighter screen can be realized.

[0100] Not limited to this, the electronic device 1 according to the first embodiment can individually read out pixel signals from the pixels 110 included in the pixel block by the first readout method. Therefore, the electronic device 1 according to the first embodiment can easily be equipped with an application for generating a 3D (Dimension) image or an application for realizing the function of a light field camera.

[0101] (How to deal with color mixing) Next, a method for dealing with color mixing between pixels that can be applied to the first embodiment will be described. In the first embodiment, the OCL 30 for a pixel set may be applied to a pixel set adjacent to the pixel set, in which each pixel 110 is provided with a color filter 112 of a different color from the color filter 112 provided to each pixel 110 included in the pixel set. In this case, depending on the chief ray incident angle (CRA (Chief Ray Angle)) of the main lens, color mixing may occur between adjacent pixel sets.

[0102] Fig. 17 is a cross-sectional view showing a first example of a configuration of a pixel 110 for suppressing color mixing between pixel sets (between pixels 110) that can be applied to the first embodiment. For convenience, Fig. 17 shows a configuration in which one OCL 31 is provided for one photoelectric conversion unit 111.

[0103] 17, the upper side of the silicon substrate 1000 is the rear side of the substrate 1000, and the lower side is the front side. That is, after the photoelectric conversion section 111 and each wiring are formed on the front side of the substrate 1000, the substrate 1000 is turned over, and a planarization film 1011 is formed on the rear side. A light shielding body 113 and an R color filter 112R are formed on the planarization film 1011. A G color filter 112G is formed to the left of the color filter 112R. An OCL31 is formed on the color filters 112R and 112G, and a protective film 1010 is formed on the OCL31.

[0104] 17, the light shield 113 and the color filter 112R are shifted to the right with respect to the photoelectric conversion unit 111, and the OCL 31 is shifted further to the right. This allows pupil correction to be performed on the light incident from the upper right direction toward the lower left direction.

[0105] According to this configuration, the color filter 112R and the light shield 113 are formed directly on the photoelectric conversion unit 111. Therefore, it is possible to suppress leakage of light incident on the photoelectric conversion unit 111 via the OCL 31 and the color filter 112R from the upper right direction to the lower left direction in FIG. 17 into the photoelectric conversion unit 111 in which the adjacent color filter 112G on the left is provided.

[0106] Fig. 18 is a cross-sectional view showing a second example of a configuration of a pixel 110 for suppressing color mixing between pixel sets (between pixels 110) applicable to the first embodiment. As with Fig. 17, Fig. 18 shows a configuration in which one OCL 31 is provided for one photoelectric conversion unit 111 for the sake of convenience. The front and back relationship of the silicon substrate 1000 is also the same as the example in Fig. 17.

[0107] In Fig. 18, the upper side of a silicon substrate 1000 is the rear surface of the substrate 1000, and the lower side is the front surface. In the example of Fig. 18, the substrate 1000 is turned over after the photoelectric conversion section 111 and each wiring are formed on the front surface side in the same manner as described above. A light shield 1020 made of a trench is formed on the rear surface side, and further a planarization film 1011 is formed. The configuration on the planarization film 1011 is the same as in the example of Fig. 17, so a description thereof will be omitted here.

[0108] 18, for example, through color filter 112R, is reflected (indicated by arrow D) by the surface of light shielding body 1020. Therefore, leakage of the light incident on photoelectric conversion unit 111 into photoelectric conversion unit 111 in which adjacent color filter 112G is provided is suppressed.

[0109] 19 and 20 are diagrams showing examples of the arrangement of the light blocking body 1020 illustrated in FIG. 18, which can be applied to the first embodiment.

[0110] Fig. 19 is a diagram showing an example of a light shield 1020a arranged on the boundary of each pixel block, which can be applied to the first embodiment. The light shield 1020a in Fig. 19 suppresses leakage of light into a pixel block in which color filters 112 of different colors are provided for each pixel 110, and is more effective than, for example, the second readout method described with reference to Fig. 16. For example, by using the light shield 1020a shown in Fig. 19, the image quality of an image based on image data based on pixel signals for each pixel block can be improved.

[0111] Fig. 20 is a diagram showing an example of a light shield 1020b arranged at the boundary of each pixel set, which can be applied to the first embodiment. The light shield 1020b in Fig. 20 suppresses light leakage between each pixel set for detecting the phase difference, and is more effective than the first readout method described with reference to Fig. 15, for example. For example, by using the light shield 1020b shown in Fig. 20, the phase difference can be detected with higher accuracy. In addition, the light shield 1020b shown in Fig. 20 can also obtain the same effect as the light shield 1020a shown in Fig. 19 described above.

[0112] [Modification of the first embodiment] Next, a modified example of the first embodiment will be described. In the modified example of the first embodiment, a pixel set in a pixel block is composed of pixels 110 adjacent in a direction different from that of the pixel set in the first embodiment described above. More specifically, taking FIG. 9 as an example, the pixel set in the first embodiment described above is composed of two pixels 110 adjacent in the horizontal direction. In contrast, in the modified example of the first embodiment, a pixel set is composed of two pixels 110 adjacent in the vertical direction.

[0113] (Specific example of pupil correction according to the modified example of the first embodiment) 21A to 21G, a pixel set according to a modified example of the first embodiment will be described, along with examples of the direction of pupil correction in areas C, L, R, CT, CB, LT, and RB shown in FIG. 13. In FIG. 21A to 21G, "None" at the bottom indicates that no pupil correction is performed in the corresponding column. Also, "Strong" at the bottom indicates that strong pupil correction is performed in the corresponding column, and "Weak" indicates that weak pupil correction is performed in the corresponding column.

[0114] Fig. 21A is a diagram showing an example of pupil correction in region C according to the modification of the first embodiment. Here, as shown in Fig. 21A, in the modification of the first embodiment, in each pixel block consisting of 2 pixels x 2 pixels, a pixel set is formed by two vertically adjacent pixels 110G, two vertically adjacent pixels 110R, and two vertically adjacent pixels 110B. One OCL 30 is provided for each pixel set.

[0115] As shown in FIG. 21A, in region C, as shown at the bottom of each column in FIG. 21A, pupil correction is not performed on each pixel set in each pixel block.

[0116] Fig. 21B is a diagram showing an example of pupil correction in region CT according to a modified example of embodiment 1. In Fig. 21B, in region CT, strong pupil correction is performed on the pixel set on the right side of each pixel block, and weak pupil correction is performed on the pixel set on the left side, toward the lower side (image height center) in Fig. 21B.

[0117] Fig. 21C is a diagram showing an example of pupil correction in region CB according to a modified example of embodiment 1. In region CB, strong pupil correction is performed on the pixel set on the right side of each pixel block, and weak pupil correction is performed on the pixel set on the left side, moving upward in Fig. 21C (toward the center of image height).

[0118] Fig. 21D is a diagram showing an example of pupil correction in region L according to a modified example of the first embodiment. In region L, pupil correction is performed with the same amount of pupil correction on pixel sets on the right and left sides of each pixel block toward the right side (center of image height) in Fig. 21D.

[0119] Fig. 21E is a diagram showing an example of pupil correction in region R according to a modified example of the first embodiment. In region R, pupil correction is performed with the same amount of pupil correction on pixel sets on the right and left sides of each pixel block toward the left side (center of image height) in Fig. 21E.

[0120] Fig. 21F is a diagram showing an example of pupil correction in region LT according to a modified example of the first embodiment. In region LT, strong pupil correction is performed on the pixel set on the right side of each pixel block toward the diagonally lower right side (center of image height) in Fig. 21F, and weak pupil correction is performed on the pixel set on the left side. In the example of Fig. 21F, pupil correction is performed in region LT in a direction that combines the direction of pupil correction in region CT in Fig. 21B (lower side) and the direction of pupil correction in region L in Fig. 21D (right side).

[0121] Fig. 21G is a diagram showing an example of pupil correction in region RB according to a modified example of the first embodiment. In region RB, strong pupil correction is performed on the pixel set on the right side of each pixel block toward the upper left side (center of image height) in Fig. 21G, and weak pupil correction is performed on the pixel set on the left side. In the example of Fig. 21G, pupil correction is performed in region RB in a direction that combines the direction of pupil correction in region CB in Fig. 21C (upper side) and the direction of pupil correction in region R in Fig. 21E (left side).

[0122] According to this modification of the first embodiment, image plane phase difference AF and parallax detection can be performed with higher accuracy based on phase difference information in the vertical direction.

[0123] [Second embodiment] Next, a second embodiment will be described. In the above-mentioned first embodiment and its modified examples, a pixel block is composed of four pixels 110, 2 pixels x 2 pixels. In contrast, in the second embodiment, n is an integer equal to or greater than 3, and a pixel block is composed of (n x n) pixels 110, ie, n pixels x n pixels. Also, a pixel set in a pixel block is composed of two pixels 110 adjacent in the horizontal direction, similar to the pixel set in the above-mentioned first embodiment.

[0124] In the second embodiment, pupil correction with three or more different pupil correction amounts can be realized in one pixel block, which makes it possible to perform image plane phase difference AF and parallax detection with high accuracy over a wider range of EPD by the main lens compared to the above-described first embodiment and its modified examples.

[0125] Fig. 22 is a diagram showing an example of a pixel configuration applicable to the second embodiment. In the pixel configuration of Fig. 22, n=4 as described above, a pixel block is a unit in which 4 pixels x 4 pixels of the same color are arranged in a grid pattern, and each pixel block is composed of 16 R pixels 110R, 16 G pixels 110G, and 16 B pixels 110B, and is arranged in a pixel array conforming to the Bayer array. Hereinafter, unless otherwise specified, such a pixel array is called a 4-division Bayer type RGB array (n=4).

[0126] More specifically, in the four-division Bayer RGB array (n=4), similarly to the above-mentioned four-division Bayer RGB array, pixel blocks each made up of R pixels 110R, G pixels 110G, and B pixels 110B are arranged in a 4×4 grid such that the ratio of the numbers of pixels 110R, pixels 110G, and pixels 110B is 1:2:1, and pixel blocks made up of pixels of the same color are not adjacent to each other. In the example of Fig. 22, pixel blocks made up of G pixels 110G are arranged to the left and below the pixel blocks made up of B pixels 110B, and pixel blocks made up of R pixels 110R are arranged diagonally to the pixel blocks made up of B pixels 110B.

[0127] (Specific example of pupil correction according to the second embodiment) 23A to 23G, a pixel set according to the second embodiment will be described, along with examples of the direction of pupil correction in areas C, L, R, CT, CB, LT, and RB shown in FIG.

[0128] 23A to 23G, "None" on the right indicates that no pupil correction is performed in the corresponding row. "Strong" on the right indicates that strong pupil correction is performed in the corresponding row, and "Strong Strong" indicates that even stronger pupil correction (called "stronger pupil correction") is performed in the corresponding row compared to "Strong." Furthermore, "Weak" on the right indicates that weak pupil correction is performed in the corresponding row, and "Weak Weak" indicates that even weaker pupil correction (called "weaker pupil correction") is performed in the corresponding row compared to "Weak."

[0129] Not limited to this, "weak weak" may be negative pupil correction, that is, pupil correction in the opposite direction to "strong strong", "strong" and "weak". For example, when the diameter of the main lens is larger than the width of the light receiving surface (pixel array section 11), it may happen that light from the edge of the main lens enters the pixel array section 11 from the opposite direction to the light in the central part of the main lens. In this way, negative pupil correction is performed on light that enters the pixel array section 11 from the opposite direction to the light in the central part of the main lens.

[0130] 23A is a diagram showing an example of pupil correction in region C according to the second embodiment. As shown in FIG. 23A, in the second embodiment, in each pixel block consisting of 4 pixels by 4 pixels, a pixel set is formed by two horizontally adjacent pixels 110G, two horizontally adjacent pixels 110R, and two horizontally adjacent pixels 110B. That is, in the second embodiment, one pixel block includes eight pixel sets. One OCL 30 is provided for each pixel set.

[0131] As shown in FIG. 23A, in region C, as shown at the right end of each column in FIG. 23A, pupil correction is not performed on each pixel set in each pixel block.

[0132] Fig. 23B is a diagram showing an example of pupil correction in region L according to the second embodiment. In region L, from the right side in Fig. 23B (center of image height), stronger pupil correction is performed in the top two pixel sets of each pixel block, stronger pupil correction is performed in the second two pixel sets from the top, weak pupil correction is performed in the third pixel set from the top, and weaker pupil correction is performed in the bottom pixel set.

[0133] Fig. 23C is a diagram showing an example of pupil correction in region R according to the second embodiment. In region R, from the left side in Fig. 23C (center of image height), stronger pupil correction is performed in the top two pixel sets of each pixel block, stronger pupil correction is performed in the second two pixel sets from the top, weaker pupil correction is performed in the third pixel set from the top, and weaker pupil correction is performed in the bottom pixel set.

[0134] Fig. 23D is a diagram showing an example of pupil correction in region CT according to the second embodiment. In region CT, pupil correction is performed using the same amount of pupil correction in pixel sets in each row of pixel blocks toward the lower side (image height center) in Fig. 23D.

[0135] Fig. 23E is a diagram showing an example of pupil correction in region CB according to the second embodiment. In region CT, pupil correction is performed with the same pupil correction amount on pixel sets in each row of pixel blocks toward the upper side (image height center) in Fig. 23E.

[0136] Fig. 23F is a diagram showing an example of pupil correction in region LT according to the second embodiment. In region LT, stronger pupil correction is performed in the top two pixel sets of each pixel block toward the diagonally lower right side (center of image height) in Fig. 23F, stronger pupil correction is performed in the second two pixel sets from the top, weaker pupil correction is performed in the third pixel set from the top, and weaker pupil correction is performed in the bottom pixel set. In the example of Fig. 23F, pupil correction is performed in region LT in a direction that combines the direction of pupil correction in region L in Fig. 23B (right side) and the direction of pupil correction in region CT in Fig. 23D (downward).

[0137] Fig. 23G is a diagram showing an example of pupil correction in region RB according to the second embodiment. In region RB, stronger pupil correction is performed in the top two pixel sets of each pixel block toward the upper left side (center of image height) in Fig. 23G, stronger pupil correction is performed in the second two pixel sets from the top, weaker pupil correction is performed in the third pixel set from the top, and weaker pupil correction is performed in the bottom pixel set. In the example of Fig. 23G, pupil correction is performed in region RB in a direction that combines the direction of pupil correction (left side) in region R in Fig. 23C and the direction of pupil correction (upper side) in region CB in Fig. 23E.

[0138] In addition to the above-mentioned pupil correction according to each direction with respect to the image height center, the amount of pupil correction can be changed according to the image height.

[0139] Also, in the second embodiment, similar to the modified example of the first embodiment described above, each pixel set can be composed of two vertically adjacent pixels 110 and one OCL 30 that is provided in common to these two pixels 110.

[0140] [Modification of the second embodiment] Next, a modified example of the second embodiment will be described. In the modified example of the second embodiment, in a pixel block configured of (n×n) pixels 110 of n×n pixels, where n is an integer equal to or greater than 3, according to the above-mentioned second embodiment, a plurality of pixel sets in which adjacent pixels 110 are arranged in different directions are mixed. For example, in one pixel block, a pixel set of two pixels 110 adjacent in the horizontal direction (called a horizontal pixel set) and a pixel set of two pixels 110 adjacent in the vertical direction (called a vertical pixel set) are mixed.

[0141] In the modified example of the second embodiment, a plurality of pixel sets in which adjacent pixels 110 are arranged in different directions are mixed in one pixel block, so that it is possible to detect the phase difference in each of the different directions. More specifically, by mixing a horizontal pixel set and a vertical pixel set in one pixel block, it is possible to detect the phase difference in each of the horizontal and vertical directions. This makes it possible to perform image plane phase difference AF and parallax detection with higher accuracy.

[0142] (Specific example of pupil correction according to the modified example of the second embodiment) 24A to 24G, a pixel set according to a modified example of the second embodiment will be described, along with examples of the directions of pupil correction in areas C, L, R, CT, CB, LT, and RB shown in FIG. 13.

[0143] In addition, in Figures 24A to 24G, "None" on the right side indicates that no pupil correction is performed in the corresponding row. "Strong" on the right side indicates that strong pupil correction is performed in the corresponding row, and "Weak" on the right side indicates that weak pupil correction is performed in the corresponding row. Also, "Target CRA" on the right side indicates that optimal pupil correction is performed for the EPD of the main lens. For example, it is considered that "Target CRA" performs pupil correction of an intermediate strength between strong pupil correction and weak pupil correction.

[0144] FIG. 24A is a diagram showing an example of pupil correction in region C according to a modification of the second embodiment. Here, as shown in FIG. 24A, in the modification of the second embodiment, n=4, and four horizontal pixel sets and four vertical pixel sets are mixed in each pixel block consisting of 4 pixels×4 pixels. In this case, in each pixel block, two horizontal pixel sets are arranged adjacent to each other in the horizontal direction in the upper row, four vertical pixel sets are arranged adjacent to each other in the horizontal direction in the middle row, and two horizontal pixel sets are arranged adjacent to each other in the horizontal direction in the lower row. One OCL 30 is provided for each pixel set.

[0145] As shown in Fig. 24A, in region C, as shown at the right end of each column in Fig. 24A, pupil correction is not performed on each horizontal pixel set in the upper and lower rows of each pixel block. Each vertical pixel set in the middle row performs optimal pupil correction for the EPD of the main lens. Since region C is located at the center of the image height, pupil correction is not actually performed on each vertical pixel set in the middle row shown in Fig. 24A.

[0146] Fig. 24B is a diagram showing an example of pupil correction in region L according to a modified example of the second embodiment. In region L, strong pupil correction is performed in the top two horizontal pixel sets of each pixel block toward the right side (image height center) in Fig. 24B, and weak pupil correction is performed in the bottom two pixel sets. Furthermore, each vertical pixel set in the middle row performs optimal pupil correction for EPD by the main lens depending on the image height of each vertical pixel set and its direction relative to the image height center.

[0147] Fig. 24C is a diagram showing an example of pupil correction in region R according to a modified example of the second embodiment. In region R, strong pupil correction is performed in the top two pixel sets of each pixel block toward the left side (image height center) in Fig. 24C, and weak pupil correction is performed in the bottom two pixel sets. Also, each vertical pixel set in the middle row performs optimal pupil correction for EPD by the main lens depending on the image height of each vertical pixel set and its direction relative to the image height center.

[0148] Fig. 24D is a diagram showing an example of pupil correction in region CT according to a modified example of embodiment 2. In region CT, pupil correction is performed with the same amount of pupil correction on each horizontal pixel set in the upper and lower rows of each pixel block and each vertical pixel set in the middle row toward the lower side (image height center) in Fig. 24D.

[0149] Fig. 24E is a diagram showing an example of pupil correction in area CB according to a modified example of embodiment 2. In area CT, pupil correction is performed with the same amount of pupil correction on each horizontal pixel set in the upper and lower rows of each pixel block and each vertical pixel set in the middle row downward in Fig. 24E (toward the center of image height).

[0150] Fig. 24F is a diagram showing an example of pupil correction in the region LT according to a modified example of the second embodiment. In the region LT, strong pupil correction is performed in the two horizontal pixel sets in the upper row of each pixel block toward the diagonally lower right side (image height center) in Fig. 24F, and weak pupil correction is performed in the two pixel sets in the lower row. In addition, each vertical pixel set in the middle row performs optimal pupil correction for EPD by the main lens according to the image height of each vertical pixel set and its direction relative to the image height center. In the example of Fig. 24F, pupil correction is performed in the region LT in a direction that combines the direction of pupil correction in the region L in Fig. 24B (right side) and the direction of pupil correction in the region CT in Fig. 24D (downward).

[0151] Fig. 24G is a diagram showing an example of pupil correction in region RB according to a modified example of the second embodiment. In region RB, strong pupil correction is performed in the two horizontal pixel sets in the upper row of each pixel block toward the upper left side (image height center) in Fig. 24G, and weak pupil correction is performed in the two pixel sets in the lower row. In addition, each vertical pixel set in the middle row performs optimal pupil correction for EPD by the main lens according to the image height of each vertical pixel set and its direction relative to the image height center. In the example of Fig. 24G, pupil correction is performed in region RB in a direction that combines the direction of pupil correction in region R in Fig. 24C (left side) and the direction of pupil correction in region CB in Fig. 24E (upper side).

[0152] In addition to the above-mentioned pupil correction according to each direction with respect to the image height center, the amount of pupil correction can be changed according to the image height.

[0153] [Third embodiment] Next, as a third embodiment, an application example of the image sensor 4 according to the first embodiment and its modified example, and the second embodiment and its modified example according to the present disclosure will be described. Fig. 25 is a diagram showing a usage example of the image sensor 4 according to the above-mentioned first embodiment and its modified example, and the second embodiment and its modified example.

[0154] Each of the above-mentioned imaging elements 4 can be used in various cases for sensing light such as visible light, infrared light, ultraviolet light, and X-rays, for example, as described below.

[0155] -Digital cameras, mobile devices with camera functions, and other devices that take images for viewing. Devices used for traffic purposes, such as on-board sensors that photograph the front, rear, surroundings, and interior of a vehicle for safe driving such as automatic stopping, and for recognition of the driver's condition, surveillance cameras that monitor moving vehicles and roads, and distance measuring sensors that measure distances between vehicles. A device used in home appliances such as TVs, refrigerators, and air conditioners to capture user gestures and operate the appliance in accordance with those gestures. - Equipment used for medical or healthcare purposes, such as endoscopes and devices that take blood vessel images by receiving infrared light. - Equipment used for security purposes, such as surveillance cameras for crime prevention and cameras for person authentication. - Equipment used for beauty purposes, such as skin measuring devices that take pictures of the skin and microscopes that take pictures of the scalp. Equipment used for sports, such as action cameras and wearable cameras for sports purposes. Agricultural equipment, such as cameras for monitoring fields and crop conditions.

[0156] [Further application examples of the technology disclosed herein] The technology according to the present disclosure (the technology) can be applied to various products. For example, the technology according to the present disclosure may be applied to an endoscopic surgery system.

[0157] (Example of application to an in-body information acquisition system) FIG. 26 is a block diagram showing an example of a schematic configuration of a patient's in-vivo information acquisition system using a capsule endoscope to which the technology according to the present disclosure (the present technology) can be applied.

[0158] The in-vivo information acquisition system 10001 includes a capsule endoscope 10100 and an external control device 10200 .

[0159] The capsule endoscope 10100 is swallowed by a patient during an examination. The capsule endoscope 10100 has an imaging function and a wireless communication function, and while moving inside an organ such as the stomach or intestines by peristalsis or the like until it is naturally discharged from the patient, sequentially captures images of the inside of the organ (hereinafter also referred to as in-vivo images) at predetermined intervals, and sequentially wirelessly transmits information about the in-vivo images to an external control device 10200 outside the body.

[0160] The external control device 10200 comprehensively controls the operation of the in-vivo information acquisition system 10001. In addition, the external control device 10200 receives information about the in-vivo image transmitted from the capsule endoscope 10100, and generates image data for displaying the in-vivo image on a display device (not shown) based on the received information about the in-vivo image.

[0161] In this manner, the in-vivo information acquiring system 10001 can obtain in-vivo images capturing images of the state inside the patient's body at any time from when the capsule endoscope 10100 is swallowed to when it is expelled.

[0162] The configurations and functions of the capsule endoscope 10100 and the external control device 10200 will be described in more detail.

[0163] The capsule endoscope 10100 has a capsule-shaped housing 10101, which houses a light source unit 10111, an imaging unit 10112, an image processing unit 10113, a wireless communication unit 10114, a power supply unit 10115, a power supply unit 10116, and a control unit 10117.

[0164] The light source unit 10111 is composed of a light source such as an LED (Light Emitting Diode), and irradiates the imaging field of the imaging unit 10112 with light.

[0165] The imaging unit 10112 is composed of an imaging element and an optical system consisting of multiple lenses provided in front of the imaging element. Reflected light (hereinafter referred to as observation light) of light irradiated onto the body tissue to be observed is collected by the optical system and enters the imaging element. In the imaging unit 10112, the imaging element photoelectrically converts the observation light incident thereon, and generates an image signal corresponding to the observation light. The image signal generated by the imaging unit 10112 is provided to an image processing unit 10113.

[0166] The image processing unit 10113 is configured with a processor such as a CPU or a GPU (Graphics Processing Unit), and performs various signal processing on the image signal generated by the imaging unit 10112. The image processing unit 10113 provides the image signal after the signal processing to the wireless communication unit 10114 as RAW data.

[0167] The wireless communication unit 10114 performs predetermined processing such as modulation processing on the image signal that has been subjected to signal processing by the image processing unit 10113, and transmits the image signal to the external control device 10200 via the antenna 10114A. The wireless communication unit 10114 also receives a control signal related to drive control of the capsule endoscope 10100 from the external control device 10200 via the antenna 10114A. The wireless communication unit 10114 provides the control signal received from the external control device 10200 to the control unit 10117.

[0168] The power supply unit 10115 is composed of an antenna coil for receiving power, a power regeneration circuit for regenerating power from the current generated in the antenna coil, a boost circuit, etc. In the power supply unit 10115, power is generated using the principle of so-called non-contact charging.

[0169] The power supply unit 10116 is composed of a secondary battery, and stores the power generated by the power supply unit 10115. In order to avoid cluttering the drawing, arrows and other symbols indicating the destination of the power supply from the power supply unit 10116 are omitted in Fig. 26, but the power stored in the power supply unit 10116 is supplied to the light source unit 10111, the imaging unit 10112, the image processing unit 10113, the wireless communication unit 10114, and the control unit 10117 and can be used to drive these units.

[0170] The control unit 10117 is composed of a processor such as a CPU, and appropriately controls the driving of the light source unit 10111, the imaging unit 10112, the image processing unit 10113, the wireless communication unit 10114, and the power supply unit 10115 in accordance with control signals transmitted from the external control device 10200.

[0171] The external control device 10200 is composed of a processor such as a CPU or a GPU, or a microcomputer or a control board in which a processor and a storage element such as a memory are mixed. The external control device 10200 controls the operation of the capsule endoscope 10100 by transmitting a control signal to the control unit 10117 of the capsule endoscope 10100 via the antenna 10200A. In the capsule endoscope 10100, for example, the light irradiation conditions for the observation target in the light source unit 10111 can be changed by the control signal from the external control device 10200. Furthermore, the imaging conditions (for example, the frame rate and exposure value in the imaging unit 10112) can be changed by the control signal from the external control device 10200. Furthermore, the content of the processing in the image processing unit 10113 and the conditions for the wireless communication unit 10114 to transmit an image signal (for example, the transmission interval, the number of transmitted images, etc.) may be changed by the control signal from the external control device 10200.

[0172] The external control device 10200 also applies various image processing to the image signal transmitted from the capsule endoscope 10100 to generate image data for displaying the captured in-vivo image on a display device. As the image processing, for example, various signal processing such as development processing (demosaic processing), high image quality processing (band enhancement processing, super-resolution processing, noise reduction processing, camera shake correction processing, etc.), and enlargement processing (electronic zoom processing) can be performed alone or in combination. The external control device 10200 controls the driving of the display device to display the captured in-vivo image based on the generated image data. Alternatively, the external control device 10200 may record the generated image data in a recording device (not shown) or print out the image data on a printing device (not shown).

[0173] An example of an in-vivo information acquisition system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to, for example, the imaging unit 10112 among the configurations described above. By applying the imaging element 4 according to the present disclosure to the imaging unit 10112, it becomes possible to perform good autofocus even when zooming, etc., and it becomes possible to acquire higher quality in-vivo images, etc.

[0174] (Application example to endoscopic surgery system) The technology according to the present disclosure may be further applied to an endoscopic surgery system. Fig. 27 is a diagram showing an example of a schematic configuration of an endoscopic surgery system to which the technology according to the present disclosure (the present technology) can be applied.

[0175] 27 shows a state in which an operator (doctor) 11131 is performing surgery on a patient 11132 on a patient bed 11133 using an endoscopic surgery system 11000. As shown in the figure, the endoscopic surgery system 11000 is composed of an endoscope 11100, other surgical tools 11110 such as an insufflation tube 11111 and an energy treatment tool 11112, a support arm device 11120 that supports the endoscope 11100, and a cart 11200 on which various devices for endoscopic surgery are mounted.

[0176] The endoscope 11100 is composed of a lens barrel 11101, a region of a predetermined length from the tip of which is inserted into a body cavity of a patient 11132, and a camera head 11102 connected to the base end of the lens barrel 11101. In the illustrated example, the endoscope 11100 is configured as a so-called rigid scope having a rigid lens barrel 11101, but the endoscope 11100 may be configured as a so-called flexible scope having a flexible lens barrel.

[0177] An opening into which an objective lens is fitted is provided at the tip of the lens barrel 11101. A light source device 11203 is connected to the endoscope 11100, and light generated by the light source device 11203 is guided to the tip of the lens barrel by a light guide extending inside the lens barrel 11101, and is irradiated via the objective lens toward an observation target in the body cavity of the patient 11132. The endoscope 11100 may be a direct-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0178] An optical system and an image sensor are provided inside the camera head 11102, and reflected light (observation light) from an observation target is collected on the image sensor by the optical system. The observation light is photoelectrically converted by the image sensor to generate an electrical signal corresponding to the observation light, i.e., an image signal corresponding to an observation image. The image signal is transmitted to a camera control unit (CCU) 11201 as RAW data.

[0179] The CCU 11201 is configured with a CPU, a GPU, etc., and performs overall control of the operations of the endoscope 11100 and the display device 11202. Furthermore, the CCU 11201 receives an image signal from the camera head 11102, and performs various types of image processing on the image signal, such as development processing (demosaic processing), for displaying an image based on the image signal.

[0180] Under the control of the CCU 11201, the display device 11202 displays an image based on an image signal that has been subjected to image processing by the CCU 11201.

[0181] The light source device 11203 is composed of a light source such as an LED (Light Emitting Diode), and supplies the endoscope 11100 with irradiation light when photographing an operation site or the like.

[0182] The input device 11204 is an input interface for the endoscopic surgery system 11000. A user can input various information and instructions to the endoscopic surgery system 11000 via the input device 11204. For example, the user inputs an instruction to change the imaging conditions (type of irradiated light, magnification, focal length, etc.) of the endoscope 11100.

[0183] The treatment tool control device 11205 controls the driving of the energy treatment tool 11112 for cauterizing tissue, incising, sealing blood vessels, etc. The insufflation device 11206 sends gas into the body cavity of the patient 11132 via the insufflation tube 11111 to inflate the body cavity for the purpose of securing the field of view of the endoscope 11100 and securing the working space of the surgeon. The recorder 11207 is a device capable of recording various information related to the surgery. The printer 11208 is a device capable of printing various information related to the surgery in various formats such as text, image, or graph.

[0184] The light source device 11203 that supplies irradiation light to the endoscope 11100 when photographing the surgical site can be composed of a white light source composed of, for example, an LED, a laser light source, or a combination of these. When the white light source is composed of a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, so that the white balance of the captured image can be adjusted in the light source device 11203. In this case, it is also possible to capture images corresponding to each of the RGB colors in a time-division manner by irradiating the observation object with laser light from each of the RGB laser light sources in a time-division manner and controlling the driving of the image sensor of the camera head 11102 in synchronization with the irradiation timing. According to this method, a color image can be obtained without providing a color filter to the image sensor.

[0185] The light source device 11203 may be controlled to change the intensity of the light it outputs at predetermined time intervals. The driving of the image sensor of the camera head 11102 may be controlled in synchronization with the timing of the change in the light intensity to obtain images in a time-division manner, and the images may be synthesized to generate an image with a high dynamic range that is free of so-called blackout and whiteout.

[0186] The light source device 11203 may be configured to supply light of a predetermined wavelength band corresponding to the special light observation. In the special light observation, for example, by utilizing the wavelength dependency of light absorption in body tissue, a narrow band light is irradiated compared to the irradiated light (i.e., white light) during normal observation, and a predetermined tissue such as blood vessels on the mucous membrane surface is photographed with high contrast, so-called narrow band imaging is performed. Alternatively, in the special light observation, a fluorescent observation may be performed in which an image is obtained by fluorescence generated by irradiating an excitation light. In the fluorescent observation, it is possible to irradiate an excitation light to a body tissue and observe the fluorescence from the body tissue (autofluorescence observation), or to locally inject a reagent such as indocyanine green (ICG) into the body tissue and irradiate the body tissue with excitation light corresponding to the fluorescent wavelength of the reagent to obtain a fluorescent image. The light source device 11203 may be configured to supply narrow band light and / or excitation light corresponding to such special light observation.

[0187] FIG. 28 is a block diagram showing an example of the functional configuration of the camera head 11102 and the CCU 11201 shown in FIG.

[0188] The camera head 11102 has a lens unit 11401, an imaging unit 11402, a drive unit 11403, a communication unit 11404, and a camera head control unit 11405. The CCU 11201 has a communication unit 11411, an image processing unit 11412, and a control unit 11413. The camera head 11102 and the CCU 11201 are connected to each other by a transmission cable 11400 so as to be able to communicate with each other.

[0189] The lens unit 11401 is an optical system provided at the connection portion with the lens barrel 11101. Observation light taken in from the tip of the lens barrel 11101 is guided to the camera head 11102 and enters the lens unit 11401. The lens unit 11401 is configured by combining a plurality of lenses including a zoom lens and a focus lens.

[0190] The imaging unit 11402 is composed of an imaging element. The imaging element constituting the imaging unit 11402 may be one (so-called single-plate type) or multiple (so-called multi-plate type). When the imaging unit 11402 is composed of a multi-plate type, for example, each imaging element may generate an image signal corresponding to each of RGB, and a color image may be obtained by combining the image signals. Alternatively, the imaging unit 11402 may be configured to have a pair of imaging elements for acquiring image signals for the right eye and the left eye corresponding to 3D (Dimensional) display. By performing 3D display, the surgeon 11131 can more accurately grasp the depth of the biological tissue in the surgical site. Note that when the imaging unit 11402 is composed of a multi-plate type, multiple lens units 11401 may be provided corresponding to each imaging element.

[0191] Furthermore, the imaging unit 11402 does not necessarily have to be provided in the camera head 11102. For example, the imaging unit 11402 may be provided inside the lens barrel 11101, immediately behind the objective lens.

[0192] The driving unit 11403 is configured by an actuator, and moves the zoom lens and focus lens of the lens unit 11401 by a predetermined distance along the optical axis under the control of the camera head control unit 11405. This allows the magnification and focus of the image captured by the imaging unit 11402 to be appropriately adjusted.

[0193] The communication unit 11404 is configured by a communication device for transmitting and receiving various information to and from the CCU 11201. The communication unit 11404 transmits the image signal obtained from the imaging unit 11402 to the CCU 11201 via the transmission cable 11400 as RAW data.

[0194] Furthermore, the communication unit 11404 receives a control signal for controlling the driving of the camera head 11102 from the CCU 11201, and supplies the control signal to the camera head control unit 11405. The control signal includes information on the imaging conditions, such as information specifying the frame rate of the captured image, information specifying the exposure value at the time of capturing the image, and / or information specifying the magnification and focus of the captured image.

[0195] The image capturing conditions such as the frame rate, exposure value, magnification, and focus may be appropriately specified by a user, or may be automatically set by the control unit 11413 of the CCU 11201 based on an acquired image signal. In the latter case, the endoscope 11100 is equipped with a so-called AE (Auto Exposure) function, AF (Auto Focus) function, and AWB (Auto White Balance) function.

[0196] The camera head control unit 11405 controls the driving of the camera head 11102 based on a control signal received from the CCU 11201 via the communication unit 11404.

[0197] The communication unit 11411 is configured with a communication device for transmitting and receiving various information to and from the camera head 11102. The communication unit 11411 receives an image signal transmitted from the camera head 11102 via the transmission cable 11400.

[0198] Furthermore, the communication unit 11411 transmits, to the camera head 11102, a control signal for controlling the driving of the camera head 11102. The image signal and the control signal can be transmitted by electrical communication, optical communication, or the like.

[0199] The image processing unit 11412 performs various types of image processing on the image signal, which is RAW data sent from the camera head 11102 .

[0200] The control unit 11413 performs various controls related to imaging of the surgical site etc. by the endoscope 11100 and display of the captured image obtained by imaging the surgical site etc. For example, the control unit 11413 generates a control signal for controlling the driving of the camera head 11102.

[0201] Further, the control unit 11413 causes the display device 11202 to display the captured image showing the surgical site, etc., based on the image signal that has been image-processed by the image processing unit 11412. At this time, the control unit 11413 may recognize various objects in the captured image using various image recognition techniques. For example, the control unit 11413 can recognize surgical tools such as forceps, specific living body parts, bleeding, mist when the energy treatment tool 11112 is used, etc., by detecting the shape and color of the edge of an object included in the captured image. When the control unit 11413 causes the display device 11202 to display the captured image, it may use the recognition result to superimpose various types of surgery support information on the image of the surgical site. By superimposing and presenting the surgery support information to the surgeon 11131, the burden on the surgeon 11131 can be reduced and the surgeon 11131 can proceed with the surgery reliably.

[0202] The transmission cable 11400 connecting the camera head 11102 and the CCU 11201 is an electrical signal cable for communication of electrical signals, an optical fiber for optical communication, or a composite cable of these.

[0203] Here, in the example of FIG. 28, communication is performed wired using the transmission cable 11400, but communication between the camera head 11102 and the CCU 11201 may be performed wirelessly.

[0204] An example of an endoscopic surgery system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to, for example, the endoscope 11100 and the imaging unit 11402 of the camera head 11102 among the configurations described above. By applying the imaging element 4 according to the present disclosure to the imaging unit 11402, it becomes possible to perform good autofocus even when zooming or the like is performed, and it is possible to obtain a higher quality captured image. This reduces the burden on the surgeon 11131 and enables the surgeon 11131 to reliably proceed with the surgery.

[0205] Although an endoscopic surgery system has been described as an example here, the technology disclosed herein may also be applied to other systems, such as a microsurgery system.

[0206] (Example of application to moving objects) The technology disclosed herein may also be applied to devices mounted on various types of moving objects, such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility, airplanes, drones, ships, and robots.

[0207] FIG. 29 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a moving object control system to which the technology according to the present disclosure can be applied.

[0208] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 29, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also, as functional configurations of the integrated control unit 12050, a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (interface) 12053 are illustrated.

[0209] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle according to various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0210] The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as head lamps, back lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves or signals of various switches transmitted from a portable device that replaces a key may be input to the body system control unit 12020. The body system control unit 12020 receives the input of these radio waves or signals and controls the door lock device, power window device, lamps, and the like of the vehicle.

[0211] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture an image outside the vehicle and receives the captured image. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing of people, cars, obstacles, signs, characters on the road surface, etc. based on the received image. The outside-vehicle information detection unit 12030 performs image processing on the received image, for example, and performs object detection processing or distance detection processing based on the result of the image processing.

[0212] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal according to the amount of the received light. The imaging unit 12031 can output the electrical signal as an image, or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0213] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing.

[0214] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the outside-of-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, etc.

[0215] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on the driver's operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle acquired by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0216] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside-vehicle information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside-vehicle information detection unit 12030, and perform cooperative control for the purpose of preventing glare, such as switching from high beams to low beams.

[0217] The audio / video output unit 12052 transmits at least one of audio and image output signals to an output device capable of visually or audibly notifying the passengers of the vehicle or the outside of the vehicle of information. In the example of Fig. 29, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as the output device. The display unit 12062 may include at least one of an on-board display and a head-up display, for example.

[0218] Fig. 30 is a diagram showing an example of the installation position of the imaging unit 12031. In Fig. 30, a vehicle 12100 has imaging units 12101, 12102, 12103, 12104, and 12105 as the imaging unit 12031.

[0219] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided at positions such as the front nose, side mirrors, rear bumper, back door, and the upper part of the windshield in the vehicle interior of the vehicle 12100. The imaging unit 12101 provided at the front nose and the imaging unit 12105 provided at the upper part of the windshield in the vehicle interior mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided at the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided at the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The images of the front acquired by the imaging units 12101 and 12105 are mainly used to detect a preceding vehicle, a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.

[0220] 30 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, image data captured by the imaging units 12101 to 12104 are superimposed to obtain an overhead image of the vehicle 12100 viewed from above.

[0221] At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera made up of a plurality of imaging elements, or may be an imaging element having pixels for detecting a phase difference.

[0222] For example, the microcomputer 12051 can extract, as a preceding vehicle, a three-dimensional object that is the closest to the vehicle 12100 on the travel path of the vehicle 12100 and travels at a predetermined speed (for example, 0 km / h or more) in approximately the same direction as the vehicle 12100, by calculating the distance to each three-dimensional object in the imaging ranges 12111-12114 and the change over time of this distance (relative speed with respect to the vehicle 12100) based on the distance information obtained from the imaging units 12101-12104. Furthermore, the microcomputer 12051 can set a vehicle distance to be secured in advance in front of the preceding vehicle, and perform automatic brake control (including follow-up stop control) and automatic acceleration control (including follow-up start control). In this way, cooperative control can be performed for the purpose of automatic driving that travels autonomously without relying on the driver's operation.

[0223] For example, the microcomputer 12051 classifies and extracts three-dimensional object data on three-dimensional objects, such as two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects, based on the distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. Then, the microcomputer 12051 determines a collision risk indicating the degree of danger of collision with each obstacle, and when the collision risk is equal to or greater than a set value and there is a possibility of collision, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0224] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the captured images of the imaging units 12101 to 12104. The recognition of such a pedestrian is performed, for example, by a procedure of extracting feature points in the captured images of the imaging units 12101 to 12104 as infrared cameras, and a procedure of performing pattern matching processing on a series of feature points that indicate the contour of an object to determine whether or not the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular contour line for emphasis on the recognized pedestrian. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.

[0225] An example of a vehicle control system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to, for example, the imaging unit 12031 among the configurations described above. By applying the imaging element 4 according to the present disclosure to the imaging unit 12031, it becomes possible to perform good autofocus even when zooming or the like is performed, and it becomes possible to obtain a higher quality captured image.

[0226] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0227] The present technology can also be configured as follows. (1) A plurality of photoelectric conversion elements arranged in a lattice pattern; a plurality of lenses provided in a one-to-one correspondence with each of pairs of elements each consisting of two or more photoelectric conversion elements arranged adjacent to each other among the plurality of photoelectric conversion elements; A light receiving unit having The light receiving unit is Among a plurality of pixel sets each including the set of elements and a lens among the plurality of lenses that is provided in the set of elements, at least two of the pixel sets adjacent to each other have different pupil correction amounts. Image sensor. (2) The light receiving unit is Each pixel block includes n×n (n is an integer of 2 or more) photoelectric conversion elements that receive light of the same wavelength band among the plurality of photoelectric conversion elements and are arranged in a lattice pattern, the pixel blocks are arranged according to a pattern in which the photoelectric conversion elements to be arranged among the plurality of pixel blocks receive light in the same wavelength band, and the pixel blocks are not adjacent to each other; At least one of the plurality of pixel blocks includes two of the pixel sets having different pupil correction amounts. The imaging element according to (1) above. (3) The n is an integer equal to or greater than 3, and at least one pixel block among the plurality of pixel blocks includes three or more pixel sets each having a different pupil correction amount. The imaging element according to (2) above. (4) The n is an integer equal to or greater than 3, and at least one of the plurality of pixel blocks includes two pixel sets in which adjacent photoelectric conversion elements among the plurality of photoelectric conversion elements included in the light receiving unit are aligned in different directions on the array. The imaging element according to (2) or (3). (5) A control unit that controls reading of the plurality of photoelectric conversion elements, The control unit is In each of the plurality of pixel blocks, a readout mode for reading out each of a plurality of outputs corresponding one-to-one to each of the photoelectric conversion elements arranged therein is a mode in which the plurality of outputs are read out individually; a mode in which the plurality of outputs are synthesized within a pixel block in which each photoelectric conversion element corresponding to the plurality of outputs is disposed, and the synthesized output is read out as one output; have The imaging element according to any one of (2) to (4). (6) wherein n is 2; The imaging element according to (2) above. (7) The light receiving unit is The optical fiber further includes a filter provided on an incident surface of each of the plurality of photoelectric conversion elements to limit a wavelength band of incident light, and a first light shield provided around the filter. The imaging element according to any one of (2) to (6). (8) The positions of the filter and the first light shielding body with respect to the photoelectric conversion elements corresponding to the filter and the first light shielding body among the plurality of photoelectric conversion elements are different from each other in at least two of the pixel sets adjacent to each other, each of which includes the photoelectric conversion element, so that the pupil correction amounts are different from each other. The imaging element according to (7) above. (9) The light receiving unit is The substrate on which the plurality of photoelectric conversion elements are formed further includes a second light shield formed by a groove formed in a depth direction with respect to an incidence surface of each of the plurality of photoelectric conversion elements, The second light blocking body is between each of the plurality of pixel blocks The imaging element according to any one of (2) to (8). (10) The second light shielding body further includes: between each of the pixel sets The imaging element according to (9) above. (11) The positions of the lenses included in the pixel sets with respect to the sets of elements included in the pixel sets are different between at least two of the pixel sets adjacent to each other, so that the pupil correction amounts are different from each other. The imaging element according to any one of (1) to (10). (12) The amount of pupil correction and the direction in which pupil correction is performed based on the amount of pupil correction are further varied depending on a position of the light receiving surface including each of the entrance surfaces of the plurality of photoelectric conversion elements with respect to a center of image height corresponding to an optical axis of a main lens that irradiates light onto the light receiving surface. The imaging element according to any one of (1) to (11). (13) The light receiving unit is a mask for restricting light incident on the set of elements in the pixel set; The positions of the mask with respect to the set of elements included in the pixel set are different between at least two of the pixel sets adjacent to each other, so that the pupil correction amount is different. The imaging element according to any one of (1) to (12). (14) A plurality of photoelectric conversion elements arranged in a lattice pattern; a plurality of lenses provided in a one-to-one correspondence with each of pairs of elements each consisting of two or more photoelectric conversion elements arranged adjacent to each other among the plurality of photoelectric conversion elements; A light receiving unit having an optical unit that guides external light to the light receiving unit; an image processing unit that performs image processing based on a plurality of outputs read out from each of the plurality of photoelectric conversion elements to generate image data; a storage unit that stores the image data generated by the image processing unit; Equipped with The light receiving unit is Among a plurality of pixel sets each including the set of elements and a lens among the plurality of lenses that is provided in the set of elements, at least two of the pixel sets adjacent to each other have different pupil correction amounts. electronic equipment. [Explanation of symbols]

[0228] 1 Electronic equipment 4. Image sensor 5. Image Processing Section 11 Pixel array section 30,31 OCL 50,113,1020a,1020b Shading body 110, 110a, 110b, 110a', 110b', 110R, 110G, 110G1, 110G2, 110G3, 110G4, 110B, 110w Pixels 111,111a,111b,111a',111b',111G1,111G2,111G3,111G4,111wa,111wb Photoelectric conversion section 112,112a,112b,112G1,112G2,112G3,112G4,112R Color filters

Claims

1. A plurality of photoelectric conversion elements arranged in a lattice pattern; a plurality of lenses provided in a one-to-one correspondence with each pair of elements each including two photoelectric conversion elements arranged adjacent to each other among the plurality of photoelectric conversion elements; A light receiving unit having The light receiving unit is a plurality of pixel sets each including the set of elements and a lens among the plurality of lenses that is provided in the set of elements, the pupil correction amounts of at least two of the pixel sets adjacent to each other being different from each other; Each pixel block includes n×n (n is 2 or 4) photoelectric conversion elements that receive light of the same wavelength band among the plurality of photoelectric conversion elements and are arranged in a lattice pattern, the pixel blocks are arranged according to a pattern in which the photoelectric conversion elements to be arranged among the plurality of pixel blocks receive light in the same wavelength band, and the pixel blocks are not adjacent to each other; At least one of the plurality of pixel blocks includes two of the pixel sets having different pupil correction amounts, When the n is 4, at least one pixel block among the plurality of pixel blocks includes four pixel sets having different pupil correction amounts, or two pixel sets having different directions in which adjacent photoelectric conversion elements among the plurality of photoelectric conversion elements included in the light receiving unit are aligned on the array. Image sensor.

2. A control unit that controls reading of the plurality of photoelectric conversion elements, The control unit is A readout mode for reading out a plurality of outputs corresponding one-to-one to each of the photoelectric conversion elements arranged in each of the plurality of pixel blocks, a mode in which the plurality of outputs are read out individually; a mode in which the plurality of outputs are synthesized within a pixel block in which each photoelectric conversion element corresponding to the plurality of outputs is disposed, and the synthesized output is read out as one output; have The imaging device according to claim 1 .

3. The light receiving unit is The optical fiber further includes a filter provided on an incident surface of each of the plurality of photoelectric conversion elements to limit a wavelength band of incident light, and a first light shield provided around the filter. The imaging device according to claim 1 .

4. The positions of the filter and the first light shielding body with respect to the photoelectric conversion elements corresponding to the filter and the first light shielding body among the plurality of photoelectric conversion elements are different from each other in at least two of the pixel sets adjacent to each other, each of which includes the photoelectric conversion element, so that the pupil correction amounts are different from each other. The imaging device according to claim 3 .

5. The light receiving unit is The substrate on which the plurality of photoelectric conversion elements are formed further includes a second light shield formed of a groove in a depth direction relative to an incidence surface of each of the plurality of photoelectric conversion elements, The second light blocking body is between each of the plurality of pixel blocks The imaging device according to claim 1 .

6. The second light shielding body further includes: between each of the pixel sets The imaging device according to claim 5 .

7. The positions of the lenses included in the pixel sets with respect to the image height center of the set of elements included in the pixel sets are different between at least two of the pixel sets adjacent to each other, so that the pupil correction amounts are different from each other. The imaging device according to claim 1 .

8. The amount of pupil correction and the direction in which pupil correction is performed based on the amount of pupil correction are further varied depending on a position of the light receiving surface including each of the entrance surfaces of the plurality of photoelectric conversion elements with respect to a center of image height corresponding to an optical axis of a main lens that irradiates light onto the light receiving surface. The imaging device according to claim 1 .

9. The light receiving unit is a mask for restricting light incident on the set of elements in the pixel set; The positions of the groups of elements included in the pixel sets with respect to the center of the image height of the mask are different from each other in at least two of the pixel sets adjacent to each other, so that the pupil correction amounts are different. The imaging device according to claim 1 .

10. A plurality of photoelectric conversion elements arranged in a lattice pattern; a plurality of lenses provided in a one-to-one correspondence with each pair of elements each including two photoelectric conversion elements arranged adjacent to each other among the plurality of photoelectric conversion elements; A light receiving unit having an optical unit that guides external light to the light receiving unit; an image processing unit that performs image processing based on a plurality of outputs read out from each of the plurality of photoelectric conversion elements to generate image data; a storage unit that stores the image data generated by the image processing unit; Equipped with The light receiving unit is a plurality of pixel sets each including the set of elements and a lens among the plurality of lenses that is provided in the set of elements, the pupil correction amounts of at least two of the pixel sets adjacent to each other being different from each other; Each pixel block includes n×n (n is 2 or 4) photoelectric conversion elements that receive light of the same wavelength band among the plurality of photoelectric conversion elements and are arranged in a lattice pattern, the pixel blocks are arranged according to a pattern in which the photoelectric conversion elements to be arranged among the plurality of pixel blocks receive light in the same wavelength band, and the pixel blocks are not adjacent to each other; At least one of the plurality of pixel blocks includes two of the pixel sets having different pupil correction amounts, When the n is 4, at least one pixel block among the plurality of pixel blocks includes four pixel sets having different pupil correction amounts, or two pixel sets having different directions in which adjacent photoelectric conversion elements among the plurality of photoelectric conversion elements included in the light receiving unit are aligned on the array. electronic equipment.

Images