Imaging element and imaging device

Abstract

Provided is an imaging element and an imaging device.　This imaging element has a first pixel array arranged repeatedly in a first direction and a second direction. When M is a natural number greater than 2 and N is a natural number greater than 2, the first pixel array is formed by arranging M pixels in the first direction and N pixels in the second direction. The first pixel array includes a first pixel which is included in a second pixel array formed by arranging M-1 pixels in the first direction and N-1 pixels in the second direction, and a second pixel other than the first pixel. The first pixel and the second pixel have different functions.

Description

Image pickup element and image pickup device

[0001] The technology of the present disclosure relates to an imaging element and an imaging device.

[0002] Japanese Patent Application Laid-Open No. 2006-129999 describes an imaging element having an imaging pixel group and a phase difference detection pixel group.

[0003] Patent Document 2 describes an imaging element in which imaging pixel rows and phase difference detection pixel rows are alternately arranged, and transparent filters are provided for the phase difference pixels.

[0004] Patent Document 3 describes an imaging element having a pixel array pattern consisting of high-sensitivity pixels and low-sensitivity pixels.

[0005] Patent Document 4 describes an imaging element that includes a pixel for detecting a phase difference, a pixel including a polarizer, and a pixel including a white filter.

[0006] Japanese Patent Publication No. 2015-216519 Japanese Patent Publication No. 2021-158668 Japanese Patent Publication No. 2013-038504 Japanese Patent Publication No. 2012-212978

[0007] An imaging element and an imaging apparatus according to one embodiment of the technique of the present disclosure are as follows.

[0008] (1) An imaging element in which a first pixel array is repeatedly arranged in a first direction and a second direction, wherein M is a natural number equal to or greater than 3 and N is a natural number equal to or greater than 3, the first pixel array is configured by arranging M pixels in the first direction and N pixels in the second direction, the first pixel array includes a first pixel included in a second pixel array configured by arranging M-1 pixels in the first direction and N-1 pixels in the second direction, and a second pixel other than the first pixel, and the first pixel and the second pixel have different functions.

[0009] (2) The imaging element according to (1), wherein the second pixels are arranged consecutively in at least one of the first direction and the second direction.

[0010] (3) The imaging element according to (1) or (2), wherein the second pixels include first-type pixels having a pixel structure different from that of the first pixels.

[0011] (4) The imaging element according to (3), wherein the first type pixels include pixels for detecting a phase difference.

[0012] (5) The imaging element according to (4), wherein the phase difference detection pixels are arranged at positions other than those at which the pixels of the second pixel array are arranged in the second direction.

[0013] (6) The imaging element according to (4) or (5), wherein the phase difference detection pixels are arranged at positions other than those at which the pixels of the second pixel array are arranged in the first direction.

[0014] (7) The imaging element according to any one of (4) to (6), wherein the phase difference detection pixel has a microlens with a shape different from that of the first pixel.

[0015] (8) The imaging element according to (7), wherein the phase difference detection pixels share a microlens.

[0016] (9) The imaging element according to (3), wherein the first type pixels include polarization pixels for acquiring a polarization image.

[0017] (10) The imaging element according to (3), wherein the first type pixels include pixels having an aperture area different from that of the first pixels.

[0018] (11) The imaging element according to (10), wherein the pixel having an aperture area different from that of the first pixel has an aperture area smaller than that of the first pixel.

[0019] (12) The imaging element according to (1) or (2), wherein the second pixels include second-type pixels having spectral characteristics different from those of the first pixels.

[0020] (13) The imaging device according to (12), wherein the second type pixels include pixels that have a luminance filter or do not have a filter.

[0021] (14) The imaging element according to (1) or (2), wherein the second pixels include third-type pixels having sensitivity characteristics different from those of the first pixels.

[0022] (15) The imaging element according to (14), wherein the first pixels include a plurality of types of pixels corresponding to a plurality of colors, and the third type pixels include a plurality of types of pixels corresponding to the plurality of colors.

[0023] (16) The imaging element according to (14) or (15), wherein the third type pixels have lower sensitivity than the first pixels.

[0024] (17) An imaging element according to any one of (1) to (16), wherein the first pixel includes a plurality of types of pixels corresponding to a plurality of colors, and the second pixel includes a plurality of types of pixels corresponding to the plurality of colors, and the imaging element comprises: a first conversion circuit that converts an analog signal output from the first pixel into a digital signal; and a second conversion circuit that converts an analog signal output from the plurality of types of pixels of the second pixel into a digital signal, wherein the first conversion circuit and the second conversion circuit have different noise characteristics.

[0025] (18) An imaging element according to any one of (1) to (16), comprising: a first conversion circuit that converts a pixel signal output from the first pixel into a digital signal; and a second conversion circuit that converts a pixel signal output from the second pixel into a digital signal, wherein the first conversion circuit and the second conversion circuit have different signal conversion speeds.

[0026] (19) The imaging element according to (3), wherein the first type pixels include at least one of a pixel for distance measurement and a pixel for detecting a change in luminance.

[0027] (20) The imaging element according to any one of (1) to (19), wherein the arrangement pattern of the first pixels in the second pixel array is a pattern that allows imaging of a color image.

[0028] (21) An imaging device including the imaging element according to any one of (1) to (20) and a processor.

[0029] (22) The imaging device according to (21), wherein the processor performs control to change the exposure time between at least a part of the second pixels and the first pixels.

[0030] (23) The imaging device according to (22), wherein the arrangement pattern of at least a part of the second pixels is a pattern that allows imaging of a color image.

[0031] (24) The imaging device according to (23), wherein the arrangement pattern of the first pixels in the second pixel array is a pattern that allows imaging of a color image.

[0032] (25) An imaging element in which a first pixel array is repeatedly arranged in a first direction and a second direction, L being an odd number equal to or greater than 3, the first pixel array being configured by L pixels arranged in the second direction being arranged in the first direction, the first pixel array including a first pixel included in a second pixel array being configured by L-1 pixels arranged in the second direction being arranged in the first direction, and a second pixel other than the first pixel, the first pixel and the second pixel having different functions.

[0033] FIG. 1 is a diagram showing a schematic configuration of a digital camera 100 that is an embodiment of an imaging device relating to the technique of the present disclosure. FIG. 2 is a plan view schematic showing a schematic configuration of the image sensor 5 shown in FIG. 1. FIG. 3 is a schematic view showing a partially enlarged image sensing surface 60 of the image sensor 5 shown in FIG. 2. FIG. 4 is a cross-sectional view schematic of first pixels 61R, first pixels 61G, and first pixels 61B. FIG. 5 is a diagram showing an example in which some of the five second pixels 61F in the first pixel array P1 shown in FIG. 3 are first-type pixels having a pixel structure different from that of the first pixels 61 included in the second pixel array P2. FIG. 6 is a cross-sectional view schematic of first-type pixels 61FXr and 61FXl shown in FIG. 5. FIG. 7 is a diagram showing an example in which another part of the five second pixels 61F in the first pixel array P1 shown in FIG. 3 are first-type pixels having a pixel structure different from that of the first pixels 61 included in the second pixel array P2. FIG. 8 is a diagram showing an example in which all of the five second pixels 61F in the first pixel array P1 shown in FIG. 3 are first-type pixels having a pixel structure different from that of the first pixels 61 included in the second pixel array P2. FIG. 9 is a diagram showing a modified example of the pixel structure of the phase difference detection pixels. FIG. 10 is a diagram showing a modified example of the pixel arrangement shown in FIG. 8. FIG. 11 is a diagram showing an example in which all of the five second pixels 61F in the first pixel array P1 shown in FIG. 3 are polarization pixels for acquiring a polarization image having a pixel structure different from that of the first pixels 61 included in the second pixel array P2. FIG. 12 is a schematic cross-sectional view of the polarization pixel 61FP shown in FIG. 11. FIG. 13 is a diagram showing an example in which the five second pixels 61F in the first pixel array P1 shown in FIG. 3 are other first-type pixels having a pixel structure different from that of the first pixels 61 included in the second pixel array P2. Fig. 14 is a cross-sectional schematic diagram of the first-type pixels 61FR, 61FG, and 61FB shown in Fig. 13 and the first pixels 61R, 61G, and 61B shown in Fig. 13. Fig. 15 is a cross-sectional schematic diagram showing a modified example of the first-type pixels 61FR, 61FG, and 61FB shown in Fig. 13 and the first pixels 61R, 61G, and 61B shown in Fig. 13. Fig. 16 is a diagram of the first-type pixels 61FR, 61FG, and 61FB shown in Fig. 15 as viewed from the microlens ML side.FIG. 17 is a diagram showing an example in which five second pixels 61F in the first pixel array P1 shown in FIG. 3 are second-type pixels having spectral characteristics different from those of the first pixels 61 included in the second pixel array P2. FIG. 18 is a schematic diagram showing a first example of the cross-sectional configuration of the second-type pixels 61FL shown in FIG. 17. FIG. 19 is a schematic diagram showing a second example of the cross-sectional configuration of the second-type pixels 61FL shown in FIG. 17. FIG. 20 is a schematic diagram showing a third example of the cross-sectional configuration of the second-type pixels 61FL shown in FIG. 17. FIG. 21 is a diagram showing an example in which five second pixels 61F in the first pixel array P1 shown in FIG. 3 are third-type pixels having sensitivity characteristics different from those of the first pixels 61 included in the second pixel array P2. FIG. 22 is a diagram showing an example in which some of the five second pixels 61F in the first pixel array P1 shown in FIG. 3 have the same configuration as the first pixels 61 included in the second pixel array P2. Fig. 23 is a diagram showing a configuration example of an analog-to-digital conversion circuit included in the signal processing circuit 64 of the image sensor 5 including the image sensor surface 60 with the pixel arrangement shown in Fig. 3. Fig. 24 is a diagram showing another example in which some of the five second pixels 61F in the first pixel array P1 shown in Fig. 3 have the same configuration as the first pixels 61 included in the second pixel array P2. Fig. 25 is a schematic diagram showing a modified example of the image sensor surface 60 of the image sensor 5 shown in Fig. 2, and is a partially enlarged view of the image sensor surface 60. Fig. 26 shows the external appearance of a smartphone 200. Fig. 27 is a block diagram showing the configuration of the smartphone 200 shown in Fig. 26.

[0034] Fig. 1 is a diagram showing a schematic configuration of a digital camera 100, which is an embodiment of an imaging device relating to the technique of the present disclosure. The digital camera 100 shown in Fig. 1 includes a lens device 40 having an imaging lens 1, an aperture 2, a lens driver 8 that drives the imaging lens 1, an aperture driver 9 that drives the aperture 2, and a lens controller 4 that controls the lens driver 8 and the aperture driver 9, and a main body 100A.

[0035] The main body 100A includes an image sensor 5, a system control unit 11 that controls the overall electrical control system of the digital camera 100, an operation unit 14, a display device 22, a memory 16 that includes RAM (Random Access Memory) and ROM (Read Only Memory), a memory control unit 15 that controls the storage of data in the memory 16 and the reading of data from the memory 16, a digital signal processing unit 17, and an external memory control unit 20 that controls the storage of data in a storage medium 21 and the reading of data from the storage medium 21.

[0036] The lens device 40 may be detachable from the main body 100 A, or may be integrated with the main body 100 A. The imaging lens 1 includes a focus lens.

[0037] The focus lens is a lens for adjusting the focus of an imaging optical system including the imaging lens 1 and the aperture 2, and is composed of a single lens or multiple lenses. When the focus lens moves in the optical axis direction, the position of the principal point of the focus lens (hereinafter also referred to as the focus lens position) changes along the optical axis, thereby changing the focal position on the subject side. Note that a liquid lens whose principal point position in the optical axis direction can be changed by electrical control may also be used as the focus lens.

[0038] The lens control unit 4 of the lens device 40 controls the lens drive unit 8 based on the lens drive signal transmitted from the system control unit 11 to change the focus lens position. The lens control unit 4 of the lens device 40 controls the diaphragm drive unit 9 based on the drive control signal transmitted from the system control unit 11 to change the opening amount of the diaphragm 2.

[0039] The image sensor 5 captures an image of a subject through the imaging optical system provided between the image sensor 5 and the subject. The image sensor 5 has an imaging surface 60 (see FIG. 2 ) on which a plurality of pixels are arranged two-dimensionally, and converts an image of the subject formed on the imaging surface 60 by the imaging optical system into an image signal using the plurality of pixels and outputs the image signal. The output of the pixels included in the image sensor 5 is referred to as a pixel signal, and a collection of pixel signals is referred to as an image signal.

[0040] For example, a CMOS (complementary metal-oxide semiconductor) image sensor or a CCD (charge coupled device) image sensor is used as the imaging element 5. In the following, an example in which the imaging element 5 is a CMOS image sensor will be described.

[0041] The system control unit 11 controls the entire digital camera 100, and its hardware structure is made up of various processors that execute programs and perform processing. The programs executed by the system control unit 11 (including the control program for the image sensor 5) are stored in the ROM (non-transitory storage medium) of the memory 16.

[0042] The various types of processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes programs to perform various processes, a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), or a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically to perform specific processing. More specifically, the structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0043] The system control unit 11 may be configured with one of various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA).

[0044] The system control unit 11 drives the image sensor 5 and the lens device 40 to output, as an image signal, an image of a subject captured through the imaging optical system of the lens device 40. The image signal output from the image sensor 5 is processed by the digital signal processing unit 17 to generate captured image data that is data suitable for display on the display device 22 or data suitable for storage in the storage medium 21.

[0045] An instruction signal from the user is input to the system control unit 11 through the operation unit 14. The operation unit 14 includes a touch panel integrated with the display surface 22b, various buttons, and the like.

[0046] The display device 22 includes a display surface 22b configured by an organic EL (electroluminescence) panel, a liquid crystal panel, or the like, and a display controller 22a that controls the display on the display surface 22b.

[0047] The memory control unit 15 , digital signal processing unit 17 , external memory control unit 20 , and display controller 22 a are interconnected by a control bus 24 and a data bus 25 , and are controlled by commands from the system control unit 11 .

[0048] Fig. 2 is a schematic plan view showing the general configuration of the image sensor 5 shown in Fig. 1. The image sensor 5 includes an image sensor surface 60 on which pixel rows 62, each consisting of a plurality of pixels 61 arranged in a row direction X, are arranged in a column direction Y intersecting the row direction X, a drive circuit 63 that drives the pixels 61 arranged on the image sensor surface 60, and a signal processing circuit 64 that processes pixel signals read out to signal lines from each pixel 61 in the pixel rows 62 arranged on the image sensor surface 60.

[0049] In the example of Fig. 2, the row direction X and the column direction Y are orthogonal to each other. One of the row directions X is referred to as a right direction XR, and the other of the row directions X is referred to as a left direction XL. One of the column directions Y is referred to as an upward direction YU, and the other of the column directions Y is referred to as a downward direction YD.

[0050] It can also be said that on the imaging surface 60, a plurality of pixel columns, each of which is made up of a plurality of pixels 61 aligned in a column direction Y, are arranged in a row direction X. The row direction X constitutes a first direction, and the column direction Y constitutes a second direction.

[0051] 3 is a partially enlarged schematic diagram of the imaging surface 60 of the imaging element 5 shown in FIG. 2. In the following description, M is a natural number equal to or greater than 3, and N is a natural number equal to or greater than 3. The upper limit values ​​of M and N are preferably, but not limited to, 8.

[0052] On the imaging surface 60, first pixel arrays P1 each configured by arranging M pixels 61 in the row direction X (three pixels in the example of FIG. 3 ) and N pixels 61 in the column direction Y (three pixels in the example of FIG. 3 ) are repeatedly arranged in the row direction X and the column direction Y. The first pixel array P1 is configured by arranging N pixel groups each made up of M pixels 61 arranged in the row direction X in the column direction Y. On the imaging surface 60, a plurality of first pixel arrays P1 arranged in the row direction X are arranged in the column direction Y.

[0053] The first pixel array P1 includes a second pixel array P2 configured by arranging M-1 pixels 61 in the row direction (two pixels in the example of FIG. 3) and N-1 pixels 61 in the column direction Y (two pixels in the example of FIG. 3). The second pixel array P2 is configured by arranging N-1 pixels 61 in the column direction Y, each pixel group consisting of M-1 pixels 61 arranged in the row direction X. The pixels 61 included in the second pixel array P2 are also referred to as first pixels 61. The pixels 61 included in the first pixel array P1 other than the first pixels 61 are also referred to as second pixels 61F.

[0054] The second pixel array P2 includes multiple types of first pixels 61 corresponding to multiple wavelength bands. In the example of Fig. 3, the second pixel array P2 includes a first pixel 61R (block marked with the letter "R" in the figure) corresponding to the wavelength band of red light, two first pixels 61G (block marked with the letter "G" in the figure) corresponding to the wavelength band of green light, and a first pixel 61B (block marked with the letter "B" in the figure) corresponding to the wavelength band of blue light. The first pixels 61R, 61G, and 61B each receive light in the corresponding wavelength band and output a pixel signal according to the amount of light received.

[0055] The arrangement pattern of the first pixels 61 in the second pixel array P2 is a pattern that allows capturing a color image. In the example of Fig. 3, the first pixels 61 in the second pixel array P2 are arranged based on a Bayer pattern. The arrangement pattern of the first pixels 61 in the second pixel array P2 is not limited to the Bayer pattern, and various other patterns can be adopted.

[0056] The first pixel array P1 includes first pixels 61 included in the second pixel array P2 (in the example of FIG. 3 , a total of four first pixels 61: a first pixel 61R, two first pixels 61G, and a first pixel 61B) and second pixels 61F (five blocks marked with the letter "F" in the figure) other than the first pixels 61. The second pixels 61F are arranged to surround the second pixel array P2, and in the example of FIG. 3 , three second pixels 61F are arranged consecutively in the row direction X and three second pixels 61F are arranged consecutively in the column direction Y.

[0057] The first pixels 61R, 61G, and 61B have different functions from the second pixel 61F. The five second pixels 61F included in the first pixel array P1 may all have the same function, or may include multiple types of second pixels 61F with different functions.

[0058] The difference in function between the two pixels is obtained by at least one of the following: a difference in pixel structure between the two pixels, a difference in spectral characteristics between the two pixels, a difference in sensitivity characteristics between the two pixels, and a difference in the intended use of the pixel signals output from the two pixels.

[0059] For example, the pixel signal of the first pixel 61 included in the second pixel array P2 is used for the purpose of recording captured image data. The pixel signal of the second pixel 61F included in the first pixel array P1 is used for the purpose of quality control of the captured image data to be recorded (control of noise, sharpness, dynamic range, color vividness, etc.). Alternatively, the pixel signal of the second pixel 61F included in the first pixel array P1 is used, for example, to generate a live view image, to determine the material, color, or unevenness of an object, to obtain captured image data with exposure or noise characteristics different from the captured image data, to obtain distance information of a subject, or to detect moving parts of a subject.

[0060] 4 is a cross-sectional schematic diagram of the first pixel 61R, the first pixel 61G, and the first pixel 61B. Each of the first pixel 61R, the first pixel 61G, and the first pixel 61B includes a microlens ML that collects light from a subject, a photoelectric conversion unit PD that converts the light collected by the microlens ML into an electric charge, a spectral filter CF that transmits light in a specific wavelength band and is provided between the photoelectric conversion unit PD and the microlens ML, and a light-shielding film LS that is provided between the spectral filter CF and the photoelectric conversion unit PD. The photoelectric conversion unit PD is a photodiode formed in a semiconductor substrate such as silicon, but may also be composed of an organic material film or the like that is disposed above the semiconductor substrate.

[0061] The spectral filter CF (referred to as R filter in FIG. 4) included in the first pixel 61R transmits red light, the spectral filter CF (referred to as G filter in FIG. 4) included in the first pixel 61G transmits green light, and the spectral filter CF (referred to as B filter in FIG. 4) included in the first pixel 61B transmits blue light.

[0062] The differences in the image structure mentioned above include the presence or absence of a spectral filter CF, the presence or absence of a functional element provided between the microlens ML and the photoelectric conversion unit PD, differences in the shape or size of the opening in the light-shielding film LS, differences in the shape of the microlens ML, etc.

[0063] 3 , color captured image data can be generated based on the pixel signals of the first pixels 61 in the second pixel array P2 without performing interpolation processing or the like on pixel signals at the positions of the first pixels 61. This can improve the quality of the captured image data and reduce the processing load. Furthermore, because the second pixels 61F are provided around the second pixel array P2, using the pixel signals of the second pixels 61F enables quality control of the captured image data, thereby improving the quality of the captured image data. Furthermore, using the pixel signals of the second pixels 61F can also provide detailed information, such as the movement, color, or unevenness of the subject being captured, thereby realizing multi-functionality of the digital camera 100.

[0064] Hereinafter, a specific example of the five second pixels 61F in the first pixel array P1 will be described.

[0065] Figure 5 is a diagram showing an example in which some of the five second pixels 61F in the first pixel array P1 shown in Figure 3 are first-type pixels having a pixel structure different from that of the first pixels 61 included in the second pixel array P2.

[0066] 5 , first-type pixels 61FXr for phase difference detection (blocks marked with "FXr" in the figure) and first-type pixels 61FXl for phase difference detection (blocks marked with "FXl" in the figure) are arranged in the first pixel array P1 other than the positions where the first pixels 61 of the second pixel array P2 are arranged in the column direction Y. In the example of Fig. 5 , it is preferable that the remaining two second pixels 61F in the first pixel array P1 have functions different from those of the first-type pixels 61FXr and 61FXl.

[0067] Fig. 6 is a cross-sectional schematic diagram of the first-type pixels 61FXr and 61FXl shown in Fig. 5. As shown in Fig. 6, the first-type pixels 61FXr and 61FXl each have a configuration in which the size of the opening in the light-shielding film LS is changed compared to the configuration of the first pixel 61G. The opening in the light-shielding film LS of the first-type pixel 61FXr is configured such that, for example, the right half of the opening in the light-shielding film LS of the first pixel 61G is closed. The opening in the light-shielding film LS of the first-type pixel 61FXl is configured such that, for example, the left half of the opening in the light-shielding film LS of the first pixel 61G is closed.

[0068] With such a configuration of the light-shielding film LS, the first-type pixel 61FXr receives one of a pair of light beams that have passed through two different portions aligned in the row direction X of the pupil region of the imaging optical system of the lens device 40. The first-type pixel 61FXl receives the other of the pair of light beams. The first pixel 61 included in the second pixel array P2 receives both of the pair of light beams. The system control unit 11 detects a phase difference based on pixel signals from the first-type pixel 61FXr and the first-type pixel 61FXl, and controls the focus lens position based on the phase difference.

[0069] The spectral filters CF included in the first-type pixels 61FXr and 61FXl may be filters other than G filters. Also, instead of the spectral filters CF, luminance filters may be provided.

[0070] The luminance filter has spectral characteristics that correlate with the luminance component of light, and examples of such filters include a neutral density (ND) filter, a transparent filter, a white filter, and a gray filter. The luminance filter can transmit light with more wavelength components than the spectral filter CF. Therefore, by providing a luminance filter instead of the spectral filter CF, it is possible to improve the accuracy of phase difference detection using the pixel signals of the first-type pixels 61FXr and 61FXl.

[0071] The first-type pixels 61FXr and 61FXl may be configured without the spectral filter CF and the luminance filter. This configuration also allows light of more wavelength components to pass through than the spectral filter CF. This improves the accuracy of phase difference detection using the pixel signals of the first-type pixels 61FXr and 61FXl. When the first-type pixels 61FXr and 61FXl are configured with a luminance filter instead of the spectral filter CF, or when they are configured without the spectral filter CF and the luminance filter, they can also be referred to as second-type pixels having spectral characteristics different from those of the first pixels 61.

[0072] 5 , first-type pixels 61FXr and first-type pixels 61FXl are provided in pixel rows 62 other than the pixel row 62 including the first pixels 61 included in the second pixel array P2. This makes it easy to control the exposure time to be different between the pixel row 62 including the first-type pixels 61FXr and first-type pixels 61FXl and the pixel row 62 including the first pixels 61 included in the second pixel array P2.

[0073] For example, by lengthening the exposure time of the pixel row 62 that includes the first-type pixels 61FXr and 61FXl, it is possible to improve the accuracy of detecting the phase difference using the pixel signals output from the pixel row 62 that includes the first-type pixels 61FXr and 61FXl. By capturing an image of a subject using the pixel row 62 that includes the first pixels 61 included in the second pixel array P2 while controlling the focus lens position based on this phase difference, it is possible to improve the sharpness of the captured image data obtained by processing the pixel signals output from this pixel row 62.

[0074] Figure 7 is a diagram showing an example in which another portion of the five second pixels 61F in the first pixel array P1 shown in Figure 3 is a first-type pixel having a pixel structure different from that of the first pixels 61 included in the second pixel array P2.

[0075] In the example of Figure 7, in the first pixel array P1, first-type pixels 61FXr and first-type pixels 61FXl for phase difference detection having the configuration shown in Figure 6 are arranged in places other than the positions in the row direction X where the first pixels 61 of the second pixel array P2 are arranged.

[0076] 7, a first-type pixel 61FXr is arranged to the left of the upper left first pixel 61G included in the second pixel array P2, and a first-type pixel 61FXl is arranged to the left of the first pixel 61B included in the second pixel array P2. In the example of Fig. 7, it is preferable that the remaining three second pixels 61F in the first pixel array P1 have functions different from those of the first-type pixels 61FXr and 61FXl.

[0077] 7 , a pixel row 62 including a first pixel 61 included in the second pixel array P2 includes a first-type pixel 61FXr and a first-type pixel 61FXl. Therefore, it is possible to simultaneously read out pixel signals from the first-type pixels 61FXr and 61FXl and pixel signals from the first pixels 61 included in the second pixel array P2. This makes it possible to capture images with the subject in focus, even when performing continuous shooting or the like.

[0078] Fig. 8 is a diagram showing an example in which all of the five second pixels 61F in the first pixel array P1 shown in Fig. 3 are first-type pixels (pixels for detecting a phase difference) having a pixel structure different from that of the first pixels 61 included in the second pixel array P2. The pixel arrangement shown in Fig. 8 is a configuration that combines the pixel arrangement shown in Fig. 5 and the pixel arrangement shown in Fig. 7. The configuration shown in Fig. 8 makes it possible to obtain both the advantages described in Fig. 5 and Fig. 7.

[0079] In the example of FIG. 6, the shape of the light-shielding film LS makes it possible for the first-type pixels 61FXr and 61FXl to detect a phase difference, but this can also be achieved by the shape of the microlenses ML.

[0080] 9 , the aperture size of the light-shielding film LS of the first-type pixel 61FXr and the first-type pixel 61FXl may be the same as that of the first pixel 61 included in the second pixel array P2, but instead, the adjacent first-type pixels 61FXr and 61FXl may share a microlens ML, with the shape of the microlens ML being different from that of the first pixel 61. Even as shown in FIG. 9 , one of a pair of light beams that pass through two different portions aligned in the row direction X of the pupil region of the imaging optical system of the lens device 40 can be received by the first-type pixel 61FXr, and the other of the pair of light beams can be received by the first-type pixel 61FXl. According to the configuration of FIG. 9 , the aperture areas of the first-type pixels 61FXr and 61FXl are increased, thereby improving the accuracy of phase difference detection.

[0081] Fig. 10 is a diagram showing a modified example of the pixel arrangement shown in Fig. 8. The pixel arrangement shown in Fig. 10 is configured such that the first-type pixel 61FXr located to the left of the upper left first pixel 61G in the second pixel array P2 in Fig. 8 is replaced with a first-type pixel 61FYu for detecting a phase difference, and the first-type pixel 61FXl located to the left of the first pixel 61B in the second pixel array P2 is replaced with a first-type pixel 61FYd for detecting a phase difference. The first-type pixels 61FYu and 61FYd have a pixel structure capable of detecting a phase difference in the column direction Y.

[0082] 10 , a part of the microlens ML is shown by a dashed line when the microlens ML is shared by the adjacent first-type pixels 61FXr and 61FXl, and when the microlens ML is shared by the adjacent first-type pixels 61FYu and 61FYd. The pixel arrangement shown in FIG. 10 makes it possible to detect phase differences in two directions.

[0083] Figure 11 is a diagram showing an example in which all of the five second pixels 61F in the first pixel array P1 shown in Figure 3 are polarization pixels for obtaining a polarization image, which have a pixel structure different from that of the first pixels 61 included in the second pixel array P2.

[0084] In the example of Fig. 11 , all of the pixels 61 other than those in the second pixel array P2 are polarization pixels 61FP. Fig. 12 is a schematic cross-sectional view of the polarization pixel 61FP shown in Fig. 11 . As shown in Fig. 12 , the polarization pixel 61FP has a configuration in which the spectral filter CF in the first pixel 61 included in the second pixel array P2 is replaced with a polarizer PF, and has a different pixel structure from each of the first pixels 61 included in the second pixel array P2. The first pixel array P1 may include multiple types of polarization pixels 61FP that include polarizers PF with different vibration directions of transmitted light. The polarization pixels FP constitute first-type pixels.

[0085] 11 , by configuring the second pixel 61F as a polarization pixel 61FP, it is possible to generate color captured image data based on pixel signals output from the second pixel array P2 and generate at least one polarization image based on pixel signals output from the polarization pixel 61FP. Using the polarization image makes it possible to determine the material, color, and unevenness of objects included in a subject. This enables image processing of the captured image data and subject recognition, etc., using the determination results, thereby improving the quality of captured image data and achieving highly accurate focusing control, etc.

[0086] Although an example in which all second pixels 61F are polarization pixels 61FP has been shown here, this is not limiting. For example, at least one of the two second pixels 61F in the first pixel array P1 in Fig. 5 may be a polarization pixel 61FP. Alternatively, at least one of the three second pixels 61F in the first pixel array P1 in Fig. 7 may be a polarization pixel 61FP.

[0087] In the pixel arrangement shown in Fig. 11 , at least one of the polarization pixels FP may be replaced with at least one of a pixel for distance measurement and a pixel for brightness change detection. The pixel for distance measurement is configured by a pixel that acquires the distance to a subject using TOF (Time Of Flight). The pixel for brightness change detection is configured by, for example, a pixel used in an event-based vision sensor. Both the pixel for distance measurement and the pixel for brightness change detection have a pixel structure different from that of the first pixel 61 included in the second pixel array P2 and constitute first-type pixels.

[0088] FIG. 13 is a diagram showing an example in which the five second pixels 61F in the first pixel array P1 shown in FIG. 3 are replaced by first-type pixels 61FR (blocks marked with "FR" in the figure), first-type pixels 61FG (blocks marked with "FG" in the figure), and first-type pixels 61FB (blocks marked with "FB" in the figure) that have a pixel structure different from that of the first pixels 61 included in the second pixel array P2.

[0089] Fig. 14 is a cross-sectional schematic diagram of the first-type pixels 61FR, 61FG, and 61FB shown in Fig. 13 and the first pixels 61R, 61G, and 61B shown in Fig. 13. The first-type pixels 61FR, 61FG, and 61FB have the same pixel structure except for the different transmission wavelength bands of the spectral filters CF, and therefore only one pixel is shown in Fig. 14. Similarly, only one pixel is shown for the first pixels 61R, 61G, and 61B.

[0090] The spectral filter CF of the first-type pixel 61FR has the same spectral characteristics as the spectral filter CF included in the first pixel 61R. The spectral filter CF of the first-type pixel 61FG has the same spectral characteristics as the spectral filter CF included in the first pixel 61G. The spectral filter CF of the first-type pixel 61FB has the same spectral characteristics as the spectral filter CF included in the first pixel 61B.

[0091] As shown in FIG. 14, the first type pixels 61FR, 61FG, and 61FB have the same pixel structure as the first pixels 61R, 61G, and 61B, except that the opening area of ​​the light-shielding film LS of the first type pixels 61FR, 61FG, and 61FB is smaller than the opening area of ​​the light-shielding film LS of the first pixels 61R, 61G, and 61B.

[0092] In addition, the positional relationship between the center of the opening of the light-shielding film LS and the center of the microlens ML in the first type pixel 61FR, the first type pixel 61FG, and the first type pixel 61FB is the same as the positional relationship between the center of the opening of the light-shielding film LS and the center of the microlens ML in the first pixel 61R, the first pixel 61G, and the first pixel 61B.

[0093] 13 , it is possible to capture a first image of the same subject using the first pixels 61R, 61G, and 61B, and a second image using a different exposure from the first image captured using the first-type pixels 61FR, 61FG, and 61FB. In this way, it is possible to capture images of the same subject in two different patterns using different exposures, thereby realizing multi-functionality of the digital camera 100.

[0094] Note that, as long as the first pixel array P1 includes at least one first-type pixel 61FR, one first-type pixel 61FG, and one first-type pixel 61FB, color imaging is possible using the first-type pixels 61FR, 61FG, and 61FB. Therefore, for example, in FIG. 13 , the first-type pixel 61FG located in the upper right corner of the first pixel array P1 and the first-type pixel 61FR located to the left of it can be replaced with a second pixel 61F with a different function. Examples of the second pixel 61F with a different function that can be used include a pixel for detecting a phase difference, a polarization pixel, a pixel for measuring distance, or a pixel for detecting a change in brightness, as described above.

[0095] Fig. 15 is a schematic diagram showing a cross-sectional modification of the first-type pixels 61FR, 61FG, and 61FB shown in Fig. 13 and the first pixels 61R, 61G, and 61B shown in Fig. 13. The first-type pixels 61FR, 61FG, and 61FB have the same pixel structure as the first pixels 61R, 61G, and 61B, except that a mesh-shaped filter MF is added between the spectral filter CF and the light-shielding film LS.

[0096] Fig. 16 is a view of the first-type pixels 61FR, 61FG, and 61FB shown in Fig. 15 as viewed from the microlens ML side. As shown in Fig. 16, the mesh filter MF is a filter in which a plurality of openings Mfa are formed, and the portions other than the openings Mfa are light-shielding regions. The total area of ​​the plurality of openings Mfa is smaller than the opening area of ​​the light-shielding film LS in each of the first pixels 61R, 61G, and 61B. Even with the above configuration, it is possible to capture images of the same subject in two patterns with different exposures, thereby achieving multi-functionality for the digital camera 100.

[0097] Fig. 17 is a diagram showing an example in which five second pixels 61F in the first pixel array P1 shown in Fig. 3 are replaced with second-type pixels 61FL (blocks marked with "FL" in the diagram) having spectral characteristics different from those of the first pixels 61 included in the second pixel array P2. Fig. 18 is a schematic diagram showing a first example of the cross-sectional configuration of the second-type pixels 61FL shown in Fig. 17.

[0098] 18 , the second-type pixels 61FL have a pixel structure in which the spectral filters CF in the first pixels 61 included in the second pixel array P2 are replaced with luminance filters LF. As described above, the luminance filters LF have spectral characteristics that correlate with the luminance component of light, and include ND filters, transparent filters, white filters, gray filters, etc. The luminance filters LF can transmit light with more wavelength components than the spectral filters CF.

[0099] 17 and 18 , the second-type pixels 61FL can acquire more luminance components than the first pixels 61 included in the second pixel array P2. Therefore, by using the pixel signals of the second-type pixels 61FL, it becomes possible to perform processing to improve the quality of captured image data generated based on the pixel signals of the first pixels 61 included in the second pixel array P2, such as noise reduction.

[0100] Fig. 19 is a schematic diagram showing a second example of the cross-sectional configuration of the second-type pixel 61FL shown in Fig. 17. As shown in Fig. 19, the second-type pixel 61FL has a pixel structure in which the spectral filter CF in the first pixel 61 included in the second pixel array P2 is deleted. Because the second-type pixel 61FL is not provided with the spectral filter CF, it can transmit light of more wavelength components than the first pixel 61 included in the second pixel array P2.

[0101] The pixel structure shown in Fig. 19 can acquire more luminance components than the pixel structure shown in Fig. 18. Therefore, by using the pixel signals of the second-type pixels 61FL, it is possible to perform processing to improve the quality of captured image data generated based on the pixel signals of the first pixels 61 included in the second pixel array P2, such as further reducing noise.

[0102] Fig. 20 is a schematic diagram showing a third example of the cross-sectional configuration of the second-type pixel 61FL shown in Fig. 17. As shown in Fig. 20, the second-type pixel 61FL has a pixel structure in which the spectral filter CF in the first pixel 61 included in the second pixel array P2 is replaced with a specific filter CFa. The specific filter CFa has spectral characteristics different from those of the spectral filter CF of the first pixel 61 included in the second pixel array P2.

[0103] The specific filter CFa is, for example, a filter that transmits infrared light, a filter that transmits ultraviolet light, a filter that transmits cyan light, a filter that transmits yellow light, a filter that transmits magenta light, a filter having narrower or broader spectral characteristics than the spectral filter CF of the first pixel 61R, a filter having narrower or broader spectral characteristics than the spectral filter CF of the first pixel 61G, or a filter having narrower or broader spectral characteristics than the spectral filter CF of the first pixel 61B.

[0104] In the example of FIG. 17 , there are five second-type pixels 61FL, and therefore, as the five second-type pixels 61FL, a configuration can be exemplified in which second-type pixels 61FL including specific filters CFa that transmit infrared light, second-type pixels 61FL including specific filters CFa that transmit ultraviolet light, second-type pixels 61FL including specific filters CFa that transmit cyan light, second-type pixels 61FL including specific filters CFa that transmit yellow light, and second-type pixels 61FL including specific filters CFa that transmit magenta light are provided.

[0105] 20 , the second-type pixels 61FL can distinguish the color of a subject that cannot be distinguished by the first pixels 61 of the second pixel array P2. Therefore, by using the pixel signals of the second-type pixels 61FL, it is possible to emphasize (make vivid) a color of a specific wavelength, for example, in captured image data generated based on the pixel signals of the first pixels 61 included in the second pixel array P2. Furthermore, by using the pixel signals of the second-type pixels 61FL, it is possible to identify an object that cannot be identified by the second pixel array P2.

[0106] Fig. 21 is a diagram showing an example in which five second pixels 61F in the first pixel array P1 shown in Fig. 3 are third-type pixels having sensitivity characteristics different from those of the first pixels 61 included in the second pixel array P2. In the example of Fig. 21, the first pixel array P1 includes two third-type pixels 61Fr (blocks marked with "Fr" in the figure), two third-type pixels 61Fg (blocks marked with "Fg" in the figure), and one third-type pixel 61Fb (block marked with "Fb" in the figure).

[0107] The third-type pixels 61Fr have the same spectral characteristics as the first pixels 61R but have lower sensitivity. The third-type pixels 61Fg have the same spectral characteristics as the first pixels 61G but have lower sensitivity. The third-type pixels 61Fb have the same spectral characteristics as the first pixels 61B but have lower sensitivity. The difference in sensitivity between the two pixels can be achieved by modifying the light-shielding film LS, the microlens ML, or the spectral filter CF.

[0108] 21 , it is possible to capture a first image of the same subject using the first pixels 61R, 61G, and 61B, and a third image of the same subject using the third type pixels 61Fr, 61Fg, and 61Fb, which has a different sensitivity from the first image. Therefore, for example, by using image signals obtained in the first image capture and image signals obtained in the third image capture, it is possible to capture images with a wider dynamic range.

[0109] Note that as long as the first pixel array P1 includes at least one third-type pixel 61Fr, one third-type pixel 61Fg, and one third-type pixel 61Fb, it is possible to widen the dynamic range of the image captured by the second pixel array P2. Therefore, for example, in FIG. 21 , the third-type pixel 61Fg located in the upper right corner of the first pixel array P1 and the third-type pixel 61Fr located to the left of it can be replaced with second pixels 61F with other functions. Examples of the second pixels 61F with other functions that can be used include the first-type pixels (phase difference detection pixels, polarization pixels, distance measurement pixels, or luminance change detection pixels) or second-type pixels described above.

[0110] Fig. 22 is a diagram showing an example in which some of the five second pixels 61F in the first pixel array P1 shown in Fig. 3 have the same configuration as the first pixels 61 included in the second pixel array P2. In the example of Fig. 22, in the first pixel array P1, a pixel 61fg having the same spectral characteristics, sensitivity characteristics, and pixel structure as the first pixel 61G is arranged to the left of the upper left first pixel 61G, a pixel 61fb having the same spectral characteristics, sensitivity characteristics, and pixel structure as the first pixel 61B is arranged to the left of the first pixel 61B, and a pixel 61fr having the same spectral characteristics, sensitivity characteristics, and pixel structure as the first pixel 61R is arranged above the pixel 61fg.

[0111] 22 shows an example configuration of an analog-to-digital conversion circuit included in the signal processing circuit 64 of the image sensor 5. The signal processing circuit 64 includes a second conversion circuit 64A that is provided corresponding to a specific pixel column consisting of only pixels 61fr, 61fg, and 61fb and converts an analog signal into a digital signal, and a first conversion circuit 64B that is provided corresponding to pixel columns other than this specific pixel column and converts an analog signal into a digital signal.

[0112] The second conversion circuit 64A and the first conversion circuit 64B have different noise characteristics. The first conversion circuit 64B has noise characteristics such that the noise level included in the digital signal at an ISO (International Organization for Standardization) sensitivity setting equal to or higher than a threshold is a first level proportional to the ISO sensitivity. The second conversion circuit 64A has noise characteristics such that the noise level included in the digital signal at an ISO sensitivity setting equal to or higher than the threshold is a second level lower than the first level. Therefore, when converting an analog signal to a digital signal using the second conversion circuit 64A, noise included in the pixel signal can be reduced more effectively when capturing an image at a high ISO sensitivity setting than when converting an analog signal to a digital signal using the first conversion circuit 64B.

[0113] 22 , even in a dark imaging environment with a high ISO sensitivity, it is possible to obtain a first pixel signal obtained by imaging the same subject using the first pixels 61R, 61G, and 61B, and a low-noise second pixel signal obtained by imaging the same subject using the pixels 61fr, 61fg, and 61fb. Therefore, low-noise captured image data can be generated by using these first pixel signals and second pixel signals.

[0114] In FIG. 22, the arrangement pattern of the pixels 61fr, 61fg, and 61fb may be a Bayer pattern, similar to the second pixel arrangement P2.

[0115] Fig. 23 is a diagram showing an example configuration of an analog-to-digital conversion circuit included in the signal processing circuit 64 of the image sensor 5 that includes the imaging surface 60 having the pixel arrangement shown in Fig. 3. In Fig. 23, the various configurations described above can be employed for each second pixel 61F of the first pixel array P1.

[0116] The signal processing circuit 64 shown in Figure 23 includes a second conversion circuit 64C that is provided corresponding to a second specific pixel column consisting only of second pixels 61F and converts an analog signal into a digital signal, and a first conversion circuit 64D that is provided corresponding to a pixel column other than this second specific pixel column and converts an analog signal into a digital signal.

[0117] The second conversion circuit 64C and the first conversion circuit 64D have different signal conversion speeds. The second conversion circuit 64C has a faster signal conversion speed than the first conversion circuit 64D. According to the configuration of the signal processing circuit 64 shown in FIG. 23 , pixel signals from some of the second pixels 61F can be read out faster than from the first pixels 61 in the second pixel array P2. This allows functions using pixel signals from some of the second pixels 61F to be realized at high speed. For example, by applying the configuration of the signal processing circuit 64 shown in FIG. 23 to the pixel arrangement shown in FIG. 7 , pixel signals can be read out from the first-type pixels 61FXr and 61FXl at a higher speed than from the second pixel array P2. As a result, the focus position can be controlled at a higher speed.

[0118] 23, two conversion circuits may be provided for a pixel column corresponding to the first conversion circuit 64D: a conversion circuit AD1 that converts the pixel signal of the first pixel 61 in the pixel column into a digital signal, and a conversion circuit AD2 that converts the pixel signal of the second pixel 61F in the pixel column into a digital signal. In this case, the conversion circuit AD1 has the same configuration as the first conversion circuit 64D. The conversion circuit AD2 has the same configuration as the second conversion circuit 64C. In this way, the pixel signals of all the second pixels 61F can be read out faster than in the second pixel array P2.

[0119] Fig. 24 is a diagram showing another example in which some of the five second pixels 61F in the first pixel array P1 shown in Fig. 3 have the same configuration as the first pixels 61 included in the second pixel array P2. In the example of Fig. 24, two types of first pixel arrays P1, namely, first pixel arrays P1a and first pixel arrays P1b, are provided as the first pixel array P1, and the first pixel arrays P1a and P1b are arranged alternately in the row direction X and the column direction Y.

[0120] A pixel 61fg having the same spectral characteristics, sensitivity characteristics, and pixel structure as the first pixel 61G is arranged immediately above the upper left first pixel 61G in the first pixel array P1a, and pixels 61fr having the same spectral characteristics, sensitivity characteristics, and pixel structure as the first pixel 61R are arranged on both the left and right sides of this pixel 61fg.

[0121] In the first pixel array P1b, a pixel 61fg is arranged immediately above the upper left first pixel 61G, and on both the left and right sides of this pixel 61fg, pixels 61fb are arranged which have the same spectral characteristics, sensitivity characteristics, and pixel structure as the first pixel 61B.

[0122] The second pixel 61F in FIG. 24 can employ any of the various configurations described above.

[0123] 24 , RG pixel rows in which pixels 61fr and pixels 61fg are alternately arranged in the row direction X, and GB pixel rows in which pixels 61fg and pixels 61fb are alternately arranged in the row direction X, are arranged alternately in the column direction Y, sandwiching two pixel rows including the first pixels 61 of the second pixel array P2. The arrangement pattern of the pixels included in these RG pixel rows and GB pixel rows is an arrangement pattern based on the Bayer pattern, which is a pattern that allows color images to be captured.

[0124] The system control unit 11 performs control to change the first exposure time of each pixel 61 in the RG pixel rows and the BG pixel rows and the second exposure time of each pixel 61 in pixel rows other than the RG pixel rows and the BG pixel rows. The system control unit 11 also performs control to generate a live view image based on pixel signals read out from each pixel 61 in the RG pixel rows and the BG pixel rows, and controls to generate captured image data for recording based on pixel signals read out from each pixel 61 in pixel rows other than the RG pixel rows and the BG pixel rows.

[0125] For example, by making the second exposure time longer than the first exposure time, the system control unit 11 can update the live view image frequently even while capturing images for recording using long exposure times.

[0126] 24, it is possible to change the exposure time of all pixels 61 other than those in the second pixel array P2 from the exposure time of the pixels 61 in the second pixel array P2. As a result, for example, while an image of a subject is being captured using the second pixel array P2, a live view image can be generated and displayed based on pixel signals read out from the pixels 61 in the RG pixel row and the BG pixel row, and further, other functions such as focus position control can be executed based on the pixel signal read out from the second pixel 61F.

[0127] Such control of the exposure time can be applied regardless of the function of the second pixels 61F included in the first pixel array P1. For example, in the pixel arrangement shown in Fig. 24 , the pixels 61fr may be changed to first-type pixels 61FR or third-type pixels 61Fr, the pixels 61fg may be changed to first-type pixels 61FG or third-type pixels 61Fg, and the pixels 61fb may be changed to first-type pixels 61FB or third-type pixels 61Fb.

[0128] 25 is a schematic diagram showing a modification of the imaging surface 60 of the imaging element 5 shown in FIG. 2, and is a partially enlarged view of the imaging surface 60. In the following description, L is assumed to be an odd number equal to or greater than 3. The upper limit of L is preferably set to 7, although it is not limited to this value.

[0129] 25 , first pixel arrays PX, each configured by L pixels 61 (L=3 in the example of FIG. 25 ) arranged in the column direction Y and arranged in the row direction X, are repeatedly arranged in the row direction X and the column direction Y on the imaging surface 60. A plurality of first pixel arrays PX arranged in the row direction X are arranged in the column direction Y on the imaging surface 60. In the example of FIG. 25 , the first pixel array PX is configured by arranging L−1 pixel groups in the row direction X, each group consisting of L pixels 61 arranged in the column direction Y.

[0130] The first pixel array PX includes a second pixel array PY configured by arranging L-1 pixels 61 (two in the example of FIG. 25 ) arranged in the column direction Y in the row direction X. In the example of FIG. 25 , the second pixel array PY is configured by arranging L-1 pixels 61 in the row direction X, each pixel group consisting of L-1 pixels 61 arranged in the column direction Y. The second pixel array PY has the same configuration as the second pixel array P2. The pixels 61 in the first pixel array PX other than the second pixel array PY are pixels with different functions from the pixels 61 included in the second pixel array PY (the same as the second pixel 61F described above).

[0131] 25 , color captured image data can be generated based on the pixel signal of pixel 61 in the second pixel array PY without performing interpolation processing or the like on the pixel signal at the position of pixel 61. This can improve the quality of the captured image data and reduce the processing load. Furthermore, since the second pixel 61F is provided, using the pixel signal of the second pixel 61F enables quality control of the captured image data, thereby improving the quality of the captured image data. Furthermore, using the pixel signal of the second pixel 61F can also obtain detailed information such as the movement, color, or unevenness of the subject being captured, thereby realizing multi-functionality of the digital camera 100.

[0132] Next, the configuration of a smartphone, which is another embodiment of an imaging device according to the technique of the present disclosure, will be described.

[0133] Fig. 26 shows the external appearance of a smartphone 200. The smartphone 200 shown in Fig. 26 has a flat housing 201, and is provided with a display input unit 204 on one surface of the housing 201, which is an integrated unit of a display panel 202 as a display unit and an operation panel 203 as an input unit.

[0134] Such a housing 201 also includes a speaker 205, a microphone 206, an operation unit 207, and a camera unit 208. Note that the configuration of the housing 201 is not limited to this, and for example, it is also possible to adopt a configuration in which the display unit and the input unit are independent, or a configuration having a folding structure or a sliding mechanism.

[0135] FIG. 27 is a block diagram showing the configuration of the smartphone 200 shown in FIG.

[0136] As shown in FIG. 27 , the main components of the smartphone include a wireless communication unit 210, a display input unit 204, a call unit 211, an operation unit 207, a camera unit 208, a memory unit 212, an external input / output unit 213, a GNSS (Global Navigation Satellite System) receiving unit 214, a motion sensor unit 215, a power supply unit 216, and a main control unit 220.

[0137] The smartphone 200 also has, as its main function, a wireless communication function for performing mobile wireless communication via a base station device BS (not shown) and a mobile communication network NW (not shown).

[0138] The wireless communication unit 210 performs wireless communication with a base station device BS accommodated in the mobile communication network NW in accordance with instructions from the main control unit 220. Using this wireless communication, various file data such as audio data and image data, e-mail data, etc. are sent and received, and web data, streaming data, etc. are received.

[0139] The display input unit 204 is a so-called touch panel that, under the control of the main control unit 220, displays images (still images and moving images) or text information, etc. to visually convey information to the user and detects user operations on the displayed information, and is equipped with a display panel 202 and an operation panel 203.

[0140] The display panel 202 uses a liquid crystal display (LCD), an organic electroluminescence display (OELD), or the like as a display device.

[0141] The operation panel 203 is a device placed so that an image displayed on the display surface of the display panel 202 can be seen, and detects one or more coordinates operated by a user's finger or a stylus. When this device is operated by the user's finger or a stylus, a detection signal generated by the operation is output to the main control unit 220. Next, the main control unit 220 detects the operation position (coordinates) on the display panel 202 based on the received detection signal.

[0142] As shown in Figure 27, the display panel 202 and operation panel 203 of a smartphone 200, which is an example of one embodiment of the imaging device of the present invention, are integrated to form a display input unit 204, and the operation panel 203 is positioned so that it completely covers the display panel 202.

[0143] When such an arrangement is adopted, operation panel 203 may also have a function for detecting user operations in areas outside display panel 202. In other words, operation panel 203 may have a detection area for the overlapping portion that overlaps display panel 202 (hereinafter referred to as a display area), and a detection area for the remaining outer edge portion that does not overlap display panel 202 (hereinafter referred to as a non-display area).

[0144] The size of the display area and the size of the display panel 202 may be exactly the same, but they do not necessarily have to be the same. The operation panel 203 may also have two sensitive areas: an outer edge portion and an inner portion. The width of the outer edge portion is designed appropriately depending on the size of the housing 201, etc.

[0145] Furthermore, the position detection method used in the operation panel 203 may be a matrix switch method, a resistive film method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, a capacitance method, or the like, and any method may be used.

[0146] The call unit 211 is equipped with a speaker 205 or a microphone 206, and converts the user's voice input through the microphone 206 into voice data that can be processed by the main control unit 220 and outputs it to the main control unit 220, or decodes voice data received by the wireless communication unit 210 or the external input / output unit 213 and outputs it from the speaker 205.

[0147] Also, as shown in FIG. 26, for example, the speaker 205 can be mounted on the same surface as the display input unit 204, and the microphone 206 can be mounted on the side of the housing 201.

[0148] The operation unit 207 is a hardware key using a key switch or the like, and receives instructions from the user. For example, as shown in Fig. 26 , the operation unit 207 is a push-button switch mounted on the side of the housing 201 of the smartphone 200, which turns on when pressed with a finger or the like, and turns off when the finger is released by the restoring force of a spring or the like.

[0149] The storage unit 212 stores the control program and control data of the main control unit 220, application software, address data associated with names or telephone numbers of communication partners, data of sent and received e-mails, web data downloaded by web browsing, downloaded content data, and also temporarily stores streaming data, etc. The storage unit 212 is composed of an internal storage unit 217 built into the smartphone and an external storage unit 218 having a removable external memory slot.

[0150] The internal memory unit 217 and the external memory unit 218 constituting the memory unit 212 are realized using storage media such as a flash memory type, a hard disk type, a multimedia card micro type, a card-type memory (e.g., MicroSD (registered trademark) memory, etc.), a RAM (Random Access Memory), a ROM (Read Only Memory), etc.

[0151] The external input / output unit 213 serves as an interface with all external devices connected to the smartphone 200, and is used to directly or indirectly connect to other external devices via communication (e.g., Universal Serial Bus (USB), IEEE 1394, Bluetooth (registered trademark), RFID (Radio Frequency Identification), Infrared Data Association (IrDA) (registered trademark), UWB (Ultra Wideband) (registered trademark), ZigBee (registered trademark), etc.) or a network (e.g., Ethernet (registered trademark), wireless LAN (Local Area Network), etc.).

[0152] Examples of external devices connected to the smartphone 200 include a wired / wireless headset, a wired / wireless external charger, a wired / wireless data port, a memory card connected via a card socket, a SIM (Subscriber Identity Module Card) / UIM (User Identity Module Card) card, an external audio / video device connected via an audio / video I / O (Input / Output) terminal, a wirelessly connected external audio / video device, a wired / wirelessly connected smartphone, a wired / wirelessly connected personal computer, a wired / wirelessly connected personal computer, and earphones.

[0153] The external input / output unit 213 can transmit data received from such external devices to each component inside the smartphone 200, or transmit data inside the smartphone 200 to external devices.

[0154] The GNSS receiving unit 214 receives GNSS signals transmitted from the GNSS satellites ST1 to STn in accordance with instructions from the main control unit 220, executes positioning calculation processing based on the received multiple GNSS signals, and detects a position consisting of the latitude, longitude, and altitude of the smartphone 200. When the GNSS receiving unit 214 can acquire position information from the wireless communication unit 210 or the external input / output unit 213 (for example, a wireless LAN), it can also detect the position using the position information.

[0155] The motion sensor unit 215 includes, for example, a three-axis acceleration sensor, and detects the physical movement of the smartphone 200 in accordance with instructions from the main control unit 220. By detecting the physical movement of the smartphone 200, the direction of movement or acceleration of the smartphone 200 is detected. The detection result is output to the main control unit 220.

[0156] The power supply unit 216 supplies power stored in a battery (not shown) to each unit of the smartphone 200 in accordance with instructions from the main control unit 220 .

[0157] The main control unit 220 includes a microprocessor, operates according to the control program and control data stored in the storage unit 212, and controls all the units of the smartphone 200. The microprocessor of the main control unit 220 has the same functions as the system control unit 11. The main control unit 220 also includes a mobile communication control function that controls all the units of the communication system to perform voice communication or data communication via the wireless communication unit 210, and an application processing function.

[0158] The application processing function is realized by the main control unit 220 operating in accordance with the application software stored in the storage unit 212. Examples of the application processing function include an infrared communication function that controls the external input / output unit 213 to perform data communication with a connected device, an email function that sends and receives emails, and a web browsing function that views web pages.

[0159] The main control unit 220 also has an image processing function for displaying video on the display input unit 204 based on image data (still image or moving image data) such as received data or downloaded streaming data.

[0160] The image processing function refers to a function in which the main control unit 220 decodes the image data, performs image processing on the decoded result, and displays the image on the display input unit 204 .

[0161] Furthermore, the main control unit 220 executes display control for the display panel 202 and operation detection control for detecting user operations via the operation unit 207 and the operation panel 203 .

[0162] By executing display control, the main control unit 220 displays software keys such as icons or scroll bars for starting application software, or displays a window for creating an e-mail.

[0163] The scroll bar refers to a software key for receiving an instruction to move the displayed portion of an image, such as a large image that cannot fit in the display area of ​​the display panel 202 .

[0164] In addition, by executing operation detection control, the main control unit 220 detects user operations through the operation unit 207, accepts operations on the above icons and input of character strings into the input field of the above window through the operation panel 203, or accepts requests to scroll the displayed image through the scroll bar.

[0165] Furthermore, by executing operation detection control, the main control unit 220 determines whether the operation position on the operation panel 203 is an overlapping portion (display area) that overlaps the display panel 202 or an outer edge portion (non-display area) that does not overlap the display panel 202, and is equipped with a touch panel control function that controls the sensitive area of ​​the operation panel 203 or the display position of the software key.

[0166] The main control unit 220 can also detect a gesture operation on the operation panel 203 and execute a preset function in response to the detected gesture operation.

[0167] Gesture operation is not a simple touch operation as in the past, but rather refers to an operation of drawing a path with a finger or the like, specifying multiple positions simultaneously, or combining these to draw a path for at least one of multiple positions.

[0168] The camera unit 208 includes the lens device 40, the image sensor 5, and the digital signal processing unit 17 shown in FIG.

[0169] The captured image data generated by the camera unit 208 can be stored in the storage unit 212 or output via the external input / output unit 213 or the wireless communication unit 210 .

[0170] In the smartphone 200 shown in FIG. 27, the camera unit 208 is mounted on the same surface as the display input unit 204, but the mounting position of the camera unit 208 is not limited to this, and it may be mounted on the back surface of the display input unit 204.

[0171] The camera unit 208 can also be used for various functions of the smartphone 200. For example, an image acquired by the camera unit 208 can be displayed on the display panel 202, or an image from the camera unit 208 can be used as one of the operation inputs for the operation panel 203.

[0172] Furthermore, when the GNSS receiving unit 214 detects a position, it can also detect the position by referring to an image from the camera unit 208. Furthermore, it can also refer to an image from the camera unit 208 to determine the optical axis direction of the camera unit 208 of the smartphone 200 or determine the current usage environment without using a triaxial acceleration sensor or by using the image in combination with a triaxial acceleration sensor. Of course, the image from the camera unit 208 can also be used in application software.

[0173] In addition, image data of still images or videos can be added with location information acquired by the GNSS receiving unit 214, audio information acquired by the microphone 206 (which may be converted to text information by the main control unit, etc.), posture information acquired by the motion sensor unit 215, etc., and stored in the memory unit 212 or output via the external input / output unit 213 or wireless communication unit 210.

[0174] Although various embodiments have been described above, it goes without saying that the present invention is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present invention. Furthermore, the components of the above embodiments may be combined in any manner as long as they do not deviate from the spirit of the invention.

[0175] This application is based on a Japanese patent application (Patent Application No. 2024-046434) filed on March 22, 2024, the contents of which are incorporated herein by reference.

[0176] REFERENCE SIGNS LIST 1 imaging lens 2 aperture 4 lens control unit 5 imaging element 8 lens driving unit 9 aperture driving unit 11 system control unit 14, 207 operation unit 15 memory control unit 16 memory 17 digital signal processing unit 20 external memory control unit 21 storage medium 22 display device 22a display controller 22b display surface 24 control bus 25 data bus 40 lens device 60 imaging surface 61, 61fb, 61fg, 61fr pixel 61, 61B, 61G, 61R first pixel 61FB, 61FG, 61FR, 61FYd, 61FXl, 61FXr, 61FYu first type pixel 61F second pixel 61FL second type pixel 61FP polarization pixel 61Fb, 61Fg, 61Fr third type pixel 62 Pixel row 63 Drive circuit 64 Signal processing circuit 64A, 64C Second conversion circuit 64B, 64D First conversion circuit 100 Digital camera 100A Main body 200 Smartphone 201 Housing 202 Display panel 203 Operation panel 204 Display input unit 205 Speaker 206 Microphone 208 Camera unit 210 Wireless communication unit 211 Call unit 212 Memory unit 213 External input / output unit 214 GNSS receiving unit 215 Motion sensor unit 216 Power supply unit 217 Internal memory unit 218 External memory unit 220 Main control unit P1, P1a, P1b, PX First pixel array P2, PY Second pixel array

Claims

1. An imaging element in which a first pixel array is repeatedly arranged in a first direction and a second direction, wherein M is a natural number greater than or equal to 3 and N is a natural number greater than or equal to 3, the first pixel array is configured by arranging M pixels in the first direction and N pixels in the second direction, the first pixel array includes a first pixel included in a second pixel array configured by arranging M-1 pixels in the first direction and N-1 pixels in the second direction, and a second pixel other than the first pixel, and the first pixel and the second pixel have different functions.

2. An imaging element according to claim 1, wherein the second pixels are arranged consecutively in at least one of the first direction and the second direction.

3. An imaging element according to claim 2, wherein the second pixels include first-type pixels having a pixel structure different from that of the first pixels.

4. An imaging device according to claim 3, wherein the first type pixels include pixels for detecting a phase difference.

5. An imaging element according to claim 4, wherein the phase difference detection pixels are arranged in a position other than the position at which the pixels of the second pixel array are arranged in the second direction.

6. An imaging element according to claim 4, wherein the phase difference detection pixels are arranged in a position other than the position at which the pixels of the second pixel array are arranged in the first direction.

7. An imaging device according to claim 4, wherein the pixel for detecting a phase difference has a microlens having a shape different from that of the first pixel.

8. An imaging device according to claim 7, wherein the phase difference detection pixels share a microlens.

9. An imaging device according to claim 3, wherein the first type pixels include polarization pixels for acquiring a polarization image.

10. An imaging device according to claim 3, wherein the first type pixels include pixels having an aperture area different from that of the first pixels.

11. An imaging element according to claim 10, wherein the pixel having an aperture area different from that of the first pixel has an aperture area smaller than that of the first pixel.

12. An imaging device according to claim 2, wherein the second pixels include second-type pixels having spectral characteristics different from those of the first pixels.

13. An imaging device according to claim 12, wherein the second type pixels include pixels that have a luminance filter or do not have a filter.

14. An imaging device according to claim 2, wherein the second pixels include third-type pixels having sensitivity characteristics different from those of the first pixels.

15. An imaging element according to claim 14, wherein the first pixels include a plurality of types of pixels corresponding to a plurality of colors, and the third type of pixels include a plurality of types of pixels corresponding to the plurality of colors.

16. An imaging device according to claim 15, wherein the third type pixels have lower sensitivity than the first type pixels.

17. An imaging element according to claim 1, wherein the first pixels include a plurality of types of pixels corresponding to a plurality of colors, and the second pixels include a plurality of types of pixels corresponding to the plurality of colors, and the imaging element comprises: a first conversion circuit that converts an analog signal output from the first pixels into a digital signal; and a second conversion circuit that converts an analog signal output from the plurality of types of pixels of the second pixels into a digital signal, wherein the first conversion circuit and the second conversion circuit have different noise characteristics.

18. An imaging element according to claim 1, comprising: a first conversion circuit that converts a pixel signal output from the first pixel into a digital signal; and a second conversion circuit that converts a pixel signal output from the second pixel into a digital signal, wherein the first conversion circuit and the second conversion circuit have different signal conversion speeds.

19. An imaging device according to claim 3, wherein the first type pixels include at least one of pixels for distance measurement and pixels for detecting changes in luminance.

20. An imaging device according to claim 1, wherein the arrangement pattern of the first pixels in the second pixel array is a pattern that allows imaging of a color image.

21. An imaging device comprising an imaging element according to any one of claims 1 to 20 and a processor.

22. An imaging device according to claim 21, wherein the processor controls to change the exposure time between at least some of the second pixels and the first pixels.

23. An imaging device according to claim 22, wherein the arrangement pattern of at least a portion of the second pixels is a pattern that allows imaging of a color image.

24. An imaging device according to claim 23, wherein the arrangement pattern of the first pixels in the second pixel array is a pattern that allows imaging of a color image.

Images