Image capture control device, image capturing device, image capturing element, image capture control method, and image capture control program

Abstract

Provided are an image capture control device, an image capturing device, an image capturing element, an image capture control method and an image capture control program.　The image capture control device is provided with a processor that controls the image capturing element, wherein: the image capturing element includes a plurality of pixels arranged in a first direction and a second direction; the plurality of pixels include pixels corresponding to a plurality of colors, and include a first pixel group in which a converting unit for converting electric charges into voltage is shared, and N pixels corresponding to the same color are arranged adjacent to each other in the first direction, where N is a natural number of 2 or more; the pixels corresponding to the same color are not arranged adjacent to each other in the second direction; and the processor performs first control to add together the electric charges of the first pixel group using the converting unit on the basis of imaging conditions, and to convert the added electric charges into a voltage and read out the voltage.

Description

Imaging control device, imaging device, imaging element, imaging control method, and imaging control program

[0001] The technology of the present disclosure relates to an imaging control device, an imaging device, an imaging element, an imaging control method, and an imaging control program.

[0002] In Patent Document 1, a first set consisting of first green pixels and a second set consisting of second green pixels are arranged in a checkerboard pattern in a pixel array portion. A first microlens is arranged on the first green pixel, and a second microlens is arranged on the second green pixel. The first microlens is shared by the first green pixels, and the second microlens is shared by the second green pixels. An imaging element is described.

[0003] Patent Document 2 describes an image sensor having a pixel array including a plurality of pixels arranged in row and column directions, and a readout circuit that generates image data and phase data based on sensing signals received from the pixel array.

[0004] Patent Document 3 describes a solid-state imaging device in which a plurality of pixel cells including three types of pixel cells, namely, a G pixel cell including a photoelectric conversion portion for detecting green light, an R pixel cell including a photoelectric conversion portion for detecting red light, and a B pixel cell including a photoelectric conversion portion for detecting blue light, are arranged two-dimensionally.

[0005] Japanese Patent Application Laid-Open No. 2016-139988, Japanese Patent Application Laid-Open No. 2024-14855, International Publication No. 2013 / 145888

[0006] The technology of the present disclosure is as follows.

[0007] (1) An imaging control device comprising a processor for controlling an image sensor, wherein the image sensor includes a plurality of pixels arranged in a first direction and a second direction, the plurality of pixels include pixels corresponding to a plurality of colors, and also includes a first pixel group in which N pixels corresponding to the same color are arranged adjacently in the first direction, sharing a conversion unit that converts charge to voltage, and in the second direction, the pixels corresponding to the same color are not arranged adjacently, and the processor performs a first control based on imaging conditions, which involves adding the charges of the first pixel group using the conversion unit, converting the added charge to a voltage, and reading it out.

[0008] (2) The imaging control device described in (1), wherein, in the second direction, pixels corresponding to the same color and having the same function are not arranged next to each other.

[0009] (3) An imaging control device as described in (1) or (2), wherein the imaging conditions include a frame rate.

[0010] (4) An imaging control device as described in (3), wherein the processor performs the first control when the frame rate is higher than the first threshold.

[0011] (5) An imaging control device according to (4), wherein the processor reads out the charge of the first pixel group by converting it to a voltage without adding it in the conversion unit when the frame rate is less than or equal to the first threshold.

[0012] (6) An imaging control device according to any one of (3) to (5), wherein the imaging conditions further include the brightness of the subject.

[0013] (7) An imaging control device according to (6), wherein the processor performs the first control when the frame rate is higher than the first threshold and the brightness of the subject is less than or equal to the second threshold.

[0014] (8) An imaging control device according to (7), wherein the processor reads out the charge of the first pixel group by converting it to a voltage without adding it in the conversion unit when the frame rate is higher than the first threshold and the brightness of the subject is brighter than the second threshold.

[0015] (9) An imaging control device according to any one of (1) to (8), wherein the processor performs a second control to convert the charge of some of the pixels in the first pixel group into a voltage using the conversion unit and read it out, and determines the pixel from which to read the charge in the second control based on the position of the first pixel group.

[0016] (10) An imaging control device according to any one of (1) to (9), wherein the processor performs a third control which averages and reads out the charges of N pixels of the same color that are arranged in the second direction.

[0017] (11) An imaging control device according to (10), wherein in the third control, when the N pixels to be averaged and the pixels arranged between them are grouped, at least one pixel is arranged between the groups, and the processor, when performing the third control, converts the charge of the pixels arranged between the groups into a voltage using the conversion unit and reads it out.

[0018] (12) An imaging control device according to (10) or (11), wherein the processor switches the position of the pixels to be averaged on a frame-by-frame basis in the third control.

[0019] (13) An imaging control device according to any one of (10) to (12), wherein the processor switches the corresponding color of the pixels to be averaged on a frame-by-frame basis in the third control.

[0020] (14) An imaging control device according to any one of (10) to (13), wherein the processor performs a decimation control in which it does not read out the charge of the pixels arranged between the N pixels that are to be averaged when performing the third control.

[0021] (15) An imaging control device according to (14), wherein the pixels from which charge is not read out by the above-mentioned decimation control include pixels that have a different function from the pixels that are averaged by the above-mentioned third control.

[0022] (16) An imaging control device as described in (15), wherein pixels with different functions include pixels that have a brightness filter or pixels that do not have a filter.

[0023] (17) An imaging control device according to (15) or (16), wherein the pixels with different functions include pixels for phase difference detection.

[0024] (18) An imaging control device according to any one of (15) to (17), wherein the pixels averaged by the third control include those corresponding to multiple colors, and the pixels with different functions include pixels corresponding to colors different from the multiple colors.

[0025] (19) An imaging control device according to any one of (15) to (18), wherein the pixels with different functions include pixels with different sensitivities than the pixels that are averaged by the third control.

[0026] (20) An imaging control device according to any one of (15) to (19), wherein pixels with different functions include pixels for acquiring noise in the dark.

[0027] (21) An imaging control device according to any one of (15) to (20), wherein n is a natural number of 3 or more, and m is a natural number of 4 or more and a value different from n, the processor is capable of operating in a mode in which charge reading is not performed at a rate of 1 for n pixels arranged in the second direction, and a mode in which charge reading is not performed at a rate of 1 for m pixels arranged in the second direction, and pixels with different functions are provided at a rate of 1 for n × m pixels arranged in the second direction.

[0028] (22) An imaging control device according to any one of (14) to (21), wherein in the third control, when the N pixels to be averaged and the pixels arranged between them are grouped, at least one pixel is arranged between the groups, and the processor, when performing the third control, performs a control to convert the charge of the pixels arranged between the groups into a voltage using the conversion unit and read it out.

[0029] (23) An imaging device comprising an imaging control device described in any one of (1) to (22), and the image sensor.

[0030] (24) An image sensor comprising a plurality of pixels arranged in a first direction and a second direction, wherein the plurality of pixels comprises pixels corresponding to a plurality of colors, N is a natural number of 2 or more, and N pixels that share a conversion unit for converting charge to voltage and that correspond to the same color are arranged adjacent to each other in the first direction, in the second direction the pixels that correspond to the same color and have a first function are not arranged adjacent to each other, and pixels that have a second function different from the first function are arranged at predetermined intervals in the second direction.

[0031] (25) An imaging control method for controlling an image sensor, wherein the image sensor includes a plurality of pixels arranged in a first direction and a second direction, the plurality of pixels include pixels corresponding to a plurality of colors, and includes a first pixel group in which N pixels corresponding to the same color are arranged adjacently in the first direction, sharing a conversion unit that converts charge to voltage, and in the second direction, the pixels corresponding to the same color are not arranged adjacently, and a first control is performed in which the charge of the first pixel group is added in the conversion unit based on imaging conditions, and the added charge is converted to a voltage and read out.

[0032] (26) An imaging control program for controlling an image sensor, wherein the image sensor includes a plurality of pixels arranged in a first direction and a second direction, the plurality of pixels include pixels corresponding to a plurality of colors, and includes a first pixel group in which N pixels corresponding to the same color are arranged adjacently in the first direction, sharing a conversion unit that converts charge to voltage, and in the second direction, the pixels corresponding to the same color are not arranged adjacently, and the imaging control program causes a processor to execute a first control step in which, based on imaging conditions, the charge of the first pixel group is added by the conversion unit, the added charge is converted to voltage and read out.

[0033] Figure 1 is a diagram showing the schematic configuration of a digital camera 100, which is one embodiment of an imaging device. Figure 2 is a schematic plan view showing the schematic configuration of the image sensor 5 shown in Figure 1. Figure 3 is a schematic diagram showing a partially enlarged view of the imaging surface 60 of the image sensor 5 shown in Figure 2. Figure 4 is a schematic plan view showing the schematic configuration of pixels 61 in the image sensor 5 shown in Figure 2. Figure 5 is a schematic cross-sectional view of the pixel 61 along line A-A shown in Figure 4. Figure 6 is a schematic diagram showing an example of sharing of floating diffusion 61D. Figure 7 is a schematic diagram for explaining the full pixel readout mode. Figure 8 is a schematic diagram for explaining the full pixel readout mode. Figure 9 is a schematic diagram for explaining the full pixel readout mode. Figure 10 is a schematic diagram for explaining the full pixel readout mode. Figure 11 is a schematic diagram for explaining the horizontal summation readout mode. Figure 12 is a schematic diagram for explaining the horizontal summation readout mode. Figure 13 is a schematic diagram for explaining the horizontal decimation readout mode. Figure 14 is a schematic diagram for explaining the horizontal decimation readout mode. Figure 15 is a flowchart illustrating the operation of the system control unit 11. Figure 16 is a schematic diagram illustrating an example of changing the position of the pixel 61 that reads out the pixel signal based on the positions of the RG pixel group and the GB pixel group. Figure 17 is a schematic diagram illustrating an example of changing the position of the pixel 61 that reads out the pixel signal based on the positions of the RG pixel group and the GB pixel group. Figure 18 is a schematic diagram showing a modified configuration of the floating diffusion 61D. Figure 19 is a schematic diagram illustrating the averaging readout mode. Figure 20 is a schematic diagram illustrating the decimated averaging readout mode. Figure 21 is a schematic diagram illustrating a first modified configuration of the decimated averaging readout mode. Figure 22 is a schematic diagram illustrating a second modified configuration of the decimated averaging readout mode. Figure 23 is a schematic diagram illustrating a third modified configuration of the decimated averaging readout mode. Figure 24 is a schematic diagram showing a modified pixel arrangement of the image sensor 5. Figure 25 is a schematic diagram showing another modified pixel arrangement of the image sensor 5. Figure 26 is a schematic diagram showing a first configuration example of the functional pixels 61F in the image sensor 5 shown in Figure 24. Figure 27 is a schematic diagram showing a modified example of the image sensor 5 shown in Figure 26.Figure 28 is a schematic diagram showing a second configuration example of the functional pixels 61F in the image sensor 5 shown in Figure 24. Figure 29 is a schematic diagram showing a third configuration example of the functional pixels 61F in the image sensor 5 shown in Figure 24. Figure 30 is a schematic diagram showing a fourth configuration example of the functional pixels 61F in the image sensor 5 shown in Figure 24. Figure 31 is a schematic diagram showing a fifth configuration example of the functional pixels 61F in the image sensor 5 shown in Figure 24. Figure 32 is a schematic diagram showing a modified version of the image sensor 5 shown in Figure 31. Figure 33 is a schematic diagram showing another modified version of the pixel arrangement of the image sensor 5. Figure 34 shows the external appearance of the smartphone 200. Figure 35 is a block diagram showing the configuration of the smartphone 200 shown in Figure 34.

[0034] Figure 1 is a diagram showing the schematic configuration of a digital camera 100, which is one embodiment of an imaging device. The digital camera 100 shown in Figure 1 comprises a lens device 40 having an imaging lens 1, an aperture 2, a lens drive unit 8 that drives the imaging lens 1, an aperture drive unit 9 that drives the aperture 2, and a lens control unit 4 that controls the lens drive unit 8 and the aperture drive unit 9, and a main body 100A.

[0035] The main unit 100A includes an image sensor 5, a system control unit 11 that provides overall control of the entire electrical control system of the digital camera 100, an operation unit 14, a display device 22, a memory 16 including RAM (Random Access Memory) and ROM (Read-only memory), a memory control unit 15 that controls data storage in the memory 16 and data reading from the memory 16, a digital signal processing unit 17, and an external memory control unit 20 that controls data storage in the storage medium 21 and data reading from the storage medium 21.

[0036] The lens device 40 may be detachable from the main body 100A, or it may be integrated with the main body 100A. The imaging lens 1 may include at least one of a focus lens and a zoom lens that is movable in the optical axis direction.

[0037] The focusing lens is a lens used to adjust the focus of the imaging optical system, which includes the imaging lens 1 and the aperture 2, and is composed of a single lens or multiple lenses. When the focusing lens moves in the optical axis direction, the position of the principal point of the focusing lens (hereinafter also referred to as the focusing lens position) changes along the optical axis direction, thereby changing the focal position on the subject side. In addition, a liquid lens that can change the position of its principal point in the optical axis direction by electrical control may be used as the focusing lens.

[0038] A zoom lens is a lens used to change the focal length of an imaging optical system, which includes an imaging lens 1 and an aperture 2. It consists of a single lens or multiple lenses. The zoom magnification is changed by moving the zoom lens along the optical axis.

[0039] The lens control unit 4 of the lens device 40 controls the lens drive unit 8 based on the lens drive signal transmitted from the system control unit 11 to change the focus lens position and zoom lens position. The lens control unit 4 of the lens device 40 controls the aperture drive unit 9 based on the drive control signal transmitted from the system control unit 11 to change the aperture amount (F number) of the aperture 2.

[0040] The image sensor 5 captures an image of a subject through an imaging optical system that includes an imaging lens 1 and an aperture 2. The image sensor 5 has an imaging surface 60 (see Figure 2) in which multiple pixels are arranged in two dimensions, and the imaging optical system converts the image of the subject formed on this imaging surface 60 into an image signal using these multiple pixels and outputs it.

[0041] The image sensor 5 may be, for example, a CMOS (complete metal-oxide semiconductor) image sensor or a CCD (charge coupled device) image sensor. The following example describes a case where the image sensor 5 is a CMOS image sensor.

[0042] The system control unit 11 provides overall control of the digital camera 100 and performs various processes such as controlling the image sensor 5 or the lens device 40. The system control unit 11 and the memory 16 constitute the image capture control device.

[0043] In this embodiment, each process (each control) of the system control unit 11 is executed by an arbitrary computer. Also, an arbitrary computer may execute these processes by a processor, a program, or a combination thereof. An arbitrary computer may be a general-purpose computer, a computer for a specific purpose, a system such as a workstation, or other hardware elements capable of executing a program.

[0044] The processor may be constituted by one or more hardware, and the type of hardware is not limited. For example, the processor may be constituted by hardware such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a programmable logic device such as an FPGA (Field Programmable Gate Array), an application-specific circuit for executing specific processes such as an ASIC (Application Specific Integrated Circuit), a GPU (Graphic Processing Unit), or an NPU (Neural Processing Unit). Also, the processor has each part (Unit) or each means for executing various processes in this embodiment. Further, the type of hardware may be a combination of different types of hardware. When a plurality of hardware are configured to execute one or more processes of a certain processor, the plurality of hardware may exist in physically separate devices from each other, or may exist in the same device. Also, in any embodiment, the order of each process by the processor is not limited to the order described above, and may be changed as appropriate. Note that the hardware is constituted by an electric circuit (circuitry) combining circuit elements such as semiconductor elements.

[0045] Furthermore, this embodiment may be implemented by hardware, software, firmware, microcode, or a combination thereof. Software, firmware, and microcode are composed of programs. Also, the program may be, for example, a group of program modules, and each function may be realized by a processor configured to execute each function. The program may be program code or a plurality of code segments stored in one or more non-transitory computer-readable media (such as storage media or other storage). The program may be divided and stored in a plurality of non-transitory computer-readable media existing in physically separate devices. The program code or code segment may represent any combination of procedures, functions, subprograms, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. The program code or code segment may be connected to other code segments or hardware circuits by transmitting and receiving information, data, arguments, parameters, or the content of memory.

[0046] The system control unit 11 drives the imaging device 5 and the lens device 40, and outputs the subject image captured through the imaging optical system of the lens device 40 as an image signal. The image signal output from the imaging device 5 is processed by the digital signal processing unit 17 to generate imaging image data that is suitable for display on the display device 22 or for storage in the storage medium 21.

[0047] An instruction signal from the user is input to the system control unit 11 through the operation unit 14. The operation unit 14 includes a touch panel integrated with the display surface 22b and various buttons and the like.

[0048] The display device 22 includes a display surface 22b composed of an organic EL (electroluminescence) panel or a liquid crystal panel, etc., and a display controller 22a that controls the display on the display surface 22b.

[0049] The memory control unit 15, the digital signal processing unit 17, the external memory control unit 20, and the display controller 22a are interconnected by a control bus 24 and a data bus 25, and are controlled by commands from the system control unit 11.

[0050] Figure 2 is a schematic plan view showing the general configuration of the image sensor 5 shown in Figure 1. The image sensor 5 comprises an imaging surface 60 in which multiple pixel rows 62, each consisting of multiple pixels 61 arranged in the row direction X, are arranged in the column direction Y which intersects (orthogonal in the example shown) the row direction X; a drive circuit 63 for driving the pixels 61 arranged on the imaging surface 60; and a signal processing circuit 64 for processing the pixel signals read out from each pixel 61 of the pixel rows 62 arranged on the imaging surface 60 to the signal lines. The row direction X constitutes the first direction. The column direction Y constitutes the second direction.

[0051] The pixel signal read from pixel 61 to the signal line is an analog signal. The signal processing circuit 64 includes a converter that converts the analog signal to a digital signal. The pixel signal read from pixel 61 is digitally converted by the signal processing circuit 64 and output as a digital signal to the outside of the image sensor 5.

[0052] Figure 3 is a schematic diagram showing a partially enlarged view of the imaging surface 60 of the image sensor 5 shown in Figure 2. The multiple pixels 61 arranged on the imaging surface 60 include pixels corresponding to each of multiple (three in this embodiment) colors (wavelength bands).

[0053] Specifically, the imaging surface 60 is provided with pixels 61R corresponding to red (blocks labeled "R" in the figure), pixels 61G corresponding to green (blocks labeled "G" in the figure), and pixels 61B corresponding to blue (blocks labeled "B" in the figure).

[0054] The imaging surface 60 has a Bayer-like arrangement of pixels: a red pixel group Gr, which includes two adjacent pixels 61R in the row direction X; a green pixel group Gg, which includes two adjacent pixels 61G in the row direction X; and a blue pixel group Gb, which includes two adjacent pixels 61B in the row direction X. The red pixel group Gr, the green pixel group Gg, and the blue pixel group Gb each constitute a first pixel group. The red pixel group Gr, the green pixel group Gg, and the blue pixel group Gb may each include three or more pixels 61 of the same color that are adjacent in the row direction X.

[0055] On the imaging surface 60, RG pixel rows, in which red pixel group Gr and green pixel group Gg are arranged alternately in the row direction X, and GB pixel rows, in which green pixel group Gg and blue pixel group Gb are arranged alternately in the row direction X, are arranged alternately in the column direction Y. Each pixel 61 on the imaging surface 60 receives light of the corresponding color and outputs a pixel signal corresponding to the amount of that light.

[0056] Multiple pixels 61 are arranged such that, in the column direction Y, pixels 61 corresponding to the same color and function are not adjacent to each other. That is, next to pixel 61R in the column direction Y, there is a pixel 61G which has the same function as pixel 61R but corresponds to a different color. Next to pixel 61B in the column direction Y, there is a pixel 61G which has the same function as pixel 61B but corresponds to a different color. Next to pixel 61G in the column direction Y, there is a pixel 61R or pixel 61B which has the same function as pixel 61G but corresponds to a different color. In this specification, "same" in color and function means substantially the same and may include errors in common sense.

[0057] All pixels 61 in the red pixel group Gr, the green pixel group Gg, and the blue pixel group Gb have the same function. Specifically, the pixel signals output from pixels 61 in the red pixel group Gr, the green pixel group Gg, and the blue pixel group Gb serve as the reference for each pixel data in the captured image data, and can be said to have an imaging function. The imaging function constitutes the first function.

[0058] As will be discussed later, the pixels in the image sensor 5 can have various functions other than imaging, such as phase difference detection used for focus control, dark noise detection used for correcting dark shading, dynamic range expansion, or improved color reproduction. Each of these functions—phase difference detection, dark noise acquisition, dynamic range expansion, and improved color reproduction—constitutes a second function distinct from the first function.

[0059] Figure 4 is a schematic plan view showing the general configuration of pixels 61 in the image sensor 5 shown in Figure 2. Figure 5 is a schematic cross-sectional view of pixel 61 along line A-A shown in Figure 4.

[0060] As shown in Figure 4, the pixel 61 includes a photoelectric conversion unit 61A, a charge holding unit 61H, a charge transfer unit 61C, a floating diffusion 61D, and a readout circuit 61E.

[0061] The photoelectric conversion unit 61A receives light that has passed through the imaging optical system of the lens device 40 and generates and stores an electric charge corresponding to the amount of light received. The photoelectric conversion unit 61A is composed of a photodiode or the like. The charge holding unit 61H is composed of impurity regions within the semiconductor substrate.

[0062] The charge transfer unit 61C controls the height of the potential barrier between the photoelectric conversion unit 61A and the charge holding unit 61H. The charge transfer unit 61C is composed of an impurity region in the semiconductor substrate and an electrode formed above this impurity region.

[0063] For example, the potential barrier is raised at the start of exposure to begin exposure of the photoelectric conversion unit 61A, and the potential barrier is lowered at the end of exposure. Through this control, the charge generated and accumulated in the photoelectric conversion unit 61A during the exposure period is transferred to the charge holding unit 61H almost simultaneously with the end of exposure. Subsequently, the potential barrier is returned to its original value, thereby holding the charge generated in the photoelectric conversion unit 61A during the exposure period in the charge holding unit 61H.

[0064] Furthermore, the photoelectric conversion unit 61A can be provided with a potential gradient that decreases toward the charge holding unit 61H. When this configuration is adopted, by lowering the potential barrier at the exposure start timing, the charge generated in the photoelectric conversion unit 61A by exposure is not accumulated there but moves to the charge holding unit 61H. Then, when the potential barrier is raised at the exposure end timing, the exposure period ends and the charge generated in the photoelectric conversion unit 61A during the exposure period is held in the charge holding unit 61H.

[0065] In this way, by controlling the charge transfer unit 61C, the charge generated in the photoelectric conversion unit 61A can be held in the charge holding unit 61H.

[0066] The floating diffusion 61D is for converting electric charge into a signal, and the charge from the charge holding unit 61H is transferred to it. The floating diffusion 61D constitutes the conversion unit.

[0067] The readout circuit 61E is a circuit that reads a signal corresponding to the potential of the floating diffusion 61D as a pixel signal onto the signal line 65. The readout circuit 61E is driven by the drive circuit 63.

[0068] As shown in Figure 5, a P-well layer 71 is formed on the surface of the N-type substrate 70, and a photoelectric conversion section 61A is formed on the surface portion of the P-well layer 71.

[0069] The photoelectric conversion unit 61A is composed of an N-type impurity layer 73 and a P-type impurity layer 74 formed thereon. The semiconductor substrate is composed of an N-type substrate 70 and a P-well layer 71.

[0070] On the surface of the P-well layer 71, a charge-holding portion 61H made of an N-type impurity layer is formed, slightly separated from the photoelectric conversion portion 61A.

[0071] A transfer electrode 76 is formed above the region 75 of the P-well layer 71 between the charge holding portion 61H and the photoelectric conversion portion 61A, via an oxide film.

[0072] The region 75 and the transfer electrode 76 constitute the charge transfer section 61C. In the example shown in Figure 4, the transfer electrode 76 is formed above the charge holding section 61H, but the transfer electrode 76 only needs to be formed at least above the region 75.

[0073] By controlling the potential of the transfer electrode 76 to form a channel in region 75, the aforementioned potential barrier can be lowered. The potential of the transfer electrode 76 is controlled by the drive circuit 63.

[0074] On the surface of the P-well layer 71, a floating diffusion 61D made of an N-type impurity layer is formed, slightly separated from the charge-holding portion 61H.

[0075] A readout electrode 72 is formed above the P-well layer 71 between the charge-holding portion 61H and the floating diffusion 61D, via an oxide film.

[0076] By controlling the potential of the readout electrode 72 and forming a channel in the region between the charge holding unit 61H and the floating diffusion 61D, the charge from the charge holding unit 61H can be transferred to the floating diffusion 61D. The potential of the readout electrode 72 is controlled by the drive circuit 63.

[0077] The readout circuit 61E consists of a reset transistor 77 for resetting the potential of the floating diffusion 61D, an output transistor 78 for converting the potential of the floating diffusion 61D into a pixel signal and outputting it, and a selection transistor 79 for selectively reading the pixel signal output from the output transistor 78 to the signal line 65. The configuration of the readout circuit is an example and is not limited to this. The readout circuit 61E may also be shared by multiple pixels 61.

[0078] A light-shielding film is provided on the pixel 61, and areas other than the photoelectric conversion unit 61A are shielded from light by this light-shielding film.

[0079] The structure of the pixel 61 shown in Figures 4 and 5 is just one example and is not limited thereto.

[0080] The drive circuit 63 shown in Figure 2 independently drives the transfer electrode 76, read electrode 72, and read circuit 61E of each pixel 61 to perform actions such as resetting the photoelectric conversion unit 61A (discharging the charge accumulated in the photoelectric conversion unit 61A), holding the charge generated in the photoelectric conversion unit 61A in the charge holding unit 61H, and reading out the pixel signal to the signal line 65 according to the charge held in the charge holding unit 61H.

[0081] The photoelectric conversion unit 61A is reset by forming a channel in the semiconductor substrate below the transfer electrode 76 and a channel in the semiconductor substrate below the read electrode 72, and then discharging the charge from the floating diffusion 61D using the reset transistor 77.

[0082] The signal processing circuit 64 shown in Figure 2 performs correlated double sampling on the pixel signals read from each pixel 61 of the pixel row 62 to the signal line 65, converts the pixel signals after correlated double sampling into digital signals, and outputs them to the data bus 25 (see Figure 1). The signal processing circuit 64 is controlled by the system control unit 11. The digital signal processing unit 17 performs signal processing such as demosaicing and gamma correction on the group of pixel signals output from the image sensor 5 to the data bus 25 to generate image data.

[0083] In the image sensor 5, the floating diffusion 61D is shared by multiple pixels 61. Figure 6 is a schematic diagram showing an example of the sharing of the floating diffusion 61D. In this embodiment, as shown in Figure 6, one floating diffusion 61D is shared by four pixels 61 (hereinafter also referred to as the RG pixel group) that constitute the red pixel group Gr and the green pixel group Gg adjacent to each other in the column direction Y. In addition, the floating diffusion 61D is shared by four pixels 61 (hereinafter also referred to as the GB pixel group) that constitute the green pixel group Gg and the blue pixel group Gb adjacent to each other in the column direction Y.

[0084] The system control unit 11 controls the drive circuit 63 to drive the image sensor 5 in multiple drive modes.

[0085] The multiple drive modes include a full-pixel readout mode in which pixel signals are read individually from each of the four pixels 61 in both the RG pixel group and the GB pixel group.

[0086] The multiple drive modes include a horizontal summation readout mode in which, for each of the RG pixel group and GB pixel group, the charges of two pixels 61 aligned in the row direction X are added by a floating diffusion 61D shared by the two pixels 61, and the added charges are converted into a pixel signal and read out. The control that drives the image sensor 5 in the horizontal summation readout mode constitutes the first control.

[0087] The multiple drive modes include a horizontal decimation drive mode in which, for each of the RG pixel group and GB pixel group, pixel signals are individually read from one of two pixels 61 aligned in the row direction X and the adjacent pixel 61 in the column direction Y of that pixel 61, and pixel signals are not read from the other two pixels 61. The control that drives the image sensor 5 in the horizontal decimation readout mode constitutes the second control.

[0088] Figures 7 to 10 are schematic diagrams illustrating the all-pixel readout mode. In the all-pixel readout mode, the system control unit 11 first transfers the charge held in the charge holding unit 61H of the upper left pixel 61 in each of the RG pixel group and GB pixel group to the floating diffusion 61D, as shown in Figure 7, and converts this charge into a pixel signal for readout. The white arrows in the figures schematically show the state of charge movement. The same applies to the following figures.

[0089] Next, the system control unit 11 resets the floating diffusion 61D, and then, as shown in Figure 8, transfers the charge held in the charge holding unit 61H of the upper right pixel 61 in each of the RG pixel group and GB pixel group to the floating diffusion 61D, converts this charge into a pixel signal, and reads it out.

[0090] Next, the system control unit 11 resets the floating diffusion 61D, and then, as shown in Figure 9, transfers the charge held in the charge holding unit 61H of the lower left pixel 61 in each of the RG pixel group and GB pixel group to the floating diffusion 61D, converts this charge into a pixel signal, and reads it out.

[0091] Finally, after resetting the floating diffusion 61D, the system control unit 11 transfers the charge held in the charge holding unit 61H of the lower right pixel 61 in each of the RG pixel group and GB pixel group to the floating diffusion 61D, as shown in Figure 10, and converts this charge into a pixel signal for reading.

[0092] The pixel signals read out in this manner are, for example, averaged by digital processing of the pixel signals of the same color components read out from each pixel group, such as the red pixel group Gr, the green pixel group Gg, and the blue pixel group Gb. This makes it possible to generate image data with a high signal-to-noise ratio (SNR). Furthermore, since the pixel signal is read out from one pixel 61 using one floating diffusion 61D, the pixel signal is less likely to saturate even when the brightness of the subject is high.

[0093] Figure 11 is a schematic diagram illustrating the horizontal summation readout mode. In the horizontal summation readout mode, the system control unit 11 first transfers the charges held in the charge holding units 61H of the two upper pixels 61 in each of the RG pixel group and GB pixel group to the floating diffusion 61D and adds them together, then converts the added charges into a pixel signal and reads it out.

[0094] Next, the system control unit 11 resets the floating diffusion 61D, and then, as shown in Figure 12, transfers the charges held in the charge holding units 61H of the two lower pixels 61 in each of the RG pixel group and GB pixel group to the floating diffusion 61D, adds them up, converts the added charges into a pixel signal, and reads it out.

[0095] The pixel signals read out in this manner are, for example, averaged by digital processing of the pixel signals of the same color components read out from each pixel group, such as the red pixel group Gr, the green pixel group Gg, and the blue pixel group Gb. This makes it possible to generate image data with a high signal-to-noise ratio (SNR). Furthermore, the pixel signal readout speed can be doubled compared to the all-pixel readout mode. It is also possible to omit the above averaging, in which case, even when the brightness of the subject is low, the signal value of the pixel signal is increased, enabling high-sensitivity imaging.

[0096] Figure 13 is a schematic diagram illustrating the horizontal decimation readout mode. In the horizontal decimation readout mode, the system control unit 11 first transfers the charge held in the charge holding unit 61H of the upper left pixel 61 in each of the RG pixel group and GB pixel group to the floating diffusion 61D, as shown in Figure 13, and converts this charge into a pixel signal for readout.

[0097] Next, the system control unit 11 resets the floating diffusion 61D, and then, as shown in Figure 14, transfers the charge held in the charge holding unit 61H of the lower left pixel 61 in each of the RG pixel group and GB pixel group to the floating diffusion 61D, converts this charge into a pixel signal, and reads it out.

[0098] In horizontal decimation readout mode, the readout speed of the pixel signal can be doubled compared to full pixel readout mode. Also, compared to horizontal addition readout mode, since the pixel signal is read out using one floating diffusion 61D from one pixel 61, the pixel signal is less likely to saturate even when the subject is bright.

[0099] Thus, the full-pixel readout mode, the horizontal addition readout mode, and the horizontal decimation readout mode each have their own advantages. For this reason, it is preferable for the system control unit 11 to drive the image sensor 5 in an appropriate mode from the full-pixel readout mode, the horizontal addition readout mode, and the horizontal decimation readout mode based on the imaging conditions.

[0100] Figure 15 is a flowchart illustrating the operation of the system control unit 11.

[0101] The system control unit 11 determines whether the frame rate setting is higher than the first threshold TH1 (step S1). If the frame rate setting is less than or equal to the first threshold TH1 (step S1: NO), the system control unit 11 drives the image sensor 5 in full pixel readout mode (step S3). That is, the charge of all pixels 61 is converted to voltage and read out without being added by the floating diffusion 61D. The full pixel readout mode has the slowest pixel signal readout speed of the three modes. Therefore, when the frame rate is low, the frame rate can be set in this mode. In addition, the number of pixel data in the captured image data can be maximized.

[0102] If the frame rate setting is higher than the first threshold TH1 (step S1: YES), the system control unit 11 drives the image sensor 5 in either the horizontal decimation readout mode or the horizontal summation readout mode based on the brightness of the subject. Both the horizontal summation readout mode and the horizontal decimation readout mode have a faster pixel signal readout speed than the full pixel readout mode. Therefore, when the frame rate is high, adopting these modes can enable imaging at a high frame rate.

[0103] If the determination in step S1 is YES, the system control unit 11 determines whether the brightness of the subject is less than or equal to the second threshold TH2 (step S2).

[0104] The system control unit 11 drives the image sensor 5 in horizontal decimation readout mode (step S4) when the brightness of the subject exceeds the second threshold TH2 (step S2: NO). That is, in each of the RG pixel group and GB pixel group, some pixels 61 are read out by converting the charge into voltage without adding it using floating diffusion 61D.

[0105] The horizontal decimation readout mode is less prone to pixel signal saturation than the horizontal summation readout mode. Therefore, when the brightness of the subject exceeds the second threshold TH2, the quality of the captured image data can be improved by driving the image sensor 5 in the horizontal decimation readout mode, which is less prone to pixel signal saturation.

[0106] The system control unit 11 drives the image sensor 5 in horizontal summation readout mode (step S3) when the brightness of the subject is below the second threshold TH2 (step S2: YES). In this way, when the subject is dark, driving the image sensor 5 in horizontal summation readout mode can improve the signal-to-noise ratio of the captured image data.

[0107] The system control unit 11 may, for example, switch the drive mode based on the brightness of the subject, regardless of the frame rate. For example, the system control unit 11 may drive the image sensor 5 in horizontal summation readout mode when the brightness of the subject is below a second threshold, and drive the image sensor 5 in full pixel readout mode or horizontal decimation readout mode when the brightness of the subject exceeds the second threshold.

[0108] The system control unit 11 may, for example, switch the drive mode based on the frame rate, regardless of the brightness of the subject. For example, the system control unit 11 may drive the image sensor 5 in horizontal summation readout mode or horizontal decimation readout mode when the frame rate is higher than a first threshold, and drive the image sensor 5 in full pixel readout mode when the frame rate is less than or equal to the first threshold.

[0109] In the explanation of the horizontal decimation readout mode shown in Figures 13 and 14, it was assumed that pixel signals were read individually from the two leftmost pixels 61 in each of the RG and GB pixel groups. However, the positions of the pixels 61 from which the pixel signals are read in each of the RG and GB pixel groups do not need to be fixed across the entire imaging surface 60. For example, the positions of the pixels 61 from which the pixel signals are read may be determined based on the respective positions of the RG and GB pixel groups on the imaging surface 60.

[0110] Figures 16 and 17 are schematic diagrams illustrating an example of changing the position of a pixel 61 that reads out a pixel signal based on the positions of the RG pixel group and the GB pixel group.

[0111] As shown in Figure 16, the system control unit 11 transfers the charge held in the charge holding unit 61H of the upper left pixel 61 to the floating diffusion 61D in each of the RG pixel group and GB pixel group located to the left of the center of the row direction X of the imaging plane 60, and converts this charge into a pixel signal for reading. In addition, the system control unit 11 transfers the charge held in the charge holding unit 61H of the upper right pixel 61 to the floating diffusion 61D in each of the RG pixel group and GB pixel group located to the right of the center of the row direction X of the imaging plane 60, and converts this charge into a pixel signal for reading.

[0112] Next, the system control unit 11 resets the floating diffusion 61D, and as shown in Figure 17, for each of the RG pixel group and GB pixel group located to the left of the center of the row direction X of the imaging plane 60, the system control unit 11 transfers the charge held in the charge holding unit 61H of the lower left pixel 61 to the floating diffusion 61D, converts this charge into a pixel signal, and reads it out. Also, for each of the RG pixel group and GB pixel group located to the right of the center of the row direction X of the imaging plane 60, the system control unit 11 transfers the charge held in the charge holding unit 61H of the lower right pixel 61 to the floating diffusion 61D, converts this charge into a pixel signal, and reads it out.

[0113] Figures 16 and 17 show the direction of subject light from the center of the row direction X of the imaging surface 60 using white arrow LDs. To the left of the center of the row direction X of the imaging surface 60, subject light is directed to the left. To the right of the center of the row direction X of the imaging surface 60, subject light is directed to the right. Of two adjacent pixels 61 in the row direction X, the pixel 61 located downstream with respect to the direction of subject light propagation may experience color mixing from the pixel 61 located upstream.

[0114] In the examples shown in Figures 16 and 17, among the two pixels 61 arranged in the row direction X in each of the RG pixel group and GB pixel group, the pixel signal is read from the pixel 61 located downstream with respect to the direction of light propagation. For each pixel 61 from which the pixel signal is read, there is a corresponding pixel 61 of the same color located upstream of that pixel 61. Therefore, different color components do not mix in the pixel signal, preventing color mixing and improving the image quality of the captured image data.

[0115] In the example shown in Figure 6, four pixels 61 share one floating diffusion 61D, but this is not the only configuration. Figure 18 is a schematic diagram showing a modified configuration of the floating diffusion 61D.

[0116] In the configuration shown in Figure 18, two pixels 61, each constituting the red pixel group Gr, the green pixel group Gg, and the blue pixel group Gb, share one floating diffusion 61D. Compared to the configuration shown in Figure 6, the configuration shown in Figure 18 allows the pixel signal readout speed to be halved in all three modes: full pixel readout mode, horizontal summation readout mode, and horizontal decimation readout mode.

[0117] The driving modes of the image sensor 5 may include an averaging readout mode that performs averaging readout, which averages the charges of multiple pixels 61 corresponding to the same color arranged in the column direction Y and reads them out as a pixel signal. The control that causes the image sensor 5 to perform averaging readout constitutes a third control.

[0118] Furthermore, the driving modes of the image sensor 5 may include a decimated averaging readout mode that performs both the averaged readout described above and decimated readout that does not read out pixel signals corresponding to the charge of the pixels 61 placed between the plurality of pixels 61 that are to be averaged.

[0119] Figure 19 is a schematic diagram illustrating the averaging readout mode. In the averaging readout mode, the system control unit 11 averages the charges of two pixels 61 that are at the same position in the row direction X in two adjacent RG pixel rows in the column direction Y and reads them out as a pixel signal. The system control unit 11 also averages the charges of two pixels 61 that are at the same position in the row direction X in two adjacent GB pixel rows in the column direction Y and reads them out as a pixel signal. In Figure 19, the two pixels 61 whose pixel signals are averaged are connected by a straight line with black circles at both ends. The same applies to subsequent figures.

[0120] There are two methods for averaging the charges of two pixels 61 and reading them out as a pixel signal: one method of obtaining an averaged pixel signal by analog processing inside the image sensor 5, and another method of obtaining an averaged pixel signal by digital processing outside the image sensor 5.

[0121] For example, the charge of two pixels 61 that are at the same position in the row direction X is simultaneously transferred to the floating diffusion 61D of each pixel 61, and the voltage converted by each floating diffusion 61D is simultaneously read out to a signal line common to these two pixels 61 and converted into a digital signal. In this way, a pixel signal that is an average of the voltages corresponding to the respective charges of the two pixels 61 is output from the image sensor 5.

[0122] For example, after reading out the pixel signal from each pixel 61 in full-pixel readout mode, the averaged pixel signal can be obtained by digitally averaging the pixel signals output from two pixels 61 that are at the same position in the row direction X.

[0123] In this way, by driving the image sensor 5 in averaged readout mode, the readout speed of the pixel signal can be increased compared to the full pixel readout mode, for example, when obtaining an averaged pixel signal through analog processing within the image sensor 5. By combining the horizontal summation readout mode or horizontal decimation readout mode with the averaged readout mode, the readout speed of the pixel signal can be increased even further, enabling imaging at even higher frame rates. In addition, the signal-to-noise ratio (SNR) can be improved by averaging the pixel signal.

[0124] Figure 20 is a schematic diagram illustrating the decimation-averaging readout mode. In the decimation-averaging readout mode, the system control unit 11 averages the charges of two pixels 61 that are at the same position in the row direction X in two adjacent RG pixel rows in the column direction Y, and reads them out as a pixel signal. The system control unit 11 also does not read out pixel signals from pixels 61 in the GB pixel row between these two RG pixel rows. This control of not reading out pixel signals from pixels 61 in the GB pixel row between two RG pixel rows constitutes decimation control. In Figure 20, pixel rows 62 from which pixel signals are not read out are shown with dashed lines. The same applies to subsequent figures.

[0125] In the driving example shown in Figure 20, when two adjacent RG pixel rows to be averaged in the column direction Y and a GB pixel row whose pixel signal is not read between these two RG pixel rows are grouped together, one GB pixel row is placed between adjacent groups in the column direction Y. In the decimation averaging readout mode, the system control unit 11 reads the pixel signal individually from each pixel 61 contained in the GB pixel row between these groups.

[0126] According to the driving example shown in Figure 20, since there are pixel rows 62 from which pixel signals are not read out, the readout speed can be further increased. In addition, because there is a mix of pixel rows where the pixel signals are averaged in the column direction Y and pixel rows from which the pixel signals are read out individually, false resolution and jaggies can be suppressed.

[0127] Furthermore, if two adjacent RG pixel rows to be averaged in the column direction Y are grouped together with a GB pixel row from which the pixel signal between these two RG pixel rows is not read, then GR pixel rows, RG pixel rows, and GB pixel rows may be arranged between adjacent groups in the column direction Y. In this case, the pixel signals can be read from each of the GR pixel rows, RG pixel rows, and GB pixel rows.

[0128] Figure 21 is a schematic diagram illustrating the first modified example of the decimation-averaging readout mode. In the example shown in Figure 21, the positions of the two RG pixel rows targeted for pixel signal averaging are changed compared to the example in Figure 20. In the example in Figure 20, pixel signal averaging is performed on the top RG pixel row and the RG pixel row immediately below it, and on the third RG pixel row from the top and the RG pixel row immediately below it. On the other hand, in the example in Figure 21, pixel signal averaging is performed on the second RG pixel row from the top and the RG pixel row immediately below it, and on the fourth RG pixel row from the top and the RG pixel row immediately below it.

[0129] In the decimated average readout mode, the system control unit 11 switches between the drive shown in Figure 20 and the drive shown in Figure 21 for each frame. For example, in even frames, the image sensor 5 is driven as shown in Figure 20, and in odd frames, the image sensor 5 is driven as shown in Figure 21.

[0130] By switching between the drive shown in Figure 20 and the drive shown in Figure 21 on a frame-by-frame basis, false resolution and jaggies can be further suppressed.

[0131] Figure 22 is a schematic diagram illustrating a second modified example of the decimation-averaging readout mode. In the example shown in Figure 22, the types of the two pixel rows targeted for pixel signal averaging are changed compared to the example in Figure 20. In the example in Figure 20, pixel signal averaging is performed on two RG pixel rows, but in the example in Figure 22, pixel signal averaging is performed on two GB pixel rows that are adjacent in the column direction Y.

[0132] In the decimated average readout mode, the system control unit 11 switches between the drive shown in Figure 20 and the drive shown in Figure 22 for each frame. For example, in even frames, the image sensor 5 is driven as shown in Figure 20, and in odd frames, the image sensor 5 is driven as shown in Figure 22.

[0133] In this way, by switching between the drive shown in Figure 20 and the drive shown in Figure 22 on a frame-by-frame basis, false colors can be suppressed.

[0134] Figure 23 is a schematic diagram illustrating a third variation of the decimated averaging readout mode. In the example shown in Figure 23, the system control unit 11 divides two adjacent RG pixel rows in the column direction Y and the GB pixel row between these two RG pixel rows into a first group, and two adjacent GB pixel rows in the column direction Y and the GB pixel row between these two GB pixel rows into a second group, and reads out the pixel signals for each of these groups. There is no pixel row 62 between the first group and the second group.

[0135] As shown in Figure 23, the system control unit 11 averages the charges of two pixels 61 that are at the same position in the row direction X in the two RG pixel rows constituting the first group and reads them out as a pixel signal. The system control unit 11 does not read out a pixel signal from the pixels 61 of the GB pixel row constituting the first group.

[0136] Furthermore, the system control unit 11 averages the charges of two pixels 61 that are at the same position in the row direction X in the two GB pixel rows that constitute the second group and reads them out as a pixel signal. In addition, the system control unit 11 does not read out pixel signals from the pixels 61 of the RG pixel rows that constitute the second group.

[0137] As shown in the driving example in Figure 23, the number of pixel rows 62 from which pixel signals are not read out is greater than in the driving examples shown in Figures 20 to 22, allowing for a higher frame rate. Also, compared to the driving example in Figure 19, there are no pixels 61 between the two pixels 61 that are subject to averaging that are averaged with other pixels 61. Therefore, even when imaging a subject with a specific pattern of horizontal lines arranged vertically, it is possible to prevent the horizontal lines from becoming double lines and improve the image quality.

[0138] Figure 24 is a schematic diagram showing a modified pixel arrangement of the image sensor 5. In the image sensor 5 shown in Figure 24, in the configuration shown in Figure 3, each pixel 61 in the GB pixel row between two adjacent RG pixel rows in the column direction Y is replaced with a functional pixel 61F (a block labeled "F" in the figure) which has a different function from pixels 61R, 61G, and 61B, and each pixel 61 in the RG pixel row between two adjacent GB pixel rows in the column direction Y is replaced with a functional pixel 61F. A pixel row 62 in which functional pixels 61F are arranged in the row direction X is referred to as an F pixel row.

[0139] The image sensor 5 shown in Figure 24 comprises a group of RG pixel rows, F pixel rows, and RG pixel rows arranged in this order in the column direction Y, and a group of GB pixel rows, F pixel rows, and GB pixel rows arranged in this order in the column direction Y, with these groups arranged alternately in the column direction Y. In the case of the image sensor 5 shown in Figure 24, the floating diffusion 61D may or may not be shared among multiple functional pixels 61F in the F pixel row.

[0140] With the image sensor 5 shown in Figure 24, for example, by performing the decimation and averaging readout shown in Figure 23, pixel signals can be read out only from pixels 61 other than the functional pixels 61F. It is also possible to read out pixel signals only from the functional pixels 61F. Therefore, while making the image sensor 5 multifunctional, pixel signals can be read out at high speed from the imaging pixels 61, thereby achieving multifunctionality and a high frame rate.

[0141] Figure 25 is a schematic diagram showing another modified example of the pixel arrangement of the image sensor 5. The image sensor 5 shown in Figure 25 has a configuration suitable for, for example, performing the decimation-averaging readout shown in Figure 20.

[0142] In the image sensor 5 shown in Figure 25, each pixel 61 in the GB pixel row between the two pixel rows 62 that are averaged as shown in Figure 20 is replaced with a functional pixel 61F. In the configuration shown in Figure 25, the arrangement of the floating diffusion 61D is preferably as shown in Figure 18. Even with the configuration shown in Figure 25, multi-functionality and a high frame rate can be achieved. Furthermore, the configuration shown in Figure 25 allows for an increase in the number of pixels 61 used for imaging compared to the configuration shown in Figure 24.

[0143] Figure 26 is a schematic diagram showing a first configuration example of the functional pixel 61F in the image sensor 5 shown in Figure 24. In the configuration shown in Figure 26, the functional pixel 61F includes a phase difference detection pixel 610R corresponding to red, a phase difference detection pixel 610G corresponding to green, and a phase difference detection pixel 610B corresponding to blue.

[0144] A functional pixel 61F sandwiched between two adjacent pixels 61R in the column direction Y is a phase difference detection pixel 610G. A functional pixel 61F sandwiched between two adjacent pixels 61B in the column direction Y is a phase difference detection pixel 610G. A functional pixel 61F sandwiched between two adjacent pixels 61G in the column direction Y is either a phase difference detection pixel 610R or a phase difference detection pixel 610B.

[0145] According to the image sensor 5 shown in Figure 26, a phase difference can be detected based on the pixel signals read from the phase difference detection pixels 610R, 610G, and 610B, and the focus lens can be controlled based on this phase difference.

[0146] Figure 27 is a schematic diagram showing a modified version of the image sensor 5 shown in Figure 26. In the image sensor 5 shown in Figure 26, a functional pixel 61F sandwiched between two adjacent pixels 61R in the column direction Y is a phase difference detection pixel 610R. A functional pixel 61F sandwiched between two adjacent pixels 61B in the column direction Y is a phase difference detection pixel 610B. A functional pixel 61F sandwiched between two adjacent pixels 61G in the column direction Y is a phase difference detection pixel 610G.

[0147] According to the image sensor 5 shown in Figure 27, a phase difference can be detected based on the pixel signals read from the phase difference detection pixels 610R, 610G, and 610B, and the focus lens can be controlled based on this phase difference.

[0148] Figure 28 is a schematic diagram showing a second configuration example of the functional pixel 61F in the image sensor 5 shown in Figure 24. In the configuration shown in Figure 28, the functional pixel 61F is a phase difference detection pixel 610W.

[0149] In the image sensor 5 shown in Figure 28, each pixel 61 other than the phase difference detection pixel 610W has a spectral filter above the photoelectric conversion unit 61A that transmits the color corresponding to each pixel 61. On the other hand, the phase difference detection pixel 610W has either a luminance filter above the photoelectric conversion unit 61A or does not have a spectral filter.

[0150] Luminance filters have spectral characteristics correlated with the luminance component of light, and include ND (Neutral Density) filters, transparent filters, white filters, or gray filters. Luminance filters can transmit light of more wavelengths than spectral filters. Therefore, by using a luminance filter instead of a spectral filter, the accuracy of phase difference detection using the pixel signal of the phase difference detection pixel 610W can be improved.

[0151] The phase difference detection pixel 610W may be configured without a spectral filter and a luminance filter. This configuration also improves the accuracy of phase difference detection.

[0152] Figure 29 is a schematic diagram showing a third configuration example of the functional pixel 61F in the image sensor 5 shown in Figure 24. In the configuration shown in Figure 29, the functional pixel 61F is a pixel 610S for acquiring noise in dark conditions.

[0153] The pixel 610S for acquiring noise in the dark is configured such that, for example, no opening is provided in the light-shielding film formed above the photoelectric conversion unit 61A, so that subject light does not enter the photoelectric conversion unit 61A.

[0154] According to the image sensor 5 shown in Figure 29, the quality of the captured image data can be improved by correcting the dark shading of the pixel signal read out from the pixel 61 which has an imaging function, based on the pixel signal (dark noise) read out from the dark noise acquisition pixel 610S.

[0155] Figure 30 is a schematic diagram showing a fourth configuration example of the functional pixels 61F in the image sensor 5 shown in Figure 24. In the configuration shown in Figure 30, the functional pixels 61F include a yellow pixel 610Y corresponding to yellow, which is a different color from pixels 61R, 61G, and 61B, and a cyan pixel 610C corresponding to cyan, which is a different color from pixels 61R, 61G, and 61B.

[0156] According to the image sensor 5 shown in Figure 30, by using the pixel signals read out from the yellow pixel 610Y and the cyan pixel 610C, the color reproducibility of the image data generated from the pixel signals read out from the pixel 61 which has an imaging function can be improved.

[0157] Figure 31 is a schematic diagram showing a fifth configuration example of the functional pixels 61F in the image sensor 5 shown in Figure 24. In the configuration shown in Figure 31, the functional pixels 61F are provided as a low-sensitivity pixel 610r corresponding to red and having lower sensitivity than pixel 61R, a low-sensitivity pixel 610g corresponding to green and having lower sensitivity than pixel 61G, and a low-sensitivity pixel 610b corresponding to blue and having lower sensitivity than pixel 61B.

[0158] In the image sensor 5 shown in Figure 31, a functional pixel 61F sandwiched between two adjacent pixels 61R in the column direction Y is a low-sensitivity pixel 610r. A functional pixel 61F sandwiched between two adjacent pixels 61B in the column direction Y is a low-sensitivity pixel 610b. A functional pixel 61F sandwiched between two adjacent pixels 61G in the column direction Y is a low-sensitivity pixel 610g.

[0159] According to the image sensor 5 shown in Figure 31, by using the pixel signals read out from the low-sensitivity pixels 610r, 610g, and 610b, the dynamic range of the image data generated from the pixel signals read out from the pixel 61 which has imaging function can be expanded.

[0160] Figure 32 is a schematic diagram showing a modified version of the image sensor 5 shown in Figure 31. In the image sensor 5 shown in Figure 32, a functional pixel 61F sandwiched between two adjacent pixels 61R in the column direction Y is a low-sensitivity pixel 610g. A functional pixel 61F sandwiched between two adjacent pixels 61B in the column direction Y is a low-sensitivity pixel 610g. A functional pixel 61F sandwiched between two adjacent pixels 61G in the column direction Y is either a low-sensitivity pixel 610r or a low-sensitivity pixel 610b.

[0161] According to the image sensor 5 shown in Figure 32, by using the pixel signals read out from the low-sensitivity pixels 610r, 610g, and 610b, the dynamic range of the image data generated from the pixel signals read out from the pixel 61 which has imaging function can be expanded.

[0162] Figure 33 is a schematic diagram showing another modified pixel arrangement of the image sensor 5. The image sensor 5 shown in Figure 33 has a configuration suitable for use in combination with, for example, the decimated average readout shown in Figure 20 and the decimated average readout shown in Figure 23.

[0163] When n is a natural number greater than or equal to 3, and m is a natural number greater than or equal to 4 and different from n, the decimation-averaging readout shown in Figure 20 can be described as a mode in which charge is not read out at a rate of one out of m (m=4) pixels 61 arranged in the column direction Y. Similarly, the decimation-averaging readout shown in Figure 23 can be described as a mode in which charge is not read out at a rate of one out of n (n=3) pixels 61 arranged in the column direction Y.

[0164] In the image sensor 5 shown in Figure 33, one functional pixel 61F is provided for every n × m (12) pixels 61 arranged in the column direction Y. With this image sensor 5, both the decimated average readout shown in Figure 20 and the decimated average readout shown in Figure 23 can be performed, enabling optimal driving depending on the situation.

[0165] Next, we will describe the configuration of a smartphone, which is another embodiment of the imaging device of the present invention.

[0166] Figure 34 shows the external appearance of the smartphone 200. The smartphone 200 shown in Figure 34 has a flat casing 201, and one side of the casing 201 is equipped with a display input unit 204 which is an integrated display panel 202 as a display unit and an operation panel 203 as an input unit.

[0167] Furthermore, such a housing 201 includes a speaker 205, a microphone 206, an operating unit 207, and a camera unit 208. However, the configuration of the housing 201 is not limited to this; for example, a configuration in which the display unit and input unit are independent, or a configuration having a folding structure or a sliding mechanism, can also be adopted.

[0168] Figure 35 is a block diagram showing the configuration of the smartphone 200 shown in Figure 34.

[0169] As shown in Figure 35, the main components of the smartphone include a wireless communication unit 210, a display input unit 204, a call unit 211, an operation unit 207, a camera unit 208, a storage unit 212, an external input / output unit 213, a GNSS (Global Navigation Satellite System) receiving unit 214, a motion sensor unit 215, a power supply unit 216, and a main control unit 220.

[0170] Furthermore, the smartphone 200 has a primary function of providing wireless communication via a base station device BS (not shown) and a mobile communication network NW (not shown).

[0171] The wireless communication unit 210 performs wireless communication with base station equipment BS connected to the mobile communication network NW, in accordance with instructions from the main control unit 220. This wireless communication is used to send and receive various file data such as voice data and image data, email data, etc., and to receive web data or streaming data, etc.

[0172] The display input unit 204 is a so-called touch panel that, under the control of the main control unit 220, displays images (still images and moving images) or text information to visually convey information to the user and detects user operations on the displayed information, and comprises a display panel 202 and an operation panel 203.

[0173] The display panel 202 uses an LCD (Liquid Crystal Display), an OLED (Organic Electro-Luminescence Display), or the like as a display device.

[0174] The operation panel 203 is a device that detects one or more coordinates operated by the user's finger or stylus, and is positioned so as to be visible on the display surface of the display panel 202. When this device is operated by the user's finger or stylus, it outputs a detection signal generated by the operation to the main control unit 220. The main control unit 220 then detects the operation position (coordinates) on the display panel 202 based on the received detection signal.

[0175] As shown in Figure 35, in the smartphone 200, which is illustrated as one embodiment of the imaging device of the present invention, the display panel 202 and the operation panel 203 are integrated to form a display input unit 204, but the operation panel 203 is positioned to completely cover the display panel 202.

[0176] When such an arrangement is adopted, the operation panel 203 may also have a function to detect user operations in areas outside the display panel 202. In other words, the operation panel 203 may have a detection area for the overlapping portion that overlaps with the display panel 202 (hereinafter referred to as the display area) and a detection area for the outer edge portion that does not overlap with the display panel 202 (hereinafter referred to as the non-display area).

[0177] The size of the display area and the size of the display panel 202 may be made to match perfectly, but it is not necessary for them to match. Furthermore, the operation panel 203 may have two sensitive areas: an outer edge portion and an inner portion. The width of the outer edge portion is designed appropriately according to the size of the housing 201, etc.

[0178] Furthermore, the position detection methods used in the control panel 203 include matrix switch methods, resistive film methods, surface acoustic wave methods, infrared methods, electromagnetic induction methods, and capacitive methods, and any of these methods can be adopted.

[0179] The communication unit 211 is equipped with a speaker 205 or a microphone 206, and converts the user's voice input through the microphone 206 into audio data that can be processed by the main control unit 220 and outputs it to the main control unit 220, or decodes audio data received by the wireless communication unit 210 or the external input / output unit 213 and outputs it from the speaker 205.

[0180] Furthermore, as shown in Figure 34, for example, the speaker 205 can be mounted on the same side as the display input unit 204, and the microphone 206 can be mounted on the side of the housing 201.

[0181] The operation unit 207 is a hardware key using a key switch or the like, which receives instructions from the user. For example, as shown in Figure 34, the operation unit 207 is mounted on the side of the casing 201 of the smartphone 200 and is a push-button type switch that turns on when pressed with a finger or the like, and turns off when the finger is released due to a restoring force such as a spring.

[0182] The memory unit 212 stores the control program and control data of the main control unit 220, application software, address data associated with the name or telephone number of the communication partner, sent and received email data, web data downloaded through web browsing, downloaded content data, and also temporarily stores streaming data. The memory unit 212 is composed of an internal memory unit 217 built into the smartphone and an external memory unit 218 with a removable external memory slot.

[0183] The internal storage units 217 and external storage units 218 that constitute the storage unit 212 are implemented using storage media such as flash memory type, hard disk type, multimedia card micro type, card type memory (for example, MicroSD® memory), RAM (Random Access Memory), and ROM (Read Only Memory).

[0184] The external input / output unit 213 serves as an interface for all external devices connected to the smartphone 200, and is intended for direct or indirect connection to other external devices via communication (e.g., Universal Serial Bus (USB), IEEE 1394, Bluetooth®, RFID (Radio Frequency Identification), Infrared Data Association (IrDA)®, UWB (Ultra Wideband)®, ZigBee®, etc.) or network (e.g., Ethernet®, Wireless LAN (Local Area Network), etc.).

[0185] External devices that can be connected to the smartphone 200 include, for example, wired / wireless headsets, wired / wireless external chargers, wired / wireless data ports, memory cards connected via card sockets, SIM (Subscriber Identity Module Card) / UIM (User Identity Module Card) cards, external audio / video equipment connected via audio / video I / O (Input / Output) terminals, wirelessly connected external audio / video equipment, wired / wireless connected smartphones, wired / wireless connected personal computers, wired / wireless connected personal computers, earphones, etc.

[0186] The external input / output unit 213 can transmit data received from such external devices to the various internal components of the smartphone 200, or enable data from inside the smartphone 200 to be transmitted to external devices.

[0187] The GNSS receiving unit 214 receives GNSS signals transmitted from GNSS satellites ST1 to STn in accordance with instructions from the main control unit 220, performs positioning calculation processing based on the received GNSS signals, and detects the position of the smartphone 200, consisting of latitude, longitude, and altitude. When the GNSS receiving unit 214 can acquire position information from the wireless communication unit 210 or the external input / output unit 213 (for example, wireless LAN), it can also use that position information to detect the position.

[0188] The motion sensor unit 215 includes, for example, a three-axis acceleration sensor, and detects the physical movement of the smartphone 200 according to the instructions of the main control unit 220. By detecting the physical movement of the smartphone 200, the direction of movement or acceleration of the smartphone 200 is detected. The detection results are output to the main control unit 220.

[0189] The power supply unit 216 supplies power stored in a battery (not shown) to each part of the smartphone 200 according to the instructions of the main control unit 220.

[0190] The main control unit 220 is equipped with a microprocessor and operates according to the control program and control data stored in the memory unit 212, and comprehensively controls each part of the smartphone 200. The microprocessor of the main control unit 220 has the same functions as the system control unit 11. In addition, the main control unit 220 is equipped with a mobile communication control function that controls each part of the communication system for voice communication or data communication via the wireless communication unit 210, and an application processing function.

[0191] The application processing function is realized by the operation of the main control unit 220 according to the application software stored in the memory unit 212. Examples of application processing functions include an infrared communication function that controls the external input / output unit 213 to communicate data with a counterpart device, an email function that sends and receives emails, and a web browsing function that displays web pages.

[0192] Furthermore, the main control unit 220 is equipped with image processing functions, such as displaying video on the display input unit 204 based on image data (still image or moving image data) such as received data or downloaded streaming data.

[0193] The image processing function refers to the function in which the main control unit 220 decodes the image data, applies image processing to the decoded result, and displays the image on the display input unit 204.

[0194] Furthermore, the main control unit 220 performs display control for the display panel 202 and operation detection control to detect user operations through the operation unit 207 and the operation panel 203.

[0195] By executing the display control, the main control unit 220 displays software keys such as icons or scroll bars for launching application software, or displays a window for composing an email.

[0196] A scroll bar is a software key that accepts instructions to move the display portion of an image, such as a large image that does not fit within the display area of ​​the display panel 202.

[0197] Furthermore, by executing operation detection control, the main control unit 220 detects user operations through the operation unit 207, accepts operations on the icons and input of strings into the input fields of the window through the operation panel 203, or accepts requests to scroll the displayed image through the scroll bar.

[0198] Furthermore, by executing operation detection control, the main control unit 220 determines whether the operation position on the operation panel 203 is in the overlapping portion (display area) that overlaps with the display panel 202 or in the outer edge portion (non-display area) that does not overlap with the display panel 202, and has a touch panel control function that controls the display position of the sensitive area of ​​the operation panel 203 or the software key.

[0199] Furthermore, the main control unit 220 can detect gesture operations on the operation panel 203 and execute pre-set functions in response to the detected gesture operations.

[0200] Gesture control refers to operations that differ from traditional simple touch operations, such as drawing a path with a finger or other object, specifying multiple locations simultaneously, or combining these to draw a path from at least one of multiple locations.

[0201] The camera unit 208 includes the lens device 40, image sensor 5, and digital signal processing unit 17 shown in Figure 1.

[0202] The image data generated by the camera unit 208 can be stored in the storage unit 212 or output via the external input / output unit 213 or the wireless communication unit 210.

[0203] In the smartphone 200 shown in Figure 35, the camera unit 208 is mounted on the same side as the display input unit 204, but the mounting position of the camera unit 208 is not limited to this, and it may also be mounted on the back of the display input unit 204.

[0204] Furthermore, the camera unit 208 can be used for various functions of the smartphone 200. For example, images acquired by the camera unit 208 can be displayed on the display panel 202, or images from the camera unit 208 can be used as one of the inputs for the operation panel 203.

[0205] Furthermore, when the GNSS receiver 214 detects position, it can also detect position by referring to the image from the camera unit 208. Moreover, by referring to the image from the camera unit 208, it is possible to determine the optical axis direction of the camera unit 208 of the smartphone 200, or to determine the current usage environment, either without using a 3-axis accelerometer or in combination with a 3-axis accelerometer. Of course, the image from the camera unit 208 can also be used within the application software.

[0206] In addition, position information acquired by the GNSS receiver 214, audio information acquired by the microphone 206 (which may be converted to text information by the main control unit, etc.), posture information acquired by the motion sensor unit 215, etc., can be added to still image or video image data and stored in the storage unit 212 or output through the external input / output unit 213 or the wireless communication unit 210.

[0207] Although various embodiments have been described above, it goes without saying that the present invention is not limited to these examples. It is clear to those skilled in the art that various modifications or alterations can be conceived within the scope of the claims, and these will naturally also fall within the technical scope of the present invention. Furthermore, the components in the above embodiments may be combined in any way without departing from the spirit of the invention.

[0208] This application is based on a Japanese patent application (JP 2024-163912) filed on September 20, 2024, the contents of which are incorporated by reference within this application.

[0209] 1 Imaging lens 2 Aperture 4 Lens control unit 5 Image sensor 8 Lens drive unit 9 Aperture drive unit 11 System control unit 14, 207 Operation unit 15 Memory control unit 16 Memory 17 Digital signal processing unit 20 External memory control unit 21 Storage medium 22 Display device 22a Display controller 22b Display surface 24 Control bus 25 Data bus 40 Lens device 60 Imaging surface 61, 61B, 61G, 61R Pixel 61F Functional pixel 61A Photoelectric conversion unit 61C Charge transfer unit 61D Floating diffusion 61E Readout circuit 61H Charge holding unit 62 Pixel row 63 Drive circuit 64 Signal processing circuit 65 Signal line 70 N-type substrate 71 P-well layer 72 Readout electrode 73 N-type impurity layer 74 P-type impurity layer 75 Area 76 Transfer electrode 77 Reset transistor 78 Output transistor 79 Selection transistor 100 Digital camera 100A Main unit 200 Smartphone 201 Housing 202 Display panel 203 Operation panel 204 Display input unit 205 Speaker 206 Microphone 208 Camera unit 210 Wireless communication unit 211 Call unit 212 Memory unit 213 External input / output unit 214 GNSS receiver unit 215 Motion sensor unit 216 Power supply unit 217 Internal memory unit 218 External memory unit 220 Main control unit 610B, 610G, 610R, 610W Pixel for phase difference detection 610C Cyan pixel 610S Pixel for dark noise acquisition 610Y Yellow pixel 610b, 610g, 610r Low-sensitivity pixels

Claims

1. An imaging control device comprising a processor for controlling an image sensor, wherein the image sensor includes a plurality of pixels arranged in a first direction and a second direction, the plurality of pixels include pixels corresponding to a plurality of colors, and includes a first pixel group in which N pixels corresponding to the same color are arranged adjacently in the first direction, sharing a conversion unit that converts charge to voltage, and in the second direction, the pixels corresponding to the same color are not arranged adjacently, and the processor performs a first control based on imaging conditions, which involves adding the charges of the first pixel group using the conversion unit, converting the added charge to a voltage, and reading it out.

2. An imaging control device according to claim 1, wherein, in the second direction, pixels corresponding to the same color and having the same function are not adjacent to each other.

3. An imaging control device according to claim 2, wherein the imaging conditions include a frame rate.

4. An imaging control device according to claim 3, wherein the processor performs the first control when the frame rate is higher than a first threshold.

5. An imaging control device according to claim 4, wherein the processor converts the charge of the first pixel group into a voltage without adding it in the conversion unit and reads it out when the frame rate is less than or equal to the first threshold.

6. An imaging control device according to claim 3, wherein the imaging conditions further include the brightness of the subject.

7. An imaging control device according to claim 6, wherein the processor performs the first control when the frame rate is higher than a first threshold and the brightness of the subject is less than or equal to a second threshold.

8. An imaging control device according to claim 7, wherein the processor converts the charge of the first pixel group into a voltage without adding it in the conversion unit and reads it out when the frame rate is higher than the first threshold and the brightness of the subject is brighter than the second threshold.

9. An imaging control device according to claim 1, wherein the processor performs a second control which converts the charge of some of the pixels in the first pixel group into a voltage using the conversion unit and reads it out, and determines the pixel from which to read the charge in the second control based on the position of the first pixel group.

10. An imaging control device according to claim 1, wherein the processor performs a third control, which averages and reads out the charges of N pixels of the same color that are arranged in the second direction.

11. An imaging control device according to claim 10, wherein in the third control, when the N pixels to be averaged and the pixels arranged between them are grouped, at least one of the pixels is arranged between the groups, and the processor, when performing the third control, converts the charge of the pixels arranged between the groups into a voltage using the conversion unit and reads it out.

12. An imaging control device according to claim 10, wherein the processor switches the position of the pixels to be averaged on a frame-by-frame basis in the third control.

13. An imaging control device according to claim 10, wherein the processor switches the corresponding color of the pixels to be averaged on a frame-by-frame basis in the third control.

14. An imaging control device according to claim 10, wherein the processor performs a decimation control in which it does not read out the charge of the pixels arranged between the N pixels that are to be averaged when performing the third control.

15. An imaging control device according to claim 14, wherein the pixels from which charge is not read out in the decimation control include pixels with a different function from the pixels that are averaged in the third control.

16. An imaging control device according to claim 15, wherein the pixels with different functions include pixels having a brightness filter or pixels without a filter.

17. An imaging control device according to claim 15, wherein the pixels with different functions include pixels for phase difference detection.

18. An imaging control device according to claim 15, wherein the pixels averaged by the third control include those corresponding to a plurality of colors, and the pixels with different functions include pixels corresponding to colors different from the plurality of colors.

19. An imaging control device according to claim 15, wherein the pixels with different functions include pixels with different sensitivities from the pixels that are averaged by the third control.

20. An imaging control device according to claim 15, wherein the pixels with different functions include pixels for acquiring noise in the dark.

21. An imaging control device according to claim 15, wherein n is a natural number of 3 or more, m is a natural number of 4 or more and is different from n, the processor is operable in a mode in which charge reading is not performed at a rate of 1 for n pixels arranged in the second direction, and in a mode in which charge reading is not performed at a rate of 1 for m pixels arranged in the second direction, and the pixels with different functions are provided at a rate of 1 for n × m pixels arranged in the second direction.

22. An imaging control device according to claim 14, wherein, in the third control, when the N pixels to be averaged and the pixels arranged between them are grouped, at least one of the pixels is arranged between the groups, and the processor, when performing the third control, performs a control to convert the charge of the pixels arranged between the groups into a voltage using the conversion unit and read it out.

23. An imaging device comprising an imaging control device according to any one of claims 1 to 22, and the image sensor.

24. An image sensor comprising a plurality of pixels arranged in a first direction and a second direction, wherein the plurality of pixels comprises pixels corresponding to a plurality of colors, N being a natural number of 2 or more, sharing a conversion unit that converts charge to voltage, and N pixels corresponding to the same color being arranged adjacently in the first direction, in the second direction, pixels corresponding to the same color and having a first function are not arranged adjacently, and pixels having a second function different from the first function are arranged at predetermined intervals in the second direction.

25. An imaging control method for controlling an image sensor, wherein the image sensor includes a plurality of pixels arranged in a first direction and a second direction, the plurality of pixels include pixels corresponding to a plurality of colors, and includes a first pixel group in which N pixels corresponding to the same color are arranged adjacently in the first direction, sharing a conversion unit that converts charge to voltage, and in the second direction, the pixels corresponding to the same color are not arranged adjacently, and a first control is performed in which the charge of the first pixel group is added in the conversion unit based on imaging conditions, and the added charge is converted to a voltage and read out.

26. An imaging control program for controlling an image sensor, wherein the image sensor includes a plurality of pixels arranged in a first direction and a second direction, the plurality of pixels include pixels corresponding to a plurality of colors, and includes a first pixel group in which N pixels corresponding to the same color are arranged adjacently in the first direction, sharing a conversion unit that converts charge to voltage, and in the second direction, the pixels corresponding to the same color are not arranged adjacently, and the imaging control program causes a processor to execute a first control step in which, based on imaging conditions, the charge of the first pixel group is added in the conversion unit, the added charge is converted to voltage and read out.

Images