Image-capturing control device, image-capturing device, image-capturing control method, and image-capturing control program

Abstract

Provided are an image-capturing control device, an image-capturing device, an image-capturing control method, and an image-capturing control program.　The image-capturing control device comprises a processor that controls an image-capturing element including a plurality of pixels. The plurality of pixels each include a photoelectric conversion unit, a charge-holding unit that holds charges transferred from the photoelectric conversion unit, and a conversion unit that converts the charges into a signal. The processor performs first control involving causing the charge-holding unit to hold charges generated in the photoelectric conversion unit and ending exposure of the plurality of pixels, performs second control on a first pixel among the plurality of pixels, and performs third control on a second pixel among the plurality of pixels. The second control involves transferring the charges in the charge-holding unit to the conversion unit after a first time elapses from the end of the exposure. The third control involves discharging the charges in the charge-holding unit during a period from the end of the exposure until the first time elapses, and transferring said charges in the charge-holding unit to the conversion unit after the first time elapses from the end of the exposure.

Description

Imaging control device, imaging device, imaging control method, and imaging control program

[0001] The present invention relates to an imaging control device, an imaging device, an imaging control method, and an imaging control program.

[0002] Patent Document 1 describes a solid-state imaging device comprising: a pixel array having a plurality of unit pixels; a drive unit that drives each of the plurality of unit pixels so that the exposure periods are aligned; and a signal processing circuit that processes pixel signals output from each of the plurality of unit pixels to a signal line based on the driving by the drive unit, wherein each of the plurality of unit pixels comprises: a photoelectric conversion unit; a first transfer transistor connected to the photoelectric conversion unit; a memory unit that is connected to the photoelectric conversion unit via the first transfer transistor and holds signal charges generated in the photoelectric conversion unit; a second transfer transistor connected to the memory unit; a storage unit that is connected to the memory unit via the second transfer transistor and stores charges transferred from the memory unit; an amplifying transistor that causes a pixel signal of a voltage value based on the charges stored in the storage unit to appear on the signal line; a first discharge transistor connected to the photoelectric conversion unit; and a second discharge transistor connected to the photoelectric conversion unit.

[0003] Patent Document 2 describes a solid-state imaging device having a normal pixel arranged within a pixel section capable of performing a global shutter, the normal pixel including at least a photoelectric conversion section and a memory section adjacent to the photoelectric conversion section, and a leakage light correction pixel arranged within the pixel section to correct image quality degradation caused by leakage light into the memory section.

[0004] Patent document 3 describes an imaging element that includes a photoelectric conversion unit that converts light into electric charges, a first charge holding unit that holds the electric charges converted in the photoelectric conversion unit, a second charge holding unit that holds electric charges for correcting a signal based on the electric charges held in the first charge holding unit, and a storage unit that accumulates the electric charges held in at least one of the first charge holding unit and the second charge holding unit.

[0005] Japanese Patent Publication No. 2021-125716 Japanese Patent Publication No. 2011-029835 Japanese Patent Publication No. 2024-054607

[0006] This specification describes at least the following:

[0007] (1) An imaging control device comprising a processor that controls an imaging element including a plurality of pixels, wherein the pixels include a photoelectric conversion unit, a charge holding unit that holds charge sent from the photoelectric conversion unit, and a conversion unit that converts the charge into a signal, wherein the processor performs first control to hold charge generated in the photoelectric conversion unit in the charge holding unit to terminate exposure of the plurality of pixels, performs second control on a first pixel of the plurality of pixels, and performs third control on a second pixel of the plurality of pixels, wherein the second control is control to send charge from the charge holding unit to the conversion unit a first time after the end of the exposure, and the third control is control to discharge charge from the charge holding unit between the end of the exposure and the first time after the end of the exposure, and send charge from the charge holding unit to the conversion unit a first time after the end of the exposure.

[0008] (2) The imaging control device according to (1), wherein in the third control, the charge in the charge holding section is discharged immediately after the exposure is completed.

[0009] (3) The imaging control device according to (1) or (2), wherein the first pixel includes pixels corresponding to a plurality of colors, and the second pixel includes pixels corresponding to the plurality of colors.

[0010] (4) The imaging control device according to any one of (1) to (3), wherein the number of the second pixels is smaller than the number of the first pixels.

[0011] (5) The imaging control device according to any one of (1) to (4), wherein the second pixels are arranged periodically.

[0012] (6) The imaging control device according to any one of (1) to (5), wherein all of the plurality of pixels can be set as the first pixel, and all of the plurality of pixels can be set as the second pixel.

[0013] (7) The imaging control device according to (6), wherein the first pixel and the second pixel have the same structure.

[0014] (8) An imaging control device according to any one of (1) to (7), wherein the processor is operable in a first mode in which the first control and the second control are performed on the plurality of pixels, and a second mode in which the processor performs the first control on the plurality of pixels, the second control on the first pixel, and the third control on the second pixel.

[0015] (9) The imaging control device according to (8), wherein the processor operates in either the first mode or the second mode based on imaging conditions, subject conditions, or a drive mode of the imaging element.

[0016] (10) The imaging control device according to (9), wherein the imaging conditions include an exposure time.

[0017] (11) The imaging control device according to (10), wherein the processor operates in the second mode when the exposure time is equal to or less than a time threshold.

[0018] (12) The imaging control device according to any one of (9) to (11), wherein the imaging conditions include imaging sensitivity.

[0019] (13) The imaging control device according to (12), wherein the processor operates in the second mode when the imaging sensitivity is equal to or greater than a sensitivity threshold.

[0020] (14) The imaging control device according to any one of (9) to (13), wherein the subject conditions include brightness of the subject.

[0021] (15) The imaging control device according to (14), wherein the processor operates in the second mode when the brightness of the subject is equal to or greater than a brightness threshold.

[0022] (16) The imaging control device according to any one of (9) to (15), wherein the subject condition includes a movement of the subject.

[0023] (17) The imaging control device according to (16), wherein the processor operates in the second mode when the amount of motion of the subject is equal to or greater than a motion threshold.

[0024] (18) An imaging control device according to any one of (9) to (17), wherein the drive mode includes a thinning mode in which signals are thinned out and read out from the plurality of pixels, or an addition mode in which signals are added up and read out from the plurality of pixels, and the processor operates in the second mode in a mode other than the thinning mode or the addition mode.

[0025] (19) The imaging control device according to any one of (1) to (18), wherein the processor is capable of executing the second control and the third control in a plurality of modes in which the second pixels are arranged differently.

[0026] (20) The imaging control device according to (19), wherein the processor operates in one of the plurality of modes based on imaging conditions, subject conditions, or a drive mode of the imaging element.

[0027] (21) The imaging control device according to (20), wherein the imaging conditions include an exposure time.

[0028] (22) The imaging control device according to (21), wherein the processor operates in the mode in which the number of the second pixels is greater when the exposure time is equal to or less than a time threshold than when the exposure time exceeds the time threshold.

[0029] (23) The imaging control device according to any one of (20) to (22), wherein the imaging conditions include imaging sensitivity.

[0030] (24) The imaging control device according to (23), wherein when the imaging sensitivity is equal to or greater than a sensitivity threshold, the processor operates in the mode in which the number of the second pixels is greater than when the imaging sensitivity is less than the sensitivity threshold.

[0031] (25) The imaging control device according to any one of (20) to (24), wherein the subject conditions include brightness of the subject.

[0032] (26) The imaging control device according to (25), wherein the processor operates in the mode in which the number of the second pixels is greater when the brightness of the subject is equal to or greater than a brightness threshold than when the brightness of the subject is less than the brightness threshold.

[0033] (27) The imaging control device according to any one of (20) to (26), wherein the subject condition includes a movement of the subject.

[0034] (28) The imaging control device according to (27), wherein the processor operates in the mode in which the number of the second pixels is greater when the amount of motion of the subject is equal to or greater than a motion threshold than when the amount of motion of the subject is less than the motion threshold.

[0035] (29) An imaging control device according to any one of (20) to (28), wherein the drive mode includes a thinning mode in which signals are thinned out and read out from the plurality of pixels, or an addition mode in which signals are added up and read out from the plurality of pixels, and the processor operates in the thinning mode or the addition mode in a mode in which the number of the second pixels is smaller than in modes other than the thinning mode or the addition mode.

[0036] (30) An imaging device including: the imaging control device according to any one of (1) to (29); and the imaging element.

[0037] (31) An imaging control method for controlling an imaging element including a plurality of pixels, wherein the pixels include a photoelectric conversion unit, a charge holding unit that holds charges sent from the photoelectric conversion unit, and a conversion unit that converts the charges into signals; the imaging control method performs first control to hold charges generated in the photoelectric conversion unit in the charge holding unit and terminate exposure of the plurality of pixels, performs second control on a first pixel of the plurality of pixels, and performs third control on a second pixel of the plurality of pixels; the second control is control to send charges from the charge holding unit to the conversion unit one hour after the end of the exposure, and the third control is control to discharge charges from the charge holding unit between the end of the exposure and the first hour, and send charges from the charge holding unit to the conversion unit one hour after the end of the exposure.

[0038] (32) An imaging control program for controlling an imaging element including a plurality of pixels, wherein the pixels include a photoelectric conversion unit, a charge holding unit that holds charge sent from the photoelectric conversion unit, and a conversion unit that converts the charge into a signal; the imaging control program causing a computer to execute steps of: performing first control to hold charge generated in the photoelectric conversion unit in the charge holding unit and terminate exposure of the plurality of pixels; performing second control on a first pixel of the plurality of pixels; and performing third control on a second pixel of the plurality of pixels; wherein the second control is control to send charge from the charge holding unit to the conversion unit one hour after the end of the exposure; and the third control is control to discharge charge from the charge holding unit between the end of the exposure and the first hour, and to send charge from the charge holding unit to the conversion unit one hour after the end of the exposure.

[0039] FIG. 1 is a diagram showing the general configuration of a digital camera 100, which is an embodiment of an imaging device. FIG. 2 is a schematic plan view showing the general configuration of the image sensor 5 shown in FIG. 1. FIG. 3 is a schematic view showing a partially enlarged image capturing surface 60 of the image sensor 5 shown in FIG. 2. FIG. 4 is a schematic plan view showing the general configuration of a pixel 61 in the image sensor 5 shown in FIG. 2. FIG. 5 is a schematic cross-sectional view of the pixel 61 shown in FIG. 4 taken along line A-A. FIG. 6 is a timing chart showing the operation of the digital camera 100 shown in FIG. 1 in an image capturing mode. FIG. 7 is a schematic diagram showing an example of the setting of noise detection pixels. FIG. 8 is a timing chart (part 1) illustrating, for example, the driving of noise detection pixels and imaging pixels included in the second pixel row 62 from the top shown in FIG. 7. FIG. 9 is a timing chart (part 2) illustrating the driving of noise detection pixels and imaging pixels included in the second pixel row 62 from the top shown in FIG. 7. FIG. 10 is a diagram showing another example of the setting of noise detection pixels (part 1). Fig. 11 is a diagram showing another example (part 2) of setting noise detection pixels. Fig. 12 shows the appearance of a smartphone 200. Fig. 13 is a block diagram showing the configuration of the smartphone 200 shown in Fig. 12.

[0040] Fig. 1 is a diagram showing the schematic configuration of a digital camera 100, which is an embodiment of an imaging device. The digital camera 100 shown in Fig. 1 includes a lens device 40 having an imaging lens 1, an aperture 2, a lens driver 8 that drives the imaging lens 1, an aperture driver 9 that drives the aperture 2, and a lens controller 4 that controls the lens driver 8 and the aperture driver 9, and a main body 100A.

[0041] The main body 100A includes an image sensor 5, a system control unit 11 that controls the overall electrical control system of the digital camera 100, an operation unit 14, a display device 22, a memory 16 that includes RAM (Random Access Memory) and ROM (Read Only Memory), a memory control unit 15 that controls the storage of data in the memory 16 and the reading of data from the memory 16, a digital signal processing unit 17, and an external memory control unit 20 that controls the storage of data in a storage medium 21 and the reading of data from the storage medium 21.

[0042] The lens device 40 may be detachable from the main body 100A, or may be integrated with the main body 100A. The imaging lens 1 may include at least one of a focus lens and a zoom lens that are movable in the optical axis direction.

[0043] The focus lens is a lens for adjusting the focus of an imaging optical system including the imaging lens 1 and the aperture 2, and is composed of a single lens or multiple lenses. When the focus lens moves in the optical axis direction, the position of the principal point of the focus lens (hereinafter also referred to as the focus lens position) changes along the optical axis, thereby changing the focal position on the subject side. Note that a liquid lens whose principal point position in the optical axis direction can be changed by electrical control may also be used as the focus lens.

[0044] The zoom lens is a lens for changing the focal length of an imaging optical system including an imaging lens 1 and an aperture 2, and is composed of a single lens or multiple lenses. The zoom magnification is changed by moving the zoom lens in the optical axis direction.

[0045] The lens control unit 4 of the lens device 40 controls the lens drive unit 8 based on the lens drive signal transmitted from the system control unit 11 to change the focus lens position and the zoom lens position. The lens control unit 4 of the lens device 40 controls the diaphragm drive unit 9 based on the drive control signal transmitted from the system control unit 11 to change the aperture size (F-number) of the diaphragm 2.

[0046] The image sensor 5 captures an image of a subject through an imaging optical system including an imaging lens 1 and an aperture 2. The image sensor 5 has an imaging surface 60 (see FIG. 2 ) on which a plurality of pixels are arranged two-dimensionally, and converts an image of a subject formed on the imaging surface 60 by the imaging optical system into an image signal using the plurality of pixels and outputs the image signal.

[0047] The imaging element 5 is, for example, a complementary metal-oxide semiconductor (CMOS) image sensor.

[0048] The system control unit 11 performs various processes and controls, and controls the entire digital camera 100. The system control unit 11 and the memory 16 constitute an imaging control device.

[0049] The system control unit 11 controls the image sensor 5 and the lens device 40 to output, as an image signal, an image of a subject captured through the imaging optical system of the lens device 40. The image signal output from the image sensor 5 is processed by the digital signal processing unit 17 to generate captured image data that is data suitable for display on the display device 22 or data suitable for storage in the storage medium 21.

[0050] An instruction signal from the user is input to the system control unit 11 through the operation unit 14. The operation unit 14 includes a touch panel integrated with the display surface 22b, various buttons, and the like.

[0051] The display device 22 includes a display surface 22b configured by an organic EL (electroluminescence) panel, a liquid crystal panel, or the like, and a display controller 22a that controls the display on the display surface 22b.

[0052] The memory control unit 15 , digital signal processing unit 17 , external memory control unit 20 , and display controller 22 a are interconnected by a control bus 24 and a data bus 25 , and are controlled by commands from the system control unit 11 .

[0053] Fig. 2 is a schematic plan view showing the general configuration of the image sensor 5 shown in Fig. 1. The image sensor 5 includes an image sensor surface 60 on which pixel rows 62, each consisting of a plurality of pixels 61 arranged in a row direction X, are arranged in a column direction Y that intersects with the row direction X (orthogonal in the illustrated example), a drive circuit 63 that drives the pixels 61 arranged on the image sensor surface 60, and a signal processing circuit 64 that processes pixel signals read out to signal lines from each pixel 61 in the pixel rows 62 arranged on the image sensor surface 60.

[0054] The pixel signals read out from the pixels 61 to the signal lines are analog signals. The signal processing circuit 64 includes a converter that converts the analog signals into digital signals. The pixel signals read out from the pixels 61 are converted into digital signals by the signal processing circuit 64 and output as digital signals to the outside of the image sensor 5. The board on which the image sensor 5 is mounted may be provided with a processing circuit that processes the digital pixel signals output from the image sensor 5. The board on which the system control unit 11 is provided and the board on which the image sensor 5 is provided may be the same board.

[0055] Fig. 3 is a partially enlarged schematic diagram of the imaging surface 60 of the imaging element 5 shown in Fig. 2. The plurality of pixels 61 arranged on the imaging surface 60 includes pixels corresponding to each of a plurality of colors (three in this embodiment).

[0056] Specifically, pixels 61R (blocks marked with the letter "R" in the figure) corresponding to the red wavelength band, pixels 61G (blocks marked with the letters "Ga" and "Gb" in the figure) corresponding to the green wavelength band, and pixels 61B (blocks marked with the letter "B" in the figure) corresponding to the blue wavelength band are provided on the imaging surface 60. Each pixel 61 provided on the imaging surface 60 receives light in the corresponding wavelength band and outputs a pixel signal according to the amount of light.

[0057] On the imaging surface 60, GR pixel rows in which pixels 61R and pixels 61G are alternately arranged in the row direction X, and BG pixel rows in which pixels 61G and pixels 61B are alternately arranged in the row direction X are alternately arranged in the column direction Y. In the example of Fig. 3, the multiple colors are three colors: red, blue, and green, but they may also be three colors: cyan, magenta, and yellow, or four or more colors. The arrangement of the pixels corresponding to the multiple colors is not limited to the Bayer arrangement illustrated in Fig. 3, and may be another arrangement such as a stripe arrangement.

[0058] Pixel 61G includes pixel 61Ga (a block marked with the letter "Ga" in the figure) in which pixel 61R is arranged on both sides in the row direction X and pixel 61B is arranged on both sides in the column direction Y, and pixel 61Gb (a block marked with the letter "Gb" in the figure) in which pixel 61B is arranged on both sides in the row direction X and pixel 61R is arranged on both sides in the column direction Y.

[0059] Fig. 4 is a plan view showing a schematic configuration of a pixel 61 in the image sensor 5 shown in Fig. 2. Fig. 5 is a cross-sectional view of the pixel 61 taken along line AA shown in Fig. 4.

[0060] As shown in FIG. 4, the pixel 61 includes a photoelectric conversion unit 61A, a charge holding unit 61F, a charge transfer unit 61C, a floating diffusion 61D, and a readout circuit 61E.

[0061] The photoelectric conversion unit 61A receives light that has passed through the imaging optical system of the lens device 40 and generates and accumulates electric charges according to the amount of received light. The photoelectric conversion unit 61A is composed of a photodiode, etc. The charge holding unit 61F is composed of an impurity region in a semiconductor substrate.

[0062] The charge transfer unit 61C controls the height of the potential barrier between the photoelectric conversion unit 61A and the charge holding unit 61F. The charge transfer unit 61C is composed of an impurity region in a semiconductor substrate and an electrode formed above this impurity region.

[0063] For example, the potential barrier is raised at the start of exposure to start exposure of the photoelectric conversion unit 61A, and the potential barrier is lowered at the end of exposure. By this control, the charge generated in the photoelectric conversion unit 61A during the exposure period and stored therein is transferred to the charge holding unit 61F almost simultaneously with the end of exposure. Then, the potential barrier is returned to its original state, allowing the charge generated in the photoelectric conversion unit 61A during the exposure period to be held in the charge holding unit 61F.

[0064] It is also possible to provide the photoelectric conversion unit 61A with a potential gradient that decreases toward the charge holding unit 61F. When this configuration is adopted, the potential barrier is lowered at the timing of the start of exposure, so that the charge generated in the photoelectric conversion unit 61A by exposure moves to the charge holding unit 61F without being stored there. Thereafter, when the potential barrier is raised at the timing of the end of exposure, the exposure period ends and the charge generated in the photoelectric conversion unit 61A during the exposure period is held in the charge holding unit 61F.

[0065] In this way, the charge generated in the photoelectric conversion unit 61A can be held in the charge holding unit 61F by controlling the charge transfer unit 61C.

[0066] The floating diffusion 61D converts the electric charge into a signal, and the electric charge is transferred from the electric charge holding portion 61F to the floating diffusion 61D. The floating diffusion 61D constitutes a conversion portion.

[0067] The readout circuit 61E is a circuit that reads out a signal corresponding to the potential of the floating diffusion 61D as a pixel signal to a signal line 65. The readout circuit 61E is driven by a drive circuit 63.

[0068] As shown in FIG. 5, a P-well layer 71 is formed on the surface of an N-type substrate 70, and a photoelectric conversion section 61A is formed on the surface of the P-well layer 71.

[0069] The photoelectric conversion section 61A is composed of an N-type impurity layer 73 and a P-type impurity layer 74 formed thereon. The N-type substrate 70 and the P-well layer 71 form a semiconductor substrate.

[0070] A charge holding portion 61F made of an N-type impurity layer is formed on the surface of the P-well layer 71 at a slight distance from the photoelectric conversion portion 61A.

[0071] A transfer electrode 76 is formed above a region 75 of the P-well layer 71 between the charge holding portion 61F and the photoelectric conversion portion 61A, with an oxide film interposed therebetween.

[0072] 5, the transfer electrode 76 is formed up to above the charge holding portion 61F, but it is sufficient that the transfer electrode 76 is formed at least above the region 75.

[0073] The above-mentioned potential barrier can be lowered by controlling the potential of the transfer electrode 76 to form a channel in the region 75. The potential of the transfer electrode 76 is controlled by the drive circuit 63.

[0074] A floating diffusion 61D made of an N-type impurity layer is formed on the surface of the P-well layer 71 at a slight distance from the charge holding portion 61F.

[0075] A read electrode 72 is formed above the P-well layer 71 between the charge holding portion 61F and the floating diffusion 61D via an oxide film.

[0076] By controlling the potential of the readout electrode 72 and forming a channel in the region between the charge retention portion 61F and the floating diffusion 61D, the charge in the charge retention portion 61F can be transferred to the floating diffusion 61D. The potential of the readout electrode 72 is controlled by the drive circuit 63.

[0077] The readout circuit 61E is composed of a reset transistor 77 for resetting the potential of the floating diffusion 61D, an output transistor 78 for converting the potential of the floating diffusion 61D into a pixel signal and outputting it, and a selection transistor 79 for selectively reading out the pixel signal output from the output transistor 78 to the signal line 65. The configuration of the readout circuit is an example and is not limited to this. The readout circuit 61E may be shared by multiple pixels 61.

[0078] A light-shielding film is provided in the pixel 61, and the area other than the photoelectric conversion portion 61A is shielded from light by this light-shielding film.

[0079] The structure of the pixel 61 shown in FIGS. 4 and 5 is an example, and the present invention is not limited to this.

[0080] The drive circuit 63 shown in Figure 2 independently drives the transfer electrode 76, readout electrode 72, and readout circuit 61E of each pixel 61 to reset the photoelectric conversion unit 61A (discharge the charge accumulated in the photoelectric conversion unit 61A), hold the charge generated in the photoelectric conversion unit 61A in the charge holding unit 61F, and read out a pixel signal corresponding to the charge held in the charge holding unit 61F to the signal line 65.

[0081] The photoelectric conversion unit 61A is reset by forming a channel in the semiconductor substrate below the transfer electrode 76 and also forming a channel in the semiconductor substrate below the readout electrode 72, and then discharging the charge in the floating diffusion 61D using the reset transistor 77.

[0082] 2 performs correlated double sampling on pixel signals read out to signal lines 65 from each pixel 61 in a pixel row 62, converts the pixel signals after correlated double sampling into digital signals, and outputs the digital signals to the data bus 25 (see FIG. 1). The signal processing circuit 64 is controlled by the system control unit 11. The digital signal processing unit 17 performs signal processing such as demosaic processing and gamma correction processing on the pixel signals output from the image sensor 5 to the data bus 25 to generate captured image data.

[0083] The system control unit 11 can drive the image sensor 5 in each of global reset driving, global shutter driving, and rolling readout driving.

[0084] The global reset driving is a driving method in which the photoelectric conversion units 61A of all pixels 61 formed on the imaging surface 60 of the imaging element 5 are simultaneously reset, and exposure of all pixels 61 is simultaneously started.

[0085] Global shutter driving is a driving method in which the charges generated in the photoelectric conversion units 61A of each pixel 61 by exposure started in all pixels 61 by global reset driving are simultaneously held in the charge holding units 61F of each pixel 61, thereby simultaneously ending exposure in all pixels 61. Control for causing the drive circuit 63 to execute global shutter driving constitutes first control.

[0086] The rolling readout driving is a driving method in which pixel signals corresponding to the charges held in the charge holding units 61F of all the pixels 61 by the global shutter driving are read out sequentially for each pixel row 62.

[0087] Fig. 6 is a timing chart showing the operation of the digital camera 100 shown in Fig. 1 in the imaging mode. Fig. 6 shows the drive timing of the photoelectric conversion units 61A and charge holding units 61F of each pixel row 62 of the image sensor 5. In Fig. 6, the vertical axis indicates the position of the pixel row 62 in the column direction Y.

[0088] A straight line GR shown in FIG. 6 indicates the timing at which each photoelectric conversion unit 61A included in a pixel row 62 is reset by global reset driving.

[0089] The lines GS and ST shown in Figure 6 indicate the timing at which the charges generated in each photoelectric conversion unit 61A included in the pixel row 62 due to exposure started by global reset driving are held in the charge holding unit 61F by global shutter driving.

[0090] The period enclosed by the lines GR and GS is the exposure period EX of the image sensor 5 when capturing an image for storage in the storage medium 21 .

[0091] A straight line RO shown in FIG. 6 indicates the timing at which a pixel signal corresponding to the charge held in the charge holding section 61F is output from the image sensor 5 by rolling readout driving.

[0092] When the global shutter driving is performed at time t1, the charges generated in each photoelectric conversion unit 61A during the exposure period EX are held in the charge holding unit 61F (line ST in FIG. 6).

[0093] The system control unit 11 performs global shutter driving indicated by the line GS, and then performs rolling readout driving indicated by the line RO. In this rolling readout driving, the system control unit 11 selects pixel rows 62 in order, for example, from the top to the bottom of the imaging surface 60, and reads pixel signals from the selected pixel rows 62.

[0094] The time T1 between the lines ST and RO is the time between the first timing when charge is held in the charge holding unit 61F (i.e., exposure ends) and the second timing when the charge in the charge holding unit 61F is transferred to the floating diffusion 61D and a pixel signal corresponding to the charge is read out in each pixel row 62. The time T1 is a predetermined value, but varies depending on the pixel row 62. The time T1 constitutes the first time.

[0095] The period during which the rolling readout driving indicated by the straight line RO is performed will be referred to below as the signal readout period. The pixel signals output from the image sensor 5 during this signal readout period are processed by the digital signal processor 17 to become captured image data, which is then stored in the storage medium 21.

[0096] The charge holding portion 61F is light-shielded. However, if strong light is irradiated onto the light-shielding film above the charge holding portion 61F during the signal readout period, the light may cause noise charges to be generated below the area irradiated by the light due to leakage light.

[0097] In this embodiment, in order to detect noise charges, the system control unit 11 sets a portion of all pixels 61 provided on the imaging surface 60 as pixels for detecting noise charges (hereinafter referred to as noise detection pixels).

[0098] Of all the pixels 61, the noise detection pixels form second pixels, and the pixels 61 other than the noise detection pixels (hereinafter referred to as imaging pixels) form first pixels.

[0099] In the image sensor 5, all pixels 61 have the same pixel structure. The same pixel structure means that the size and layout of the charge holding portions 61F are designed so that the noise charges generated in the charge holding portions 61F are equivalent when light is incident on the light-shielding film under the same conditions.

[0100] The noise detection pixels and the imaging pixels have the same pixel structure, but their uses can be changed by changing the driving method. That is, all of the pixels 61 can be set as noise detection pixels (first pixels). Also, all of the pixels 61 can be set as imaging pixels (second pixels).

[0101] Fig. 7 is a schematic diagram showing an example of setting noise detection pixels. Fig. 7 shows some of the multiple pixels 61 arranged on the imaging surface 60. For example, the pixels 61 surrounded by a thick frame in the figure are set as noise detection pixels. In the example of Fig. 7, some pixels 61B, some pixels 61R, some pixels 61Ga, and some pixels 61Gb are set as noise detection pixels.

[0102] The amount of noise charge generated in each of the charge holding units 61F of pixels 61B, 61R, and 61G may vary depending on the color component of light that causes the noise charge. Therefore, by setting each of pixels 61B, 61R, and 61G as a noise detection pixel, it becomes possible to accurately detect the noise charge generated in each of pixels 61B, 61R, and 61G.

[0103] Furthermore, there are two types of pixel 61G (pixel 61Ga and pixel 61Gb) that differ in the arrangement of colors detected by the surrounding pixels 61. The amount of noise charge generated may differ between pixel 61Ga and pixel 61Gb due to the influence of color mixing from the surrounding pixels 61. Therefore, for pixel 61G, by setting each of pixel 61Ga and pixel 61Gb as a noise detection pixel, it becomes possible to accurately detect the noise charge generated in pixel 61G.

[0104] 7, of all the pixels 61, the number of pixels 61 set as noise detection pixels is preferably smaller than the number of pixels 61 set as imaging pixels. As will be described later, the noise detection pixels cannot read out pixel signals corresponding to the charges generated in the photoelectric conversion unit 61A during the exposure period. For this reason, high-quality captured image data can be generated by limiting the number of noise detection pixels to the minimum necessary. Furthermore, it is preferable that the noise detection pixels are evenly and discretely arranged across the entire imaging surface 60.

[0105] The system control unit 11 can generate captured image data in which the influence of noise charges is reduced by correcting the pixel signals output from the imaging pixels based on the signals output from the noise detection pixels.

[0106] 7 , of all the pixels 61, the pixels 61 set as noise detection pixels are preferably arranged periodically in the row direction X and the column direction Y. This makes it possible to correct the pixel signals output from the imaging pixels with high accuracy.

[0107] 8 and 9 are timing charts illustrating the drive details of the noise detection pixels and imaging pixels included in, for example, the second pixel row 62 from the top shown in Fig. 7. Fig. 8 shows the drive details of the imaging pixels, and Fig. 9 shows the drive details of the noise detection pixels. Figs. 8 and 9 show the drive details for the period from a timing slightly before time t1 shown in Fig. 6 to the line RO.

[0108] 8 and 9, the upper part shows the locations of the charges generated in the photoelectric conversion unit 61A during the exposure period EX. "PD" in the figures indicates the photoelectric conversion unit 61A. "MEM" in the figures indicates the charge holding unit 61F. "FD" in the figures indicates the floating diffusion 61D.

[0109] "TRX" in the diagram indicates the voltage applied to the transfer electrode 76. "TG" in the diagram indicates the voltage applied to the readout electrode 72. When the voltage applied to the transfer electrode 76 is at a high level, a channel is formed below the transfer electrode 76. When the voltage applied to the readout electrode 72 is at a high level, a channel is formed below the readout electrode 72. "RST" in the diagram indicates the voltage applied to the gate electrode of the reset transistor 77. When a high-level voltage is applied to the gate electrode of the reset transistor 77, the reset transistor 77 turns on. "SEL" in the diagram indicates the voltage applied to the gate electrode of the selection transistor 79. When a high-level voltage is applied to the gate electrode of the selection transistor 79, the selection transistor 79 turns on.

[0110] As shown in FIG. 8, the system control unit 11 performs global shutter driving for the imaging pixels at time t1 to complete exposure, and then performs reset noise readout control before time t3 after the time T1 has elapsed.

[0111] In this read control, between time t1 and time t3, the system control unit 11 turns on the reset transistor 77 to reset the floating diffusion 61D, and then turns on the selection transistor 79 to read out a signal (so-called reset noise) corresponding to the potential of the floating diffusion 61D using the read circuit 61E.

[0112] After the above readout control is completed, the system control unit 11 applies a high-level voltage to the transfer electrode 76 at time t3 to transfer the charge in the charge holding unit 61F to the floating diffusion 61D. Next, the system control unit 11 turns on the selection transistor 79 to control the readout circuit 61E to read out a signal corresponding to the potential of the floating diffusion 61D. The control to send the charge in the charge holding unit 61F to the floating diffusion 61D at time t3 (time T1 has elapsed since the end of exposure) is referred to as second control.

[0113] In this way, the system control unit 11 reads out the pixel signal of the imaging pixel (first pixel) by performing the first control and then the second control for that imaging pixel. This pixel signal includes not only a signal corresponding to the charge generated in the photoelectric conversion unit 61A during the exposure period EX, but also a signal corresponding to the noise charge if the noise charge is generated in the charge holding unit 61F during the period from the end of exposure to time t3.

[0114] As shown in FIG. 9, for the noise detection pixels, the system control unit 11 performs global shutter drive at time t1 to end exposure, and then performs charge discharge control to discharge the charge held in the charge holding unit 61F by the global shutter drive until time t3 after the time T1 has elapsed.

[0115] In the charge discharge control, the system control unit 11 sets the transfer electrode 76 and the readout electrode 72 to a high level immediately after time t1, preferably simultaneously with time t1, thereby discharging the charge held in the charge holding unit 61F to the drain of the reset transistor 77 via the floating diffusion 61D.

[0116] At time t2, which is between time t1 and time t3, the system control unit 11 sets the readout electrode 72 and the transfer electrode 76 to a low level. This ends the charge discharge control. The start timing of the charge discharge control is preferably immediately after time t1, and most preferably simultaneously with time t1. This start timing may be any time between time t1 and time t3, and noise charges can be detected with high accuracy by starting the control as close to time t1 as possible. For example, noise charges can be detected with high accuracy by starting the charge discharge control closer to time t1 than the midpoint between time t1 and time t3. This start timing should be close enough to time t1 to ensure sufficient noise charge detection accuracy.

[0117] After the charge discharge control ends, shortly before time t3, the system control unit 11 turns on the selection transistor 79 and controls the read circuit 61E to read out a signal (reset noise) corresponding to the potential of the floating diffusion 61D.

[0118] At time t3, the system control unit 11 applies a high-level voltage to the transfer electrode 76 to transfer the charge in the charge holding unit 61F to the floating diffusion 61D. The charge transferred here includes noise charge generated in the charge holding unit 61F between time t2 and time t3, in addition to the reset noise of the floating diffusion 61D generated by the charge discharge control.

[0119] Next, the system control unit 11 turns on the selection transistor 79 and controls the readout circuit 61E to read out a signal corresponding to the potential of the floating diffusion 61D. The control of discharging charge between time t1 and time t3, and then transferring the charge of the charge holding unit 61F to the floating diffusion 61D at time t3, is referred to as third control.

[0120] In this way, the system control unit 11 performs the first control and then the third control on the noise detection pixel (second pixel), thereby reading out a signal corresponding to the noise charge generated in the charge holding unit 61F.

[0121] The signal read out from the noise detection pixel by the third control does not include a signal corresponding to the charge generated in the photoelectric conversion unit 61A of the noise detection pixel during the exposure period EX. Therefore, by using the signal read out from this noise detection pixel, it is possible to correct with high precision the noise included in the pixel signal read out from the imaging pixels surrounding this noise detection pixel.

[0122] If the noise charge generated in the charge holding unit 61F has only a minor effect on the captured image data, it may not be necessary to detect it. In the digital camera 100, all pixels 61 can be set as imaging pixels. Therefore, in situations where detection of noise charge is not necessary, all pixels 61 can be set as imaging pixels and an image can be captured, thereby improving the quality of the captured image data and reducing the processing load for noise correction.

[0123] For example, it is preferable that the system control unit 11 be operable in a first mode in which all pixels 61 are set as imaging pixels to perform imaging, and a second mode in which some of all pixels 61 are set as noise detection pixels to perform imaging, as shown in Figure 7.

[0124] The first mode is a mode in which the first control is performed on all pixels 61, and then the second control is performed sequentially for each pixel row 62. The second mode is a mode in which the first control is performed on all pixels 61, and then the second control is performed sequentially for each pixel row 62 on the imaging pixels and the third control is performed on the noise detection pixels.

[0125] The system control unit 11 preferably operates in either the first mode or the second mode based on the image capturing conditions, the subject conditions, or the drive mode of the image sensor 5 .

[0126] An example of an imaging condition is exposure time. When the exposure time is short, the ratio of the amount of noise charge generated in the charge holding unit 61F to the amount of charge generated in the photoelectric conversion unit 61A during the exposure period EX becomes larger than when the exposure time is long. When this ratio is large, it is highly likely that pixel signal correction will be necessary.

[0127] Therefore, it is preferred that the system controller 11 operates in the second mode when the exposure time is equal to or less than a predetermined time threshold, and in the first mode when the exposure time exceeds the time threshold.

[0128] One example of an imaging condition is imaging sensitivity. For example, consider the case of capturing an image of a night scene including a high-brightness subject. In this case, if the high-brightness subject moves during the period from the end of exposure until time T1 has elapsed, and light from the high-brightness subject is incident on the imaging surface 60, the range in which noise charges are generated will expand in accordance with the movement of the high-brightness subject. As a result, noise charges will be generated even in pixels 61 that were not exposed to light from the high-brightness subject during the exposure period EX. When imaging sensitivity is high, these pixels 61 often capture dark areas. Therefore, if imaging sensitivity is high, the signal corresponding to the noise charge will be amplified, making noise more noticeable in the pixel signal output from these pixels 61.

[0129] Therefore, it is preferable that the system control unit 11 operates in the second mode when the imaging sensitivity is equal to or greater than a predetermined sensitivity threshold, and operates in the first mode when the imaging sensitivity is less than the sensitivity threshold.

[0130] An example of a subject condition is the brightness of the subject. For example, if the subject includes a high-brightness subject, a large amount of noise charge may be generated in the area of ​​the imaging surface 60 where light from the high-brightness subject is incident, and therefore, it is highly likely that pixel signal correction will be required.

[0131] Therefore, it is preferable that the system control unit 11 operates in the second mode when the brightness of the subject is equal to or greater than a predetermined brightness threshold, and operates in the first mode when the brightness of the subject is less than the brightness threshold. The brightness of the subject may be determined using pixel signals acquired in live view imaging performed before the exposure period EX.

[0132] Even when a high-brightness subject is included in the subjects, the high-brightness subject may be stationary. When the high-brightness subject is stationary, the pixel 61 on which light from the high-brightness subject is incident is likely to have its pixel signal output saturated, so the effect of noise can be ignored. Therefore, it is unlikely that pixel signal correction is required for this pixel 61.

[0133] On the other hand, when a high-brightness subject is moving, as described above, the range in which noise charges are generated expands, and noise charges may be generated even in pixels 61 whose pixel signals have not been saturated by exposure. In other words, when there is a moving high-brightness subject, there is a greater need to correct the pixel signals.

[0134] Therefore, it is preferable that the system control unit 11 operates in the second mode when the amount of motion of the subject is equal to or greater than a predetermined motion threshold, and operates in the first mode when the amount of motion of the subject is less than the motion threshold. The amount of motion of the subject may be determined using pixel signals acquired in live view imaging performed before the exposure period EX.

[0135] The system control unit 11 may operate in the second mode when the brightness of the subject is equal to or greater than the brightness threshold and the amount of movement of the subject is equal to or greater than the motion threshold, and may operate in the first mode when the brightness of the subject is equal to or greater than the brightness threshold and the amount of movement of the subject is less than the motion threshold.

[0136] Examples of the drive mode include a thinning mode in which pixel signals are thinned out and read out from all of the pixels 61, and an addition mode in which pixel signals are added up and read out from all of the pixels 61. In the thinning mode or addition mode, the number of pixel signals output from the image sensor 5 is smaller than in modes other than these modes (modes in which thinning out or addition is not performed).

[0137] If the system controller 11 operates in the second mode in the thinning mode or the addition mode, some of the originally small number of pixel signals will become signals corresponding to noise charges. Therefore, it is preferable that the system controller 11 operates in the second mode when the drive mode is a mode other than the thinning mode or the addition mode, and operates in the first mode when the drive mode is the thinning mode or the addition mode.

[0138] In the explanation so far, the system control unit 11 has been described as operating in either the first mode or the second mode based on the imaging conditions, the subject conditions, or the drive mode of the image sensor 5. As a modification of this, the system control unit 11 may be further operable in a third mode in which the arrangement of noise detection pixels is changed in the second mode, and may operate in either the second mode or the third mode based on the imaging conditions, the subject conditions, or the drive mode of the image sensor 5.

[0139] Different arrangements of noise detection pixels include cases where the total number of noise detection pixels is the same but the positions of the noise detection pixels (e.g., the density in the column direction Y) are different, cases where the total number of noise detection pixels is different, and cases where both the total number and positions of the noise detection pixels are different.

[0140] In the following, the third mode will be described as a mode in which the total number of noise detection pixels is greater than that of the second mode. In this case, the third mode can detect noise charges with higher accuracy than the second mode because the total number of noise detection pixels is greater.

[0141] On the other hand, in the second mode, the total number of noise detection pixels is small, so the number of pixel signals read out can be increased, improving the quality of the captured image data. Even in the second mode, noise charges can be detected. Therefore, if it is determined that there is a large amount of noise charge, pixel signals can be corrected, thereby improving the quality of the captured image data.

[0142] It is preferable that the system control unit 11 operates in the third mode with a large number of noise detection pixels when the exposure time is equal to or less than the time threshold, and operates in the second mode with a small number of noise detection pixels when the exposure time exceeds the time threshold.

[0143] It is preferable that the system control unit 11 operates in the third mode, which has a large number of noise detection pixels, when the imaging sensitivity is equal to or greater than the sensitivity threshold, and operates in the second mode, which has a small number of noise detection pixels, when the imaging sensitivity is less than the sensitivity threshold.

[0144] It is preferable that the system control unit 11 operates in the third mode with a large number of noise detection pixels when the brightness of the subject is equal to or greater than the brightness threshold, and operates in the second mode with a small number of noise detection pixels when the brightness of the subject is less than the brightness threshold.

[0145] It is preferable that the system control unit 11 operates in the third mode with a large number of noise detection pixels when the amount of motion of the subject is equal to or greater than the motion threshold, and operates in the second mode with a small number of noise detection pixels when the amount of motion of the subject is less than the motion threshold.

[0146] It is preferable that the system control unit 11 operates in the third mode, which has a larger number of noise detection pixels, when the drive mode is a mode other than the thinning mode or the addition mode, and operates in the second mode, which has a smaller number of noise detection pixels, when the drive mode is the thinning mode or the addition mode.

[0147] In the thinning mode, the system control unit 11 sets the noise detection pixels so that the noise detection pixels are arranged discretely in pixel rows 62 that include imaging pixels from which pixel signals are to be read out.

[0148] 10 , in the case of thinning readout in which pixel signals of pixels 61 enclosed by dashed lines are not read out, noise detection pixels are set at a rate of, for example, one in three pixel rows 62 from which pixel signals are to be read out (one in six pixel rows when viewed across the entire imaging surface 60). The number of noise detection pixels in the example of FIG. 10 is smaller than that in the example of FIG. 7 .

[0149] For example, when performing vertical two-pixel additive readout in which pixel signals of two pixels 61 of the same color in the column direction Y are added together as shown in Fig. 11 , the system control unit 11 sets noise detection pixels at a rate of, for example, one in every six pixel rows 62 from which pixel signals are to be read out. The number of noise detection pixels in the example of Fig. 11 is smaller than in the example of Fig. 7. In Fig. 11, double-headed arrows indicate pixel rows to be added together.

[0150] Here, the second mode and the third mode are exemplified as modes with different noise detection pixel arrangements, but there may be three or more modes. The system control unit 11 may operate in any one of the first mode, the second mode, and the third mode based on the imaging conditions, the subject conditions, the drive mode, etc.

[0151] Next, the configuration of a smartphone, which is another embodiment of an imaging device according to the technique of the present disclosure, will be described.

[0152] Fig. 12 shows the external appearance of a smartphone 200. The smartphone 200 shown in Fig. 12 has a flat housing 201, and is provided with a display input unit 204 on one surface of the housing 201, which is an integrated unit of a display panel 202 as a display unit and an operation panel 203 as an input unit.

[0153] Such a housing 201 also includes a speaker 205, a microphone 206, an operation unit 207, and a camera unit 208. Note that the configuration of the housing 201 is not limited to this, and for example, it is also possible to adopt a configuration in which the display unit and the input unit are independent, or a configuration having a folding structure or a sliding mechanism.

[0154] FIG. 13 is a block diagram showing the configuration of the smartphone 200 shown in FIG.

[0155] As shown in FIG. 13 , the main components of the smartphone include a wireless communication unit 210, a display input unit 204, a call unit 211, an operation unit 207, a camera unit 208, a memory unit 212, an external input / output unit 213, a GNSS (Global Navigation Satellite System) receiving unit 214, a motion sensor unit 215, a power supply unit 216, and a main control unit 220.

[0156] The smartphone 200 also has, as its main function, a wireless communication function for performing mobile wireless communication via a base station device BS (not shown) and a mobile communication network NW (not shown).

[0157] The wireless communication unit 210 performs wireless communication with a base station device BS accommodated in the mobile communication network NW in accordance with instructions from the main control unit 220. Using this wireless communication, various file data such as audio data and image data, e-mail data, etc. are sent and received, and web data, streaming data, etc. are received.

[0158] The display input unit 204 is a so-called touch panel that, under the control of the main control unit 220, displays images (still images and moving images) or text information, etc. to visually convey information to the user and detects user operations on the displayed information, and is equipped with a display panel 202 and an operation panel 203.

[0159] The display panel 202 uses a liquid crystal display (LCD), an organic electroluminescence display (OELD), or the like as a display device.

[0160] The operation panel 203 is a device placed so that an image displayed on the display surface of the display panel 202 can be seen, and detects one or more coordinates operated by a user's finger or a stylus. When this device is operated by the user's finger or a stylus, a detection signal generated by the operation is output to the main control unit 220. Next, the main control unit 220 detects the operation position (coordinates) on the display panel 202 based on the received detection signal.

[0161] As shown in Figure 13, the display panel 202 and operation panel 203 of a smartphone 200, which is an example of one embodiment of the imaging device of the present invention, are integrated to form a display input unit 204, and the operation panel 203 is positioned so that it completely covers the display panel 202.

[0162] When such an arrangement is adopted, operation panel 203 may also have a function for detecting user operations in areas outside display panel 202. In other words, operation panel 203 may have a detection area for the overlapping portion that overlaps display panel 202 (hereinafter referred to as a display area), and a detection area for the remaining outer edge portion that does not overlap display panel 202 (hereinafter referred to as a non-display area).

[0163] The size of the display area and the size of the display panel 202 may be exactly the same, but they do not necessarily have to be the same. The operation panel 203 may also have two sensitive areas: an outer edge portion and an inner portion. The width of the outer edge portion is designed appropriately depending on the size of the housing 201, etc.

[0164] Furthermore, the position detection method used in the operation panel 203 may be a matrix switch method, a resistive film method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, a capacitance method, or the like, and any method may be used.

[0165] The call unit 211 is equipped with a speaker 205 or a microphone 206, and converts the user's voice input through the microphone 206 into voice data that can be processed by the main control unit 220 and outputs it to the main control unit 220, or decodes voice data received by the wireless communication unit 210 or the external input / output unit 213 and outputs it from the speaker 205.

[0166] Also, as shown in FIG. 12, for example, the speaker 205 can be mounted on the same surface as the display input unit 204, and the microphone 206 can be mounted on the side of the housing 201.

[0167] The operation unit 207 is a hardware key using a key switch or the like, and receives instructions from a user. For example, as shown in Fig. 12, the operation unit 207 is a push-button switch mounted on the side surface of the housing 201 of the smartphone 200, which turns on when pressed with a finger or the like, and turns off when the finger is released by the restoring force of a spring or the like.

[0168] The storage unit 212 stores the control program and control data of the main control unit 220, application software, address data associated with names or telephone numbers of communication partners, data of sent and received e-mails, web data downloaded by web browsing, downloaded content data, and also temporarily stores streaming data, etc. The storage unit 212 is composed of an internal storage unit 217 built into the smartphone and an external storage unit 218 having a removable external memory slot.

[0169] The internal memory unit 217 and the external memory unit 218 constituting the memory unit 212 are realized using storage media such as a flash memory type, a hard disk type, a multimedia card micro type, a card-type memory (e.g., MicroSD (registered trademark) memory, etc.), a RAM (Random Access Memory), a ROM (Read Only Memory), etc.

[0170] The external input / output unit 213 serves as an interface with all external devices connected to the smartphone 200, and is used to directly or indirectly connect to other external devices via communication (e.g., Universal Serial Bus (USB), IEEE 1394, Bluetooth (registered trademark), RFID (Radio Frequency Identification), Infrared Data Association (IrDA) (registered trademark), UWB (Ultra Wideband) (registered trademark), ZigBee (registered trademark), etc.) or a network (e.g., Ethernet (registered trademark), wireless LAN (Local Area Network), etc.).

[0171] Examples of external devices connected to the smartphone 200 include a wired / wireless headset, a wired / wireless external charger, a wired / wireless data port, a memory card connected via a card socket, a SIM (Subscriber Identity Module Card) / UIM (User Identity Module Card) card, an external audio / video device connected via an audio / video I / O (Input / Output) terminal, a wirelessly connected external audio / video device, a wired / wirelessly connected smartphone, a wired / wirelessly connected personal computer, a wired / wirelessly connected personal computer, and earphones.

[0172] The external input / output unit 213 can transmit data received from such external devices to each component inside the smartphone 200, or transmit data inside the smartphone 200 to external devices.

[0173] The GNSS receiving unit 214 receives GNSS signals transmitted from the GNSS satellites ST1 to STn in accordance with instructions from the main control unit 220, executes positioning calculation processing based on the received multiple GNSS signals, and detects a position consisting of the latitude, longitude, and altitude of the smartphone 200. When the GNSS receiving unit 214 can acquire position information from the wireless communication unit 210 or the external input / output unit 213 (for example, a wireless LAN), it can also detect the position using the position information.

[0174] The motion sensor unit 215 includes, for example, a three-axis acceleration sensor, and detects the physical movement of the smartphone 200 in accordance with instructions from the main control unit 220. By detecting the physical movement of the smartphone 200, the direction of movement or acceleration of the smartphone 200 is detected. The detection result is output to the main control unit 220.

[0175] The power supply unit 216 supplies power stored in a battery (not shown) to each unit of the smartphone 200 in accordance with instructions from the main control unit 220 .

[0176] The main control unit 220 includes a microprocessor, operates according to the control program and control data stored in the storage unit 212, and controls all the units of the smartphone 200. The microprocessor of the main control unit 220 has the same functions as the system control unit 11. The main control unit 220 also includes a mobile communication control function that controls all the units of the communication system to perform voice communication or data communication via the wireless communication unit 210, and an application processing function.

[0177] The application processing function is realized by the main control unit 220 operating in accordance with the application software stored in the storage unit 212. Examples of the application processing function include an infrared communication function that controls the external input / output unit 213 to perform data communication with a connected device, an email function that sends and receives emails, and a web browsing function that views web pages.

[0178] The main control unit 220 also has an image processing function for displaying video on the display input unit 204 based on image data (still image or moving image data) such as received data or downloaded streaming data.

[0179] The image processing function refers to a function in which the main control unit 220 decodes the image data, performs image processing on the decoded result, and displays the image on the display input unit 204 .

[0180] Furthermore, the main control unit 220 executes display control for the display panel 202 and operation detection control for detecting user operations via the operation unit 207 and the operation panel 203 .

[0181] By executing display control, the main control unit 220 displays software keys such as icons or scroll bars for starting application software, or displays a window for creating an e-mail.

[0182] The scroll bar refers to a software key for receiving an instruction to move the displayed portion of an image, such as a large image that cannot fit in the display area of ​​the display panel 202 .

[0183] In addition, by executing operation detection control, the main control unit 220 detects user operations through the operation unit 207, accepts operations on the above icons and input of character strings into the input field of the above window through the operation panel 203, or accepts requests to scroll the displayed image through the scroll bar.

[0184] Furthermore, by executing operation detection control, the main control unit 220 determines whether the operation position on the operation panel 203 is an overlapping portion (display area) that overlaps the display panel 202 or an outer edge portion (non-display area) that does not overlap the display panel 202, and is equipped with a touch panel control function that controls the sensitive area of ​​the operation panel 203 or the display position of the software key.

[0185] The main control unit 220 can also detect a gesture operation on the operation panel 203 and execute a preset function in response to the detected gesture operation.

[0186] Gesture operation is not a simple touch operation as in the past, but rather refers to an operation of drawing a path with a finger or the like, specifying multiple positions simultaneously, or combining these to draw a path for at least one of multiple positions.

[0187] The camera unit 208 includes the lens device 40, the image sensor 5, and the digital signal processing unit 17 shown in FIG.

[0188] The captured image data generated by the camera unit 208 can be stored in the storage unit 212 or output via the external input / output unit 213 or the wireless communication unit 210 .

[0189] In the smartphone 200 shown in FIG. 13, the camera unit 208 is mounted on the same surface as the display input unit 204, but the mounting position of the camera unit 208 is not limited to this, and it may be mounted on the back surface of the display input unit 204.

[0190] The camera unit 208 can also be used for various functions of the smartphone 200. For example, an image acquired by the camera unit 208 can be displayed on the display panel 202, or an image from the camera unit 208 can be used as one of the operation inputs for the operation panel 203.

[0191] Furthermore, when the GNSS receiving unit 214 detects a position, it can also detect the position by referring to an image from the camera unit 208. Furthermore, it can also refer to an image from the camera unit 208 to determine the optical axis direction of the camera unit 208 of the smartphone 200 or determine the current usage environment without using a triaxial acceleration sensor or by using the image in combination with a triaxial acceleration sensor. Of course, the image from the camera unit 208 can also be used in application software.

[0192] In addition, image data of still images or videos can be added with location information acquired by the GNSS receiving unit 214, audio information acquired by the microphone 206 (which may be converted to text information by the main control unit, etc.), posture information acquired by the motion sensor unit 215, etc., and stored in the memory unit 212 or output via the external input / output unit 213 or wireless communication unit 210.

[0193] In this embodiment, each process (each control) is executed by a computer. The computer may execute these processes using a processor, a program, or a combination thereof. The computer may be a general-purpose computer, a computer for specific applications, a system such as a workstation, or other hardware element capable of executing a program.

[0194] The processor may be configured with one or more pieces of hardware, and the type of hardware is not limited. For example, the processor may be configured with hardware such as a programmable logic device such as a central processing unit (CPU), a micro processing unit (MPU), a field programmable gate array (FPGA), a dedicated circuit for executing specific processing such as an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or a neural processing unit (NPU). The processor also has various units or means for executing various processes in this embodiment. The type of hardware may also be a combination of different types of hardware. When multiple pieces of hardware are configured to execute one or more processes of a certain processor, the multiple pieces of hardware may be located in devices physically separated from each other, or may be located in the same device. Furthermore, in any embodiment, the order of each process performed by the processor is not limited to the order described above and may be changed as appropriate. The hardware is configured by an electric circuit (circuitry) or the like that combines circuit elements such as semiconductor elements.

[0195] Furthermore, the present embodiment may be implemented by hardware, software, firmware, microcode, or a combination thereof. Software, firmware, and microcode may be configured by a program. A program may also be, for example, a group of program modules, each function of which may be implemented by a processor configured to perform the respective function. The program may be program code or multiple code segments stored in one or more non-transitory computer-readable media (e.g., storage media or other storages). The program may be stored in multiple non-transitory computer-readable media that reside in physically separate devices. Program code or a code segment may represent a procedure, a function, a subprogram, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. Program code or a code segment may be connected to another code segment or a hardware circuit by sending or receiving information, data, arguments, parameters, or memory contents.

[0196] Although various embodiments have been described above, it goes without saying that the present invention is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present invention. Furthermore, the components of the above embodiments may be combined in any manner as long as they do not deviate from the spirit of the invention.

[0197] This application is based on a Japanese patent application (Patent Application No. 2024-142997) filed on August 23, 2024, the contents of which are incorporated herein by reference.

[0198] REFERENCE SIGNS LIST 1 imaging lens 2 aperture 4 lens control unit 5 imaging element 8 lens driving unit 9 aperture driving unit 11 system control unit 14 operation unit 15 memory control unit 16 memory 17 digital signal processing unit 20 external memory control unit 21 storage medium 22 display device 22a display controller 22b display surface 24 control bus 25 data bus 40 lens device 60 imaging surface 61, 61B, 61G, 61R, 61Ga, 61Gb pixel 61F charge holding unit 61A photoelectric conversion unit 61C charge transfer unit 61D floating diffusion 61E readout circuit 62 pixel row 63 drive circuit 64 signal processing circuit 65 signal line 70 N-type substrate 71 P-well layer 72 readout electrode 73 N-type impurity layer 74 P-type impurity layer 75 Region 76 Transfer electrode 77 Reset transistor 78 Output transistor 79 Selection transistor 100 Digital camera 100A Main body

Claims

1. An imaging control device comprising a processor that controls an imaging element including a plurality of pixels, wherein the pixels include a photoelectric conversion unit, a charge holding unit that holds charge sent from the photoelectric conversion unit, and a conversion unit that converts the charge into a signal, wherein the processor performs a first control to hold charge generated in the photoelectric conversion unit in the charge holding unit and terminate exposure of the plurality of pixels, a second control to a first pixel among the plurality of pixels, and a third control to a second pixel among the plurality of pixels, wherein the second control is control to send charge from the charge holding unit to the conversion unit a first time after the end of the exposure, and the third control is control to discharge charge from the charge holding unit between the end of the exposure and the first time, and to send charge from the charge holding unit to the conversion unit a first time after the end of the exposure.

2. An imaging control device according to claim 1, wherein in the third control, the charge in the charge holding section is discharged immediately after the end of the exposure.

3. An imaging control device according to claim 2, wherein the first pixels include pixels corresponding to a plurality of colors, and the second pixels include pixels corresponding to the plurality of colors.

4. An imaging control device according to claim 3, wherein the number of the second pixels is less than the number of the first pixels.

5. An imaging control device according to claim 1, wherein the second pixels are arranged periodically.

6. An imaging control device according to claim 1, wherein all of the plurality of pixels can be set as the first pixel, and all of the plurality of pixels can be set as the second pixel.

7. An imaging control device according to claim 6, wherein the first pixel and the second pixel have the same structure.

8. An imaging control device according to any one of claims 1 to 7, wherein the processor is operable in a first mode in which the first control and the second control are performed on the plurality of pixels, and a second mode in which the processor performs the first control on the plurality of pixels, the second control on the first pixel, and the third control on the second pixel.

9. An imaging control device according to claim 8, wherein the processor operates in either the first mode or the second mode based on imaging conditions, subject conditions, or a drive mode of the imaging element.

10. An imaging control device according to claim 9, wherein the imaging conditions include an exposure time.

11. An imaging control device according to claim 10, wherein the processor operates in the second mode when the exposure time is equal to or less than a time threshold.

12. An imaging control device according to claim 9, wherein the imaging conditions include imaging sensitivity.

13. An imaging control device according to claim 12, wherein the processor operates in the second mode when the imaging sensitivity is equal to or greater than a sensitivity threshold.

14. An imaging control device according to claim 9, wherein the subject conditions include brightness of the subject.

15. An imaging control device according to claim 14, wherein the processor operates in the second mode when the brightness of the subject is equal to or greater than a brightness threshold.

16. An imaging control device according to claim 9, wherein the subject conditions include subject movement.

17. An imaging control device according to claim 16, wherein the processor operates in the second mode when the amount of motion of the subject is equal to or greater than a motion threshold.

18. An imaging control device according to claim 9, wherein the drive modes include a thinning mode in which signals are thinned out and read out from the plurality of pixels, or an addition mode in which signals are added up and read out from the plurality of pixels, and the processor operates in the second mode when in a mode other than the thinning mode or the addition mode.

19. An imaging control device according to any one of claims 1 to 7, wherein the processor is capable of executing the second control and the third control in a plurality of modes in which the arrangement of the second pixels differs.

20. An imaging control device according to claim 19, wherein the processor operates in one of the plurality of modes based on imaging conditions, subject conditions, or a drive mode of the imaging element.

21. An imaging control device according to claim 20, wherein the imaging conditions include an exposure time.

22. An imaging control device according to claim 21, wherein the processor operates in the mode in which the number of second pixels is greater when the exposure time is equal to or less than a time threshold than when the exposure time exceeds the time threshold.

23. An imaging control device according to claim 20, wherein the imaging conditions include imaging sensitivity.

24. An imaging control device according to claim 23, wherein the processor operates in the mode in which the number of the second pixels is greater when the imaging sensitivity is equal to or greater than the sensitivity threshold than when the imaging sensitivity is less than the sensitivity threshold.

25. An imaging control device according to claim 20, wherein the subject conditions include brightness of the subject.

26. An imaging control device according to claim 25, wherein the processor operates in the mode in which the number of second pixels is greater when the brightness of the subject is equal to or greater than a brightness threshold than when the brightness of the subject is less than the brightness threshold.

27. An imaging control device according to claim 20, wherein the subject conditions include subject movement.

28. An imaging control device according to claim 27, wherein the processor operates in the mode in which the number of second pixels is greater when the amount of motion of the subject is equal to or greater than a motion threshold than when the amount of motion of the subject is less than the motion threshold.

29. An imaging control device according to claim 20, wherein the drive modes include a thinning mode in which signals are thinned out and read out from the plurality of pixels, or an addition mode in which signals are added up and read out from the plurality of pixels, and the processor operates in the thinning mode or the addition mode in a mode in which the number of second pixels is smaller than in modes other than the thinning mode or the addition mode.

30. An imaging device comprising the imaging control device according to any one of claims 1 to 7 and the imaging element.

31. An imaging control method for controlling an imaging element including a plurality of pixels, wherein the pixels include a photoelectric conversion unit, a charge holding unit that holds charge sent from the photoelectric conversion unit, and a conversion unit that converts the charge into a signal; the imaging control method performs first control to hold charge generated in the photoelectric conversion unit in the charge holding unit and terminate exposure of the plurality of pixels, performs second control on a first pixel of the plurality of pixels, and performs third control on a second pixel of the plurality of pixels; the second control is control to send charge from the charge holding unit to the conversion unit a first time after the end of the exposure; and the third control is control to discharge charge from the charge holding unit between the end of the exposure and the first time, and to send charge from the charge holding unit to the conversion unit a first time after the end of the exposure.

32. An imaging control program for controlling an imaging element including a plurality of pixels, wherein the pixels include a photoelectric conversion unit, a charge holding unit that holds charge sent from the photoelectric conversion unit, and a conversion unit that converts the charge into a signal; the program causes a computer to execute the steps of: performing first control to hold charge generated in the photoelectric conversion unit in the charge holding unit and terminate exposure of the plurality of pixels; performing second control on a first pixel of the plurality of pixels; and performing third control on a second pixel of the plurality of pixels; the second control is control to send charge from the charge holding unit to the conversion unit a first time after the end of the exposure; and the third control is control to discharge charge from the charge holding unit between the end of the exposure and the first time after the end of the exposure, and to send charge from the charge holding unit to the conversion unit the first time after the end of the exposure.

Images