Imaging control device, imaging device, imaging control method, and imaging control program

Abstract

Provided are an imaging control device, an imaging device, an imaging control method, and an imaging control program.　The imaging control device comprises a processor that controls an imaging element including a plurality of pixels. Each of the pixels include a photoelectric conversion unit, a charge holding unit that holds a charge sent from the photoelectric conversion unit, and a conversion unit that converts the charge into a signal. The processor: performs a first control for sending to the conversion unit a charge held in the charge holding unit of each pixel in a first region among the plurality of pixels; performs a second control for sending to the conversion unit a charge held in the charge holding unit of each pixel in a second region among the plurality of pixels; performs, between the first control and the second control, a third control for sending to the conversion unit a charge held in the charge holding unit of each pixel in the first region; and performs a detection process for detecting, on the basis of a signal obtained from the conversion unit by means of the third control, noise occurring in the charge holding unit.

Description

Imaging control device, imaging device, imaging control method, and imaging control program

[0001] The present invention relates to an imaging control device, an imaging device, an imaging control method, and an imaging control program.

[0002] Patent Document 1 describes an imaging device that includes a signal processing unit that performs correction processing on pixel signals read in a first mode in which pixel signals are read out by thinning out lines from a pixel array unit in which pixels that generate pixel signals are arranged in a matrix, and pixel signals read in a second mode in which pixel signals are read out in the first mode, including the lines that were thinned out in the first mode.

[0003] Patent Document 2 describes an imaging device having a pixel section in which multiple pixels, each including a photoelectric conversion section that generates a signal according to the amount of incident light, are arranged two-dimensionally, and a readout drive section that performs a readout operation on a readout unit group basis, with one or more pixels in the pixel section as readout unit groups, and that obtains an imaging signal, the imaging device having a global shutter function that makes the exposure start timing and exposure period the same for all target pixels in the pixel section, a light amount distribution detection means that detects the distribution of the amount of light incident on the pixel section of the solid-state imaging element, and a drive control means that sets the readout order of the readout unit groups to the readout drive section based on the light amount distribution information of the pixel section detected by the light amount distribution detection means.

[0004] Japanese Patent Publication No. 2020-061605 Japanese Patent Publication No. 2008-042714

[0005] This specification describes at least the following:

[0006] (1) An imaging control device comprising a processor that controls an imaging element including a plurality of pixels, wherein the pixels include a photoelectric conversion unit, a charge holding unit that holds charges sent from the photoelectric conversion unit, and a conversion unit that converts the charges into signals, wherein the processor: performs first control to send charges held in the charge holding units of a first region of the plurality of pixels to the conversion unit; performs second control to send charges held in the charge holding units of a second region of the plurality of pixels to the conversion unit; performs third control between the first control and the second control to send charges held in the charge holding units of the first region to the conversion unit; and performs detection processing to detect noise occurring in the charge holding units based on a signal obtained from the conversion unit by the third control.

[0007] (2) The imaging control device according to (1), wherein the third control is a control for sending the charges held in the charge holding unit in a part of the first region to the conversion unit.

[0008] (3) An imaging control device according to (2), wherein, when the processor determines that the noise is occurring through the detection process, it performs a fourth control of sending the charges held in at least a part of the charge holding unit of the other part of the first region other than the part of the first region, or the charges held in the charge holding unit of the part of the first region, to the conversion unit.

[0009] (4) The imaging control device according to (3), wherein the fourth control is a control for sending the charges held in the charge holding unit of at least a part of the other part of the first region to the conversion unit.

[0010] (5) The imaging control device according to (4), wherein the fourth control is a control for sending the charge held in the charge holding unit of the other part of the first region to the conversion unit.

[0011] (6) The imaging control device according to any one of (3) to (5), in which the processor performs the fourth control after the second control.

[0012] (7) An imaging control device according to (1), wherein the processor further performs the third control at least once after the second control, and in the detection process, detects noise occurring in the charge holding unit based on a signal obtained from the conversion unit by multiple executions of the third control.

[0013] (8) An imaging control device according to (7), wherein the processor performs a fifth control to send the charge held in the charge holding unit of a third region of the plurality of pixels to the conversion unit, and determines whether to perform the third control after the fifth control based on a first signal obtained from the conversion unit by the third control between the first control and the second control, and a second signal obtained from the conversion unit by the third control between the second control and the fifth control.

[0014] (9) The imaging control device according to (8), wherein the processor determines the movement of the subject that is the source of the noise based on the first signal and the second signal, and determines whether to perform the third control after the fifth control based on the movement.

[0015] (10) An imaging control device according to any one of (7) to (9), wherein the processor determines a range of pixels for sending charge to the conversion unit in the third control performed after the second control, based on a third signal obtained from the conversion unit by the first control and a first signal obtained from the conversion unit by the third control performed between the first control and the second control.

[0016] (11) The imaging control device according to (10), wherein the processor determines the movement of the subject that is the source of the noise based on the first signal and the third signal, and determines the range based on the movement.

[0017] (12) An imaging control device according to any one of (7) to (11), wherein the processor determines whether to perform the third control after the second control based on a third signal obtained from the conversion unit by the first control and a first signal obtained from the conversion unit by the third control performed between the first control and the second control.

[0018] (13) The imaging control device according to (12), wherein the processor determines the movement of the subject that is the source of the noise based on the first signal and the third signal, and determines whether to perform the third control after the second control based on the movement.

[0019] (14) An imaging device including: the imaging control device according to any one of (1) to (13); and the imaging element.

[0020] (15) An imaging control method for controlling an imaging element including a plurality of pixels, wherein the pixels include a photoelectric conversion unit, a charge holding unit that holds charges sent from the photoelectric conversion unit, and a conversion unit that converts the charges into a signal; the imaging control method performs first control to send charges held in the charge holding units of a first region of the plurality of pixels to the conversion unit; performs second control to send charges held in the charge holding units of a second region of the plurality of pixels to the conversion unit; performs third control between the first control and the second control to send charges held in the charge holding units of the first region to the conversion unit; and performs detection processing to detect noise occurring in the charge holding units based on a signal obtained from the conversion unit by the third control.

[0021] (16) An imaging control program for controlling an imaging element including a plurality of pixels, wherein the pixels include a photoelectric conversion unit, a charge holding unit that holds charges sent from the photoelectric conversion unit, and a conversion unit that converts the charges into signals; the imaging control program causes a processor to execute the following steps: performing first control to send charges held in the charge holding units of a first region of the plurality of pixels to the conversion unit; performing second control to send charges held in the charge holding units of a second region of the plurality of pixels to the conversion unit; performing third control between the first control and the second control to send charges held in the charge holding units of the first region to the conversion unit; and performing detection processing to detect noise occurring in the charge holding units based on a signal obtained from the conversion unit by the third control.

[0022] FIG. 1 is a diagram showing a schematic configuration of a digital camera 100, which is an embodiment of an imaging device. FIG. 2 is a plan view schematic showing a schematic configuration of the image sensor 5 shown in FIG. 1. FIG. 3 is a plan view schematic showing a schematic configuration of a pixel 61 in the image sensor 5 shown in FIG. 2. FIG. 4 is a cross-sectional schematic view of the pixel 61 shown in FIG. 3 taken along line A-A. FIG. 5 is a timing chart showing operation of the digital camera 100 shown in FIG. 1 in an imaging mode. FIG. 6 is a timing chart showing another example of operation of the digital camera 100 shown in FIG. 1 in an imaging mode. FIG. 7 is a diagram illustrating a group of signals acquired by rolling readout driving. FIG. 8 is a diagram illustrating a case where the moving speed of a high-brightness subject is slower than in the case of FIG. 7. FIG. 9 is a diagram illustrating a case where a high-brightness subject is stationary. FIG. 10 is a flowchart for explaining the example of operation shown in FIG. 6. FIG. 11 is a flowchart for explaining a first modified example of the operation of the system control unit 11. FIG. 12 is a diagram illustrating a case where a high-brightness subject is moving. FIG. 13 is a flowchart for explaining a second modified example of the operation of the system control unit 11. Fig. 14 is an image diagram of a group of signals acquired by rolling readout driving when a high-brightness subject moves in the row direction X on the imaging surface 60. Fig. 15 shows the appearance of the smartphone 200. Fig. 16 is a block diagram showing the configuration of the smartphone 200 shown in Fig. 15.

[0023] Fig. 1 is a diagram showing the schematic configuration of a digital camera 100, which is an embodiment of an imaging device. The digital camera 100 shown in Fig. 1 includes a lens device 40 having an imaging lens 1, an aperture 2, a lens driver 8 that drives the imaging lens 1, an aperture driver 9 that drives the aperture 2, and a lens controller 4 that controls the lens driver 8 and the aperture driver 9, and a main body 100A.

[0024] The main body 100A includes an image sensor 5, a system control unit 11 that controls the overall electrical control system of the digital camera 100, an operation unit 14, a display device 22, a memory 16 that includes RAM (Random Access Memory) and ROM (Read Only Memory), a memory control unit 15 that controls the storage of data in the memory 16 and the reading of data from the memory 16, a digital signal processing unit 17, and an external memory control unit 20 that controls the storage of data in a storage medium 21 and the reading of data from the storage medium 21.

[0025] The lens device 40 may be detachable from the main body 100A, or may be integrated with the main body 100A. The imaging lens 1 may include at least one of a focus lens and a zoom lens that are movable in the optical axis direction.

[0026] The focus lens is a lens for adjusting the focus of an imaging optical system including the imaging lens 1 and the aperture 2, and is composed of a single lens or multiple lenses. When the focus lens moves in the optical axis direction, the position of the principal point of the focus lens (hereinafter also referred to as the focus lens position) changes along the optical axis, thereby changing the focal position on the subject side. Note that a liquid lens whose principal point position in the optical axis direction can be changed by electrical control may also be used as the focus lens.

[0027] The zoom lens is a lens for changing the focal length of an imaging optical system including an imaging lens 1 and an aperture 2, and is composed of a single lens or multiple lenses. The zoom magnification is changed by moving the zoom lens in the optical axis direction.

[0028] The lens control unit 4 of the lens device 40 controls the lens drive unit 8 based on the lens drive signal transmitted from the system control unit 11 to change the focus lens position and the zoom lens position. The lens control unit 4 of the lens device 40 controls the diaphragm drive unit 9 based on the drive control signal transmitted from the system control unit 11 to change the aperture size (F-number) of the diaphragm 2.

[0029] The image sensor 5 captures an image of a subject through an imaging optical system including an imaging lens 1 and an aperture 2. The image sensor 5 has an imaging surface 60 (see FIG. 2 ) on which a plurality of pixels are arranged two-dimensionally, and converts an image of a subject formed on the imaging surface 60 by the imaging optical system into an image signal using the plurality of pixels and outputs the image signal.

[0030] The imaging element 5 is, for example, a complementary metal-oxide semiconductor (CMOS) image sensor.

[0031] The system control unit 11 performs various processes and controls, and controls the entire digital camera 100. The system control unit 11 and the memory 16 constitute an imaging control device.

[0032] The system control unit 11 controls the image sensor 5 and the lens device 40 to output, as an image signal, an image of a subject captured through the imaging optical system of the lens device 40. The image signal output from the image sensor 5 is processed by the digital signal processing unit 17 to generate captured image data that is data suitable for display on the display device 22 or data suitable for storage in the storage medium 21.

[0033] An instruction signal from the user is input to the system control unit 11 through the operation unit 14. The operation unit 14 includes a touch panel integrated with the display surface 22b, various buttons, and the like.

[0034] The display device 22 includes a display surface 22b configured by an organic EL (electroluminescence) panel, a liquid crystal panel, or the like, and a display controller 22a that controls the display on the display surface 22b.

[0035] The memory control unit 15 , digital signal processing unit 17 , external memory control unit 20 , and display controller 22 a are interconnected by a control bus 24 and a data bus 25 , and are controlled by commands from the system control unit 11 .

[0036] Fig. 2 is a schematic plan view showing the general configuration of the image sensor 5 shown in Fig. 1. The image sensor 5 includes an image sensor surface 60 on which pixel rows 62, each consisting of a plurality of pixels 61 arranged in a row direction X, are arranged in a column direction Y that intersects with the row direction X (orthogonal in the illustrated example), a drive circuit 63 that drives the pixels 61 arranged on the image sensor surface 60, and a signal processing circuit 64 that processes pixel signals read out to signal lines from each pixel 61 in the pixel rows 62 arranged on the image sensor surface 60.

[0037] The pixel signals read out from the pixels 61 to the signal lines are analog signals. The signal processing circuit 64 includes a converter that converts the analog signals into digital signals. The pixel signals read out from the pixels 61 are converted into digital signals by the signal processing circuit 64 and output as digital signals to the outside of the image sensor 5. The board on which the image sensor 5 is mounted may be provided with a processing circuit that processes the digital pixel signals output from the image sensor 5. The board on which the system control unit 11 is provided and the board on which the image sensor 5 is provided may be the same board.

[0038] In the following description, the upper end of the imaging surface 60 in the column direction Y in FIG. 2 will be referred to as the upper end, and the lower end of the imaging surface 60 in the column direction Y will be referred to as the lower end.

[0039] The pixels 61 arranged on the imaging surface 60 are classified into a plurality of fields (regions). For example, let M be an integer equal to or greater than 0 (0, 1, 2, 3, ...), let n be the number of fields set on the imaging surface 60 (here, n = 3), and let k be a value between 1 and n. In the example of FIG. 2 , of all pixel rows 62 arranged on the imaging surface 60, the (n × M + k)th pixel row 62 counting from the top end of the imaging surface 60 is designated as field Fk. As shown in FIG. 2 , the pixels 61 arranged on the imaging surface 60 are classified into field F1, field F2, and field F3. This field division is an example and is not limited to this. Field F1 constitutes the first region. Field F2 constitutes the second region. Field F3 constitutes the third region.

[0040] Fig. 3 is a plan view showing a schematic configuration of a pixel 61 in the image sensor 5 shown in Fig. 2. Fig. 4 is a cross-sectional view of the pixel 61 taken along line AA shown in Fig. 3.

[0041] As shown in FIG. 3, the pixel 61 includes a photoelectric conversion unit 61A, a charge holding unit 61F, a charge transfer unit 61C, a floating diffusion 61D, and a readout circuit 61E.

[0042] The photoelectric conversion unit 61A receives light that has passed through the imaging optical system of the lens device 40 and generates and accumulates electric charges according to the amount of received light. The photoelectric conversion unit 61A is composed of a photodiode, etc. The charge holding unit 61F is composed of an impurity region in a semiconductor substrate.

[0043] The charge transfer unit 61C controls the height of the potential barrier between the photoelectric conversion unit 61A and the charge holding unit 61F. The charge transfer unit 61C is composed of an impurity region in a semiconductor substrate and an electrode formed above this impurity region.

[0044] For example, the potential barrier is raised at the start of exposure to start exposure of the photoelectric conversion unit 61A, and the potential barrier is lowered at the end of exposure. By this control, the charge generated in the photoelectric conversion unit 61A during the exposure period and stored therein is transferred to the charge holding unit 61F almost simultaneously with the end of exposure. Then, the potential barrier is returned to its original state, allowing the charge generated in the photoelectric conversion unit 61A during the exposure period to be held in the charge holding unit 61F.

[0045] It is also possible to provide the photoelectric conversion unit 61A with a potential gradient that decreases toward the charge holding unit 61F. When this configuration is adopted, the potential barrier is lowered at the timing of the start of exposure, so that the charge generated in the photoelectric conversion unit 61A by exposure moves to the charge holding unit 61F without being stored there. Thereafter, when the potential barrier is raised at the timing of the end of exposure, the exposure period ends and the charge generated in the photoelectric conversion unit 61A during the exposure period is held in the charge holding unit 61F.

[0046] In this way, the charge generated in the photoelectric conversion unit 61A can be held in the charge holding unit 61F by controlling the charge transfer unit 61C.

[0047] The floating diffusion 61D converts the electric charge into a signal, and the electric charge is transferred from the electric charge holding portion 61F to the floating diffusion 61D. The floating diffusion 61D constitutes a conversion portion.

[0048] The readout circuit 61E is a circuit that reads out a signal corresponding to the potential of the floating diffusion 61D as a pixel signal to a signal line 65. The readout circuit 61E is driven by a drive circuit 63.

[0049] As shown in FIG. 4, a P-well layer 71 is formed on the surface of an N-type substrate 70, and a photoelectric conversion section 61A is formed on the surface of the P-well layer 71.

[0050] The photoelectric conversion section 61A is composed of an N-type impurity layer 73 and a P-type impurity layer 74 formed thereon. The N-type substrate 70 and the P-well layer 71 form a semiconductor substrate.

[0051] A charge holding portion 61F made of an N-type impurity layer is formed on the surface of the P-well layer 71 at a slight distance from the photoelectric conversion portion 61A.

[0052] A transfer electrode 76 is formed above a region 75 of the P-well layer 71 between the charge holding portion 61F and the photoelectric conversion portion 61A, with an oxide film interposed therebetween.

[0053] 4, the transfer electrode 76 is formed up to above the charge holding portion 61F, but it is sufficient that the transfer electrode 76 is formed at least above the region 75.

[0054] The above-mentioned potential barrier can be lowered by controlling the potential of the transfer electrode 76 to form a channel in the region 75. The potential of the transfer electrode 76 is controlled by the drive circuit 63.

[0055] A floating diffusion 61D made of an N-type impurity layer is formed on the surface of the P-well layer 71 at a slight distance from the charge holding portion 61F.

[0056] A read electrode 72 is formed above the P-well layer 71 between the charge holding portion 61F and the floating diffusion 61D via an oxide film.

[0057] By controlling the potential of the readout electrode 72 and forming a channel in the region between the charge retention portion 61F and the floating diffusion 61D, the charge in the charge retention portion 61F can be transferred to the floating diffusion 61D. The potential of the readout electrode 72 is controlled by the drive circuit 63.

[0058] The readout circuit 61E is composed of a reset transistor 77 for resetting the potential of the floating diffusion 61D, an output transistor 78 for converting the potential of the floating diffusion 61D into a pixel signal and outputting it, and a selection transistor 79 for selectively reading out the pixel signal output from the output transistor 78 to the signal line 65. The configuration of the readout circuit is an example and is not limited to this. The readout circuit 61E may be shared by multiple pixels 61.

[0059] A light-shielding film is provided in the pixel 61, and the area other than the photoelectric conversion portion 61A is shielded from light by this light-shielding film.

[0060] The structure of the pixel 61 shown in FIGS. 3 and 4 is an example, and the present invention is not limited to this.

[0061] The drive circuit 63 shown in Figure 2 independently drives the transfer electrode 76, readout electrode 72, and readout circuit 61E of each pixel 61 to reset the photoelectric conversion unit 61A (discharge the charge accumulated in the photoelectric conversion unit 61A), hold the charge generated in the photoelectric conversion unit 61A in the charge holding unit 61F, and read out a pixel signal corresponding to the charge held in the charge holding unit 61F to the signal line 65.

[0062] The photoelectric conversion unit 61A is reset by forming a channel in the semiconductor substrate below the transfer electrode 76 and also forming a channel in the semiconductor substrate below the readout electrode 72, and then discharging the charge in the floating diffusion 61D using the reset transistor 77.

[0063] 2 performs correlated double sampling on pixel signals read out to signal lines 65 from each pixel 61 in a pixel row 62, converts the pixel signals after correlated double sampling into digital signals, and outputs the digital signals to the data bus 25 (see FIG. 1). The signal processing circuit 64 is controlled by the system control unit 11. The digital signal processing unit 17 performs signal processing such as demosaic processing and gamma correction processing on the pixel signals output from the image sensor 5 to the data bus 25 to generate captured image data.

[0064] The system control unit 11 can drive the image sensor 5 in each of global reset driving, global shutter driving, and rolling readout driving.

[0065] The global reset driving is a driving method in which the photoelectric conversion units 61A of all pixels 61 formed on the imaging surface 60 of the imaging element 5 are simultaneously reset, and exposure of all pixels 61 is simultaneously started.

[0066] Global shutter driving is a driving method in which the charge generated in the photoelectric conversion unit 61A of each pixel 61 by exposure initiated in all pixels 61 by global reset driving is simultaneously held in the charge holding unit 61F of each pixel 61, thereby ending exposure in all pixels 61 simultaneously.

[0067] The rolling readout driving is a driving method in which signals corresponding to the charges held in the charge holding section 61F are read out sequentially for each pixel row 62.

[0068] Fig. 5 is a timing chart showing the operation of the digital camera 100 shown in Fig. 1 in the imaging mode. Fig. 5 shows the drive timing of the photoelectric conversion units 61A and charge holding units 61F of each pixel row 62 of the image sensor 5. In Fig. 6, the vertical axis indicates the position of the pixel row 62 in the column direction Y.

[0069] A straight line GR shown in FIG. 5 indicates the timing at which each photoelectric conversion unit 61A included in a pixel row 62 is reset by global reset driving.

[0070] The lines GS and ST shown in Figure 5 indicate the timing at which the charges generated in each photoelectric conversion unit 61A included in the pixel row 62 due to exposure started by global reset driving are held in the charge holding unit 61F by global shutter driving.

[0071] The period enclosed by the lines GR and GS is the exposure period EX of the image sensor 5 when capturing an image for storage in the storage medium 21 .

[0072] The lines R1, R2, and R3 shown in Figure 5 indicate the timing at which pixel signals corresponding to the charges held in the charge holding section 61F by global shutter driving are output from the image sensor 5 by rolling readout driving.

[0073] At time t0, the system control unit 11 performs control to execute global reset driving indicated by the straight line GR, and simultaneously resets the photoelectric conversion units 61A in all pixel rows 62 formed on the imaging surface 60. As a result, exposure starts at the same timing in all pixel rows 62 formed on the imaging surface 60.

[0074] After that, when a predetermined exposure time has elapsed, the system control unit 11 performs control to execute global shutter driving, indicated by the line GS, at time t1. This global shutter driving simultaneously transfers electric charges from the photoelectric conversion units 61A to the charge holding units 61F in all pixel rows 62 formed on the imaging surface 60, and the electric charges are held in the charge holding units 61F, as indicated by the line ST. This completes exposure at the same time for all pixel rows 62 formed on the imaging surface 60. In Figure 5, the period enclosed by the lines GR and GS is indicated as the exposure period EX.

[0075] The system control unit 11 performs control to execute global shutter driving indicated by line GS, and then controls to execute rolling readout driving indicated by line R1. This rolling readout driving selects pixel rows 62 in one of fields F1, F2, and F3 (field F1 in this example) sequentially from the top to the bottom of the imaging surface 60. Then, charges are transferred from the charge holding unit 61F of the selected pixel row 62 to the floating diffusion 61D, and a pixel signal corresponding to this charge is read out by the readout circuit 61E.

[0076] A set of pixel signals read out from field F1 by the rolling readout driving indicated by line R1 is referred to as pixel signal group G1. Control for causing drive circuit 63 to execute the rolling readout driving indicated by line R1 constitutes first control. Pixel signal group G1 constitutes third signal.

[0077] After completing the readout of pixel signal group G1 at time t2, the system control unit 11 performs control to execute the rolling readout drive indicated by line R2 at a subsequent time t3. This rolling readout drive selects pixel rows 62 in one of two fields (field F2 in this example) among fields F1, F2, and F3 that were not readout by the rolling readout drive of line R1, sequentially from the top to the bottom of the imaging surface 60. Then, charges are transferred from the charge holding unit 61F of the selected pixel row 62 to the floating diffusion 61D, and a signal corresponding to this charge is read out by the readout circuit 61E.

[0078] A set of pixel signals read out from field F2 by the rolling readout driving indicated by line R2 is referred to as pixel signal group G2. Control for causing drive circuit 63 to execute the rolling readout driving indicated by line R2 constitutes second control.

[0079] When the system control unit 11 finishes reading out pixel signals G2 at time t4, it performs control to execute the rolling readout drive indicated by line R3. This rolling readout drive selects pixel rows 62 in fields F1, F2, and F3 that were not read out by the rolling readout drives of lines R1 and R2 (here, field F3) in order from the top to the bottom of the imaging surface 60. Then, charges are transferred from the charge holding unit 61F of the selected pixel row 62 to the floating diffusion 61D, and a signal corresponding to this charge is read out by the readout circuit 61E.

[0080] A set of pixel signals read out from field F3 by the rolling readout driving indicated by line R3 is referred to as pixel signal group G3. The control for causing drive circuit 63 to execute the rolling readout driving indicated by line R3 constitutes fifth control.

[0081] The digital signal processing unit 17 processes the pixel signal groups G1, G2, and G3 read out in this manner to generate captured image data, and stores the data in the storage medium 21.

[0082] The period enclosed by the lines ST and R3 is referred to as the signal readout period. Although the charge storage unit 61F is light-shielded, if strong light is irradiated onto the light-shielding film above the charge storage unit 61F during this signal readout period, the light may leak and generate noise charges below the area irradiated by the light.

[0083] In this embodiment, in order to determine whether noise (noise caused by noise charge) that may affect the quality of the captured image data has occurred, the system control unit 11 controls the reading of a signal corresponding to the charge of the charge holding unit 61F of field F1 after the pixel signal has been read out during the period between line R1 and line R2.

[0084] Specifically, when the system control unit 11 finishes reading out the pixel signals G1 at time t2, it performs control to execute a rolling readout drive indicated by a straight line R1a. This rolling readout drive selects some pixel rows 62 in field F1 (for example, the (3×M+1)th pixel row 62 counting from the top in field F1) in order from the top to the bottom of the imaging surface 60. Then, charges are transferred from the charge holding unit 61F of the selected pixel row 62 to the floating diffusion 61D, and a signal corresponding to this charge is read out by the readout circuit 61E.

[0085] The set of signals read out from field F1 by the rolling readout driving indicated by line R1a is referred to as noise signal group N1. Control for causing drive circuit 63 to execute the rolling readout driving indicated by line R1a constitutes third control. This third control may be performed between the first control and the second control.

[0086] The noise signal group N1 obtained by this third control is a signal corresponding to the charge generated and held in the charge holding unit 61F during the period between the lines R1 and R1a. Therefore, by using this noise signal group N1, the amount of noise charge generated during the period between the lines ST and R1 can be determined. In this embodiment, the system control unit 11 performs a detection process to detect noise (noise charge) generated in the charge holding unit 61F based on the noise signal group N1.

[0087] In this detection process, the system control unit 11 determines that noise that has a significant impact on the captured image data has occurred if, for example, the signals that make up the noise signal group N1 include a predetermined number or more of signals that are at a level above a threshold value, and determines that no noise has occurred in any other cases.

[0088] When it is determined that noise is occurring, the system control unit 11 performs rolling readout driving indicated by a straight line R1b after completing rolling readout driving indicated by a straight line R3, as shown in FIG.

[0089] This rolling readout driving selects pixel rows 62 other than a portion of field F1 (for example, in field F1, the (3×M+2)th pixel row 62 counting from the top and the (3×M+3)th pixel row 62 counting from the top) in order from the top to the bottom of imaging surface 60. Then, charges are transferred from charge holding portions 61F of the selected pixel rows 62 to floating diffusions 61D, and signals corresponding to these charges are read out by readout circuit 61E.

[0090] A set of signals read out from field F1 by the rolling readout driving indicated by line R1b is referred to as noise signal group N2. The control for causing drive circuit 63 to execute the rolling readout driving indicated by line R1b constitutes fourth control.

[0091] After the rolling readout drive indicated by the straight line R1b is completed, the system control unit 11 can improve the quality of the captured image data by correcting the noise caused by the noise charges generated in the charge holding unit 61F, which are contained in the pixel signal group G1, pixel signal group G2, and pixel signal group G3, based on the noise signal group N1 and the noise signal group N2.

[0092] On the other hand, if the system control unit 11 determines that no noise is occurring, it does not perform the rolling readout driving indicated by the straight line R1b in Fig. 5. By detecting the occurrence of noise based on the noise signal group N1 in this way, it is possible to shorten the time until imaging is completed when it is determined that no noise is occurring.

[0093] If the system control unit 11 determines that noise is occurring, it may start the same rolling readout driving as indicated by line R1a at time t5, instead of the rolling readout driving indicated by line R1b. This also makes it possible to correct noise caused by noise charges generated in the charge holding unit 61F. However, obtaining noise signal groups N1 and N2 allows for more accurate noise correction.

[0094] Furthermore, in the rolling readout driving indicated by the line R1b, the system control unit 11 may read signals from only a portion of the other portion of the field F1, rather than from all of the other portion. This also makes it possible to correct noise caused by noise charges generated in the charge holding unit 61F. This also shortens the time required to complete imaging.

[0095] Fig. 6 is a timing chart showing another example of operation in imaging mode of the digital camera 100 shown in Fig. 1. The timing chart shown in Fig. 6 is similar to Fig. 5 in that a rolling readout drive indicated by a line R1a is added between the rolling readout drive indicated by a line R2 and the rolling readout drive indicated by a line R3, and the rolling readout drive indicated by a line R1b is replaced with the rolling readout drive indicated by a line R1a.

[0096] In the operation example of FIG. 6 , the system control unit 11 acquires a first noise signal group N1 (hereinafter also referred to as noise signal group N1-1) at time t3, acquires a second noise signal group N1 (hereinafter also referred to as noise signal group N1-2) at time t5, and acquires a third noise signal group N1 (hereinafter also referred to as noise signal group N1-3) at time t7. Based on these three noise signal groups N1, the system control unit 11 detects noise (noise caused by noise charges) that may affect the quality of the captured image data. The noise signal group N1-1 constitutes a first signal. The noise signal group N1-2 constitutes a second signal.

[0097] 7 is an image diagram of a signal group acquired by rolling readout driving. Here, an example is shown in which a dark environment, such as a night scene, is captured. The hatched areas in the figure indicate areas with low signal levels. The pixel signal group G1 includes a saturated region H where the signal level reaches the upper limit. The saturated region H corresponds to a high-brightness subject that generates noise charges.

[0098] FIG. 7 illustrates an example in which, after pixel signal group G1 is read out, a high-brightness subject moves toward the lower end of the imaging surface 60. In this case, noise charges are generated in the charge storage unit 61F located along the path of movement of the high-brightness subject during the period between line R1 and the first line R1a. Therefore, as shown in FIG. 7, noise signal group N1-1 includes a region H1 with a high signal level and a shape corresponding to the movement trajectory of the high-brightness subject. Furthermore, noise signal group N1-2 includes a region H2 with a high signal level and a shape corresponding to the movement trajectory of the high-brightness subject. Because the high-brightness subject is moving, region H2 is located below region H1.

[0099] 7, it is possible to determine whether a high-brightness subject, which is a source of noise, is moving. Furthermore, it is also possible to determine the speed and direction of movement of the high-brightness subject when it is moving. Furthermore, it is also possible to determine whether a high-brightness subject is within the imaging range during the period between the second line R1a and the line R3, based on the position, speed, and direction of movement of the high-brightness subject.

[0100] Assume that a high-brightness subject is outside the imaging range during the period between the second line R1a and the line R3. In this case, the noise signal group N1-3 obtained by the rolling readout driving indicated by the third line R1a does not include a portion corresponding to the high-brightness subject. Therefore, this rolling readout driving is unnecessary.

[0101] Figure 8 is an illustration of a case where the moving speed of the high-brightness subject is slower than in the case of Figure 7. In the example of Figure 8, it can be determined from the noise signal group N1-1 and the noise signal group N1-2 that there is a high possibility that the high-brightness subject is in the imaging range during the period between the second line R1a and the line R3. If the high-brightness subject is in the imaging range during this period, the noise signal group N1-3 obtained by the rolling readout driving indicated by the third line R1a will include a portion corresponding to the high-brightness subject. For this reason, this rolling readout driving is necessary for noise correction.

[0102] If the system control unit 11 determines, based on the noise signal groups N1-1 and N1-2, that a high-brightness subject will move outside the imaging range during the period between the second line R1a and the line R3, the system control unit 11 does not perform the rolling readout driving indicated by the third line R1a. This shortens the time until imaging is completed. In this case, it is sufficient to correct the noise contained in the pixel signal groups G1, G2, and G3 based on the signals in the regions H1 and H2 of the noise signal groups N1-1 and N1-2.

[0103] On the other hand, if the system control unit 11 determines, based on the noise signal groups N1-1 and N1-2, that a high-brightness subject is within the imaging range during the period between the second line R1a and the line R3, it performs rolling readout driving as indicated by the third line R1a. In this case, it is sufficient to correct the noise contained in the pixel signal groups G1, G2, and G3 based on the noise signal groups N1-1, N1-2, and N1-3. This allows for highly accurate correction of noise caused by noise charges.

[0104] 9 is an image diagram of a stationary high-brightness subject. When the high-brightness subject is stationary, the position of the saturated region H in the pixel signal group G1 coincides with the regions H1 and H2 in the noise signal groups N1-1 and N1-2. Because the saturated region H is a region where the signal level is originally saturated, it is difficult to perform accurate correction. In other words, even if the noise signal group N1-3 is acquired, there is little need to use it for correction.

[0105] Therefore, when the system control unit 11 determines based on the noise signal groups N1-1 and N1-2 that the movement of the high-brightness subject is equal to or less than the movement threshold (in other words, the subject is stationary), it may determine that no noise that could affect the quality of the captured image data has occurred and may not perform the rolling readout driving indicated by the third line R1a. This can shorten the time until image capture is completed. In addition, in this case, the time until image capture is completed can be further shortened by not performing noise correction based on the noise signal groups N1-1 and N1-2.

[0106] Fig. 10 is a flowchart for explaining the operation example shown in Fig. 6. After the exposure is completed, the system control unit 11 reads out the pixel signal group G1 from the field F1 (step S1). Next, the system control unit 11 reads out the noise signal group N1-1 from a part of the pixel rows 62 in the field F1 (step S2).

[0107] Next, the system control unit 11 determines whether or not noise that may affect the quality of the captured image data is occurring based on the noise signal group N1-1 (step S3). For example, the system control unit 11 determines that noise is occurring when the noise signal group N1-1 includes a predetermined number or more signals whose levels are equal to or higher than a threshold value.

[0108] If the determination in step S3 is NO, the system control unit 11 reads out the pixel signal group G2 from field F2 and the pixel signal group G3 from field F3, and then ends the imaging.

[0109] If the determination in step S3 is YES, the system control unit 11 reads out the pixel signal group G2 from the field F2 (step S4), and then performs the same process as in step S2 (step S5).

[0110] Next, the system control unit 11 determines the movement of the high-brightness subject that is the source of the noise based on the noise signal group N1-2 obtained in step S5 and the noise signal group N1-1 obtained in step S2, and based on that movement, determines whether noise caused by the high-brightness subject will occur in the period between the second line R1a and line R3, i.e., the period until the pixel signals from the next field are read out (step S6).

[0111] If it is determined that no high-brightness subject is present in the imaging range during the period between the second line R1a and line R3, the determination in step S6 is NO. If the determination in step S6 is NO, pixel signal group G3 is read out from field F3 in step S9. Thereafter, pixel signal group G1, pixel signal group G2, and pixel signal group G3 are corrected based on noise signal group N1-1 and noise signal group N1-2 as necessary, and imaging is completed.

[0112] If it is determined that a high-brightness subject is present in the imaging range during the period between the second line R1a and the line R3, the determination in step S6 is YES. If the determination in step S6 is YES, the system control unit 11 reads out the pixel signal group G3 from the field F3 (step S7), and then performs the same process as in step S2 to obtain the noise signal group N1-3 (step S8).

[0113] Next, the system control unit 11 corrects the pixel signal groups G1, G2, and G3 based on the noise signal groups N1-1, N1-2, and N1-3, and ends the imaging.

[0114] In step S6, it is determined whether noise occurs before the pixel signals of field F3 are read out based on the movement of the high-brightness subject, but this is not limited to this. For example, it is possible that noise caused by a high-brightness subject is included in the noise signal group N1-1, and that noise is absent from the noise signal group N1-2. This would be the case if the high-brightness subject were a car headlight and the headlights were turned off.

[0115] Therefore, if the system control unit 11 determines in step S6 that the high-brightness subject has disappeared, it determines that no noise will occur until the pixel signals of field F3 are read out, and proceeds to step S9. Alternatively, if the system control unit 11 determines in step S6 that the high-brightness subject has not disappeared, it may proceed to step S7, or may proceed to either step S7 or step S9 based on the movement of the high-brightness subject.

[0116] Fig. 11 is a flowchart for explaining a first modified example of the operation of the system control unit 11. The flowchart shown in Fig. 11 is similar to the flowchart shown in Fig. 10 except that step S6a is added between step S3 and step S4.

[0117] In step S6a, the system control unit 11 determines the movement of the high-brightness subject that is the source of noise based on the pixel signal group G1 obtained in step S1 and the noise signal group N1-1 obtained in step S2, and determines, based on that movement, whether or not a high-brightness subject is present in the imaging range during the period between the first line R1a and the line R2. If it is determined that a high-brightness subject is present in the imaging range during this period, the determination in step S6a becomes YES, and the processing from step S4 onwards is carried out.

[0118] For example, assume that the pixel signal group G1 and the noise signal group N1-1 are as shown in Figure 12. In this case, it can be determined that there is a high possibility that the high-brightness subject will move outside the imaging range during the period from when the noise signal group N1-1 is read out to when the pixel signals of field F2 are read out. Therefore, in such a case, the processes of steps S5, S6, and S8 are not performed, thereby making it possible to shorten the time until imaging is completed compared to the example of Figure 10.

[0119] The above-described determination of the movement of a high-brightness subject can also be performed using three or more noise signal groups N1, or the pixel signal group G1 and two or more noise signal groups N1.

[0120] Fig. 13 is a flowchart for explaining a second modified example of the operation of the system control unit 11. The flowchart shown in Fig. 13 is similar to the flowchart shown in Fig. 10 except that steps S11 and S12 are added between steps S3 and S4.

[0121] In step S11, the system control unit 11 determines the movement of the high-brightness subject that is the source of noise, based on the pixel signal group G1 obtained in step S1 and the noise signal group N1-1 obtained in step S2. If the system control unit 11 determines that the high-brightness subject is moving (step S11: YES), in step S12, the system control unit 11 determines the range of pixels 61 from which signals are to be read out in steps S5 and S8, based on the direction of movement of the high-brightness subject.

[0122] 14 is an image diagram of a signal group acquired by rolling readout driving when a high-brightness subject moves in the row direction X on the imaging surface 60. In the example shown in Fig. 14, it can be determined from the saturated region H of the pixel signal group G1 and the region H1 of the noise signal group N1-1 that the high-brightness subject is present only in a specific range in the column direction Y and is moving in the row direction X.

[0123] In such a case, it is possible to acquire noise caused by a high-brightness subject in the noise signal group N1-2 and the noise signal group N1-3 without reading out signals from the entire imaging surface 60. Therefore, in such a case, the system control unit 11 determines a specific range A in the column direction Y that includes the region H1 included in the noise signal group N1-1. The specific range A is narrower than the width of the imaging surface 60 in the column direction Y.

[0124] The system control unit 11 determines the specific range A in step S12 of Fig. 13. Then, in steps S4 and S7 performed thereafter, the system control unit 11 reads out signals only from the pixels 61 that are in this specific range A, among the pixels 61 from which the noise signal group N1-1 was read out in step S2.

[0125] As a result, the noise signal groups N1-2 and N1-3 have a horizontally elongated shape as shown in FIG. 14, but they include areas H2 and H3 corresponding to high-brightness subjects.

[0126] Therefore, by using the noise signal group N1-1, the noise signal group N1-2, and the noise signal group N1-3, it is possible to correct noise caused by a high-brightness subject. According to the operation shown in Fig. 13, the readout range of the noise signal group N1-2 and the noise signal group N1-3 is reduced, thereby shortening the time until imaging is completed.

[0127] Next, the configuration of a smartphone, which is another embodiment of an imaging device according to the technique of the present disclosure, will be described.

[0128] Fig. 15 shows the external appearance of a smartphone 200. The smartphone 200 shown in Fig. 15 has a flat housing 201, and is provided on one surface of the housing 201 with a display panel 202 as a display unit and a display input unit 204 that is an integrated unit of an operation panel 203 as an input unit.

[0129] Such a housing 201 also includes a speaker 205, a microphone 206, an operation unit 207, and a camera unit 208. Note that the configuration of the housing 201 is not limited to this, and for example, it is also possible to adopt a configuration in which the display unit and the input unit are independent, or a configuration having a folding structure or a sliding mechanism.

[0130] FIG. 16 is a block diagram showing the configuration of the smartphone 200 shown in FIG.

[0131] As shown in FIG. 16 , the main components of the smartphone include a wireless communication unit 210, a display input unit 204, a call unit 211, an operation unit 207, a camera unit 208, a memory unit 212, an external input / output unit 213, a GNSS (Global Navigation Satellite System) receiving unit 214, a motion sensor unit 215, a power supply unit 216, and a main control unit 220.

[0132] The smartphone 200 also has, as its main function, a wireless communication function for performing mobile wireless communication via a base station device BS (not shown) and a mobile communication network NW (not shown).

[0133] The wireless communication unit 210 performs wireless communication with a base station device BS accommodated in the mobile communication network NW in accordance with instructions from the main control unit 220. Using this wireless communication, various file data such as audio data and image data, e-mail data, etc. are sent and received, and web data, streaming data, etc. are received.

[0134] The display input unit 204 is a so-called touch panel that, under the control of the main control unit 220, displays images (still images and moving images) or text information, etc. to visually convey information to the user and detects user operations on the displayed information, and is equipped with a display panel 202 and an operation panel 203.

[0135] The display panel 202 uses a liquid crystal display (LCD), an organic electroluminescence display (OELD), or the like as a display device.

[0136] The operation panel 203 is a device placed so that an image displayed on the display surface of the display panel 202 can be seen, and detects one or more coordinates operated by a user's finger or a stylus. When this device is operated by the user's finger or a stylus, a detection signal generated by the operation is output to the main control unit 220. Next, the main control unit 220 detects the operation position (coordinates) on the display panel 202 based on the received detection signal.

[0137] As shown in Figure 16, the display panel 202 and operation panel 203 of a smartphone 200, which is an example of one embodiment of the imaging device of the present invention, are integrated to form a display input unit 204, and the operation panel 203 is positioned so that it completely covers the display panel 202.

[0138] When such an arrangement is adopted, operation panel 203 may also have a function for detecting user operations in areas outside display panel 202. In other words, operation panel 203 may have a detection area for the overlapping portion that overlaps display panel 202 (hereinafter referred to as a display area), and a detection area for the remaining outer edge portion that does not overlap display panel 202 (hereinafter referred to as a non-display area).

[0139] The size of the display area and the size of the display panel 202 may be exactly the same, but they do not necessarily have to be the same. The operation panel 203 may also have two sensitive areas: an outer edge portion and an inner portion. The width of the outer edge portion is designed appropriately depending on the size of the housing 201, etc.

[0140] Furthermore, the position detection method used in the operation panel 203 may be a matrix switch method, a resistive film method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, a capacitance method, or the like, and any method may be used.

[0141] The call unit 211 is equipped with a speaker 205 or a microphone 206, and converts the user's voice input through the microphone 206 into voice data that can be processed by the main control unit 220 and outputs it to the main control unit 220, or decodes voice data received by the wireless communication unit 210 or the external input / output unit 213 and outputs it from the speaker 205.

[0142] Also, as shown in FIG. 15, for example, the speaker 205 can be mounted on the same surface as the display input unit 204, and the microphone 206 can be mounted on the side of the housing 201.

[0143] The operation unit 207 is a hardware key using a key switch or the like, and receives instructions from the user. For example, as shown in Fig. 15 , the operation unit 207 is a push-button switch mounted on the side surface of the housing 201 of the smartphone 200, which turns on when pressed with a finger or the like, and turns off when the finger is released due to the restoring force of a spring or the like.

[0144] The storage unit 212 stores the control program and control data of the main control unit 220, application software, address data associated with names or telephone numbers of communication partners, data of sent and received e-mails, web data downloaded by web browsing, downloaded content data, and also temporarily stores streaming data, etc. The storage unit 212 is composed of an internal storage unit 217 built into the smartphone and an external storage unit 218 having a removable external memory slot.

[0145] The internal memory unit 217 and the external memory unit 218 constituting the memory unit 212 are realized using storage media such as a flash memory type, a hard disk type, a multimedia card micro type, a card-type memory (e.g., MicroSD (registered trademark) memory, etc.), a RAM (Random Access Memory), a ROM (Read Only Memory), etc.

[0146] The external input / output unit 213 serves as an interface with all external devices connected to the smartphone 200, and is used to directly or indirectly connect to other external devices via communication (e.g., Universal Serial Bus (USB), IEEE 1394, Bluetooth (registered trademark), RFID (Radio Frequency Identification), Infrared Data Association (IrDA) (registered trademark), UWB (Ultra Wideband) (registered trademark), ZigBee (registered trademark), etc.) or a network (e.g., Ethernet (registered trademark), wireless LAN (Local Area Network), etc.).

[0147] Examples of external devices connected to the smartphone 200 include a wired / wireless headset, a wired / wireless external charger, a wired / wireless data port, a memory card connected via a card socket, a SIM (Subscriber Identity Module Card) / UIM (User Identity Module Card) card, an external audio / video device connected via an audio / video I / O (Input / Output) terminal, a wirelessly connected external audio / video device, a wired / wirelessly connected smartphone, a wired / wirelessly connected personal computer, a wired / wirelessly connected personal computer, and earphones.

[0148] The external input / output unit 213 can transmit data received from such external devices to each component inside the smartphone 200, or transmit data inside the smartphone 200 to external devices.

[0149] The GNSS receiving unit 214 receives GNSS signals transmitted from the GNSS satellites ST1 to STn in accordance with instructions from the main control unit 220, executes positioning calculation processing based on the received multiple GNSS signals, and detects a position consisting of the latitude, longitude, and altitude of the smartphone 200. When the GNSS receiving unit 214 can acquire position information from the wireless communication unit 210 or the external input / output unit 213 (for example, a wireless LAN), it can also detect the position using the position information.

[0150] The motion sensor unit 215 includes, for example, a three-axis acceleration sensor, and detects the physical movement of the smartphone 200 in accordance with instructions from the main control unit 220. By detecting the physical movement of the smartphone 200, the direction of movement or acceleration of the smartphone 200 is detected. The detection result is output to the main control unit 220.

[0151] The power supply unit 216 supplies power stored in a battery (not shown) to each unit of the smartphone 200 in accordance with instructions from the main control unit 220 .

[0152] The main control unit 220 includes a microprocessor, operates according to the control program and control data stored in the storage unit 212, and controls all the units of the smartphone 200. The microprocessor of the main control unit 220 has the same functions as the system control unit 11. The main control unit 220 also includes a mobile communication control function that controls all the units of the communication system to perform voice communication or data communication via the wireless communication unit 210, and an application processing function.

[0153] The application processing function is realized by the main control unit 220 operating in accordance with the application software stored in the storage unit 212. Examples of the application processing function include an infrared communication function that controls the external input / output unit 213 to perform data communication with a connected device, an email function that sends and receives emails, and a web browsing function that views web pages.

[0154] The main control unit 220 also has an image processing function for displaying video on the display input unit 204 based on image data (still image or moving image data) such as received data or downloaded streaming data.

[0155] The image processing function refers to a function in which the main control unit 220 decodes the image data, performs image processing on the decoded result, and displays the image on the display input unit 204 .

[0156] Furthermore, the main control unit 220 executes display control for the display panel 202 and operation detection control for detecting user operations via the operation unit 207 and the operation panel 203 .

[0157] By executing display control, the main control unit 220 displays software keys such as icons or scroll bars for starting application software, or displays a window for creating an e-mail.

[0158] The scroll bar refers to a software key for receiving an instruction to move the displayed portion of an image, such as a large image that cannot fit in the display area of ​​the display panel 202 .

[0159] In addition, by executing operation detection control, the main control unit 220 detects user operations through the operation unit 207, accepts operations on the above icons and input of character strings into the input field of the above window through the operation panel 203, or accepts requests to scroll the displayed image through the scroll bar.

[0160] Furthermore, by executing operation detection control, the main control unit 220 determines whether the operation position on the operation panel 203 is an overlapping portion (display area) that overlaps the display panel 202 or an outer edge portion (non-display area) that does not overlap the display panel 202, and is equipped with a touch panel control function that controls the sensitive area of ​​the operation panel 203 or the display position of the software key.

[0161] The main control unit 220 can also detect a gesture operation on the operation panel 203 and execute a preset function in response to the detected gesture operation.

[0162] Gesture operation is not a simple touch operation as in the past, but rather refers to an operation of drawing a path with a finger or the like, specifying multiple positions simultaneously, or combining these to draw a path for at least one of multiple positions.

[0163] The camera unit 208 includes the lens device 40, the image sensor 5, and the digital signal processing unit 17 shown in FIG.

[0164] The captured image data generated by the camera unit 208 can be stored in the storage unit 212 or output via the external input / output unit 213 or the wireless communication unit 210 .

[0165] In the smartphone 200 shown in FIG. 16 , the camera unit 208 is mounted on the same surface as the display input unit 204, but the mounting position of the camera unit 208 is not limited to this, and it may be mounted on the back surface of the display input unit 204.

[0166] The camera unit 208 can also be used for various functions of the smartphone 200. For example, an image acquired by the camera unit 208 can be displayed on the display panel 202, or an image from the camera unit 208 can be used as one of the operation inputs for the operation panel 203.

[0167] Furthermore, when the GNSS receiving unit 214 detects a position, it can also detect the position by referring to an image from the camera unit 208. Furthermore, it can also refer to an image from the camera unit 208 to determine the optical axis direction of the camera unit 208 of the smartphone 200 or determine the current usage environment without using a triaxial acceleration sensor or by using the image in combination with a triaxial acceleration sensor. Of course, the image from the camera unit 208 can also be used in application software.

[0168] In addition, image data of still images or videos can be added with location information acquired by the GNSS receiving unit 214, audio information acquired by the microphone 206 (which may be converted to text information by the main control unit, etc.), posture information acquired by the motion sensor unit 215, etc., and stored in the memory unit 212 or output via the external input / output unit 213 or wireless communication unit 210.

[0169] In this embodiment, each process (each control) for driving the image sensor 5 is executed by a computer. The computer may execute these processes using a processor, a program, or a combination thereof. The computer may be a general-purpose computer, a computer for a specific application, a system such as a workstation, or other hardware element capable of executing a program.

[0170] The processor may be configured with one or more pieces of hardware, and the type of hardware is not limited. For example, the processor may be configured with hardware such as a programmable logic device such as a central processing unit (CPU), a micro processing unit (MPU), a field programmable gate array (FPGA), a dedicated circuit for executing specific processing such as an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or a neural processing unit (NPU). The processor also has various units or means for executing various processes in this embodiment. The type of hardware may also be a combination of different types of hardware. When multiple pieces of hardware are configured to execute one or more processes of a certain processor, the multiple pieces of hardware may be located in devices physically separated from each other, or may be located in the same device. Furthermore, in any embodiment, the order of each process performed by the processor is not limited to the order described above and may be changed as appropriate. The hardware is configured by an electric circuit (circuitry) or the like that combines circuit elements such as semiconductor elements.

[0171] Furthermore, the present embodiment may be implemented by hardware, software, firmware, microcode, or a combination thereof. Software, firmware, and microcode may be configured by a program. A program may also be, for example, a group of program modules, each function of which may be implemented by a processor configured to perform the respective function. The program may be program code or multiple code segments stored in one or more non-transitory computer-readable media (e.g., storage media or other storages). The program may be stored in multiple non-transitory computer-readable media that reside in physically separate devices. Program code or a code segment may represent a procedure, a function, a subprogram, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. Program code or a code segment may be connected to another code segment or a hardware circuit by sending or receiving information, data, arguments, parameters, or memory contents.

[0172] Although various embodiments have been described above, it goes without saying that the present invention is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present invention. Furthermore, the components of the above embodiments may be combined in any manner as long as they do not deviate from the spirit of the invention.

[0173] This application is based on a Japanese patent application (Patent Application No. 2024-148580) filed on August 30, 2024, the contents of which are incorporated herein by reference.

[0174] REFERENCE SIGNS LIST 1 imaging lens 2 aperture 4 lens control unit 5 imaging element 8 lens driving unit 9 aperture driving unit 11 system control unit 14, 207 operation unit 15 memory control unit 16 memory 17 digital signal processing unit 20 external memory control unit 21 storage medium 22 display device 22a display controller 22b display surface 24 control bus 25 data bus 40 lens device 60 imaging surface 61 pixel 61A photoelectric conversion unit 61F charge holding unit 61C charge transfer unit 61D floating diffusion 61E readout circuit 62 pixel row 63 drive circuit 64 signal processing circuit 65 signal line 70 N-type substrate 71 P-well layer 72 readout electrode 73 N-type impurity layer 74 P-type impurity layer 75 region 76 transfer electrode 77 reset transistor 78 Output transistor 79 Selection transistor 100 Digital camera 100A Main body 200 Smartphone 201 Housing 202 Display panel 203 Operation panel 204 Display input unit 205 Speaker 206 Microphone 208 Camera unit 210 Wireless communication unit 211 Call unit 212 Memory unit 213 External input / output unit 214 GNSS receiving unit 215 Motion sensor unit 216 Power supply unit 217 Internal memory unit 218 External memory unit 220 Main control unit F1, F2, F3 Fields GR, GS, ST, R1a, R1b, R1, R2, R3 Straight line H Saturation region H1, H2, H3 Region G1 Pixel signal group N1-1, N1-2, N1-3 Noise signal group

Claims

1. An imaging control device comprising a processor that controls an imaging element including a plurality of pixels, wherein the pixels include a photoelectric conversion unit, a charge holding unit that holds charges sent from the photoelectric conversion unit, and a conversion unit that converts the charges into signals, wherein the processor: performs first control to send charges held in the charge holding units of a first region of the plurality of pixels to the conversion unit; performs second control to send charges held in the charge holding units of a second region of the plurality of pixels to the conversion unit; performs third control between the first control and the second control to send charges held in the charge holding units of the first region to the conversion unit; and performs detection processing to detect noise occurring in the charge holding units based on signals obtained from the conversion unit by the third control.

2. An imaging control device according to claim 1, wherein the third control is a control for sending the charges held in the charge holding section in a part of the first region to the conversion section.

3. An imaging control device as described in claim 2, wherein when the processor determines that the noise is occurring through the detection process, it performs a fourth control of sending charges held in at least a part of the charge holding unit other than the part of the first region, or in the charge holding unit of the part of the first region, to the conversion unit.

4. An imaging control device according to claim 3, wherein the fourth control is a control for sending charges held in the charge holding section of at least a part of the other part of the first region to the conversion section.

5. An imaging control device according to claim 4, wherein the fourth control is a control for sending the charges held in the charge holding section of the other part of the first region to the conversion section.

6. An imaging control device according to claim 5, wherein the processor performs the fourth control after the second control.

7. An imaging control device as described in claim 1, wherein the processor further performs the third control at least once after the second control, and in the detection process, detects noise generated in the charge holding section based on signals obtained from the conversion section by multiple times of the third control.

8. An imaging control device as described in claim 7, wherein the processor performs a fifth control to send the charge held in the charge holding section of a third region of the plurality of pixels to the conversion section, and determines whether to perform the third control after the fifth control based on a first signal obtained from the conversion section by the third control between the first control and the second control, and a second signal obtained from the conversion section by the third control between the second control and the fifth control.

9. An imaging control device according to claim 8, wherein the processor determines the movement of the subject that is the source of the noise based on the first signal and the second signal, and determines whether or not to perform the third control after the fifth control based on the movement.

10. An imaging control device as described in claim 7, wherein the processor determines the range of pixels for sending charge to the conversion unit in the third control performed after the second control, based on the third signal obtained from the conversion unit by the first control and the first signal obtained from the conversion unit by the third control performed between the first control and the second control.

11. An imaging control device according to claim 10, wherein the processor determines the movement of the subject that is the source of the noise based on the first signal and the third signal, and determines the range based on the movement.

12. An imaging control device as described in claim 7, wherein the processor determines whether or not to perform the third control after the second control based on the third signal obtained from the conversion unit by the first control and the first signal obtained from the conversion unit by the third control performed between the first control and the second control.

13. An imaging control device according to claim 12, wherein the processor determines the movement of the subject that is the source of the noise based on the first signal and the third signal, and determines whether or not to perform the third control after the second control based on the movement.

14. An imaging device comprising: an imaging control device according to any one of claims 1 to 13; and the imaging element.

15. An imaging control method for controlling an imaging element including a plurality of pixels, wherein the pixels include a photoelectric conversion unit, a charge holding unit that holds charges sent from the photoelectric conversion unit, and a conversion unit that converts the charges into signals; the imaging control method performs first control to send charges held in the charge holding units of a first region of the plurality of pixels to the conversion unit; performs second control to send charges held in the charge holding units of a second region of the plurality of pixels to the conversion unit; performs third control between the first control and the second control to send charges held in the charge holding units of the first region to the conversion unit; and performs detection processing to detect noise occurring in the charge holding units based on signals obtained from the conversion unit by the third control.

16. An imaging control program for controlling an imaging element including a plurality of pixels, wherein the pixels include a photoelectric conversion unit, a charge holding unit that holds charges sent from the photoelectric conversion unit, and a conversion unit that converts the charges into signals; the imaging control program causes a processor to execute the following steps: performing first control to send charges held in the charge holding units of a first region of the plurality of pixels to the conversion unit; performing second control to send charges held in the charge holding units of a second region of the plurality of pixels to the conversion unit; performing third control between the first control and the second control to send charges held in the charge holding units of the first region to the conversion unit; and performing detection processing to detect noise occurring in the charge holding units based on a signal obtained from the conversion unit by the third control.

Images