Imaging element

Abstract

This imaging element comprises: a first photoelectric conversion unit that converts light from a light source into an electric charge; a second photoelectric conversion unit that converts light from the light source into an electric charge; and a detection unit that detects the period of a luminance change of the light source on the basis of a signal based on the electric charge converted by the first photoelectric conversion unit in a first period and a signal based on the electric charge converted by the second photoelectric conversion unit in a second period.

Description

Image sensor

[0001] This application claims priority to Japanese Patent Application No. 2023-198219, filed November 22, 2023, the contents of which are incorporated herein by reference.

[0002] An imaging device that detects flicker of a light source is known (for example, see Patent Document 1). Conventionally, it has been difficult to detect changes in the luminance of a light source.

[0003] Japanese Patent Application Laid-Open No. 2001-358994

[0004] An imaging element of a first aspect of the present invention comprises a first photoelectric conversion unit that converts light from a light source into electric charges, a second photoelectric conversion unit that converts light from the light source into electric charges, and a detection unit that detects a period of change in luminance of the light source based on a signal based on the electric charges converted by the first photoelectric conversion unit in a first period and a signal based on the electric charges converted by the second photoelectric conversion unit in a second period.

[0005] 1 is a diagram showing a schematic configuration of an imaging device 1 according to a first embodiment; FIG. 2 is a diagram showing a schematic configuration of an imaging device 1 according to a first embodiment; FIG. 3 is a diagram showing a schematic configuration of an imaging surface 31 of an imaging element 30 according to a first embodiment; FIG. 4 is a diagram showing an example of the configuration of a portion of an imaging element 30 according to a first embodiment; FIG. 5 is a circuit diagram showing a schematic configuration of a pixel 32; FIG. 6 is a timing chart of control of one pixel 32; FIG. 7 is a timing chart showing an example of a control timing chart of pixels 32 in multiple rows; FIG. 8 is a diagram (part 1) for explaining flicker detection processing according to a first embodiment; FIG. 9 is a diagram (part 2) for explaining flicker detection processing according to a first embodiment; FIG. 10 is a diagram showing a method for detecting a change in flicker over time according to a first embodiment; FIG. 11 is a diagram showing an example of exposure amount of A line-J line according to a first embodiment; FIG. 12 is a diagram showing an example of arranged differences according to a first embodiment; FIG. 13 is a diagram showing the principle of flicker detection processing according to a first embodiment; FIG. 14 is a diagram showing a timing chart of flicker detection processing according to a first embodiment; FIG. 15 is a diagram showing a timing chart of thinning processing according to a first embodiment; FIG. 16 is a diagram showing target regions of different sizes according to a first embodiment; FIG. 17 is a diagram showing a process for detecting partially occurring flicker according to a first embodiment; FIG. 18 is a diagram showing a process for detecting overall flicker according to a first embodiment. 10A and 10B are diagrams for explaining a process according to a second embodiment of the present invention;

[0006] First Embodiment An image sensor according to a first embodiment of the present invention will now be described with reference to the drawings. FIG. 1 is a diagram showing a schematic configuration of an image sensor 1 according to the first embodiment. The image sensor 1 is, for example, a digital camera. The image sensor 1 includes, for example, an image sensor optical system 10, an aperture mechanism 20, an image sensor 30, a timing generator 40, an image processing unit 60, a control unit 70, an operation unit 90, an electronic viewfinder 92, a display unit 94, a recording interface 96, and a flash unit 98. Note that some or all of the functions of the timing generator 40, the image processing unit 60, or the control unit 70 may be included in the image sensor 30.

[0007] The imaging optical system 10 is made up of a plurality of lenses and forms a subject image on the imaging surface of the imaging element 30. The plurality of lenses includes a focus adjustment lens (focus lens) that adjusts the focus.

[0008] The diaphragm mechanism 20 adjusts the amount of light incident on the image sensor 30. This adjustment is performed by an diaphragm driving unit (not shown) in response to an instruction from the control unit 70.

[0009] The imaging element 30 is, for example, an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device). In this embodiment, the imaging element 30 is a CMOS image sensor. The imaging element 30 has a plurality of pixels 32 (see FIG. 2 ) arranged in rows and columns. The pixels 32 convert incident light into electric charges to generate pixel signals. The imaging element 30 outputs the generated pixel signals to the control unit 70. The imaging element 30 is controlled by a timing generator 40 to perform photoelectric conversion using an exposure time set for each row (line).

[0010] 2 is a diagram schematically illustrating the imaging surface 31 of the imaging element 30. In an enlarged region 31a, which is an enlarged portion of the imaging surface 31, a large number of imaging elements 30 are arranged two-dimensionally (in the row and column directions). The plurality of pixels of the imaging element 30 each have, for example, R (red), G (green), and B (blue) color filters. The pixels equipped with the respective color filters are arranged in the imaging surface 31 of the imaging element 30 according to a so-called Bayer array.

[0011] FIG. 3 is a diagram showing an example of the configuration of a portion of the image sensor 30. FIG. 3 shows some of the pixels 32 included in the image sensor 30, a vertical control unit 35, and a horizontal control unit 36. FIG. 3 shows a portion of two pixel columns of the plurality of pixels 32 arranged in the column direction (vertical direction) and the row direction (horizontal direction) intersecting the column direction. In the image sensor 30, vertical signal lines 38 are provided for the plurality of pixel columns. A current source (not shown) is provided for the vertical signal line 38. The configurations of the other pixel columns are similar to the configuration of the pixel columns shown in FIG. 3.

[0012] The vertical control unit 35 is provided in common to multiple pixel rows. The vertical control unit 35 supplies signals TG (TG1A, TG1B, TG2A, TG2B), signals RST (RST1, RST2), and signals SEL (SEL1, SEL2) to each pixel to control the operation of each pixel under the control of the timing generator 40. The vertical control unit 35 supplies signals to the gates of each transistor in the pixels to control the transistor to an on or off state.

[0013] The horizontal control unit 36 ​​includes an amplifier unit and a signal processing unit. An amplifier unit is provided for each vertical signal line 38, and amplifies pixel signals input via the vertical signal lines 38 by a predetermined gain (amplification factor). The amplifier unit determines a gain according to the ISO sensitivity determined by the control unit 70, and applies the determined gain to the pixel signals. The amplifier unit outputs the amplified pixel signals to the signal processing unit.

[0014] The signal processing unit has an analog / digital conversion unit (AD conversion unit) and converts the pixel signal output from the amplifier unit into a digital signal. The signal processing unit performs signal processing on the pixel signal, and then outputs the processed pixel signal to the image processing unit 60 as image data.

[0015] 4 is a circuit diagram showing a schematic configuration of the pixel 32. The pixel 32 has a photoelectric conversion unit PD, a first transfer unit TRX, a second transfer unit TRG, a floating diffusion FD, an analog memory MEM, a reset unit RST, an amplifier unit SF, and a selection unit SEL. The first transfer unit TRX, the second transfer unit TRG, and the reset unit RST are examples of a "first discharge unit" or a "second discharge unit."

[0016] The photoelectric conversion unit PD is a photodiode. The photoelectric conversion unit PD converts incident light into electric charges and accumulates the photoelectrically converted electric charges. The first transfer unit TRX, which is a transfer transistor, is controlled by a signal TG (TG1A, TG2A in FIG. 3 ) and transfers the electric charges photoelectrically converted by the photoelectric conversion unit PD to the analog memory MEM. The analog memory MEM accumulates the electric charges transferred to the first transfer unit TRX.

[0017] The second transfer unit TRG, which is a transfer transistor, is controlled by a signal TG (TG1B, TG2B in FIG. 3 ) and transfers the charge accumulated in the analog memory MEM to the floating diffusion FD. The floating diffusion FD accumulates the charge transferred to the second transfer unit TRG. The amplifier SF amplifies and outputs a signal based on the charge accumulated in the floating diffusion FD. The reset unit RST, which is a reset transistor, is controlled by a signal RST (RST1, RST2 in FIG. 3 ) and discharges the charge accumulated in the floating diffusion FD and resets the voltage (potential) of the floating diffusion FD.

[0018] The selection unit SEL, which is a selection transistor, is controlled by a signal SEL (SEL1, SEL2 in FIG. 3 ) and electrically connects or disconnects the amplification unit SF and the vertical signal line 38. When the selection unit SEL is in the on state, it outputs a signal from the amplification unit SF to the vertical signal line 38. The amplification unit SF and the selection unit SEL constitute an output unit that generates and outputs a signal based on the charge generated by the photoelectric conversion unit PD.

[0019] As described above, a pixel signal corresponding to the charge transferred to the floating diffusion FD is output to the vertical signal line 38. The pixel signal output from the pixel 32 is an analog signal generated based on the charge photoelectrically converted by the photoelectric conversion unit PD.

[0020] Returning to the explanation of Fig. 1, the timing generator 40 instructs the image sensor 30 to store pixel signals and to read out image signals based on instructions from the control unit 70. The timing generator 40 controls each photoelectric conversion unit PD via the vertical control unit 35 so that the timing at which the charge is discharged from the photoelectric conversion unit PD of the first pixel differs from the timing at which the charge is discharged from the photoelectric conversion unit PD of the second pixel. The first pixel and the second pixel belong to different lines.

[0021] The timing generator 40 supplies drive signals to the image sensor 30 in accordance with instructions from the control unit 70 and controls the drive timing using the supplied drive signals. The timing generator 40 can execute modes such as all-pixel readout mode, thinning readout mode, vertical culling mode, and window readout mode. The all-pixel readout mode is a mode in which pixel signals from pixels on all lines of the image sensor 30 are read as image data. The thinning readout mode is a mode in which pixel signals from pixels obtained by thinning out pixels on specified lines out of all lines of the image sensor 30 are read as image data. The vertical culling mode is a mode in which pixel signals from specified pixels on the same column are added together and the pixel signals from pixels obtained by thinning out pixels on the specified lines are read as image data. The window readout (crop) mode is a mode in which pixel signals from pixels in the center of one screen (one frame) are read as pixel data.

[0022] The image processing unit 60 stores image data output by the signal processing unit of the horizontal control unit 36 ​​in a storage unit (RAM) (not shown), for example. Based on the stored image data, the image processing unit 60 calculates an AF evaluation value related to autofocus control, an AE evaluation value related to automatic exposure control, and an AWB evaluation value related to automatic white balance control. The image processing unit 60 adjusts the white balance based on the AWB evaluation value. The image processing unit 60 provides the AF evaluation value and the AE evaluation value to the control unit 70. The image processing unit 60 also performs color interpolation processing to interpolate color signals that are missing in the Bayer array, as well as processing such as gamma correction and gradation adjustment.

[0023] The control unit 70 is configured by a CPU (Central Processing Unit) and controls the overall operation of the imaging device 1. The control unit 70 controls each unit of the imaging device 1 by executing a control program pre-stored in a storage unit (ROM) (not shown). The control unit 70 includes, for example, an imaging control unit 72, an AE control unit 74, an AF control unit 76, a light adjustment unit 78, and a processing unit 80 (detection unit).

[0024] For example, when receiving an imaging instruction via the operation unit 90, the imaging control unit 72 stores image data obtained via the imaging element 30 as a captured image in a storage medium connected to the recording interface 96. The AE control unit 74 controls the sensitivity and exposure time (charge accumulation time) of the imaging element 30 and the aperture of the diaphragm mechanism 20 based on the AE evaluation value. The AF control unit 76 controls the focus lens of the imaging optical system 10 based on the AF evaluation value. The AF control unit 76 performs focus detection calculations using a known image plane phase difference detection method based on focus detection signals acquired from phase difference detection pixels provided on the image plane of the imaging element 30. The AF control unit 76 controls a focus drive mechanism (not shown) based on the AF evaluation value obtained from this focus detection calculation to adjust the focus of the imaging optical system 10. Note that the imaging device 1 may be provided with an AF sensor and perform a known phase difference focus detection calculation based on focus detection signals acquired from phase difference pixels provided in the AF sensor. The dimming unit 78 controls the strobe 98 to perform dimming.

[0025] The processing unit 80 detects the period of change in luminance of the light source based on a signal based on the charges converted by the photoelectric conversion unit of the first pixel during the first period and a signal based on the charges converted by the second photoelectric conversion unit of the second pixel during the second period (performs a detection process). Details of the detection process will be described later.

[0026] The operation unit 90 includes various operation buttons such as a release button, a recording button, a power button, etc. The operation unit 90 outputs an operation signal corresponding to an operation by a user to the control unit 70. The control unit 70 executes control corresponding to the output operation signal.

[0027] The electronic viewfinder 92 has a small display and an eyepiece, and is an observation unit for the user to observe images. The display unit 94 displays images based on image signals, shooting information such as shutter speed and aperture value, a menu screen, etc.

[0028] The recording interface 96 is an interface to which a storage medium such as a nonvolatile flash memory can be attached and detached. The recording interface 96 stores image data in the attached storage medium. The strobe 98 emits light under the control of the light adjustment unit 78.

[0029] The imaging device 1 is capable of performing a global shutter operation, which refers to an operation in which the photoelectric conversion units PD start and end charge accumulation simultaneously for all pixels.

[0030] First, the control of the global shutter operation by the pixels 32 executed by the imaging control unit 72 of the control unit 70 described above will be described using the timing charts shown in Fig. 5 and Fig. 6. Fig. 5 is a timing chart showing the control of one pixel 32.

[0031] During period T1, the control unit 70 controls the reset unit RST, the second transfer unit TRG, and the first transfer unit TRX to the ON state. This executes a discharge operation (reset operation) that simultaneously discharges the charges accumulated in the photoelectric conversion unit PD, the analog memory MEM, and the floating diffusion FD. During period T2 after period T1, the control unit 70 controls the reset unit RST, the second transfer unit TRG, and the first transfer unit TRX to the OFF state. This ends the discharge operation and starts exposure (charge accumulation). During period T3 after period T2, the control unit 70 controls the first transfer unit TRX to the ON state. This transfers the charges accumulated in the photodiode PD to the analog memory MEM, which then accumulates the charges. Thereafter, the control unit 70 controls the first transfer unit TRX to the OFF state to end the charge transfer.

[0032] In period T4 after period T3, the control unit 70 controls the second transfer unit TRG to an ON state, transfers the charge stored in the analog memory MEM to the floating diffusion FD, and accumulates the charge in the floating diffusion FD. Thereafter, the control unit 70 controls the second transfer unit TRG to an OFF state, ending the charge transfer. In period T5 after period T4, the control unit 70 controls the amplifier unit SL and the selection unit SEL to an ON state, and outputs a signal based on the charge to the vertical signal line 38. As described above, each unit in the pixel 32 is controlled, and a signal based on the charge is output. Hereinafter, the operations in periods T1 to T5 may be referred to as a first operation to a fifth operation, respectively.

[0033] 6 is a timing chart showing an example of the control timing chart for pixels 32 in multiple rows. In the example of FIG. 6, the control timing for pixels 32 included in each of rows 1 to 4 will be described. At periods T1 to T5 shown in FIG. 6, the first to fifth operations performed during periods T1 to T5 in FIG. 5 are performed, respectively. Time passes in the order of time Ta, Tb, Tc, ... Ti.

[0034] At time Ta, the pixels 32 in rows 1 to 4 perform the first operation, at time Tb, the pixels 32 in rows 1 to 4 perform the second operation, and at time Tc, the pixels 32 in rows 1 to 4 perform the third operation. At time Td, the third operation of the pixels 32 in rows 1 to 4 is completed.

[0035] Row 1 starts the fourth operation at time Td, ends the fourth operation and starts the fifth operation at time Te, and ends the fifth operation at time Tf. Row 2 starts the fourth operation at time Te, ends the fourth operation and starts the fifth operation at time Tf, and ends the fifth operation at time Tg. Row 3 starts the fourth operation at time Tf, ends the fourth operation and starts the fifth operation at time Tg, and ends the fifth operation at time Th. Row 4 starts the fourth operation at time Tg, ends the fourth operation and starts the fifth operation at time Th, and ends the fifth operation at time Ti.

[0036] As described above, each pixel is controlled to achieve a global shutter operation. In a configuration in which a global shutter operation is performed, the timing of the first operation (discharge operation) is controlled as described below, and flicker is detected. Specifically, flicker is detected by controlling the timing of the discharge operation of each pixel to be different (shifted). Below, a detection process for detecting flicker (changes in luminance of the light source) without performing a global reset in an imaging device 1 equipped with a global shutter function will be described.

[0037] [Flicker Detection Process (Part 1)] FIG. 7 is a diagram (part 1) for explaining the flicker detection process. In FIG. 7 , as an example, flicker is detected in an environment where flicker of a frequency (100 Hz or 120 Hz) determined by a public frequency (50 Hz or 60 Hz) in Japan is present. The control unit 70 divides a region including a plurality of pixels into region X and region Y. Region X and region Y are, for example, adjacent regions. The number of lines included in region X is the same as the number of lines included in region Y. The number of pixels corresponding to each of the R (red), G (green), and B (blue) color filters included in region X is the same as the number of pixels corresponding to each of the R (red), G (green), and B (blue) color filters included in region Y. Region X is an example of a "first region" in which first photoelectric conversion units are arranged in the row direction or the row and column directions, and region Y is an example of a "second region" in which second photoelectric conversion units are arranged in the row direction or the row and column directions.

[0038] The control unit 70 shifts the timing at which the pixels in region X discharge charge (hereinafter referred to as "discharge timing") from the discharge timing of the pixels in region Y. The control unit 70 differentiates the start timing at which the pixels in region X start exposure from the start timing at which the pixels in region Y start exposure (differentiating the start of the first period from the start of the second period). The shift time is the period of the flicker to be detected, and is, for example, 10 ms or 8.3 ms. The end point of the exposure time is 40 ms (or near 40 ms), which is the least common multiple of (or near) 10 ms and 8.3 ms. The control unit 70 detects flicker using the above relationship.

[0039] FIG. 8 is a diagram (part 2) for explaining the flicker detection process. For example, assume that the flicker is 100 Hz or 120 Hz, as described above. The control unit 70, for example, shifts the discharge timing for region Y by 8.3 ms (or approximately 8.3 ms) relative to the discharge timing for region X. The control unit 70 detects flicker based on the ratio between the signal corresponding to the exposure time of the pixel in region X and the signal corresponding to the exposure time of the pixel in region Y when the discharge timing is shifted. The control unit 70 determines that the flicker period is 100 Hz if the relationship in Equation 1 below holds, and that the flicker period is 120 Hz if the relationship does not hold.

[0040] The following relationship is the following (Equation 1): X = Y / 2.Δ × 3 (Equation 1)

[0041] "X" in (Equation 1) is "n x 3," and "Y" in (Equation 1) is "n x 2.Δ." "Δ" indicates a decimal multiple (a number after the decimal point) when the discharge timing is shifted by 8.3 ms. "n" is the exposure amount per exposure time of one cycle (50 Hz) of 100 Hz flicker. "X" is three times the exposure amount per exposure time of one cycle of 100 Hz flicker, and "Y" is 2.Δ times the exposure amount per exposure time of one cycle of 100 Hz flicker. In this way, the control unit 70 varies the discharge timing of the pixel charges and detects the flicker cycle using the exposure amount per exposure time of one cycle of the detected flicker.

[0042] In the above example, the control unit 70 has been described as shifting the discharge timing for area B by 8.3 ms (or approximately 8.3 ms) relative to the discharge timing for area X, but instead (or in addition), the shift time may be set to 10 ms, and the "2.△" in (Equation 1) may be set to "2" to determine the flicker period.

[0043] Furthermore, in the above example, it has been described that if Equation 1 is true, it is determined that there is 100 Hz flicker, but instead of (or in addition to) this, an equation for detecting the period of 120 Hz flicker may be found, and whether or not there is a 120 Hz flicker period may be determined based on whether or not the found equation is true. Furthermore, in the above process, if each region includes multiple pixels, the control unit 70 may use, for example, the average value of the exposure amounts output by the pixels included in the X region and the Y region, or the sum of the exposure amounts output by the pixels included in the X region and the Y region (where the number of pixels included in the X region and the number of pixels included in the Y region are the same).

[0044] As described above, the control unit 70 performs the detection process, thereby easily and accurately detecting changes in the luminance of the light source and flicker caused by blinking of the light source.

[0045] [Process for Detecting Flicker (Part 2)] Using the above concept, the control unit 70 may detect changes in flicker over time using the following method. The control unit 70 detects changes in flicker over time based on differences in the amount of exposure of each pixel. FIG. 9 is a diagram for explaining a method for detecting changes in flicker over time. For example, the control unit 70 focuses on multiple pixels included in area Z out of multiple pixels included in area AR. Hereinafter, area Z is made up of lines A to J, and the number of pixels on each line is the same.

[0046] The control unit 70 differentiates the start of exposure (first period) of a pixel of interest from the start of exposure (second period) of other pixels of interest. For example, the control unit 70 shifts the discharge timing or the timing to start exposure for each line (or for each set of lines) by a time that is sufficiently shorter than the period of the flicker to be detected. The control unit 70 shifts the discharge timing in order, for example, A line, B line, C line, ..., J line. The control unit 70 recognizes the exposure amount for each pixel in A line, B line, C line, ..., J line during the exposure time corresponding to that pixel.

[0047] The control unit 70 may, for example, use the average value of the exposure amount output by pixels belonging to the same line in a specified area in the following processing, or may use the sum of the exposure amounts output by pixels belonging to the same line in a specified area in the following processing.

[0048] FIG. 10 is a diagram showing an example of the exposure amount of line A-line J. The control unit 70 recognizes the difference in exposure amount between adjacent lines based on the exposure amount of each of line A, line B, line C, and line J. As shown in FIG. 10, the control unit 70 calculates the difference between the exposure amount of line A and the exposure amount of line B (the A-B difference) and the difference between the exposure amount of line B and the exposure amount of line C (the B-C difference). Similarly, the control unit 70 calculates the C-D difference, the D-E difference, the E-F difference, the F-G difference, the G-H difference, and the H-I difference.

[0049] The control unit 70 arranges the A-B difference, the B-C difference, ..., the I-J difference, and the exposure amount J in that order. FIG. 11 is a diagram showing an example of the arranged differences. By arranging the differences and the exposure amount J as shown in FIG. 11, the control unit 70 can recognize the flicker waveform and recognize changes in the flicker over time. Note that FIG. 11 shows an example in which the flicker waveform rises at the discharge timing of line A and returns to the position before the rise at the end point of exposure for line J.

[0050] As described above, the control unit 70 performs the detection process, thereby making it possible to easily and accurately detect flicker of any frequency.

[0051] In the above example, the discharge timing is shifted in the order of line A to line J, but the discharge timing may be arbitrary. When calculating the difference in exposure amount, the control unit 70 calculates the difference from the exposure amount of lines whose discharge timings are adjacent, and arranges the differences in the order of the discharge timings, thereby making it possible to recognize the change in flicker over time.

[0052] Furthermore, the control unit 70 may perform the above processing using any line, not limited to lines A to J. The number of times the discharge timing is shifted may be set arbitrarily, for example, depending on the desired flicker frequency. For example, the target area may be enlarged and processed. The control unit 70 may also perform multiple exposures on the target line and detect flicker based on changes in the exposure amount resulting from the multiple exposures.

[0053] FIG. 12 is a diagram illustrating the principles of flicker detection processing. In the example of FIG. 12, there are 20 lines in the target area. The image sensor 30 is controlled in accordance with vertical and horizontal synchronization signals generated under the control of the control unit 70. The control unit 70, for example, groups two adjacent lines among the 20 lines into one pixel row group and shifts the discharge timing for each pixel row group. For example, the first and second lines are grouped into a pixel row group, the third and fourth lines are grouped into a pixel row group, and so on, with the 19th and 20th lines being grouped into a pixel row group. The odd-numbered lines are lines in which pixels with R (red) color filters and pixels with G (green) color filters are alternately arranged, and the even-numbered lines are lines in which pixels with G (green) color filters and pixels with B (blue) color filters are alternately arranged. The amount of charge accumulated according to the exposure time is analog-to-digital converted (AD converted) and read as a digital value, which is then transferred to memory. The control unit 70 recognizes the change in flicker over time based on the exposure amount of the image sensor for one line of a pixel row group, as described above with reference to FIGS.

[0054] Furthermore, the control unit 70 can detect the peak phase, bottom (rising point) or cycle of the flicker based on the change in the flicker over time. Based on the detection result, the control unit 70 may control the exposure timing based on the timing at which the change in the flicker component per unit time is small. This allows an image that is not affected by flicker to be captured.

[0055] [Timing Chart (Part 1)] Fig. 13 is a timing chart of the flicker detection process. The example in Fig. 13 shows the discharge timing when all pixels are targeted for flicker detection. In the example in Fig. 13, two lines are treated as a pixel row group, as described in Fig. 12. Furthermore, a predetermined number of lines are treated as one group for flicker detection.

[0056] [Thinning-Out Processing] The control unit 70 may perform thinning processing that does not take into account certain lines in the process of detecting flicker. FIG. 14 is a diagram illustrating the thinning processing. FIG. 14 shows an area including the first line to the twelfth line. The control unit 70 treats three lines as one pixel row group and shifts the discharge timing for each pixel row group. For example, the first line to the third line are considered the first pixel row group, the fourth line to the sixth line are considered the second pixel row group, the seventh line to the ninth line are considered the third pixel row group, and the tenth line to the twelfth line are considered the fourth pixel row group. The discharge timing is sequentially delayed for the first pixel row group, the second pixel row group, the third pixel row group, and the fourth pixel row group. The odd-numbered lines are lines in which R (red) filter pixels and G (green) filter pixels are alternately arranged, and the even-numbered lines are lines in which G (green) filter pixels and B (blue) filter pixels are alternately arranged.

[0057] In calculating the exposure amount for each line, the control unit 70 uses the exposure amounts of two lines in each pixel row group, rather than the exposure amount of the middle line (the second line, the fifth line, the eighth line, and the eleventh line) of each pixel row group. In this case, the control unit 70 may, for example, calculate a luminance component from the output of the pixels included in each pixel row group and calculate a difference using the calculated luminance component, or may perform detection processing on the exposure amount of a pixel provided with a G filter in the target line.

[0058] Furthermore, the control unit 70 may treat six lines (the first line to the sixth line, the seventh line to the twelfth line, etc.) as one pixel row group. That is, one pixel row group may be generated so that it contains the same number of lines of R pixels and G pixels and the same number of lines of G pixels and B pixels.

[0059] The detection process may be performed with the number of pixels equipped with R color filters, the number of pixels equipped with G color filters, and the number of pixels equipped with B color filters included in the pixel row group set to be the same, or the detection process may be performed with at least the number of pixels equipped with G color filters included in the pixel row group set to be the same.

[0060] As described above, the control unit 70 can easily and accurately detect flicker based on the difference in the amount of exposure, using the amount of exposure for each pixel row group.

[0061] [Timing Chart (Part 2)] Figure 15 is a timing chart of the thinning process. The control unit 70 treats, for example, three lines as one pixel row group, and shifts the discharge timing for each pixel row group so that the exposure end timing for each line is the same (or nearly the same). In addition, a predetermined number of lines are treated as one group and are used to detect flicker.

[0062] [Dividing Detection Area and Emission Timing] The control unit 70 may arbitrarily set a target area for the flicker detection process. The target area is an area (divided area) that is a combination of a predetermined number of lines and a predetermined number of columns. The size of each target area may be the same or different. For example, the control unit 70 may perform the detection process on a large area during live view to detect the position of the light source or flicker, and perform the detection process on a smaller area that takes into account the subject during still image capture.

[0063] FIG. 16 is a diagram illustrating target regions of different sizes. The control unit 70 may recognize the position of the subject using a recognition process on the image displayed in live view or a depth map of the autofocus function, and set the target region based on the recognized position of the subject. For example, a region including a subject (e.g., a person) or a target region located at the same horizontal position as the subject may be set smaller than other regions. In this way, by setting the target region smaller than other regions, the control unit 70 can more accurately detect flicker of the light source irradiating the subject with light.

[0064] Furthermore, the control unit 70 may set the interval for shifting the discharge timing of an area including a subject or a target area that is located at the same horizontal position as the subject to be shorter than the interval for other areas. For example, even if a light source such as a high-frequency LED is irradiated onto the subject in stage photography, the control unit 70 can detect flicker of the light source by setting the interval for shifting the discharge timing to be shorter.

[0065] If flicker is detected, the control unit 70 may capture a still image and then perform gain adjustment or image processing on the image to reduce the effects of the flicker. The control unit 70 may also perform processing to detect flicker from different types of light sources for each target region or for each target region. For example, the control unit 70 may switch between processing to detect flicker at 100 Hz or 120 Hz and processing to detect flicker from a high-frequency LED light source for each target region.

[0066] 17 is a diagram illustrating the process of detecting partially occurring flicker. For example, if flicker is partially present due to a light source inside a building, the control unit 70 can detect the occurrence of flicker for each of multiple target regions, thereby making it possible to detect partial flicker.

[0067] 18 is a diagram illustrating a process for detecting flicker occurring throughout the entire image. For example, the control unit 70 may detect flicker and the frequency and phase (or waveform peaks and troughs) of the flicker for each of a plurality of target regions and determine whether the frequency, phase, and waveform for each target region match. For example, if the degree of match is high, the control unit 70 may determine that the reliability of the detection result is high (above a threshold value). For example, the control unit 70 may determine that the reliability of the detection result is high if the frequency of the flicker in the first target region matches the frequency of the flicker in the second target region, and that the reliability of the detection result is low if they do not match.

[0068] For example, when the reliability is high as described above, the control unit 70 may cause the image sensor 30 to perform exposure at a timing when the change in the flicker waveform is small (for example, at the peak of the waveform), thereby enabling the control unit 70 to perform exposure at a more appropriate timing.

[0069] Furthermore, the control unit 70 may switch between detecting the period of the luminance change of the light source using all pixel rows among the pixel rows included in a divided region (or effective pixel region) obtained by dividing the effective pixel region into multiple regions, and detecting the period of the luminance change of the light source using the remaining pixel rows after thinning out one or more pixel rows. For example, if it is assumed that the detection target is experiencing luminance changes with a relatively low frequency period, the detection process may be performed by thinning out the pixel rows in the divided region (or effective pixel region). However, if it is assumed that the detection target is experiencing luminance changes with a relatively high frequency period (for example, if it is assumed that flicker is occurring due to an LED light source), the detection process may be performed using all pixels in the divided region (or effective pixel region) without thinning out the pixel rows. This allows for more accurate detection of the period of the luminance change.

[0070] [Flicker Search] The control unit 70 may search for the flicker cycle by setting the size of the area to be subjected to the detection process and the time for shifting the emission timing. For example, by changing the size of the area to be subjected to the detection process, the control unit 70 can detect flicker occurring across the entire screen or flicker occurring partially, or identify the area where flicker is occurring. Furthermore, the control unit 70 can detect flicker of various frequencies by setting the time for shifting the emission timing.

[0071] The control unit 70 performs the above-described processing to detect, for example, 100 Hz or 120 Hz flicker. For example, if the processing time of the AD conversion unit of the image sensor is 12 μs, it is possible to set a region of approximately 80 lines by shifting the discharge (reset) timing between lines 2 and 80. If the region size is 20 lines, flicker can be detected in one block every 200 lines, and if the visual size is 4000 lines, the screen will be divided into 20 regions.

[0072] An example of detecting flicker from a high-frequency light source such as an LED will be described. If the flicker period of an LED light source is, for example, 2 kHz and the resolution of one period is 1 / 10, one period is 500 μs, so the resolution is 50 μs. For example, if the processing time of the AD conversion unit of the image sensor is 12 μs, 2 kHz flicker can be detected by shifting the discharge timing every four lines.

[0073] If the control unit 70 detects flicker in the above process, it controls the exposure timing of the image sensor 30 so that the flicker does not affect imaging (so that the flicker does not appear in the captured image). Furthermore, the control unit 70 may display the captured image on the display unit 94, and if the image contains flicker, it may indicate the area of ​​the image displayed on the display unit that contains the flicker in a manner that is recognizable to the user. Furthermore, if the control unit 70 detects flicker, it may correct the image of pixels in the captured image that correspond to the flicker.

[0074] The control unit 70 can determine the number of pixel rows included in each pixel row group, the number of pixel row groups used in the process of detecting the period of change in luminance of the light source to be detected, etc., based on the period of change in luminance of the detected light source, the size of the area to be subjected to the detection process, and the number of pixels or lines included in the area. The control unit 70 can also determine the size of the detection target, the number of lines included in the pixel row, the number of pixel rows, etc., depending on the scene to be captured, the position of the subject, etc. For example, the control unit 70 can differentiate the timing of discharging the charge from the first pixel (first pixel row group) from the timing of discharging the charge from the second pixel depending on the mode (e.g., all-pixel readout mode, thinning mode, vertical addition thinning mode, readout mode), thereby easily and accurately detecting the period of change in luminance of the light source.

[0075] According to the first embodiment described above, the control unit 70 can detect the cycle of the change in luminance of the light source.

[0076] Second Embodiment A second embodiment will be described below. In the second embodiment, the discharge timing is shifted for each line or for each pixel row group including multiple lines to detect the cycle of the luminance change of the light source, and light adjustment is performed based on the detection result. The second embodiment will be described below.

[0077] 19 is a diagram for explaining the processing of the second embodiment. The control unit 70 shifts the discharge timing as described in the first embodiment. The control unit 70 causes the first pixel row group, the second pixel row group, and the third pixel row group to discharge electric charges at the first discharge timing, the second discharge timing, and the third discharge timing, respectively.

[0078] The control unit 70 causes, for example, the strobe 98 (external flash) to emit a first monitor light (e.g., a low light) between the first and second discharge timings, and causes the strobe 98 to emit a second monitor light (a high light) between the second and third discharge timings. Furthermore, the control unit 70 does not cause the strobe 98 to emit light between the third discharge timing and the end of exposure. In other words, the state of constant light is maintained between the third discharge timing and the end of exposure.

[0079] The control unit 70 calculates the reflectance of the subject based on the difference between the exposure amount obtained under constant light and the exposure amount obtained when a small amount of light is emitted, and the difference between the exposure amount obtained under constant light and the exposure amount obtained when a large amount of light is emitted, and performs dimming based on the calculated reflectance.

[0080] For example, to determine the exposure amount for each monitor flash without shifting the discharge timing, a small monitor flash is emitted for the first exposure, a large monitor flash is emitted for the second exposure, and then a third exposure is performed under constant light conditions without using a strobe.In contrast, in the second embodiment, the exposure amounts for different monitor flashes can be determined in a single process as described above.

[0081] According to the second embodiment described above, the control unit 70 can detect a change in the luminance of the light source and appropriately adjust the light by shifting the discharge timing during dimming.

[0082] Third Embodiment A third embodiment will now be described. As in the first and second embodiments, the image sensor 30 has R pixels, G pixels, and B pixels, which are normal image capturing pixels, and also has an image plane phase difference for performing image plane phase difference focus detection.

[0083] The control unit 70 controls the timing of discharging the electric charges of the photoelectric conversion units PD of the pixels included in each of the multiple lines on which the image plane phase difference pixels are arranged so that the timing is simultaneous or a constant exposure time. In other words, the control unit 70 does not change the discharge timing for each line. This allows the control unit 70 to perform focus adjustment of the image plane phase difference imaging optical system 10.

[0084] According to the third embodiment described above, the control unit 70 can perform detection processing and focus adjustment.

[0085] Fourth Embodiment A fourth embodiment will now be described. In the first embodiment, processing in an image capture device employing a global shutter was described. In contrast, in the second embodiment, processing in an image capture device employing a rolling shutter will be described.

[0086] 20 is a diagram for explaining the processing of the fourth embodiment. In a rolling shutter, if the scan rate is set sufficiently shorter than the difference between discharge timings, flicker can be detected in the same way as in the processing for detecting flicker in a global shutter. Furthermore, when performing detection processing in a rolling shutter, it is preferable to reduce the area targeted by the detection processing.

[0087] According to the fourth embodiment described above, the control unit 70 can achieve the same effects as the first embodiment in the rolling shutter.

[0088] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0089] 1...imaging device, 10...imaging optical system, 30...imaging element, 32...pixel, PD...photoelectric conversion section, TRX...first transfer section, TRG...second transfer section, FD...floating diffusion, MEM...analog memory, RST...reset section, SF...amplification section, SEL...selection section, 60...image processing section, 70...control section, 80...processing section, 90...operation section, 98...strobe, 94...display section

Claims

1. An imaging element comprising: a first photoelectric conversion unit that converts light from a light source into an electric charge; a second photoelectric conversion unit that converts the light from the light source into an electric charge; and a detection unit that detects a period of change in luminance of the light source based on a signal based on the electric charge converted by the first photoelectric conversion unit in a first period and a signal based on the electric charge converted by the second photoelectric conversion unit in a second period.

2. The imaging element according to claim 1, further comprising a control unit that differentiates a start of the first period from a start of the second period.

3. An imaging element according to claim 1 or 2, further comprising a control unit that controls the interval between the start of the first period and the start of the second period to be shorter than the period of change in luminance of the light source.

4. The imaging element according to claim 2, wherein the control unit performs control so that a timing for discharging the electric charge of the first photoelectric conversion unit differs from a timing for discharging the electric charge of the second photoelectric conversion unit.

5. The imaging element according to claim 4, wherein the control unit controls the interval between the timing at which the charge of the first photoelectric conversion unit is discharged and the timing at which the charge of the second photoelectric conversion unit is discharged to an interval shorter than the period of change in luminance of the light source.

6. The imaging element of claim 2, comprising: a first discharge section for discharging electric charges from the first photoelectric conversion section; and a second discharge section for discharging electric charges from the second photoelectric conversion section, wherein the control section differentiates the timing at which the first discharge section discharges the electric charges from the first photoelectric conversion section from the timing at which the second discharge section discharges the electric charges from the second photoelectric conversion section, thereby differentiating the start of the first period from the start of the second period.

7. The imaging element of claim 1, wherein the plurality of first photoelectric conversion units are arranged in the row direction in a first region, the plurality of second photoelectric conversion units are arranged in the row direction in a second region, and the detection unit detects a period of change in luminance of the light source for each divided region obtained by dividing an effective pixel region, in which a plurality of pixels are arranged two-dimensionally, into a plurality of regions, based on a first signal based on charges generated in the plurality of first photoelectric conversion units in the first period and a second signal based on charges generated in the plurality of second photoelectric conversion units in the second period.

8. The imaging element described in claim 7, wherein the plurality of first photoelectric conversion units are arranged in row and column directions in a first region, the plurality of second photoelectric conversion units are arranged in row and column directions in a second region, and the detection unit detects a period of luminance change of the light source based on the first signal based on charges generated in the plurality of first photoelectric conversion units in the first period and the second signal based on charges generated in the plurality of second photoelectric conversion units in the second period.

9. The imaging element described in claim 7 or 8, wherein the detection unit detects a period of change in luminance of the light source based on the first signal based on charges generated in a plurality of the first photoelectric conversion units arranged in the first region, and the second signal based on charges generated in a plurality of the second photoelectric conversion units arranged in the second region adjacent to the first region.

10. The imaging element according to claim 7 or 8, wherein the detection section detects a period of change in luminance of the light source based on a ratio between the first signal and the second signal.

11. The imaging element according to claim 7 or 8, wherein the detection section detects a period of change in luminance of the light source based on a difference between the first signal and the second signal.

12. An imaging element as described in claim 1, further comprising a control unit that sets one or more pixel row groups included in a predetermined area of ​​an effective pixel area in which a plurality of pixels are arranged two-dimensionally, synchronizes the timing of discharging electric charges from photoelectric conversion units belonging to the same pixel row group, and shifts the timing of discharging electric charges from the photoelectric conversion units for each pixel row group, and the detection unit detects a period of change in luminance of the light source based on the amount of exposure to which the photoelectric conversion units included in each pixel row group are exposed.

13. The imaging element described in claim 12, wherein the control unit determines, based on the period of luminance change of the light source, either the number of pixel rows included in each pixel row group or the number of pixel row groups used in the process of detecting the period of luminance change of the light source.

14. The image sensor according to claim 12 or 13, wherein the detection section determines a difference in the amount of exposure between the pixel rows adjacent in timing to discharge the electric charge, and detects a change in luminance of the light source based on the determined difference.

15. The image sensor according to claim 14, wherein the detection section detects a period of flicker, which is a change in luminance of the light source, a peak of the flicker, or a bottom of the flicker, based on the difference.

16. The image sensor according to claim 15, wherein the control unit controls exposure timing based on a timing when a change in the flicker component is small based on the result of the detection.

17. The imaging element described in claim 12 or 13, wherein the number of photoelectric conversion units in pixels having R color filters, the number of photoelectric conversion units in pixels having G color filters, and the number of photoelectric conversion units in pixels having B color filters, which are included in the pixel row group, are the same; or the number of photoelectric conversion units in at least pixels having G color filters, which are included in the pixel row group, are the same.

18. An imaging element as described in claim 1 or 2, further comprising a control unit which changes the timing for discharging the charge of the first photoelectric conversion unit and the timing for discharging the charge of the second photoelectric conversion unit depending on the mode, the modes being: an all-pixel readout mode in which pixel signals of pixels in all pixel rows in an effective pixel area in which a plurality of pixels are arranged two-dimensionally; a thinning out mode in which pixel signals of pixels in a specified pixel row are read out; a vertical addition thinning out mode in which pixel signals of specified pixels in the same column are added and the pixel signals of pixels in a specified line are thinned out to read the pixel signals of the pixels; and a mode in which pixel signals of pixels in the center of the effective pixel area are read out.

19. The image sensor according to claim 1 or 2, wherein the detection unit alternately performs a process of detecting the period of luminance change of the light source using all pixel rows included in a divided area obtained by dividing an effective pixel area into a plurality of areas, and a process of detecting the period of luminance change of the light source using the remaining pixel rows after one or more pixel rows have been thinned out.

20. The imaging element of claim 1 or 2, further comprising a third photoelectric conversion unit that converts light from the light source into electric charges; and a fourth photoelectric conversion unit that converts light from the light source into electric charges, wherein the first photoelectric conversion unit and the second photoelectric conversion unit are included in a first divided region among a plurality of divided regions obtained by dividing an effective pixel region into a plurality of regions, and the third photoelectric conversion unit and the fourth photoelectric conversion unit are included in a second divided region among the divided regions, and the detection unit determines a reliability of the period based on the detected period of luminance change of the light source based on a signal based on the charges converted by the first photoelectric conversion unit and a signal based on the charges converted by the second photoelectric conversion unit, and based on a third signal based on the charges converted by the third photoelectric conversion unit in the first period and a fourth signal based on the charges converted by the fourth photoelectric conversion unit in the second period.

21. The image sensor according to claim 1 or 2, further comprising: a fifth photoelectric conversion unit that is included in a phase difference pixel belonging to a first line and converts light from a light source into an electric charge; and a sixth photoelectric conversion unit that is included in the phase difference pixel belonging to a second line and converts light from the light source into an electric charge; and a control unit that controls the timing of discharging the electric charge of the fifth photoelectric conversion unit and the timing of discharging the electric charge of the sixth photoelectric conversion unit to be the same, or controls the exposure time of the fifth photoelectric conversion unit and the exposure time of the sixth photoelectric conversion unit to be constant.

Images