Imaging device and control method thereof

Abstract

To provide an imaging device and its control method that suppress the occurrence of flicker fringes in images taken under a flicker light source while enabling flexible selection of shutter speed. [Solution] The imaging device has an image sensor having multiple pixels arranged in a matrix. The imaging device divides the pixel rows of the image sensor into multiple groups and generates a composite image by combining the images obtained from each group. The imaging device controls the operation of the image sensor so that the start timing of the accumulation period for the first pixel row read out in each of the multiple groups has a difference based on the period of change in ambient light brightness and the number of multiple groups.

Description

Technical Field

[0001] The present invention relates to an imaging device and a control method thereof.

Background Art

[0002] When shooting under a light source (flicker light source) that periodically repeats lighting and extinguishing like a fluorescent lamp or an LED, stripe-like luminance unevenness (flicker stripes) may occur in the captured image. Conventionally, it is known that by setting the shutter speed to an integer multiple of the blinking frequency (flicker frequency) of the flicker light source, the flicker stripes generated in the captured image can be reduced (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the method of setting the shutter speed to an integer multiple of the flicker frequency, the options for the shutter speed at the time of shooting are limited.

[0005] In one aspect of the present invention, there is provided an imaging device and a control method thereof that enable flexible selection of the shutter speed while suppressing the occurrence of flicker stripes in an image captured under a flicker light source.

Means for Solving the Problems

[0006] In one embodiment, the present invention provides an imaging device comprising: an image sensor having a plurality of pixels arranged in a matrix; control means for controlling the operation of the image sensor; and synthesis means for dividing the pixel rows of the image sensor into a plurality of groups and generating a composite image by combining the images obtained from each group, wherein the control means controls the operation of the image sensor such that the start timing of the accumulation period for the first pixel row read out in each of the plurality of groups has a difference based on the period of change in ambient light brightness and the number of the plurality of groups. [Effects of the Invention]

[0007] According to one aspect of the present invention, it is possible to provide an imaging device and a control method therefor that allows for flexible selection of shutter speed while suppressing the occurrence of flicker fringes in images captured under a flicker light source. [Brief explanation of the drawing]

[0008] [Figure 1] Block diagram showing an example of the functional configuration of the imaging device according to the embodiment. [Figure 2] Block diagram showing an example of the image sensor configuration in the first embodiment. [Figure 3] Circuit diagram showing an example of pixel configuration in the first embodiment. [Figure 4] A diagram illustrating the pixel arrangement of the image sensor in the first embodiment. [Figure 5] A diagram illustrating the readout timing of the image sensor in the first embodiment. [Figure 6] Diagram illustrating the principle of the first embodiment. [Figure 7] Flowchart relating to the operation of the imaging device according to the first embodiment [Figure 8] Flowchart relating to the operation of the imaging device according to a modified example of the first embodiment [Figure 9] A diagram illustrating the pixel arrangement of the image sensor in the second embodiment. [Figure 10] Circuit diagram showing an example of the pixel configuration of the image sensor in the second embodiment. [Figure 11]A diagram illustrating the readout operation of the image sensor in the second embodiment. [Modes for carrying out the invention]

[0009] The present invention will be described in detail below with reference to the attached drawings, based on exemplary embodiments thereof. Note that the following embodiments do not limit the invention to the claims. Furthermore, while multiple features are described in the embodiments, not all of them are essential to the invention, and the multiple features may be combined arbitrarily. In addition, in the attached drawings, the same or similar configurations are given the same reference numeral, and redundant descriptions are omitted.

[0010] ●(First Embodiment) Figure 1 is a block diagram showing an example of the functional configuration of an imaging device 100 according to an embodiment of the present invention. The lens unit 101 may be fixed to the imaging device 100 or it may be detachable. The lens unit 101 has an imaging optical system that forms an optical image of a subject on the imaging surface of the image sensor 102. The imaging optical system is composed of a plurality of lenses, including a zoom lens that changes the angle of view and a focus lens that changes the focal distance. The focus mechanism 1015 changes the focal distance of the imaging optical system by adjusting the position of the focus lens according to the control of the control unit 107. The zoom mechanism 1016 changes the angle of view of the imaging optical system by adjusting the position of the zoom lens according to the control of the control unit 107.

[0011] The aperture mechanism 1013 changes the aperture diameter according to the control unit 107. The shutter mechanism 1014 controls the operation of the mechanical shutter of the lens unit 101 according to the control unit 107. The aperture may also function as a mechanical shutter.

[0012] In FIG. 1, the dotted arrows indicate the light incident from the subject on the lens unit 101, the solid arrows indicate the communication path of the control signal in the imaging device 100, and the dashed-dotted arrows indicate the communication path of the image signal obtained from the imaging element 102, respectively. Note that the signals may be in either analog or digital format. Hereinafter, digital format signals may be referred to as data.

[0013] The imaging element 102 may be a known CCD or CMOS color image sensor having a color filter of a primary color Bayer array, for example. The imaging element 102 includes a pixel array in which a plurality of pixels are two-dimensionally arranged, and a peripheral circuit for reading signals from each pixel. Each pixel accumulates charges corresponding to the amount of incident light by photoelectric conversion. By reading out a signal having a voltage corresponding to the amount of charges accumulated during the exposure period from each pixel, a group of pixel signals (analog image signal) representing the subject image formed on the imaging surface is obtained. In the present embodiment, it is assumed that the imaging element 102 has an A / D conversion function and outputs a digital image signal (image data).

[0014] The operations (such as exposure, reading, reset, etc.) and the timing of the imaging element 102 are controlled by a control signal from the control unit 107.

[0015] The flicker detection unit 103 determines whether or not flicker stripes are generated in the image data for one frame output by the imaging element 102. Since the presence or absence of flicker stripes can be detected by a known method, detailed description thereof is omitted. Further, when it is determined that flicker stripes are generated, the flicker detection unit 103 detects the period of the flicker stripes. The flicker detection unit 103 outputs the determination result to the control unit 107.

[0016] The image synthesis unit 104 applies a synthesis process to the image data read from the image sensor 102 at different timings, and generates synthesized image data. Details of the synthesis process will be described later. The image synthesis unit 104 outputs the generated synthesized image data to the signal processing unit 105. Note that the synthesis process may be carried out as part of the image processing performed by the signal processing unit 105. In that case, the image synthesis unit 104 is unnecessary.

[0017] The signal processing unit 105 applies predetermined image processing to the synthesized image data output by the image synthesis unit 104, generates signals or image data according to the application, and acquires and / or generates various types of information. The signal processing unit 105 may be a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the signal processing unit 105 may be configured such that a processor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit) realizes a specific function by executing software. The signal processing unit 105 outputs the acquired or generated information and data to the control unit 107.

[0018] The image processing applied by the signal processing unit 105 may include, for example, preprocessing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Preprocessing may include signal amplification, reference level adjustment, defective pixel correction, and the like. Color interpolation processing is performed when a color filter is provided in the image sensor, and is a process of interpolating the values of color components not included in the individual pixel data constituting the image data. Color interpolation processing is also called demosaicing processing. Correction processing may include white balance adjustment, gradation correction, correction of image degradation caused by optical aberration of the imaging optical system (image restoration), correction of the influence of peripheral light reduction of the imaging optical system, color correction, and other processes. Detection processing may include detection of a feature region (for example, a face region or a human body region) and its movement, person recognition processing, and the like. Data processing may include processes such as region cropping, merging, and scaling. The generation of display image data and recording image data is also included in data processing. In addition, the encoding and decoding and header information generation (data file generation) processes performed by the compression / decompression unit 106 may be performed by the signal processing unit 105 as part of the data processing. The evaluation value calculation process may include processes such as generating signals and evaluation values ​​used for autofocus detection (AF), and generating evaluation values ​​used for automatic exposure control (AE). Special effects processing may include adding blur effects, changing color tones, and relighting. These are merely examples of processes that the signal processing unit 105 can apply, and do not limit the processes that the signal processing unit 105 can apply.

[0019] The compression / decompression unit 106 applies encoding processing to the image data generated by the signal processing unit 105 and stored in the RAM of the control unit 107, thereby generating encoded data. The compression / decompression unit 106 also generates a data file containing the encoded data and associated information, and outputs it to the control unit 107.

[0020] The compression / decompression unit 106 also applies a decoding process to the image data file read from the image recording unit 110 and stored in the RAM of the control unit 107, and outputs the decoded image data to the control unit 107.

[0021] The compression / decompression unit 106 applies encoding and decoding processing according to the image type (still image / video) and recording format (data format). There are no restrictions on the recording formats supported by the compression / decompression unit 106, but for still images, it may be a format conforming to a known standard such as JPEG, GIF, or DNG, and for videos, it may be a format conforming to a known standard such as MPEG or MOV. It may also support manufacturer-specific formats such as RAW format. The recording format is predetermined, for example, by user settings.

[0022] The control unit 107 is a microcontroller composed of, for example, a program-executable processor (CPU), non-volatile memory (ROM), and volatile memory (RAM). By loading the program stored in ROM into RAM and executing it with the CPU, it controls the operation of each part that makes up the imaging device 100, thereby realizing the functions of the imaging device 100.

[0023] The ROM may be electrically rewritable and stores programs executed by the CPU, various settings for the imaging device 100, GUI data, and the like. RAM is used as the CPU's main memory, a buffer for temporarily storing image data generated by the signal processing unit 105, and data files generated by the compression / decompression unit 106. A portion of the RAM may also be used as video memory for the image display unit 109. Note that ROM and RAM may be provided separately from the control unit 107.

[0024] The operation unit 108 is a general term for input devices (buttons, switches, dials, etc.) provided for the user to input various instructions to the imaging device 100. The input devices constituting the operation unit 108 have names according to the function they are assigned to. For example, the operation unit 108 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, directional keys, and a select key. The release switch is a switch for recording still images, and the control unit 107 recognizes a half-pressed state of the release switch as a shooting preparation instruction and a fully pressed state as a shooting start instruction. The control unit 107 also recognizes the video recording switch as a video recording start instruction when pressed in shooting standby mode, and as a recording stop instruction when pressed during video recording. The functions assigned to the same input device may be variable. The input devices may also be software buttons or keys using a touch display. Furthermore, the operation unit 108 may include input devices that support non-contact input methods such as voice input and eye-tracking input.

[0025] The control unit 107 monitors operations on the operation unit 108. When the control unit 107 (CPU) detects an operation on the operation unit 108, it executes an action corresponding to the detected operation.

[0026] The image display unit 109 includes a display device such as an LCD and an interface circuit for it. The image display unit 109 displays, for example, live view images, menu screens, recorded images, and information indicating the settings and status of the imaging device 100.

[0027] The image recording unit 110 is a storage device that stores images captured by the imaging device 100 as data files. The image recording unit 110 may be a recording device using a semiconductor memory card as a recording medium, but it may also be another known recording device such as a hard disk drive. The control unit 107 records the data files generated by the compression / decompression unit 106 to the image recording unit 110, or reads the data files recorded in the image recording unit 110 and stores them in RAM.

[0028] Figure 2 shows an example of the configuration of the image sensor 102. The pixel array of the image sensor 102 has multiple pixels 201 arranged in a matrix of m rows and n columns. Each pixel 201 is connected to a vertical output line 202, a transfer control signal line 203, a reset control signal line 204, and a row selection signal line 205.

[0029] The vertical scanning circuit 206 supplies control signals to the transfer control signal line 203, the reset control signal line 204, and the row selection signal line 205, respectively. The control signals supplied by the vertical scanning circuit 206 have either a low level or a high level and control the on / off state of the switch (transistor) to which the signal line is connected. The horizontal scanning circuit 208 selects the vertical output line 202.

[0030] The row circuit group 207 includes amplification circuits and A / D conversion circuits. The row circuit group 207 applies amplification and A / D conversion to the pixel signals read out through the vertical output line 202. The pixel signals for one pixel row read out by the row circuit group 207 are output from the image sensor 102 by scanning with the horizontal scanning circuit 208.

[0031] The image sensor 102 is provided with one transfer control line 203, one reset control line 204, and one row selection signal line 205 per pixel row, and two vertical output lines 202 per pixel column. In the example shown in Figure 2, the two vertical output lines 202 provided for one pixel column are connected alternately in units of two rows of pixels. For example, of the vertical output lines 202-1a and 202-1b provided for the same pixel column, vertical output line 202-1a is connected to pixels in rows 1-2, 5-6, 9-10, (and so on). Vertical output line 202-1b is connected to pixels in rows 3-4, 7-8, 11-12, (and so on).

[0032] By providing the vertical output lines 202 in this manner, the image sensor 102 can individually read out pixel signals from each of the pairs of vertical output lines 202 provided in the same pixel row. Therefore, it is possible to individually read out pixel signals for m / 2 rows and n columns from the m rows and n columns of pixels 201 that constitute the pixel array.

[0033] Figure 3 is a circuit diagram showing an example configuration of pixel 201. The photodiode (PD) 301 generates an electric charge corresponding to the amount of incident light. When the transfer switch 302, to which the transfer control line 203 is connected to the gate, is turned on, the charge generated by PD 301 is transferred to the floating diffusion (FD) 303. The FD 303 converts the charge generated by PD 301 into a voltage.

[0034] A constant current source 306 is connected to the vertical output line 202. The source follower amplifier (SF) 304 amplifies the voltage of the FD303 and outputs it as a pixel signal. The output of the SF304 is output to the vertical output line 202 when the row selection switch 305, to which the row selection signal line 205 is connected to the gate, is turned on.

[0035] The charge stored in FD303 is reset when the reset switch 307, to which the reset control signal line 204 is connected to the gate, is turned on. When the transfer switch 302 is turned on while the reset switch 307 is on, PD301 is reset.

[0036] The transfer control signal line 203, the reset control signal line 204, and the row selection signal line 205 are supplied with control signals from the vertical scanning circuit 206 to turn each switch on and off.

[0037] The readout operation using two vertical output lines provided for each pixel row will be further explained with reference to Figure 4. In this embodiment, the pixel array of the image sensor 102 is provided with a primary color Bayer array color filter. The primary color Bayer array color filter consists of four pixels, two horizontally and two vertically, with RGB color filters arranged regularly.

[0038] In Figure 4, the pixel rows connected to the vertical output lines 202-1a to 202-na in Figure 2 are labeled as Stream 1, and the pixel rows connected to the vertical output lines 202-1b to 202-nb are labeled as Stream 2. In this embodiment, the multiple pixel rows of the image sensor 102 are divided into multiple groups (streams). Controlling the operation of the image sensor 102 to output multiple images for each pixel row group in one frame period is called multi-stream readout. Controlling the operation of the image sensor 102 to output one image in one frame period is called single-stream readout. One frame period corresponds to the reciprocal of the frame rate of the video.

[0039] The pixel rows constituting Stream 1 and the pixel rows constituting Stream 2 are connected to different vertical output lines. The vertical scanning circuit 206 alternately reads out only the image rows constituting Stream 1 and only the image rows constituting Stream 2. Hereinafter, the image data consisting of the pixel signal group read out from the pixel rows constituting Stream 1 will be referred to as Stream 1 image data, and the image data consisting of the pixel signal group read out from the pixel rows constituting Stream 2 will be referred to as Stream 2 image data. The Stream 1 image data and the Stream 2 image data are output from the image sensor 102 to the image synthesis unit 104 as separate frame image data with different readout start timings.

[0040] Next, using Figures 5 and 6, we will explain the readout operation of the image sensor 102 and the image synthesis process of stream 1 image data and stream 2 image data by the image synthesis unit 104.

[0041] 501 is the reset timing for each pixel row that makes up stream 1. The reset timing is the timing at which the vertical scanning circuit 206 turns on the reset switch 307 via the reset control signal line 204.

[0042] Furthermore, 502 is the charge transfer timing for each pixel row that makes up stream 1. The reset timing is the timing at which the vertical scanning circuit 206 turns on the transfer switch 302 via the transfer control signal line 203.

[0043] Similarly, 503 is the reset timing for each pixel row constituting Stream 2, and 504 is the charge transfer timing for each pixel row constituting Stream 2.

[0044] Furthermore, the vertical synchronization signal (VD) is one of the control signals supplied from the control unit 107 to the image sensor 102. The control unit 107 controls the image sensor 102 to operate based on timing derived from the level change of the vertical synchronization signal, which can take on high and low levels. Specifically, the control unit 107 decides to start charge transfer of the first pixel row (1st row) constituting Stream 1 at the timing (time t3) when the vertical synchronization signal changes from a high level to a low level. The control unit 107 then determines the reset timing so that charge accumulation of the 1st row begins at time t1, which is the amount of the accumulation period from time t3, so that the accumulation period ends at time t3. The control unit 107 similarly determines the operation timing for the second pixel row (2nd row) and subsequent rows constituting Stream 1. The control unit 107 controls the operation of the vertical scanning circuit 206, column circuit group 207, and horizontal scanning circuit 208 of the image sensor 102 according to the determined operation timing.

[0045] The vertical synchronization signal has a periodic pulse-like level change, with the pulse generation period corresponding to one frame duration. The ambient light is assumed to be a flicker light source (flashing light source) whose brightness changes with period f. The flicker detection unit 103 detects the brightness change period f of the light source from the image data (e.g., stream 1 image data) read out from the image sensor 102.

[0046] When the control unit 107 receives notification of the brightness change period f of the light source from the flicker detection unit 103, it determines the readout start timing for the stream 2 image data. Specifically, the control unit 107 determines the readout start timing for the pixel rows constituting stream 2 to be shifted by half the period f (f / 2) from the readout start timing for the pixel rows constituting stream 1. In other words, in Figure 5, the readout start timing time t4 for the first pixel row (3rd row) constituting stream 2 satisfies the relationship t4 = t3 + f / 2 with respect to the readout start timing time t3 for the first pixel row (1st row) constituting stream 1.

[0047] Furthermore, in order to make the pixel storage time equal for the Stream 1 image data and the Stream 2 image data, the control unit 107 determines the reset timing 503 (time t2) to satisfy the relationship t3-t1=t4-t2.

[0048] By operating the image sensor 102 at the timing described in Figure 5, when the ambient light source is a flickering light source, flicker fringes are generated in the Stream 1 image data and Stream 2 image data whose phase is shifted by half the brightness change period f of the light source (i.e., out of phase). Figure 6 schematically shows the flicker fringes generated in the Stream 1 image and Stream 2 image.

[0049] The image synthesis unit 106 generates a composite image by, for example, averaging the luminance values ​​of the stream 1 image and the stream 2 image to obtain a single frame image. The image synthesis unit 106 may also perform a process to increase the vertical resolution when generating the composite image. The image synthesis unit 106 may also generate the composite image by any other method that can suppress flicker fringes, taking advantage of the fact that the flicker fringes occurring in the stream 1 image and the stream 2 image are in opposite phases. For example, the image synthesis unit 106 may generate the composite image by adding the stream 1 image and the stream 2 image together.

[0050] Regardless of the length of the storage time, the flicker fringes that occur in the Stream 1 image and the Stream 2 image are in opposite phase. Therefore, it is possible to set the storage time (shutter speed) independently of the brightness change period f of the flashing light source.

[0051] Figure 7 is a flowchart illustrating the operation of the imaging device 100 in this embodiment. The operations described below are carried out by the CPU of the control unit 107 executing a program and controlling the operation of other components as needed.

[0052] This section describes the operation during video recording, but the same processing as when generating a single frame of video can be performed during still image shooting. Furthermore, it is assumed that the control unit 107 sequentially determines the exposure conditions (accumulation period, aperture value, shooting sensitivity) based on the video recording frame rate and the AE evaluation value generated by the signal processing unit 105. In addition, if the flicker detection unit 103 does not detect the presence or absence of a blinking light source or the brightness change period f, the control unit 107 controls the operation of the image sensor 102 based on a predetermined initial value of the operation timing.

[0053] In step S702, the control unit 107 sets an operation timing (initial value) for the image sensor 102 to alternately read out stream 1 image data and stream 2 image data with equal storage time, one frame at a time, and instructs the image sensor 102 to start video recording.

[0054] In step S703, the control unit 107 reads out either stream 1 image data or stream 2 image data from the image sensor 102 and supplies it to the flicker detection unit 103.

[0055] In S704, the flicker detection unit 103 determines whether or not there are flicker fringes in the image data supplied from the image sensor 102.

[0056] In S705, the flicker detection unit 103 executes S706 if it determines that flicker fringes are occurring, and outputs a detection result to the control unit 107 indicating that flicker fringes could not be detected if it does not determine that flicker fringes are occurring. The control unit 107 executes S708 if no flicker fringes are detected by the flicker detection unit 103.

[0057] In S706, the flicker detection unit 103 detects the period or frequency of the flicker fringes. The flicker detection unit 103 outputs the detection result to the control unit 107.

[0058] In step S707, the control unit 107 determines the readout timing and the start timing of the storage period for the stream 2 image data, as explained with reference to Figure 5, based on the period or frequency of the flicker fringes detected by the flicker detection unit 103. The control unit 107 then sets the determined timing to the image sensor 102.

[0059] In S708, the control unit 107 sequentially supplies the stream 1 image data and stream 2 image data obtained from the shooting after setting the conditions for shooting the stream 2 image in S707 to the image synthesis unit 104.

[0060] In S709, the image synthesis unit 106 applies synthesis processing to the stream 1 image data and the stream 2 image data to generate a single-frame composite image with suppressed flicker fringes. The generated composite image can be used for recording to the image recording unit 110, display on the image display unit 109, or for AE processing to determine the next shooting conditions.

[0061] In S710, the control unit 107 determines whether the conditions for ending shooting have been met. The conditions for ending shooting are predetermined, one or more of which may be present, such as the detection of a specific operation on the operation unit 108 or the free capacity of the image recording unit 110 falling below a threshold. If the control unit 107 determines that the conditions for ending shooting have been met, it terminates the shooting operation; otherwise, it executes S702.

[0062] According to this embodiment, image data of multiple frames having flicker fringes with different phases is acquired from the image sensor, and by combining the image data of multiple frames, a single frame of image data with suppressed flicker fringes is generated. Since the length of the storage period when acquiring multiple frames of image data does not depend on the brightness change period or frequency of the flashing light source, a flexible storage period (shutter speed) can be selected.

[0063] Furthermore, the number of stream images n acquired from the image sensor is not limited to 2, but may be 3 or more. When reading out 3 or more streams of image data, the timing should be determined such that the difference in the start timing of the accumulation period for each stream image is equal to the brightness change period / number of streams (pixel row groups), i.e., f / n.

[0064] (modified version) Figure 8 is a flowchart showing the operation of the imaging device 100 according to a modified version of the first embodiment. In this modified version, one frame (single stream) of image data is acquired from the image sensor 102 within one cycle of the vertical synchronization signal until flicker fringes are detected.

[0065] In S802, the control unit 107 sets the image sensor 102 to either read out only the Stream 1 image data, or to read out the Stream 1 image data and Stream 2 image data line by line, with the start timing of the storage period and the start timing of the readout being the same.

[0066] Addition within the image sensor 102 can be achieved using known configurations. For example, a single FD303 may be configured to be shared by multiple pixels to be added. Alternatively, the column circuit group 207 may be configured to perform addition after A / D conversion.

[0067] The operations from S803 to S805 are the same as those from S703 to S705 in Figure 7, so the explanation is omitted. However, the image data used for flicker detection in S804 is a single-stream image data, not either the Stream 1 image data or the Stream 2 image data.

[0068] If it is determined in S805 that flicker fringes are occurring, the process proceeds to S806; otherwise, the process proceeds to S813.

[0069] In S806, the flicker detection unit 103 detects the period or frequency of the flicker fringes, similar to S706.

[0070] In S807, the control unit 107 changes the settings of the image sensor 102 to read out both the stream 1 image data and the stream 2 image data (to perform multi-stream readout), similar to the first embodiment.

[0071] Since steps S808 to S811 perform the same operations as steps S707 to S710, their explanation will be omitted.

[0072] Meanwhile, in S813, the control unit 107 determines whether the image sensor 102 is set to single-stream readout or not. If it is determined to be single-stream readout, it executes S815; otherwise, it executes S814.

[0073] In S814, the control unit 107 switches the setting of the image sensor 102 to single-stream readout.

[0074] In S815, the control unit 107 performs the shooting operation in the same manner as in S803. If the image sensor 102 is set to single-stream readout, the image synthesis unit 104 outputs the image data to the signal processing unit 105 without performing synthesis processing.

[0075] In multi-stream readout, the storage periods for stream 1 image data and stream 2 image data are different, which can cause motion blur in the composite image if a moving subject is present. In this modified example, multi-stream readout is performed only when flicker fringes are detected, thus achieving both suppression of motion blur and suppression of flicker fringes.

[0076] ●(Second Embodiment) Next, a second embodiment of the present invention will be described. Figure 9 shows the arrangement of color filters provided on the image sensor 102 in the second embodiment. In this embodiment, the image sensor 102 is configured to read out by adding the pixel signals obtained from 4 pixels arranged in a 2x2 vertical grid. For this reason, a color filter with a primary color Bayer array, where 2x2 vertical pixels are considered as one pixel, is provided.

[0077] Figure 10 is a circuit diagram showing the configuration of the four pixels 901-1 to 901-4 shown as 902 in Figure 9. Pixels 901-1 to 901-4 share one FD1003, SF1004, row selection switch 1005, and reset switch 1007. A reset control signal line 1010 is connected to the gate of the reset switch 1007. Each of pixels 901-1 to 901-4 has a PD1001, a transfer switch 1002, and a reset switch 1008.

[0078] The outputs of the transfer switch 1002 for each of the pixels 901-1 to 902-4 are all connected to the FD1003. Transfer control signal line 1011-1 is connected to the gate of the transfer switch 1002 for pixels 901-2 and 901-4, and transfer control signal line 1011-2 is connected to the gate of the transfer switch 1002 for pixels 901-3 and 901-4.

[0079] Similarly, the pixel reset control signal line 1009-1 is connected to the gate of the reset switch 1008 for pixels 901-2 and 901-4, and the pixel reset control signal line 1009-2 is connected to the gate of the reset switch 1008 for pixels 901-3 and 901-4.

[0080] Furthermore, control signals are supplied from the vertical scanning circuit 206 to the reset control signal line 1010, the transfer control signal line 1011, and the pixel reset control signal line 1009, respectively.

[0081] Figure 11 is a timing chart of the readout operation of the image sensor 102 in this embodiment. For the sake of clarity and ease of explanation, the readout operation of the noise signal and the operation of subtracting the noise signal from the pixel signal, which are generally performed in CMOS image sensors, have been omitted here.

[0082] The control unit 107 determines the timing of the control signal supplied to the transfer control signal line 1011-1 so as to start the transfer control of the first row based on the vertical synchronization signal VD. Here, as in the first embodiment, the control unit 107 decides to start the charge transfer of the first row of pixels (pixels 901-1 and 901-2 in Figure 10) to be read out first at the timing when the vertical synchronization signal changes from a high level to a low level (time t103). The control unit 107 then determines the reset timing so that the charge accumulation of the first row of pixels starts at time t101, which is the amount of the accumulation period that precedes time t103, so that the accumulation period ends at time t103. The control unit 107 also determines the same reset timing (time t101) for FD1003, which is shared by the four pixels to be added, as for the first row of pixels.

[0083] Similar to the first embodiment, the flicker detection unit 103 detects the brightness change period f of the light source from the image data read from the image sensor 102. When taking images to acquire image data used for flicker detection, the control unit 107 controls the operation of the image sensor 102 so that the accumulation periods of the four pixels to be added are equal.

[0084] When the control unit 107 receives notification of the brightness change period f of the light source from the flicker detection unit 103, it determines the timing for reading out the second row of the image in order to suppress flicker fringes. Specifically, the control unit 107 determines the start timing for reading out the pixels of the second row (pixels 901-3 and 901-4 in Figure 10) to be at a timing (t104) that is shifted by half the period f (f / 2) from the start timing for reading out the pixels of the first row (time t103).

[0085] In other words, in Figure 11, the start timing t104 for reading pixels in the second row satisfies the relationship t104 = t103 + f / 2 with respect to the start timing t103 for reading pixels in the first row.

[0086] Furthermore, in order to equalize the storage time among pixels that share FD103, the control unit 107 determines the reset timing (time t102) of the pixels in the second row so as to satisfy the relationship t103-t101=t104-t102.

[0087] Even after the first row of pixels has been read out, the reset control signal supplied from the reset control signal line 1010 remains at a low level until all pixels to be added have been read out, and the FD1003 is not reset. Therefore, by transferring the charge accumulated in the first row of pixels and the second row of pixels to be added between times t102 and t104 to the FD1003 at time t104, the charges of pixels 901-1 to 901-4 are added to the FD1003.

[0088] Subsequently, the row selection signal line 1012 is set to a high level for a predetermined time at time t105, thereby turning on the row selection switch 1005. As a result, the voltage of FD1003 is output as a pixel signal to the vertical output line 202 via SF1004.

[0089] In this embodiment, the operation of the image synthesis unit 104 in the first embodiment is realized by summation reading of the image sensor 102. Therefore, the same effects as in the first embodiment can be achieved without providing the image synthesis unit 104.

[0090] Furthermore, a composite image may be generated on the image sensor using a configuration different from one in which the FD is shared by multiple pixels. For example, the images can be added at the input stage of the A / D converter in the column circuit group 207, added using an amplifier, or added after A / D conversion.

[0091] (Other embodiments) The flashing light source in the above embodiment is a light source whose brightness changes periodically, and does not necessarily have to have a period of being off.

[0092] In the embodiment described above, the flicker detection unit 103 and the image synthesis unit 104 were provided outside the image sensor 102. However, the functions of the flicker detection unit 103 and the image synthesis unit 104 may be incorporated into the image sensor 102. For example, the image sensor 102 could be made into a known stacked structure, and the functions of the flicker detection unit 103 and the image synthesis unit 104 could be implemented in the signal processing circuit portion.

[0093] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0094] This embodiment includes the following imaging device, a control method for the imaging device, and a program. (Item 1) An image sensor having multiple pixels arranged in a matrix, Control means for controlling the operation of the image sensor, The image sensor has a synthesis means that divides the pixel rows into a plurality of groups and generates a composite image by combining the images obtained from each group, The imaging apparatus is characterized in that the control means controls the operation of the image sensor such that the start timing of the accumulation period for the first pixel row read out in each of the plurality of groups has a difference based on the period of the brightness change of ambient light and the number of the plurality of groups. (Item 2) The imaging apparatus according to item 1, characterized in that the control means sets an accumulation period of the same length for each pixel row of the image sensor. (Item 3) The imaging apparatus according to item 1 or 2, characterized in that, if the period of the brightness change is f and the number of the plurality of groups is n, the difference is equal to f / n. (Item 4) The imaging apparatus according to any one of items 1 to 3, further comprising a detection means for detecting the period of change in the brightness of the ambient light based on an image obtained by the image sensor. (Item 5) If the detection means does not detect the period of the brightness change, the control means controls the image sensor to output an image obtained from a pixel row where the start timing of the accumulation period is the same. The imaging apparatus according to item 4, characterized in that the synthesis means does not generate the synthesized image. (Item 6) The imaging device according to item 4 or 5, characterized in that the detection means is provided on the image sensor. (Item 7) The imaging apparatus according to any one of items 1 to 6, characterized in that the image sensor functions as the synthesis means. (Item 8) The imaging device according to item 7, characterized in that the imaging device is configured to add signals read from different groups of pixel rows, and outputs the composite image based on the added signals. (Item 9) An image sensor having multiple pixels arranged in a matrix, Control means for controlling the operation of the image sensor to acquire multiple images from the image sensor in which the phase of brightness changes caused by ambient light is different, The system includes a synthesis means for combining the aforementioned multiple images to generate a composite image, The imaging device is characterized in that the control means causes the timing of the accumulation period during the capture of the plurality of images to differ based on the period of the brightness change and the number of the plurality of images. (Item 10) A control method for an imaging device having an image sensor having a plurality of pixels arranged in a matrix, The pixel rows of the aforementioned image sensor are divided into multiple groups, The operation of the imaging device is controlled such that the start timing of the accumulation period for the first pixel row read in each of the aforementioned multiple groups has a difference based on the period of the ambient light brightness change and the number of the aforementioned multiple groups. A method for controlling an imaging device, characterized by generating a composite image by combining images obtained from each of the aforementioned multiple groups. (Item 11) A program for causing a computer in an imaging device having an image sensor with multiple pixels arranged in a matrix to function as each of the means of the imaging device described in any one of items 1 to 9.

[0095] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0096] 100...Imaging device, 101...Lens unit, 102...Image sensor, 103...Flicker detection unit, 104...Image synthesis unit, 105...Signal processing unit, 107...Control unit, 108...Operation unit

Claims

1. An image sensor having multiple pixels arranged in a matrix, Control means for controlling the operation of the image sensor, The image sensor has a synthesis means that divides the pixel rows into a plurality of groups and generates a composite image by combining the images obtained from each group, The imaging apparatus is characterized in that the control means controls the operation of the image sensor such that the start timing of the accumulation period for the first pixel row read out in each of the plurality of groups has a difference based on the period of the brightness change of ambient light and the number of the plurality of groups.

2. The imaging apparatus according to claim 1, characterized in that the control means sets an accumulation period of the same length for each pixel row of the image sensor.

3. The imaging apparatus according to claim 1, characterized in that, if the period of the brightness change is f and the number of the plurality of groups is n, the difference is equal to f / n.

4. The imaging apparatus according to claim 1, further comprising a detection means for detecting the period of change in the brightness of the ambient light based on an image obtained by the image sensor.

5. If the detection means does not detect the period of the brightness change, the control means controls the image sensor to output an image obtained from a pixel row where the start timing of the accumulation period is the same. The imaging apparatus according to claim 4, characterized in that the synthesis means does not generate the synthesized image.

6. The imaging apparatus according to claim 4, characterized in that the detection means is provided on the image sensor.

7. The imaging apparatus according to claim 1, characterized in that the image sensor functions as the synthesis means.

8. The imaging device according to claim 7, characterized in that the imaging device is configured to add signals read from different groups of pixel rows, and outputs the composite image based on the added signals.

9. An image sensor having multiple pixels arranged in a matrix, Control means for controlling the operation of the image sensor to acquire multiple images from the image sensor in which the phase of brightness changes caused by ambient light is different, The system includes a synthesis means for combining the aforementioned multiple images to generate a composite image, The imaging device is characterized in that the control means causes the timing of the accumulation period during the capture of the plurality of images to differ based on the period of the brightness change and the number of the plurality of images.

10. A control method for an imaging device having an image sensor having a plurality of pixels arranged in a matrix, The pixel rows of the aforementioned image sensor are divided into multiple groups, The operation of the imaging device is controlled such that the start timing of the accumulation period for the first pixel row read in each of the aforementioned multiple groups has a difference based on the period of the ambient light brightness change and the number of the aforementioned multiple groups. A method for controlling an imaging device, characterized by generating a composite image by combining images obtained from each of the aforementioned multiple groups.

11. A program for causing a computer in an imaging device having an image sensor having a plurality of pixels arranged in a matrix to function as each of the means of the imaging device according to any one of claims 1 to 9.

Images