Imaging device, imaging method, and program

Abstract

This imaging device comprises a control unit that, on the basis of the cycle of a detected flicker component and the peak timing of the flicker component, controls exposure timing of flicker-less imaging causing a specific timing within the exposure period to be synchronized with the peak timing. In addition, the imaging device also comprises a flicker correction unit that performs flicker correction to reduce flicker within a frame of image data obtained by flicker-less imaging, on the basis of the results detecting the phase relationship of a flicker light source to each line of the image data.

Description

[Technical Field]

[0001] The present technology relates to an imaging device, an imaging method, and a program, and in particular to a technology for reducing the influence of flicker. [Background technology]

[0002] In imaging devices (cameras), images captured may be affected by flicker. For example, fluorescent lights, which are widely used as indoor light sources, and LEDs (Light Emitting Diodes), which have become increasingly popular in recent years, cause periodic flickering of the illumination light due to the influence of commercial power supply frequencies. Therefore, images captured under such lighting environments suffer from degradation of image quality, such as color unevenness, due to the flickering. Patent Document 1 listed below discloses a technology for performing so-called flickerless imaging, which reduces the effects of flicker by detecting the cycle and peak timing of the flicker component and controlling the timing of exposure to synchronize with the peak timing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] WO2017 / 217137 publication Summary of the Invention [Problem to be solved by the invention]

[0004] Flickerless imaging is performed when the curtain speed is sufficiently shorter than the flicker cycle, such as when the light source flickers at a cycle of 100 Hz or 120 Hz and the mechanical shutter provides a curtain speed of approximately 1 to 4 ms. This is because, unless the curtain speed is sufficiently shorter than the flicker cycle, flicker will remain in the frame of the captured image even when flickerless imaging is performed.

[0005] However, in recent years, the use of electronic shutter speeds with long curtain speeds and opportunities to capture images in environments with high-frequency flicker caused by LED light sources have increased, and it is often not possible to achieve shooting conditions where the curtain speed is sufficiently shorter than the flicker cycle, so there is a growing demand for flicker reduction even in such cases.

[0006] Therefore, the present disclosure proposes a technique for avoiding the effects of flicker without being limited to the condition that the curtain speed is sufficiently shorter than the flicker cycle, for example. [Means for solving the problem]

[0007] The imaging device according to the present technology includes a control unit that performs exposure timing control for flickerless imaging, synchronizing a specific timing within an exposure period with the peak timing based on the period of the detected flicker component and the peak timing of the flicker component, and a flicker correction unit that performs flicker correction to reduce intra-frame flicker in image data based on the detection result of the phase relationship between each line of image data obtained by the flickerless imaging and a flicker light source. Assuming that so-called flickerless imaging is performed by synchronizing a specific timing within an exposure period with the peak timing of the flicker component, the flicker component that occurs even when such flickerless imaging is performed is corrected.

[0008] In the imaging device according to the present technology described above, the flicker correction unit may perform the flicker correction by calculating a flicker correction value for each line based on the phase of the flicker for each line and the amplitude of the flicker for each line, and performing a process to correct the image data for each line. By performing flicker correction for each line, correction suitable for the relationship between the exposure timing and the flicker period of each line can be performed.

[0009] In the imaging device according to the present technology described above, it is considered that the flicker correction unit obtains the phase of the flicker for each line and the amplitude of the flicker for each line from the flicker detection result for the flickerless imaging. Information on the phase and amplitude of flicker for each line obtained from the results of flicker detection performed for flickerless imaging is used for flicker correction.

[0010] In the imaging device according to the present technology described above, it is considered that the flicker correction section performs the flicker correction at least when the curtain speed during imaging operation is longer than a predetermined value based on a flicker cycle. Flickerless imaging is most effective when the curtain speed is sufficiently shorter than the flicker cycle. Conversely, if the condition that the curtain speed is sufficiently shorter than the flicker cycle is not met, the effect of flickerless imaging is small. In such cases, flicker correction is performed.

[0011] In the imaging device according to the present technology described above, it is conceivable that the control unit controls whether or not the flicker correction unit performs the flicker correction based on the relationship between a flicker cycle and a curtain speed. The control unit acquires information on the curtain speed and the flicker cycle, and determines whether or not to perform flicker correction depending on whether or not the relationship between them satisfies a predetermined relationship condition.

[0012] In the imaging device according to the present technology described above, it is conceivable that the control unit uses information on a flicker cycle obtained as a result of flicker detection for the flickerless imaging to determine whether or not to perform the flicker correction. Information on the flicker cycle obtained from the results of flicker detection performed for flickerless imaging is used to determine whether flicker correction should be performed.

[0013] In the imaging device according to the present technology described above, it is conceivable that the control unit uses information on the curtain speed based on the image size or shutter method selected as the imaging operation at the time of the determination process to determine whether or not to perform the flicker correction. That is, it corresponds to the fact that the curtain speed varies depending on the image size and shutter method.

[0014] In the imaging device according to the present technology described above, the specific timing may be the timing of the center of gravity of exposure. For example, the timing of the exposure center is the timing of approximately the center of the exposure period of the approximately center line in the vertical direction of the image sensor.

[0015] In the imaging device according to the present technology described above, it is conceivable that the control unit does not perform exposure timing control for the flickerless imaging when the amount of deviation between the specific timing and the peak timing is within a set margin. When a specific timing within an exposure period is synchronized with the peak timing of a flicker component for flickerless imaging, if the deviation of the specific timing is within a margin, control for synchronization is not performed.

[0016] In the imaging device according to the present technology described above, when continuous shooting or video shooting is performed while displaying a live view image, the control unit may be configured not to perform exposure timing control for the flickerless imaging if the amount of deviation between the specific timing and the peak timing is within a set margin. During continuous shooting, the specific timing of exposure is not necessarily set to coincide with the flicker peak, but correction is performed taking into account the amount of deviation.

[0017] The imaging method according to the present technology is an imaging method in which an imaging device controls exposure timing for flickerless imaging, synchronizing a specific timing within an exposure period with the peak timing based on the period of a detected flicker component and the peak timing of the flicker component, and performs flicker correction to reduce intra-frame flicker in image data based on the detection result of the phase relationship between each line of image data obtained by the flickerless imaging and a flicker light source. This allows flicker correction to be performed during flickerless imaging. The program according to the present technology is a program that causes a calculation processing device to execute the above-described flicker correction process. This makes it possible to easily realize the imaging device of the present disclosure. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a block diagram of an imaging device according to an embodiment of the present technology; [Figure 2] FIG. 10 is an explanatory diagram of the process flow during recording standby and capture. [Figure 3] FIG. 2 is a block diagram of a main part of a digital signal processing unit according to the embodiment. [Figure 4] FIG. 2 is an explanatory diagram of the phase, period, and amplitude of flicker. [Figure 5] FIG. 1 is an explanatory diagram of flickerless imaging. [Figure 6] FIG. 2 is a block diagram of a flicker detection and correction unit according to an embodiment. [Figure 7] FIG. 10 is an explanatory diagram illustrating a case where the influence of flicker occurs even in flickerless imaging. [Figure 8] 10A and 10B are explanatory diagrams of the relationship between a flicker cycle and parameters for capturing a still image according to an embodiment. [Figure 9] 10 is a flowchart of a process for setting whether or not to perform flicker correction according to an embodiment. [Figure 10] FIG. 10 is an explanatory diagram illustrating a case where a live view image becomes non-smooth. [Figure 11] 10A and 10B are explanatory diagrams of exposure and live view image display according to an embodiment. [Figure 12] 10 is a flowchart of flickerless control according to a margin according to an embodiment. [Figure 13] FIG. 10 is an explanatory diagram of an example of margin setting according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0019] The embodiments will be described below in the following order. <1. Configuration of imaging device> 2. Flicker-free imaging and flicker correction 3. Flickerless control according to margin <4. Summary and Modifications>

[0020] The meanings of some of the terms used in this disclosure will now be explained. "Image" is a term that includes still images and moving images. A "line" refers to a line of pixels arranged horizontally in one frame of a still or video image. "Capture exposure" refers to an exposure operation for recording an image that is performed in response to a user (photographer) operating the release button (shutter button) of an imaging device. A "capture image" is an image based on an image signal obtained by a capture exposure. The capture image is recorded on a recording medium as a still image, multiple still images taken in continuous shooting, or one frame of a video. "Inter-frame exposure" refers to the exposure performed between capture exposures during continuous shooting, specifically for generating live view images. Therefore, the number of pixels read from the image sensor can be smaller than that of the capture exposure (exposure for a low-resolution image).

[0021] A "live view image" or "LV image" refers to an image captured by an image sensor and displayed on a display unit so that it can be viewed by a user. In other words, it is an image that shows the scene from the subject side in real time. For example, before a release operation for a still image, low-resolution images similar to inter-frame exposure are captured, and image data for each frame of a live view image is generated. In the case of continuous shooting, a live view image is generated from both the capture exposure and the inter-frame exposure. Generally, a captured image is generated and recorded as a high-resolution image with a large number of pixels that reflects the number of pixels of the image sensor, whereas a live view image is generated and displayed as a low-resolution image that matches the number of pixels that can be displayed on the display unit.

[0022] <1. Configuration of imaging device> FIG. 1 shows an example of the configuration of an imaging device 1 according to an embodiment. In the imaging device 1, light from a subject is incident on an imaging element 12, which is configured by, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor, via an imaging optical system 11, and is photoelectrically converted by the imaging element 12, thereby obtaining an analog image signal from the imaging element 12. In particular, in the case of this embodiment, a CMOS sensor that operates by the rolling shutter method is assumed.

[0023] The imaging optical system 11 is provided with various lenses such as a zoom lens, a focus lens, a condenser lens, an aperture mechanism, a zoom lens drive mechanism, and a focus lens drive mechanism. A mechanical shutter (for example, a focal plane shutter) may also be provided.

[0024] The image sensor 12 is formed, for example, on a CMOS substrate, with multiple pixels, each having a photodiode (photogate), a transfer gate (shutter transistor), a switching transistor (address transistor), an amplification transistor, a reset transistor (reset gate), etc., arranged two-dimensionally, and also with a vertical scanning circuit, a horizontal scanning circuit, and an image signal output circuit formed thereon.

[0025] The image sensor 12 may be either a primary color system or a complementary color system, and the analog image signal obtained from the image sensor 12 is a primary color signal of each of the RGB colors or a complementary color signal. Alternatively, the image sensor 12 may be configured without a color filter, and the analog image signal obtained from the image sensor 12 may be a black and white image signal. The analog image signal from the imaging element 12 is sampled and held for each color signal in an analog signal processing unit 13 configured as an IC (Integrated circuit), and the amplitude is adjusted by AGC (Automatic Gain Control), and converted into a digital image signal by A / D (Analog to Digital) conversion. The digital image signal (hereinafter referred to as image data) from the analog signal processing unit 13 is input to a temporary storage unit . In some cases, the image pickup device 12 and the analog signal processing unit 13, or further the temporary storage unit 26, are integrated together. A frame memory, which will be described next, may be provided as the temporary storage unit 26 within the stacked image pickup device.

[0026] In this example, the temporary storage unit 26 includes two frame memories 26A and 26B. Image data from the analog signal processing unit 13 is stored alternately in frame memory 26A and frame memory 26B. That is, temporary storage unit 26 stores two consecutively captured image frames. The image data stored in temporary storage unit 26 is output to digital signal processing unit 20 sequentially, starting with the previously stored frame. That is, the image data is output alternately from frame memory 26A and frame memory 26B sequentially to digital signal processing unit 20 in the order of imaging. By providing the frame memories 26A and 26B in this way, it is possible to continuously display a live view image without blacking out, even during continuous shooting, for example.

[0027] The digital signal processing unit 20 is configured as an image processor using an arithmetic processing device such as a DSP (Digital Signal Processor). The digital signal processing unit 20 performs various signal processing on the input image data. For example, as camera processes, the digital signal processing unit 20 performs preprocessing, synchronization processing, YC generation processing, etc. The digital signal processing unit 20 also performs file creation processing on the image data that has undergone these various processes, such as compression encoding for recording or communication, formatting, and generating and adding metadata, to generate files for recording or communication. For example, image files in formats such as JPEG, TIFF (Tagged Image File Format), and GIF (Graphics Interchange Format) are generated as still image files. It is also possible to generate image files in formats such as MP4, which is used for recording MPEG-4-compliant video and audio. It is also possible to generate an image file as raw image data. Furthermore, the digital signal processing unit 20 performs resolution conversion processing on the image data that has been subjected to various signal processes, and generates image data with a lower resolution for, for example, live view display.

[0028] The digital signal processing unit 20 is also provided with a flicker detection and correction unit 25 . The flicker detection and correction unit 25 detects the cycle and peak timing of flicker so that flickerless imaging can be performed, and transmits this information to the camera control unit 21 . Flickerless imaging, the details of which will be described later, is an imaging operation that can reduce the effect on image quality (deterioration of image quality) caused by flickering that occurs from a flickering light source. The flicker detection and correction unit 25 also performs flicker correction processing on the image data obtained by flickerless imaging to further reduce flicker. Flicker correction will also be described in detail later.

[0029] The memory unit 27 represents a buffer memory for image data. The image data processed by the digital signal processing unit 20 is temporarily stored in the memory unit 27 and then transferred to the display unit 15, the recording control unit 14, or the communication unit 16 at a predetermined timing.

[0030] The recording control unit 14 performs recording and reproduction on a recording medium such as a nonvolatile memory, and performs processing to record image files such as moving image data and still image data on the recording medium. The recording control unit 14 may take a variety of actual forms. For example, the recording control unit 14 may be configured as a flash memory built into the imaging device 1 and its write / read circuit. The recording control unit 14 may also take the form of a card recording / playback unit that performs recording / playback access to a recording medium that can be attached to or detached from the imaging device 1, such as a memory card (such as a portable flash memory). The recording control unit 14 may also be realized as an HDD (Hard Disk Drive) built into the imaging device 1.

[0031] The display unit 15 is a display unit that displays various information to the photographer, and is, for example, a display panel or viewfinder such as a liquid crystal display (LCD) or an organic electroluminescence (EL) display disposed on the housing of the imaging device 1. The display unit 15 executes various displays on the display screen based on instructions from the camera control unit 21 . For example, the display unit 15 displays a reproduced image of image data read from a recording medium by the recording control unit 14. The display unit 15 is also supplied with image data of the captured image, the resolution of which has been converted for display by the digital signal processing unit 20, and displays a corresponding image, for example, a live view image. Furthermore, based on instructions from the camera control unit 21, the display unit 15 displays various operation menus, icons, messages, etc., that is, GUI (Graphical User Interface), on the screen.

[0032] The communication unit 16 performs wired or wireless data communication and network communication with external devices, such as transmitting and outputting image data (still image files and video files) and metadata to external information processing devices, display devices, recording devices, playback devices, etc. The communication unit 16 also serves as a network communication unit, and can perform various network communications such as the Internet, a home network, and a LAN (Local Area Network), and can transmit and receive various data to and from servers, terminals, and the like on the network.

[0033] The operation unit 17 collectively refers to input devices that allow the user to input various operations. Specifically, the operation unit 17 refers to various operators (keys, dials, touch panel, touch pad, etc.) provided on the housing of the imaging device 1. The operation unit 17 detects the user's operation, and sends a signal corresponding to the input operation to the camera control unit 21 .

[0034] An AE (Automatic Exposure) detection unit 18 performs detection processing for automatic exposure adjustment from the digital image signal, and supplies brightness information to a camera control unit 21 .

[0035] The camera control unit 21 is configured by a microcomputer (arithmetic processing device) equipped with a CPU (Central Processing Unit). The memory unit 19 stores information and the like used for processing by the camera control unit 21. The illustrated memory unit 19 comprehensively represents, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like. The memory unit 19 may be a memory area built into a microcomputer chip serving as the camera control unit 21, or may be configured as a separate memory chip. The camera control unit 21 controls the entire imaging device 1 by executing a program stored in the ROM or flash memory of the memory unit 19 .

[0036] For example, the camera control unit 21 instructs the digital signal processing unit 20 to perform various signal processes, controls image capturing and recording operations in response to user operations, and controls playback of recorded image files. Furthermore, the camera control unit 21 performs automatic exposure control based on the detection signal from the AE detection unit 18, such as operation control of the diaphragm mechanism, control of the shutter speed of the image sensor 12, and AGC gain control in the analog signal processing unit 13. The camera control unit 21 also controls the driving of the focus lens and zoom lens in response to autofocus control, manual focus operation, zoom operation, and the like.

[0037] The camera control unit 21 is also provided with a function as a flickerless control unit 24, for example, by software. Therefore, the camera control unit 21 corresponds to an imaging control device that performs flickerless control.

[0038] The flickerless control unit 24 performs timing control to synchronize a specific timing (e.g., the timing of the exposure center) within the exposure period with the peak timing based on the cycle of the detected flicker component and the peak timing of the flicker component, i.e., to perform control so that flickerless imaging is performed. The exposure period here refers to the effective exposure period by the mechanical shutter or electronic shutter.

[0039] Furthermore, when performing such flickerless imaging, the flickerless control unit 24 controls the flicker detection and correction unit 25 in the digital signal processing unit 20 so that flicker correction is performed. The flickerless control section 24 may cause the flicker detection and correction section 25 to always perform flicker correction, or may set whether or not to perform flicker correction depending on the conditions.

[0040] Furthermore, when the amount of deviation between the peak timing of the flicker and a specific timing within the exposure period is within a margin set as the allowable deviation from the peak timing, the flickerless control unit 24 may not perform the timing control for flickerless imaging. The margin is set within a range that maintains the flickerless imaging effect to a certain extent without shifting the exposure period. Such an operation will be described later as "flickerless control according to a margin."

[0041] The RAM in the memory unit 19 is used as a work area for the CPU of the camera control unit 21 to process various data, and is used to temporarily store data, programs, and the like. The ROM and flash memory (non-volatile memory) in the memory unit 19 are used to store the OS (Operating System) that the CPU uses to control each unit, application programs for various operations, firmware, various setting information, etc. The various setting information includes communication setting information, setting information related to imaging operations, setting information related to image processing, etc. Setting information related to imaging operations includes exposure settings, shutter speed settings, curtain speed settings for mechanical shutters or electronic shutters, mode settings, and the margin settings described above.

[0042] The driver section 22 includes, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, a motor driver for a diaphragm mechanism motor, and the like. These motor drivers apply drive currents to the corresponding drivers in response to instructions from the camera control unit 21, thereby moving the focus lens and zoom lens, opening and closing the diaphragm blades of the diaphragm mechanism, and so on.

[0043] The shutter speed and exposure timing of the image sensor 12 are determined in accordance with a timing signal from a timing generator 23 . The timing generator 23 outputs various timing signals to the image sensor 12 based on the timing control signal from the camera control unit 21. As a result, the image sensor 12 is driven based on the control of the camera control unit 21.

[0044] Here, the flow of data processing when a live view image is displayed while waiting for still image recording, and when a captured image is recorded, will be described with reference to FIGS. 2A and 2B. In each diagram, the arrow LV indicates the flow of live view image data, and the arrow CAP indicates the flow of captured image data.

[0045] First, FIG. 2A shows the flow of processing when a live view image is displayed while waiting for a release operation or a movie recording start operation. Light incident via the imaging optical system 11 is incident on the imaging element 12, which outputs an image signal through photoelectric conversion.

[0046] In this case, the image signal is a relatively low-resolution image signal for live view display. For example, when outputting an image signal for live view display, the image sensor 12 does not output all pixels, but outputs a pixel signal with a reduced number of pixels by thinning out some of the pixels. On the other hand, when outputting an image signal for recording by capture exposure, it is necessary to generate an image for recording with a high number of pixels, so pixel signals of almost all pixels of the image sensor 12 are output.

[0047] 2A, the image signal from the image sensor 12 is processed by the analog signal processing unit 13 and supplied as a digital signal to the temporary storage unit 26. The image data digitized as described above is stored alternately in frame memories 26A and 26B for each frame. The image data stored in the temporary storage unit 26 is then output to the digital signal processing unit 20, sequentially starting with the previously stored frame. The digital signal processing unit 20 performs the necessary processing to generate image data for live view display and stores it in the memory unit 27. The display unit 15 displays the LV image stored in the memory unit 27.

[0048] Figure 2B shows the flow of data processing when recording a captured image. For example, when the user performs a release operation, the process in Figure 2B is performed. When the user performs a release operation, light incident through the imaging optical system 11 is incident on the imaging element 12, and exposure processing begins. However, after the release operation, there is a certain time interval, i.e., a release lag (release time lag), until exposure processing begins in the imaging element 12. For example, this is a time of about 15 msec to 30 msec.

[0049] When the exposure process in the image sensor 12 is completed, the image sensor 12 outputs an image signal obtained by photoelectric conversion to the analog signal processor 13. In this case, the image signal is, for example, a high-resolution image signal for recording a still image. In addition, in FIGS. 2A and 2B, the arrow LV is shown as a thin arrow and the arrow CAP as a thick arrow, and this is because the thickness of the arrow represents the number of pixels of the image signal.

[0050] The image data converted into a digital signal by the analog signal processing unit 13 is processed by the digital signal processing unit 20 via the temporary storage unit 26. In this case, the digital signal processing unit 20 generates high-resolution image data for recording, and also generates low-resolution image data for live view display, and stores both in the memory unit 27. The image data for recording is then transferred to the recording control unit 14 for recording processing, and the image data for live view display is transferred to the display unit 15 for use in live view display.

[0051] As described above, live view display is performed before the release operation for recording a still image and during the release operation, but there is a phenomenon (blackout) in which the display of the live view image is interrupted after the release. A process for preventing this live view image from being interrupted due to blackout will now be described. If a release operation is performed during exposure of a live view image, the camera control unit 21 instructs the image sensor 12 to stop the exposure of the live view image and change modes. For example, the camera control unit 21 instructs the image sensor 12 to change the readout pixel count, resolution, etc., in order to perform capture exposure. Then, after preparations for capture exposure are complete, the camera control unit 21 starts the capture exposure of the image sensor 12.

[0052] In this operational flow, if the exposure for the live view image being executed is interrupted at the timing of the release operation, the live view image for that frame cannot be displayed, resulting in a blackout. The blackout continues until a capture exposure is performed and a live view image frame based on the capture exposure, such as that shown in Figure 2B, is displayed.

[0053] One method for preventing such blackouts is to not interrupt the exposure for a live view image that was being performed at the release timing. That is, the exposure for a live view image that is being performed at the release timing is not interrupted but waited until it is completed, and the image data is stored in, for example, frame memory 26A, so that the live view image of that frame can be displayed. After the exposure of the live view image is completed, preparation for the capture exposure is performed, and the capture exposure is performed after the preparation, and the image data of the capture exposure is stored in the frame memory 26B. Then, until the live view image data based on the image data of the capture exposure can be generated, the live view image can be continuously displayed using the image data in the frame memory 26A. This prevents blackouts from occurring.

[0054] There are other methods for preventing blackouts. For example, if a release operation is performed while an exposure for a live view image is being performed and the image is being written to frame memory 26B, the exposure is interrupted and the most recent image stored in frame memory 26A (e.g., image data for the frame immediately preceding the frame for which exposure was interrupted) is copied to frame memory 26B. The live view image is then displayed continuously using the image from frame memory 26B. At this time, image data for the capture exposure is written to frame memory 26A, and after the capture exposure is completed, a live view image based on the image data for that capture exposure is displayed.

[0055] By using the frame memories 26A and 26B in this way, it is possible to prevent interruption of the live view image during the release operation.

[0056] Furthermore, by using the same process, it is possible to continuously display live view images even during continuous shooting, for example. During continuous shooting, capture exposure is repeated at a predetermined cycle. If this capture exposure cycle is longer than one frame cycle based on the frame rate of the live view image, one or more inter-frame exposures are performed between capture exposures. For example, the exposure operation such as "capture exposure," "inter-frame exposure," "inter-frame exposure," "capture exposure," "inter-frame exposure," "inter-frame exposure," etc. is repeated until the continuous shooting is completed.

[0057] In this case, if control similar to that for the release timing described above is performed when switching from inter-frame exposure to capture exposure, live view images can be displayed without blackouts during continuous shooting.

[0058] Next, Fig. 3 shows an example of the configuration of a part of the digital signal processing unit 20, in particular an example of the configuration for explaining the positioning of the flicker detection and correction unit 25. Fig. 3 shows an example of the configuration for a primary color system. A primary color system is a three-chip system in which the imaging optical system 11 in Figure 1 has a separation optical system that separates light from a subject into each of the RGB colors, and the imaging element 12 has an imaging element for each of the RGB colors, or a one-chip system in which the imaging element 12 has a single imaging element on the light incident surface on which color filters for each of the RGB colors are repeatedly arranged in sequence for each pixel in the horizontal direction of the screen.

[0059] In the digital signal processing unit 20 of FIG. 3, the clamp circuit 41 clamps the black level of the input RGB primary color signals to a predetermined level, and the gain adjustment circuit 42 adjusts the gain of the RGB primary color signals after clamping according to the amount of exposure.

[0060] The flicker detection and correction units 25R, 25G, and 25B detect the cycle and peak timing of flicker components for flickerless imaging. The flicker detection and correction units 25R, 25G, and 25B can also perform flicker correction processing to reduce flicker components in RGB primary color signals.

[0061] The RGB primary color signals that have passed through flicker detection and correction units 25R, 25G, and 25B are white-balanced by a white-balance adjustment circuit 43, and the gradation of the white-balance-adjusted RGB primary color signals is converted by a gamma correction circuit 44. An output luminance signal Y and color-difference signals RY and BY are generated from the gamma-corrected RGB primary color signals by a synthesis matrix circuit 45.

[0062] In a primary color system, the luminance signal Y is generally generated after all RGB primary color signal processing is completed, as shown in Figure 3. Therefore, by reducing the flicker components in the RGB primary color signals during the RGB primary color signal processing process, it is possible to sufficiently reduce the flicker components of both the color components and the luminance component.

[0063] However, instead of detecting and reducing flicker components for each RGB primary color signal using flicker detection and correction units 25R, 25G, and 25B as shown in Figure 3, it is also possible to provide flicker detection and correction unit 25 on the output side of luminance signal Y of synthesis matrix circuit 45, for example, and configure it to detect and reduce flicker components in luminance signal Y.

[0064] 2. Flicker-free imaging and flicker correction Flickerless imaging will now be described. First, an example of flicker components caused by fluorescent lights will be explained using the NTSC system as an example with reference to Figure 4. Note that the explanation here assumes a frame rate of 60 fps (frames per second) and a commercial power frequency of 50 Hz (Hertz). The characteristics of the flicker components caused in this case are as follows: - 5 / 3 cycles occur in one screen (3 frames (or fields) are the repeating cycle). The phase changes for each line. · It can be treated as a sine wave with a frequency (100Hz) twice the commercial power frequency (50Hz).

[0065] Due to the above characteristics, when a flicker phenomenon occurs, a flicker component like that shown in Figure 4 is generated. In Figure 4, it is assumed that scanning is performed from the top (top of the screen) to the bottom (bottom of the screen).

[0066] In the image sensor 12, the exposure timing differs for each horizontal line, so the amount of light received varies depending on the horizontal line. Therefore, even if the fluorescent lamp provides spatially uniform illumination, there will be horizontal lines where the image signal value is higher than the average and horizontal lines where the image signal value is lower than the average, as shown in Figure 4. For example, in the frame in Figure 4, the flicker component (amplitude of the flicker component) reaches its highest peak on the top horizontal line in the image, i.e., the first line. Furthermore, the flicker component also reaches its highest on a horizontal line that is shifted from the first line by 3 / 5 of the total number of lines in one screen.

[0067] In this way, the flicker component can be expressed as a sine function (sine wave) having an amplitude, period, and initial phase as shown in Figure 4. The initial phase means the phase on the first line.

[0068] Furthermore, the phase of each horizontal line changes depending on the frame. That is, for each frame, the horizontal lines for which the image signal value is higher than the average value and the horizontal lines for which the image signal value is lower than the average value change. In the next frame, the initial phase becomes a sine wave with a different phase. For example, if flicker caused by a fluorescent light occurs at 100 Hz and the frame rate is 60 fps, five periods of the fluorescent light flicker will correspond to the time of three frames. Therefore, the initial phase will be the same every three frames. In this way, the flicker component varies depending on the horizontal line and frame.

[0069] Flickerless imaging is imaging that reduces the effects of such flicker that appears in images. This will be explained using the schematic diagram in Figure 5. Figure 5 shows the amplitude of a flickering light source over a 10 ms period. It also shows an exposure operation 100. The exposure operation 100 is shown schematically as a parallelogram defined by the number of horizontal lines L arranged vertically, the readout period (curtain speed) R from the beginning to the end of the effective pixels, and the exposure period (shutter speed) e.

[0070] In the case of this exposure operation 100, the exposure period is shorter than one cycle of the flicker, and the timing of the exposure center coincides with the peak timing PT of the flicker. The timing of the exposure center is the timing at approximately the center of the exposure period of the approximately center line in the vertical direction of the image sensor. By performing exposure at a timing synchronized with the peak timing PT in this way, an image with reduced flicker effects can be obtained.

[0071] Now, let us assume that the next exposure operation 101 occurs at the timing shown in the figure due to the period of the exposure operation. As described above, the flicker period and the frame period corresponding to the frame rate of the captured image do not coincide, so even if exposure operation 100 occurs at one point, exposure may also occur at another point during a period when the amplitude of the flicker is significantly reduced, such as in the valley of the flicker waveform, as in exposure operation 101. In this case, the effect of the flicker on the image will be relatively large.

[0072] Therefore, the timing of the exposure operation 101 is delayed so that the timing of the exposure center is synchronized with the peak timing PT of the flicker, as shown as exposure operation 101S. By controlling the timing in this way, the effect of flicker can be reduced in each frame. Controlling the exposure timing in this way to capture an image is called flickerless capture.

[0073] To perform this flickerless imaging, the camera control unit 21 (flickerless control unit 24) needs to detect the frequency and peak timing of the flicker. In this embodiment, the flicker detection and correction unit 25 (25R, 25G, 25B) in the digital signal processing unit 20 performs flicker detection, detects its peak timing and frequency, and notifies the camera control unit 21 (flickerless control unit 24). In addition to the above-described flicker detection, the flicker detection / correction unit 25 also performs flicker correction to reduce flicker components occurring within a frame in image data.

[0074] The flicker detection and correction unit 25 that performs such flicker detection and flicker correction will be described with reference to FIG. In the explanation of the flicker detection and correction unit 25 in Figure 6, the input image signal means the RGB primary color signal or luminance signal input to the flicker detection and correction unit 25 (25R, 25G, 25B) as shown in Figure 3, and the output image signal means the RGB primary color signal or luminance signal after processing in the flicker detection and correction unit 25 (25R, 25G, 25B).

[0075] The flicker detection and correction unit 25 in FIG. 6 includes, for example, a normalized integral value calculation block 30, a DFT (Discrete Fourier Transform) block 50, a flicker generation block 55, a frequency estimation / peak detection block 60, and an operation block 65. The normalized integral value calculation block 30 includes an integral block 31 , an integral value holding block 32 , an average value calculation block 33 , a difference calculation block 34 , and a normalization block 35 .

[0076] Integration block 31 integrates input image signal In'(x,y) across one line in the horizontal direction of the screen to calculate integration value Fn(y). The calculated integration value Fn(y) is stored and held in integration value holding block 32 for use in flicker detection in subsequent frames. Integration value holding block 32 is configured to be able to hold integration values ​​for the number of frames required for processing (for example, two frames).

[0077] The average value calculation block 33 calculates the average value AVE[Fn(y)] of, for example, three integral values ​​Fn(y), Fn_1(y), and Fn_2(y). Note that Fn_1(y) is the integral value Fn_1(y) of the same line one frame before, and Fn_2(y) is the integral value Fn_2(y) of the same line two frames before, and these integral values ​​are values ​​read out from the integral value holding block 32.

[0078] The difference calculation block 34 calculates the difference between the integral value Fn(y) supplied from the integration block 31 and the integral value Fn_1(y) of the previous frame supplied from the integral value holding block 32. In the difference value Fn(y)-Fn_1(y), the influence of the subject is sufficiently removed, so the state of the flicker component (flicker coefficient) appears more clearly than in the integral value Fn(y).

[0079] Furthermore, in the normalization block 35, a normalization process is performed by dividing the difference value Fn(y)-Fn_1(y) from the difference calculation block 34 by the average value AVE[Fn(y)] from the average value calculation block 33, and the normalized difference value gn(y) is calculated.

[0080] The DFT block 50 performs a discrete Fourier transform on data corresponding to one wavelength (L lines) of the flicker of the normalized difference value gn(y) from the normalization block 35. This estimates the amplitude γm and initial phase Φmn of the flicker component of each order. The initial phase Φmn is stored in association with a counter generated within the imaging device 1 at predetermined time intervals (for example, every 0.5 μs (microseconds)).

[0081] The initial phase Φ calculated by the DFT block 50 is supplied to the frequency estimation / peak detection block 60. Based on the input initial phase Φ, the frequency estimation / peak detection block 60 estimates at least the frequency of the flicker component (light source), in other words, the period of the flicker component, and further detects the timing of the peak of the flicker component. For example, the frequency estimation / peak detection block 60 estimates the frequency of the flicker component from the time difference based on the frame rate and the phase difference of the initial phase Φ. Furthermore, the frequency estimation / peak detection block 60 detects the timing of the peak of the flicker component, for example, from the initial phase Φ in the first frame and a counter associated with the initial phase Φ. For example, if the initial phase Φmn is 60 degrees, it is possible to determine the timing at which the peak (e.g., 90 degrees) of the flicker component, which can be approximated by a sine wave, appears using the time interval of the counter. As described above, the peak of the flicker component is the point at which the amplitude of the flicker component is maximum.

[0082] The information thus obtained by the frequency estimation / peak detection block 60 is notified to the camera control unit 21 (flickerless control unit 24). By knowing the frequency and peak timing of the flicker, the camera control unit 21 can control the timing of the imaging element for the above-mentioned flickerless imaging.

[0083] By providing such a flicker detection and correction unit 25, it is possible to detect the characteristics of the flicker component, i.e., the period and peak timing of the flicker component, based on the captured image without providing a separate sensor or the like. This makes it possible to prevent an increase in cost due to an increase in the number of parts. It also makes it possible to reduce the size of the imaging device. Note that the process of determining the characteristics of the flicker component is not limited to the method described above, and known methods can be applied.

[0084] Furthermore, the flicker detection and correction unit 25 can perform flicker correction to reduce flicker components occurring in the image signal.

[0085] The flicker generation block 55 calculates a flicker correction value for each line using estimated values ​​of the amplitude γm and initial phase Φmn of the flicker component from the DFT block 50, and so on. Then, the calculation block 65 performs a correction calculation using the flicker correction value for each line from the flicker generation block 55 for each line of the input image signal In′(x, y). This almost completely removes the flicker components contained in the input image signal In'(x,y), and the calculation block 65 outputs a signal component In(x,y) that is substantially free of flicker components as the output image signal (RGB primary color signal or luminance signal after flicker reduction processing).

[0086] As described above, flickerless imaging is performed based on the period and peak timing of the flicker component detected by the flicker detection and correction unit 25, while flicker correction is also performed by the flicker detection and correction unit 25. This will be described below.

[0087] The flickerless imaging described with reference to FIG. 5 is effective when the curtain speed R is sufficiently shorter than the flicker period. 7 shows an exposure operation 100 in which, for example, a 100 Hz flickering light source is used and the curtain speed R is sufficiently shorter than the flicker period T. In this case, if the exposure gravity center timing WT is aligned with the peak timing PT as shown in the figure, flickerless operation is achieved, as shown schematically at the bottom of the figure.

[0088] Note that because the degree of brightness unevenness changes depending on the shutter speed and flicker light source waveform, the above "sufficiently short" should be determined appropriately for each imaging device 1, but for example, with a fluorescent lamp or the like, where the light source period is 10 ms / 8.3 ms and the curtain speed is about 2 to 4 ms, it can be said to be sufficiently short. In that case, if the curtain speed R is about 20 to 40% of the period T, the effect of flickerless imaging can be fully obtained.

[0089] On the other hand, the exposure operation 102 shows a case where the condition that the curtain speed R is sufficiently shorter than the flicker period is not satisfied. In this case, even if flicker-free imaging is used to align the exposure center timing WT with the peak timing PT, as shown in the figure, the effects of flicker cannot be completely avoided. In other words, the effects of flicker remain above and below the image frame, as shown schematically at the bottom of the figure.

[0090] In recent years, the use of imaging devices with electronic shutters that have long shutter speeds and imaging environments under high-frequency LED lighting have become more common, making it increasingly difficult to suppress the effects of flicker using flickerless imaging alone in terms of the relationship between shutter speed R and period T. Therefore, in the imaging device 1 of the embodiment, even if the condition that the curtain speed R is not sufficiently short relative to the flicker cycle T is not met, the influence of flicker that appears at the top and bottom edges of the screen during flickerless imaging is reduced by performing gain correction. This is the purpose of performing flicker correction.

[0091] The imaging device 1 performs flickerless imaging and flicker correction in the following procedure. 1) The flicker detection and correction unit 25, with the configuration shown in FIG. 6, accumulates and detects a plurality of image data sent from the image sensor 12 at regular time intervals, and determines the peak timing PT and period T of the flicker. 2) The camera control unit 21 (flickerless control unit 24) controls the timing of the start of exposure of the image sensor 12 so that the center of gravity of exposure is aligned with the timing of a multiple of the period from the detected peak timing PT. 3) The image sensor 12 performs imaging (exposure / readout) based on timing control. 4) The flicker detection and correction unit 25 applies a gain to the captured image data for each line to correct flicker.

[0092] A specific method for correcting flicker will be described. First, the phase and amplitude of the flicker for each line required for flicker correction are calculated as follows.

[0093] [Phase per line] 8 shows the relationship between flicker and still image parameters during flickerless imaging. In this case, the number of horizontal lines L, curtain speed R, and exposure period e are shown for exposure operation 102, but the curtain speed R (readout time) is relatively long and does not meet the condition that it is sufficiently short compared to the flicker period T.

[0094] The phase required for flicker correction is not the phase of the actual flickering light source, but the phase on the image plane that takes into account the exposure period e. As mentioned above, when performing flickerless imaging, the timing is adjusted so that the flicker peak timing PT is at the center of the image plane. In this diagram, the exposure center timing WT coincides with the peak timing PT. In this case, the phase rad of the n-th line can be calculated, for example, by the following formula. Phase of the nth line [rad] = π + R × (2π / T) × ((n-1) / L-1 / 2)

[0095] [Amplitude per line] The amplitude for each line at the phase for each line calculated above is calculated from the waveform and maximum amplitude of the flicker calculated during detection. In other words, the amplitude for a certain line is calculated from the amplitude at peak timing PT and the phase of that line. Specifically, the flicker waveform integrated according to the shutter speed at the time of image capture is modeled, and the amplitude value corresponding to n lines can be referenced to find the amplitude. Alternatively, the flicker waveform may be compiled into a table, and the amplitude of the corresponding phase may be referenced to find the amplitude.

[0096] When the amplitude for each line is calculated in this way, the difference between the amplitude at the peak timing PT and the amplitude of that line becomes the flicker correction value to be used for correction of that line. In the flicker generation block 55 in FIG. 6, such a flicker correction value is calculated for each line. Then, the calculation block 65 performs flicker correction using the flicker correction value for each line of the input image signal In'(x, y). That is, by applying a correction gain corresponding to the reciprocal of the flicker correction value for each line, the amplitude difference due to flicker in the lines on the image is eliminated and the amplitude due to the light source is made uniform within the frame of the image.

[0097] This reduces the effects of flicker that cannot be completely eliminated even with flickerless imaging, and therefore, for example, flicker reduction effects can be achieved even when imaging under an LED light source with a short cycle. Furthermore, since flickerless imaging aligns the exposure center with the peak timing of flicker, flicker correction involves gain correction for the upper and lower edges of the image where the effects of flicker occur, which has the advantage of preventing image quality degradation in the center of the image. Furthermore, by performing flickerless imaging, the phase relationship with the peak timing PT is defined, which simplifies the calculation process of the flicker correction value for each line and does not increase the processing load.Furthermore, the flicker correction value can be calculated by using the peak timing PT and its amplitude information detected for flickerless imaging, which is also effective in not increasing the processing load significantly.

[0098] Incidentally, some imaging devices 1 employ a mechanical shutter with a fixed shutter curtain speed, while others allow you to switch between a mechanical shutter and an electronic shutter as part of the imaging operation settings, and still others perform imaging operations with different shutter curtain speeds using an electronic shutter. For example, the shutter curtain speed varies depending on the image size. When using the same image sensor 12, if the image size is small, the crop area on the image sensor 12 becomes smaller, and the shutter curtain speed becomes faster. For example, the curtain speed differs when reading from the image sensor 12 in full size (36.0 mm×24.0 mm) and when reading in APS-C size (22.4 mm×15.0 mm). Generally, electronic shutters have a longer shutter curtain speed than mechanical shutters, although in recent years there are electronic shutters that can achieve shutter curtain speeds almost equal to those of mechanical shutters. For these reasons, the curtain speed of the imaging device 1 is not necessarily constant during imaging. Furthermore, there are various flickering light sources in the imaging environment, and the flickering period T also changes depending on the environment.

[0099] Considering these factors, it is expected that there will be cases where flickerless imaging alone is sufficient and cases where it is better to perform flicker correction in addition to flickerless imaging. Therefore, it is conceivable that the camera control unit 21 (flickerless control unit 24) selects whether or not to perform flicker correction depending on the image size, shutter method, and the like.

[0100] An example of processing is shown in Figure 9. The camera control unit 21 performs the process of FIG. 9, for example, periodically or at a timing corresponding to a change in the imaging mode or shutter setting.

[0101] In step S10, the camera control unit 21 determines the curtain speed R in the current imaging operation. This can be determined from the settings that are currently in effect for capturing a still image. For example, the camera control unit 21 determines the curtain speed R during image capture from the current image size (full size / ASP-C) setting, shutter method (electronic shutter / mechanical shutter) setting, etc.

[0102] In step S11, the camera control unit 21 determines the flicker cycle T. This can be determined by acquiring information from the flicker detection and correction unit 25. For example, the flicker cycle T is determined during live view imaging before a release operation.

[0103] In step S12, the camera control unit 21 (curtain speed R) / (period T)<(1 / x) Determine whether x>1. In other words, the condition is that the curtain speed R is shorter than 1 / x of the period T. In this case, x may be set to an appropriate value as a threshold value for determining whether the curtain speed R is sufficiently shorter than the period T.

[0104] If (curtain speed R) / (cycle T)<(1 / x), then the curtain speed R is sufficiently shorter than the cycle T. Therefore, the camera control unit 21 proceeds to step S14 and sets flicker correction to OFF. In other words, flickerless imaging is performed, but flicker correction is not performed. On the other hand, if (curtain speed R) / (period T)<(1 / x) is not true, it means that the curtain speed R is not sufficiently shorter than the period T. Therefore, the camera control unit 21 proceeds to step S13 and sets flicker correction to ON. In other words, a state is established in which flickerless imaging is performed while flicker correction is also performed.

[0105] Through the above process, if the curtain speed is shorter than 1 / x of the detected flicker cycle, flicker correction is turned off, and if the curtain speed is 1 / x or more of the detected flicker cycle, flicker correction is performed. This allows flicker correction to be performed appropriately depending on the situation.

[0106] Even in the case of an imaging device 1 in which the curtain speed R does not change, it is useful to perform such control by assuming that the flicker cycle T varies depending on the imaging environment. For example, the period T may be determined periodically in step S11, and steps S12, S13, and S14 may be performed based on the relationship between the determined period T and a fixed curtain speed R.

[0107] 3. Flickerless control according to margin In continuous shooting mode, which captures multiple still images while displaying a live view, it is difficult to update the live view image at regular intervals during flickerless imaging because the exposure center timing WT must be aligned with the flicker peak timing PT. Depending on the timing of the image capture, the same live view image may be displayed multiple times in succession, causing a change in the visual frame rate, or the amount of delay until the exposed image is displayed in response to the release operation may change (latency change), resulting in a phenomenon in which the live view display becomes unstable.

[0108] If the live view image becomes jerky during continuous shooting, it may become difficult for the user to frame the subject. To avoid this problem, in flickerless imaging, in which the exposure timing of a still image is matched to the peak of the flicker, it is conceivable to allow a slight deviation between the flicker peak timing PT and the exposure center timing WT, so long as the effects of the flicker are not visible on the screen.

[0109] In this case, even if the curtain speed R is sufficiently shorter than the flicker period T and stripes do not appear at the top or bottom of the screen, by using flicker correction in combination, the above-mentioned range of allowable timing deviation can be widened, and the occurrence of an event in which the live view image is not smooth can be further reduced.

[0110] Below, a process will be described in which, as flickerless control according to a margin, a deviation between the peak timing PT and the exposure gravity center timing WT is allowed as long as it is within a set margin.

[0111] First, an example in which the smoothness of live view image display is reduced due to flickerless imaging will be described with reference to FIG. Figure 10 shows the flicker amplitude of a 121 Hz light source and the exposure behavior when capturing images at 20 fps under that light source. This is an example of behavior during continuous shooting, where capture exposures CPE are performed multiple times and two inter-frame exposures IE1 and IE2 are performed between one capture exposure CPE and the next. The exposure behavior for one frame is shown by a parallelogram determined by the curtain speed, exposure time, and number of horizontal lines, as in Figure 5.

[0112] The parallelogram shapes in the figures are merely schematic and do not strictly represent the curtain speed, exposure time, or number of lines. However, Figures 10 to 13 will be explained using an example in which the curtain speed R is sufficiently shorter than the flicker period T. As mentioned above, this is a case in which flicker correction is not necessarily required to reduce flicker, but the purpose is to explain that even in such a case, flicker correction can be performed to achieve the effect of smoothing the live view image.

[0113] The synchronization signal SensV in FIG. 10 is a synchronization signal for the imaging operation of the imaging element 12, and the timing is variably set under the control of the camera control unit 21. The synchronization signal SysV is a synchronization signal for the display operation of the display unit 15, and has vertical timing at a predetermined frame rate (20 fps in this case). The live view image is displayed for each frame at a timing according to the synchronization signal SysV.

[0114] Assume that the user performs a continuous shooting operation, and the first capture exposure CPE is performed after time t10. In this case, the timing of the first capture exposure CPE is controlled so that the exposure gravity center timing WT is synchronized with the flicker peak timing PT. Thereafter, inter-frame exposures IE1 and IE2 are performed at timings according to the frame rate of 20 fps. In the live view image, in the frame after the capture exposure CPE is completed, an LV image CPP based on the capture exposure CPE is displayed, in the next frame an LV image IP1 based on the inter-frame exposure IE1 is displayed, and in the frame after that an LV image IP2 based on the inter-frame exposure IE2 is displayed.

[0115] After time t12, it is time for the next capture exposure CPE to be performed, but if the cycle is maintained as is, the timing of the exposure center of gravity will not coincide with the peak timing PT, so timing control TS is performed to delay the start timing of the capture exposure CPE, i.e., the synchronization signal SensV is delayed. As a result, capture exposure CPE from time t12 onwards also becomes flickerless imaging in which the exposure gravity centre timing WT coincides with the peak timing PT. After that, inter-frame exposures IE1 and IE2 are performed at timings according to the frame rate of 20 fps.

[0116] However, in this case, the capture exposure CPE is delayed by the timing control TS, so that the live view image display shows the same LV image IP2 for two consecutive frame periods, because the capture exposure CPE has not yet finished at the start of the second frame after time t12.

[0117] At time t13 and time t14, by performing timing control TS to delay the synchronization signal SensV, flickerless imaging is achieved in which the exposure gravity center timing WT of the capture exposure CPE coincides with the peak timing PT. As for the live view images from time t14 onwards, the same LV image IP2 is displayed for two consecutive frame periods due to a delay in the capture exposure CPE.

[0118] In the example of FIG. 10, the plurality of still image data obtained as continuous shots by the capture exposure CPE can be images that are less affected by flicker due to flickerless imaging. However, after the above-mentioned time t12 and after time t14, the same images continue as live view images, which means that the frame rate at which the user views the image (hereinafter referred to as the "user-viewed frame rate") fluctuates. In addition, the time difference between the capture exposure CPE and the resulting LV image CPP, the time difference between the inter-frame exposure IE1 and the LV image IP1, and the time difference between the inter-frame exposure IE2 and the LV image IP2 each constitute the latency of the live view image, but in the example shown in the figure, the latency fluctuates (the slope of the arrow indicating the timing from the end of exposure to the start of display is not constant).

[0119] Such fluctuations in the user-perceived frame rate and latency can cause the user to perceive the movement of the live view image as not being smooth. In particular, when performing continuous shooting while tracking a subject in a live view image, if the movement of the moving image in the live view image is jerky, it becomes difficult to adjust the angle of view to target the subject.

[0120] Therefore, in this embodiment, during flickerless control, flickerless imaging and flicker correction are performed to reduce the effects of flicker that appears in the image to be recorded, while minimizing the loss of smoothness in the live view image.

[0121] Similar to FIG. 10, FIG. 11 shows the flicker of the 121 Hz light source, the capture exposure CPE and interframe exposures IE1 and IE2, the synchronization signal SensV, the synchronization signal SysV, and the LV images CPP, IP1, and IP2 that form each frame of the live view image. An enlarged view showing the deviation between the flicker peak timing PT and the exposure gravity center timing WT of the capture exposure CPE is shown at the top of the figure.

[0122] Assume that the user performs a continuous shooting operation, and the first capture exposure CPE is performed after time t1. As in the case of Fig. 10, the timing of the first capture exposure CPE is controlled so that the exposure gravity center timing WT is synchronized with the flicker peak timing PT. Thereafter, inter-frame exposures IE1 and IE2 are performed at timings according to the frame rate of 20 fps. In the live view image, in the frame after the capture exposure CPE is completed, an LV image CPP based on the capture exposure CPE is displayed, in the next frame an LV image IP1 based on the inter-frame exposure IE1 is displayed, and in the frame after that an LV image IP2 based on the inter-frame exposure IE2 is displayed.

[0123] The timing for the next capture exposure CPE is assumed to be a timing that maintains the exposure cycle. In other words, it is a timing when the length of the period from the start of the exposure period of the previous inter-frame exposure IE2 to the start of the exposure period of the current capture exposure CPE is the same as the length of the period from the start of the exposure period of the inter-frame exposure IE1 before last to the start of the exposure period of the previous inter-frame exposure IE2. When the capture exposure CPE that maintains such an exposure cycle is considered, the exposure center timing WT does not coincide with the peak timing PT. However, in this case, as shown in the enlarged view, the deviation of the exposure gravity center timing WT from the peak timing PT is within the margin M. The margin M is set as a range within which it can be evaluated that the effect of flicker is not visible in the image even if the exposure gravity center timing WT is shifted from the peak timing PT.

[0124] In this way, when the amount of deviation is within the margin M, timing control is not performed to delay the start timing of the capture exposure CPE so that the exposure gravity center timing WT coincides with the peak timing PT. In other words, the capture exposure CPE is executed as is while maintaining the frame rate period of 20 fps based on the synchronization signal SensV. Thereafter, similarly, inter-frame exposures IE1 and IE2 are performed at timings according to the frame rate of 20 fps.

[0125] As shown in the enlarged view, at time t3 and time t4, the deviation of the exposure center timing WT from the peak timing PT is also within the margin M. Therefore, timing control is not performed to delay the start timing of the capture exposure CPE so that the exposure center timing WT coincides with the peak timing PT. In other words, the capture exposure CPE and inter-frame exposures IE1 and IE2 are performed while maintaining the frame rate cycle of 20 fps.

[0126] For the capture exposure CPEt from time t5, it is assumed that the deviation of the exposure gravity center timing WTd from the peak timing PT falls outside the range of the margin M. In this case, timing control TS is performed to delay the start timing so that the exposure gravity center timing WT coincides with the peak timing PT, and then capture exposure CPE is performed.

[0127] In this way, whether or not to perform timing control TS for flickerless imaging is switched depending on the relationship between the amount of deviation of the exposure gravity center timing WT from the peak timing PT and the margin M. This reduces the frequency with which the user-perceived frame rate and latency of live view images change. In the example shown in the figure, the LV images CPP, IP1, and IP2 are displayed sequentially at a constant frame rate and latency until time t5, achieving smooth video display. After time t5, the timing control TS is performed, which causes fluctuations in the user-perceived frame rate and latency, but the frequency with which this occurs can be reduced compared to the operation in Figure 7, making it less likely that the user will notice a loss in the smoothness of the live view image as a moving image.

[0128] Furthermore, when the timing control TS for flickerless imaging is not performed, the image data is also an image in which the influence of flicker is hardly noticeable, so the quality of the recorded image is maintained. Furthermore, by performing flicker correction, the effects of slight flicker caused by a slight misalignment between the peak timing PT and the exposure center timing WT can be reduced, so the effects of flicker can be almost completely eliminated, enabling high-quality image recording.

[0129] An example of processing by the camera control unit 21 (flickerless control unit 24) for realizing flickerless imaging operation with such a margin M set will be described with reference to Fig. 12. In the description of Fig. 9, the flickerless control unit 24 will be abbreviated as "control unit 24".

[0130] FIG. 12 shows an example of processing by the control unit 24 during a period in which a continuous shooting operation is being performed, for example, during a period in which the user is holding down the release button in continuous shooting mode. When the continuous shooting operation is started, the control unit 24 proceeds from step S101 to step S102, and executes timing control for flickerless imaging for the first capture exposure CPE. For example, the exposure timing is controlled so that the exposure center timing WT coincides with the peak timing PT.

[0131] In step S103, the control unit 24 causes the digital signal processing unit 20 to perform image processing for recording a still image based on the capture exposure CPE and displaying a live view.

[0132] While the continuous shooting operation is continuing, the control unit 24 proceeds from step S104 to step S105. If it is the timing for capture exposure CPE based on synchronization signal SensV, control unit 24 proceeds from step S105 to step S108; otherwise, control unit 24 proceeds from step S105 to step S106. Therefore, after performing processing according to capture exposure CPE in step S103, control unit 24 controls the timing of inter-frame exposure IE1 in step S106. In this case, inter-frame exposure IE1 is performed at a period based on synchronization signal SensV, and in step S107, digital signal processor 20 is caused to perform image processing for live view display according to inter-frame exposure IE1. Thereafter, the control unit 24 proceeds to steps S104, S105, S106, and S107 to execute the inter-frame exposure IE2 and image processing for live view display corresponding thereto.

[0133] When the timing for the next capture exposure CPE arrives, the control unit 24 proceeds from step S105 to step S108. In this case, the control unit 24 calculates the amount of deviation between the exposure gravity center timing WT and the peak timing PT when the capture exposure CPE is performed at a timing that maintains the cycle based on the synchronization signal SensV. Then, in step S109, the control unit 24 compares the amount of deviation with the margin M.

[0134] If the deviation amount is greater than the margin M, the control unit 24 executes timing control for flickerless imaging in step S110 so that the exposure gravity center timing WT coincides with the peak timing PT. On the other hand, if the amount of deviation is within the margin M, the control unit 24 proceeds to step S111, and executes the capture exposure CPE while maintaining the current exposure cycle.

[0135] The above-described processes from step S103 to step S111 are repeated until it is determined in step S104 that the continuous shooting operation has ended. As a result, flickerless imaging is performed assuming a margin M as shown in FIG.

[0136] In such flickerless imaging, the margin M can be widened by performing flicker correction, as will be described below. The margin M may be set according to the amplitude and period of the flicker, and may be set within a range in which the deviation between a predetermined timing of the exposure period, such as the exposure center timing WT, and the peak timing PT can be evaluated as not causing any noticeable flicker effects on the image. This means that flicker correction can widen the range in which it can be evaluated that the effects of flicker occurring in an image are not visible, thereby widening the margin M.

[0137] FIG. 13 shows the relationship between curtain speed and margin. If flicker correction is not performed, and exposure is performed during a period in which the amplitude value of flicker is equal to or greater than a threshold value th1, for example, the effect of flicker will be hardly noticeable in the image.

[0138] The exposure operations 74, 75, and 76 at the curtain speed R are shown schematically by diagonal lines, and their exposure gravity centers are shown as exposure gravity center timings WT74, WT75, and WT76. It should be noted that the exposure operations 74, 75, and 76 do not mean that exposure is performed three times, but rather exemplify patterns in which the exposure period differs with respect to the flicker cycle.

[0139] 13 shows that, in the case of curtain speed R, if exposure is performed within the range from exposure operation 74 to exposure operation 76, the flicker amplitude value during that exposure period will be equal to or greater than threshold value th1. In other words, whether the exposure center-of-gravity timing WT is exposure center-of-gravity timing WT74, WT75, or WT76, the image will be hardly affected by flicker. Therefore, in this case, the margin M can be set as the margin M1 shown in the figure before and after the peak timing PT.

[0140] On the other hand, when flicker correction is performed, if exposure is performed during a period in which the amplitude value of the flicker is equal to or greater than the threshold value th2, for example, it is assumed that the influence of the flicker is hardly noticeable in the image. Therefore, for the same curtain speed R, if exposure is performed within the range from exposure operation 84 to exposure operation 86, the flicker amplitude value during that exposure period will be equal to or greater than threshold value th2. In other words, whether the exposure center-of-gravity timing WT is exposure center-of-gravity timing WT84, WT85, or WT86, the image will be almost completely unaffected by flicker. Therefore, in this case, the margin M can be set as the margin M2 shown in the figure before and after the peak timing PT.

[0141] Increasing the margin M in this way reduces the frequency with which timing control for flickerless imaging is performed. This in turn reduces the frequency with which the same image appears repeatedly in live view images, as explained in Figure 10. Therefore, performing flicker correction makes it possible to provide smoother live view images.

[0142] <4. Summary and Modifications> According to the imaging device 1 and imaging control device (camera control unit 21) of the above embodiment, the following effects can be obtained.

[0143] The imaging device 1 of the embodiment includes a camera control unit 21 (flickerless control unit 24) that controls the exposure timing of flickerless imaging by synchronizing a specific timing within the exposure period (for example, exposure center timing WT) with the peak timing PT based on the period T of the detected flicker component and the peak timing PT of the flicker component.The imaging device 1 also includes a flicker detection and correction unit 25 that performs flicker correction to reduce flicker within a frame of image data based on the detection result of the phase relationship between each line of image data obtained by flickerless imaging and the flicker light source. As a result, the relationship between the curtain speed R and the period T makes it possible to reduce or eliminate flicker components that cannot be completely reduced even with flickerless imaging, thereby improving the quality of the captured image data. Furthermore, by using flickerless imaging and flicker correction together, for example, the central part of the image is captured in the bright part of the flicker peak, which has the advantage of reducing the influence of image quality degradation due to flicker correction gain.

[0144] In the embodiment, it has been described that flicker correction involves calculating a flicker correction value for each line based on the phase and amplitude of the flicker for each line, and correcting image data for each line. This makes it possible to obtain a correction value according to the relationship between the exposure timing of each line and the flicker amplitude in the flicker cycle, and to perform appropriate correction for each line.

[0145] In this embodiment, the flicker detection and correction unit 25 is configured to determine the phase and amplitude of flicker for each line from the flicker detection results for flickerless imaging. That is, the information on the phase and amplitude of flicker for each line obtained from the flicker detection results performed for flickerless imaging is used for flicker correction. This allows flicker correction to be performed using the processing results for flickerless imaging, making the processing more efficient.

[0146] The flicker detection and correction unit 25 in the embodiment performs flicker correction at least when the curtain speed R during imaging operation is longer than a predetermined value based on the flicker cycle T. In other words, this is the case when the condition that the curtain speed R is sufficiently shorter than the flicker cycle T is not met. This allows flicker correction to be performed to reduce flicker when the effect of flickerless imaging is not sufficient, thereby maintaining or improving image quality. In recent years, there has been a growing market demand for flicker reduction, particularly when using an electronic shutter with a long curtain speed, or in shooting environments with high-frequency LED light sources, etc. In such environments where the curtain speed during shooting tends to be longer than the flicker cycle, the effects of flicker can be reduced without restrictions on shutter speed by simultaneously performing flickerless imaging and flicker correction. However, flicker correction may be performed at all times regardless of the relationship between the flicker cycle and the curtain speed.

[0147] In the embodiment, the flickerless control unit 24 controls whether or not the flicker detection and correction unit 25 performs flicker correction based on the relationship between the flicker period T and the screen speed R (see FIG. 9). The flickerless control unit 24 sets the flicker detection and correction unit 25 to execute or not execute flicker correction at regular intervals or when some trigger occurs, for example, by the process in Fig. 9. In this way, flicker correction can be executed when necessary depending on the imaging environment and imaging settings.

[0148] In the above embodiment, the flickerless control unit 24 uses information on the flicker cycle obtained as a result of flicker detection for flickerless imaging to determine whether or not to perform flicker correction. This allows the execution of flicker correction to be determined using the processing results for flickerless imaging, making the processing more efficient.

[0149] In the embodiment, the flickerless control unit 24 uses information on the curtain speed R based on the image size or shutter method selected as the imaging operation at the time of the determination process to determine whether or not to perform flicker correction. This makes it possible to identify the curtain speed at the time of the determination process in Fig. 9 and determine whether flicker correction should be performed. Therefore, it is possible to determine whether flicker correction should be performed in response to fluctuations in curtain speed.

[0150] In this embodiment, the specific timing to be matched with the peak timing PT in flickerless imaging is the exposure center timing WT, that is, the timing approximately in the center of the exposure period of the approximately center line in the vertical direction of the imaging element. The most appropriate flickerless effect can be achieved by aligning the timing of the approximate center of the frame with the peak timing of the flicker. Note that the exposure center does not need to be exact. Therefore, the specific timing does not need to be exactly the exposure center. Then, by aligning the exposure center with the peak of the flicker and performing gain processing as flicker correction on the upper and lower edges of the image, it is possible to prevent image quality degradation in the center of the image.

[0151] In the embodiment, an example has been given in which the flickerless control unit 24 does not perform exposure timing control for flickerless imaging when the deviation amount between a specific timing (e.g., exposure center timing WT) and the peak timing PT is within a set margin (see Figures 11 and 12). This reduces the chances of performing control that delays the start of the exposure period. As a result, it is possible to reduce the frequency of changes in the frame rate at which the user perceives image data based on exposure, and changes in the latency until image output. In addition, not performing timing control TS may reduce release lag in some cases. By performing flicker correction, the margin M can be set wide as explained in Fig. 13. This reduces the frequency of occurrence of a state where the image becomes unsmooth, and promotes improvement in display quality. This effect can be achieved even if the curtain speed is sufficiently short compared to the flicker cycle. Therefore, if exposure timing control for flickerless imaging is not performed in accordance with this margin M, it may be possible to always perform flicker correction.

[0152] In the embodiment, the operations in FIGS. 11 and 12 are performed when continuous shooting is performed while displaying a live view image. This reduces the frequency of changes in the frame rate at which the user perceives the live view image and changes in the latency until the live view image is output during continuous shooting. Therefore, even while performing flickerless imaging, it is possible to reduce the occurrence of live view images that are not smooth or have poor responsiveness. This makes it easier for the user to aim at the desired subject when continuing continuous shooting while checking the subject in the live view image.

[0153] Note that this flicker-free control according to margin M is effective not only during continuous shooting, but also during live view display before still image capture begins, during live view display in video mode, etc. In other words, even when exposure is being performed for a live view display image, timing control is performed to match the exposure timing to the peak of flicker, but timing control to match the peak is not performed if the amount of deviation is within margin M. This reduces the frequency of changes in the frame rate at which the user perceives image data based on exposure and changes in latency until image output, thereby achieving smooth live view image display.

[0154] The program according to the embodiment is a program that causes a processor such as a CPU to execute the above-described flicker correction. That is, the program of the embodiment is a program that causes a processing unit of the imaging device 1, which controls the exposure timing of flickerless imaging by synchronizing a specific timing within the exposure period (for example, the exposure center timing WT) with the peak timing PT based on the period T of the detected flicker component and the peak timing PT of the flicker component, to perform flicker correction to reduce intra-frame flicker in the image data based on the detection result of the phase relationship between each line of the image data obtained by flickerless imaging and the flicker light source. By using such a program, the digital signal processing unit 20 having the above-mentioned flicker detection and correction unit 25 can be realized by an arithmetic processing device such as a DSP or a microcomputer.

[0155] These programs can be pre-recorded on a hard disk drive (HDD) as a recording medium built into a computer or other device, or on a ROM within a microcomputer having a CPU. Alternatively, the programs can be temporarily or permanently stored (recorded) on removable recording media such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disc, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording media can be provided as so-called packaged software. Such a program can be installed onto a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0156] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0157] The present technology can also be configured as follows. (1) a control unit that performs exposure timing control for flickerless imaging by synchronizing a specific timing within an exposure period with the peak timing of the detected flicker component based on the period of the detected flicker component and the peak timing of the flicker component; a flicker correction unit that performs flicker correction to reduce flicker within a frame of image data based on a detection result of a phase relationship between each line of image data obtained by the flickerless imaging and a flicker light source. Imaging device. (2) The flicker correction unit performs the flicker correction by: A flicker correction value is calculated for each of the lines based on the phase of the flicker for each of the lines and the amplitude of the flicker for each of the lines, and a process of correcting the image data for each of the lines is performed. The imaging device according to (1) above. (3) The flicker correction unit The phase of the flicker for each line and the amplitude of the flicker for each line are obtained from the flicker detection result for the flickerless imaging. The imaging device according to (2) above. (4) The flicker correction unit The flicker correction is performed at least when the curtain speed during the imaging operation is longer than a predetermined value based on the flicker cycle. The imaging device according to any one of (1) to (3) above. (5) The control unit controls whether or not the flicker correction unit performs the flicker correction based on the relationship between the flicker cycle and the curtain speed. The imaging device according to any one of (1) to (4) above. (6) The control unit The information on the flicker cycle obtained as a result of the flicker detection for the flickerless imaging is used to determine whether or not the flicker correction is to be performed. The imaging device according to (5) above. (7) The control unit Information on the image size or shutter speed based on the shutter method selected as the imaging operation at the time of the determination process is used to determine whether or not to perform the flicker correction. The imaging device according to (5) or (6) above. (8) The imaging device according to any one of (1) to (7), wherein the specific timing is a timing of the center of gravity of exposure. (9) The control unit When the deviation between the specific timing and the peak timing is within a set margin, the exposure timing control for the flickerless imaging is not performed. The imaging device according to any one of (1) to (8) above. (10) The control unit When taking continuous shots or recording video while displaying a live view image, When the deviation between the specific timing and the peak timing is within a set margin, the exposure timing control for the flickerless imaging is not performed. The imaging device according to (9) above. (11) The imaging device Based on the period of the detected flicker component and the peak timing of the flicker component, exposure timing control for flickerless imaging is performed to synchronize a specific timing within an exposure period with the peak timing; Based on the result of detecting the phase relationship between each line of the image data obtained by the flickerless imaging and the flicker light source, flicker correction is performed to reduce flicker within a frame of the image data. Imaging method. (12) a processing unit of an imaging device that performs exposure timing control for flickerless imaging, which synchronizes a specific timing within an exposure period with the peak timing of the detected flicker component based on the period of the detected flicker component and the peak timing of the flicker component; A program for executing flicker correction to reduce flicker within a frame of image data based on the detection result of the phase relationship between each line of image data obtained by the flickerless imaging and a flicker light source. [Explanation of symbols]

[0158] 1. Imaging device 11 Imaging optical system 12 Image sensor 13 Analog signal processing section 14 Recording control section 15 Display 20 Digital signal processing section 21 Camera control unit 24 Flickerless control section 25, 25R, 25G, 25B Flicker detection and correction unit 31 Integral Block 32 Integral value holding block 33 Average Value Calculation Block 34 Differential calculation block 35 Normalization Blocks 50 DFT blocks 55 Flicker generation block 60 Frequency Estimation / Peak Detection Block 65 Operation Blocks

Claims

1. a control unit that performs exposure timing control for flickerless imaging by synchronizing a specific timing within an exposure period with the peak timing of the detected flicker component based on the period of the detected flicker component and the peak timing of the flicker component; a flicker correction unit that performs flicker correction to reduce flicker within a frame of image data based on a detection result of a phase relationship between each line of image data obtained by the flickerless imaging and a flicker light source, The control unit controls whether or not the flicker correction unit performs the flicker correction based on the relationship between the flicker cycle and the curtain speed. Imaging device.

2. The control unit The information on the flicker cycle obtained as a result of the flicker detection for the flickerless imaging is used to determine whether or not the flicker correction is to be performed. The imaging device according to claim 1 .

3. The control unit Information on the image size or shutter speed based on the shutter method selected as the imaging operation at the time of the determination process is used to determine whether or not to perform the flicker correction.

3. The imaging device according to claim 1.

4. The flicker correction unit performs the flicker correction by: A flicker correction value is calculated for each of the lines based on the phase of the flicker for each of the lines and the amplitude of the flicker for each of the lines, and a process of correcting the image data for each of the lines is performed. The imaging device according to claim 1 .

5. The flicker correction unit The phase of the flicker for each line and the amplitude of the flicker for each line are obtained from the flicker detection result for the flickerless imaging. The imaging device according to claim 4 .

6. The specific timing is the timing of the center of gravity of exposure. The imaging device according to claim 1 .

7. The control unit When the deviation between the specific timing and the peak timing is within a set margin, the exposure timing control for the flickerless imaging is not performed. The imaging device according to claim 1 .

8. The control unit When taking continuous shots or recording video while displaying a live view image, When the deviation between the specific timing and the peak timing is within a set margin, the exposure timing control for the flickerless imaging is not performed. The imaging device according to claim 7 .

9. The imaging device Based on the period of the detected flicker component and the peak timing of the flicker component, exposure timing control for flickerless imaging is performed to synchronize a specific timing within an exposure period with the peak timing; Based on the result of detecting the phase relationship between each line of the image data obtained by the flickerless imaging and the flicker light source, flicker correction is performed to reduce flicker within a frame of the image data. An imaging method, comprising: The imaging device controls whether or not to perform the flicker correction based on the relationship between the flicker cycle and the curtain speed. Imaging method.

10. a control unit of the imaging device that performs flicker correction to reduce intra-frame flicker of image data based on a detection result of a phase relationship between each line of image data obtained by flickerless imaging that synchronizes a specific timing within an exposure period with the peak timing of the detected flicker component and the flicker light source, Execute exposure timing control for the flickerless imaging; The control of whether or not to perform the flicker correction is executed based on the relationship between the flicker cycle and the curtain speed. program.

Images