Imaging device and control method thereof

Abstract

To suppress the influence of flicker at a low cost even under such a situation that flicker of non-visible light or visible light occurs.SOLUTION: A control unit 110 executes visible light flicker processing 303 after executing non-visible light flicker processing 302. In the non-visible light flicker processing 302, the control unit 110 causes detection of a non-visible light flicker cycle from flicker detection image data 300 and controls an imaging element 100 so as to prevent the occurrence of non-visible light flicker in a photographed image on the basis of the non-visible light flicker cycle. In the visible light flicker processing 303, the control unit 110 causes detection of a visible light flicker cycle from flicker detection image data 301 separated in a state where the imaging element 100 is controlled so as to prevent the occurrence of the non-visible light flicker in the photographed image, and then controls the imaging element 100 so as to prevent the occurrence of the non-visible light flicker and the visible light flicker in the photographed image on the basis of the non-visible light flicker cycle and the visible light flicker cycle.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an imaging apparatus and a control method thereof. [Background technology]

[0002] When an image capturing device using an image sensor captures an image of a subject illuminated by a fluorescent lamp or an LED (Light Emitting Diode), periodic bright and dark changes may appear in the captured image. This phenomenon is called flicker, and flicker occurs in all light sources that emit light, and is caused by the relationship between the light and dark period of the light source and the vertical synchronous frequency of the image capturing device.

[0003] In recent years, image processing using visible light images and non-visible light (infrared light) images has been considered for imaging devices such as digital cameras, security cameras, and cameras mounted on smartphones. Non-visible light images are in a wavelength range that cannot be detected by the human eye and are treated as monochrome images without color information, but they are less susceptible to the effects of haze and improve visibility in dark places. Therefore, non-visible light images can be used for image processing to reduce noise, improve contrast, remove haze, etc., for visible light images.

[0004] In an imaging element capable of acquiring a visible light image and an invisible light image in parallel, an infrared cut filter cannot be used because infrared light cannot be received if an infrared cut filter is used. Also, invisible light pixels may affect surrounding visible light pixels. Also, an imaging device equipped with an imaging element capable of acquiring a visible light image and an invisible light image in parallel may be affected by visible light flicker caused by a visible light source and invisible light flicker caused by an invisible light source.

[0005] For example, some cameras mounted on smartphones and the like measure distances using a TOF (Time Of Flight) sensor to separate people from the background and perform image processing such as background blurring. When taking pictures indoors, the imaging device is affected by visible light flicker and non-visible light flicker due to the flicker of infrared light generated by the TOF sensor of another person's smartphone and the flicker of the indoor lighting source.

[0006] Therefore, when detecting flicker from an image in such a situation, the visible light image may be affected by the invisible light flicker, which may reduce the accuracy of detecting visible light flicker. Patent Documents 1 and 2 describe techniques for detecting visible light flicker by removing the effects of invisible light flicker. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2020-129756 A [Patent Document 2] Special Publication No. 2022-524957 Summary of the Invention [Problem to be solved by the invention]

[0008] The technology disclosed in Patent Document 1 can remove invisible light flicker (noise light source) because the period of the invisible light flicker (noise light source) is known in advance. However, in an environment where the invisible light flicker period is unknown, the detection accuracy of visible light flicker may be reduced due to the influence of the invisible light flicker. Alternatively, the invisible light flicker may affect the invisible light image when it is used to process a visible light image.

[0009] The technology disclosed in Patent Document 2 detects flicker while eliminating the influence of infrared light by inserting an infrared cut filter when detecting flicker. However, the need for an infrared cut filter may increase costs.

[0010] An object of the present invention is to suppress the effects of flicker at low cost even under conditions in which flicker of invisible light or visible light occurs. [Means for solving the problem]

[0011] In order to achieve the above-mentioned object, the imaging device of the present invention has an imaging means for outputting an image including visible light and invisible light, a separation means for separating the image output by the imaging means into a visible light image and an invisible light image, a detection means for detecting flicker, and a control means for performing a second processing after performing a first processing, wherein the control means, in the first processing, causes the detection means to detect a flicker period of invisible light from the invisible light image, and controls the imaging means based on the detected flicker period of invisible light so that invisible light flicker does not occur in the image, and in the second processing, causes the detection means to detect a flicker period of visible light from the visible light image separated while controlling the imaging means so that invisible light flicker does not occur in the image, and controls the imaging means based on the flicker period of invisible light and the detected flicker period of visible light so that invisible light flicker and visible light flicker do not occur in the image. Effect of the Invention

[0012] According to the present invention, even under circumstances in which flicker of invisible light or visible light occurs, the influence of flicker can be suppressed at low cost. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram showing an example of an imaging device. [Diagram 2] 2 is a diagram showing an example of the arrangement of pixels, color filters, and infrared filters in an imaging element. FIG. [Diagram 3] 4 is a timing chart of a flicker suppression process. [Figure 4]5 is a timing chart showing an example of an operation when acquiring image data for flicker detection. [Diagram 5] 5 is a timing chart showing an example of an operation when acquiring image data for flicker detection. [Figure 6] 13 is a flowchart of invisible light flicker processing. [Figure 7] 13 is a flowchart of visible light flicker processing. [Figure 8] 13 is a flowchart of a flicker detection process. [Figure 9] FIG. 4 is a diagram showing a plurality of frames of image data for flicker detection; [Figure 10] FIG. 13 is a conceptual diagram showing how inter-frame difference absolute value image data is divided. [Figure 11] FIG. 11 is a diagram showing signal intensity versus time of inter-frame difference absolute value image data. [Figure 12] FIG. 4 is a diagram showing a plurality of frames of image data for flicker detection; [Figure 13] FIG. 2 shows signal intensity of image data versus time. [Figure 14] 4 is a timing chart of a flicker suppression process. [Figure 15] 13 is a flowchart of visible light flicker processing. [Figure 16] 13 is a flowchart of invisible light flicker processing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] (First embodiment) FIG. 1 is a block diagram showing an example of an imaging device according to a first embodiment of the present invention. A preferred embodiment of the present invention will be described below using a digital camera as an example of an imaging device. However, the present invention is also applicable to any electronic device having an imaging function. Such electronic devices include computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, robots, drones, car cameras, surveillance cameras, etc. These are merely examples, and the present invention can be implemented in other electronic devices. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined in any combination.

[0016] As shown in FIG. 1, the imaging device 1000 includes an imaging element 100, a non-visible light pixel separation unit 101, a non-visible light image correction unit 102, a visible light image correction unit 103, a control bus 104, a data bus 105, and a flicker detection unit 106. The imaging device 1000 also includes an image processing unit 107, a memory control unit 108, a memory 109, a control unit 110, a non-volatile memory control unit 111, a non-volatile memory 112, a display control unit 113, and a display unit 114. In addition, the imaging device 1000 includes an AD converter, a compression / expansion unit, and an external recording unit, which are not shown. The AD converter converts an analog signal from the sensor into a digital signal. The compression / expansion unit compresses image data, for example, in JPEG or MPEG format, or expands compressed data.

[0017] The external recording unit records, on an external medium, so-called RAW data (image data output from the visible light image correction unit 103) before being developed by the image processing unit 107. Furthermore, the external recording unit records, on an external medium, non-visible light RAW data (image data output from the non-visible light image correction unit 102) before being developed by the image processing unit 107, JPEG, MPEG, and the like.

[0018] The image sensor 100 is an optical element such as a CCD or CMOS sensor that converts a subject image received through a color filter or an infrared filter into an electrical signal to create image data. The image sensor 100 as an image capturing means outputs an image including visible light and non-visible light.

[0019] Fig. 2 is a diagram showing an example of the arrangement of pixels, color filters, and infrared filters in the image sensor 100. The image sensor 100 has a plurality of pixels arranged in a two-dimensional matrix. Units shown in Fig. 2, each consisting of a pixel in which a red color filter 200, a green color filter 201, a blue color filter 203, and an infrared filter 202 are arranged, are arranged in a two-dimensional matrix.

[0020] In the color filter, red (R), green (G), and blue (B) color regions (filter sections) are arranged in a predetermined pattern corresponding to each pixel. The color filter basically has a peak in light transmittance in the wavelength band of each color, and blocks the transmission of light in the wavelength bands of other colors. In the infrared filter, an infrared region (filter section) is arranged corresponding to each pixel. The infrared filter basically has a peak in light transmittance in the infrared wavelength band, and blocks the transmission of light in the other wavelength bands.

[0021] The non-visible light pixel separation unit 101 (FIG. 1) serving as a separation means separates the image output from the image sensor 100 into a visible light image and a non-visible light image. The non-visible light pixel separation unit 101 generates visible light image data (R', G', B') by performing calculations using the non-visible light (Ir) pixels corresponding to the infrared filters 202 according to equations (1), (2), and (3), and outputs the generated visible light image data to the visible light image correction unit 103. At this time, the non-visible light pixel separation unit 101 also replaces the non-visible light (Ir) pixels corresponding to the infrared filters 202 in equation (2) with visible light pixels, and outputs the resulting data to the visible light image correction unit 103. R' = R - Ir * αr Equation (1) G' = G-Ir * αg Equation (2) B' = B-Ir*αb Equation (3)

[0022] Here, αr, αg, and αb are coefficients indicating the ratio of how much of the Ir pixel component is subtracted from each pixel (R, G, B), and take values ​​between 0 and 1. These coefficients αr, αg, and αb are coefficients for removing infrared light that could not be blocked by the color filters. Alternatively, the coefficients αr, αg, and αb are coefficients for removing the Ir component when the Ir pixel affects each pixel (R, G, B) (the charge of the Ir pixel leaks into each pixel (R, G, B)) due to the structure of the image sensor 100.

[0023] In addition, the coefficients αr, αg, and αb can also be used as coefficients for performing flicker correction of invisible light (described later) on a line-by-line or multiple-line basis so that dark lines have the same brightness as bright lines based on the flicker period.

[0024] Furthermore, the non-visible light pixel separation unit 101 generates Ir image data (Ir') by performing calculation using equation (4) using the visible light pixels, and outputs only the Ir pixels corresponding to the infrared filter 202 to the non-visible light image correction unit 102. Alternatively, in order to equalize the number of pixels between the visible light image and the Ir image, the non-visible light pixel separation unit 101 replaces all of the visible light pixels corresponding to the filters 200, 201, and 203 in equation (4) with Ir pixels, and outputs the Ir pixels to the non-visible light image correction unit 102. Ir'=Ir-(R*βr+G*βg+B*βb)...Equation (4)

[0025] Here, the coefficients βr, βg, and βb are coefficients indicating the ratio of how much of the visible light component is subtracted from the Ir pixel, and take values ​​between 0 and 1. These coefficients βr, βg, and βb are coefficients for removing visible light that could not be cut by the infrared filter. Alternatively, the coefficients βr, βg, and βb are coefficients for removing visible light components when each pixel (R, G, B) affects the Ir pixel (the charge of each pixel (R, G, B) leaks into the Ir pixel) due to the structure of the image sensor 100.

[0026] The coefficients βr, βg, and βb can also be used as coefficients for performing flicker correction of visible light on a line-by-line or multiple-line basis so that a dark line has the same brightness as a bright line based on the flicker period.

[0027] In this embodiment, the coefficients βr, βg, and βb are described as 0. Note that the method of separating visible light and non-visible light images in the non-visible light pixel separating unit 101 is not limited to the exemplified one, and other methods or formulas may be used.

[0028] An image output by the imaging element 100 is separated into a visible light image and a non-visible light image by a non-visible light pixel separation unit 101 serving as a separation means. Non-visible light pixel separation unit 101 outputs non-visible light image data (non-visible light image data) and visible light image data (visible light image data).

[0029] The non-visible light image correction unit 102 performs processes such as pixel correction, black level correction, shading correction, and defect correction on the non-visible light image data output from the non-visible light pixel separation unit 101, and generates corrected non-visible light image data. The visible light image correction unit 103 performs processes such as pixel correction, black level correction, shading correction, and defect correction on the visible light image data output from the non-visible light pixel separation unit 101, and generates corrected visible light image data. Hereinafter, unless otherwise specified, non-visible light image data and visible light image data refer to corrected non-visible light image data and visible light image data, respectively.

[0030] In addition, a DMAC (Direct Memory Access Controller) (not shown) is connected to the data bus 105. By this DMAC, the non-visible light image data and the visible light image data are written to the memory 109 via the memory control unit .

[0031] The control bus 104 is a control bus for accessing the image sensor 100, the non-visible light pixel separation unit 101, the non-visible light image correction unit 102, the visible light image correction unit 103, the flicker detection unit 106, the image processing unit 107, and the display control unit 113 from the control unit 110. The control bus 104 also provides access to the memory control unit 108 and the non-volatile memory control unit 111.

[0032] The data bus 105 transfers data between the memory control unit 108 or the nonvolatile memory control unit 111 and the control unit 110, the non-visible light image correction unit 102, the visible light image correction unit 103, the flicker detection unit 106, the image processing unit 107, and the display control unit 113.

[0033] A DMAC (not shown) connected to data bus 105 reads out invisible light image data and visible light image data from memory 109 via memory control unit 108. Flicker detection unit 106, which serves as detection means, detects visible light flicker (hereinafter referred to as visible light flicker) from the visible light image data. In addition, flicker detection unit 106 detects invisible light flicker (hereinafter referred to as invisible light flicker) from the invisible light image data. A method of detecting flicker by flicker detection unit 106 will be described later with reference to FIG. 8.

[0034] The image processing unit 107 performs processes such as white balance adjustment, magnification chromatic aberration correction, gamma correction, luminance / color generation processing, geometric transformation, and noise reduction on the visible light image data stored in the memory 109. The image processing unit 107 also performs processes such as gamma correction, luminance / color generation processing, geometric transformation, and noise reduction on the non-visible light image data stored in the memory 109. The image processing unit 107 also combines the visible light image data and the non-visible light image data. This combination achieves noise reduction and contrast improvement.

[0035] In noise reduction, for example, a dual bilateral filter (BF) is used to remove noise from visible light image data with the aid of non-visible light image data. The dual BF uses all edge information from non-visible light image data and visible light image data, so that unclear edges in the visible light image data are interpolated, improving edge preservation and noise removal capabilities.

[0036] In contrast improvement, for example, histograms of pixel values ​​of visible light image data and non-visible light image data are obtained. If the histogram of visible light image data (horizontal axis is pixel value, vertical axis is frequency) is distributed in a narrower range than the histogram of non-visible light image data, it is determined that the contrast is low, and contrast improvement processing is performed. For example, when the ratio m in formula (5) is less than 1, this corresponds to the case where the histogram of visible light image data is distributed in a narrower range than the histogram of non-visible light image data. m = histogram range of visible light image data (horizontal axis) / histogram range of invisible light image (horizontal axis) Equation (5)

[0037] Image processing unit 107 extracts a luminance signal from the visible light image data, and further performs wavelet transform on the luminance signal to separate it into low-frequency components and high-frequency components. Image processing unit 107 obtains low-frequency components by aligning the histogram of the spatial gradient magnitude for the low-frequency components with the histogram of the gradient magnitude of the non-visible light image data. Image processing unit 107 calculates the luminance value of the low-frequency component from this low-frequency component and the value of the original low-frequency component (of the visible light image data) using equation (6). Luminance value of low frequency components = (Low frequency component with histogram aligned*m+original low frequency component*(1-m)) / 2···Equation (6)

[0038] Moreover, the image processing unit 107 calculates the luminance value of the high frequency component from equation (7). Luminance value of high frequency component = (High-frequency component of invisible light image data * m + original low-frequency component * (1-m)) / 2··· Equation (7)

[0039] Furthermore, image processing unit 107 generates an image with improved contrast by performing inverse wavelet transform on the luminance values ​​of low-frequency components and high-frequency components. A DMAC (not shown) connected to data bus 105 reads and writes non-visible light image data, visible light image data, and processed image data from and to memory 109 via memory control unit 108. Note that other methods may be used for noise reduction and contrast improvement performed by image processing unit 107 without departing from the spirit of the invention.

[0040] The memory control unit 108 reads and writes data from and to the memory 109 in response to instructions from the control unit 110, the non-visible light image correction unit 102, the visible light image correction unit 103, the flicker detection unit 106, the image processing unit 107, and the display control unit 113. The memory 109 is a storage device having a storage capacity sufficient to store a predetermined number of still images, a predetermined length of moving image, audio data, and the like, as well as constants and programs for the operation of the control unit 110, and is constituted by a DRAM or the like.

[0041] The non-volatile memory control unit 111 reads and writes data from and to the non-volatile memory 112 in response to instructions from the control unit 110. The non-volatile memory 112 is an electrically erasable and recordable memory, such as an EEPROM. Constants, programs, and the like for the operation of the control unit 110 are stored in the non-volatile memory 112.

[0042] The control unit 110 is composed of a microcomputer and the like that controls the operation of the imaging device 1000. The control unit 110 issues various instructions to each functional block that configures the imaging device 1000 and executes various control processes. The control unit 110 controls each processing unit connected via a control bus 104. The control unit 110 realizes each process by executing a program recorded in a non-volatile memory 112.

[0043] The display control unit 113 controls the display of image data and a UI (user interface) on the display unit 114. The display unit 114 is configured with a liquid crystal panel, an organic EL panel, or the like, and displays the image data, the UI, and the like.

[0044] 3 is a timing chart of the flicker suppression process. The flicker suppression process includes a process for acquiring flicker detection image data 300 and 301, a non-visible light flicker process 302, and a visible light flicker process 303.

[0045] The flicker suppression process shown in Fig. 3 is executed during LV (live view) or at the start of shooting. The flicker suppression process is executed, for example, when live view is started by turning on the power of the imaging device 1000. Alternatively, the flicker suppression process may be executed before the start of actual shooting based on a shooting instruction. Note that the flicker suppression process is executed at least once. By repeatedly executing the flicker suppression process multiple times, the flicker detection accuracy is improved.

[0046] 3, the horizontal axis represents time, and the vertical axis represents, from the top, a vertical synchronization (VD) signal for LV of the image sensor 100, image data, and flicker processing.

[0047] A process for acquiring flicker detection image data 300, a non-visible light flicker process 302 (described later in FIG. 6), a process for acquiring flicker detection image data 301, and a visible light flicker process 303 (described later in FIG. 7) are executed in this order.

[0048] After the invisible light flicker processing 302 (first processing), the visible light flicker processing 303 (second processing) is executed. Usually, the processing for acquiring the flicker detection image data 301 starts at the timing of the next VD signal after the invisible light flicker processing 302.

[0049] The outline of the flow is as follows. Flicker detection image data 300 is acquired, and invisible light flicker processing 302 is executed using the flicker detection image data 300 (described later in Fig. 6 and Fig. 8), thereby detecting the invisible light flicker cycle. Then, based on the detected invisible light flicker cycle, flicker detection image data 301 is acquired under conditions where invisible light flicker is suppressed. Then, visible light flicker processing 303 is executed using the flicker detection image data 301 (described later in Fig. 7 and Fig. 8), thereby detecting the visible light flicker cycle. Then, image sensor 100 is controlled based on the invisible light flicker cycle and the visible light flicker cycle, thereby obtaining an image in which both invisible light flicker and visible light flicker are suppressed.

[0050] 4 is a timing chart showing an example of an operation when acquiring flicker detection image data. The horizontal axis indicates time. In FIG. 4, flicker detection image data 300 is shown as an example of the acquired flicker detection image data.

[0051] In this embodiment, exposure is performed 12 times in 20 ms, and each exposure time Tsir (Tsir1 to Tsir12) is set to Tsir=1.67 ms (=20 ms / 12). All of the exposure times Tsir1 to Tsir12 have the same value.

[0052] 4 shows that the invisible light flickers at a high speed with a period Tir (2 ms in this embodiment). In order to detect the invisible light flicker, the control unit 110 operates the image sensor 100 at a high shutter speed and performs 12 exposures to acquire image data.

[0053] In this case, the control unit 110 may perform addition in the horizontal direction in the sensor and thin out the vertical direction to reduce the number of pixels to read out at high speed. The blinking period of a fluorescent lamp is twice that of a commercial AC power supply. Therefore, the exposure time is set to a common multiple of 1 / 100 and 1 / 120. Moreover, the exposure time is set to a value suitable for detecting in-plane flicker that occurs in the case of a blinking light source using PWM (Pulse Width Modulation) as seen in LED lighting and the like.

[0054] However, the above-mentioned exposure time setting value is an example, and other values ​​may be used. In addition, at the same time as reading out the flicker detection image data 300, an image for LV may be read out from the sensor using lines (thinned vertical lines) that were not used for flicker detection, and used for LV display.

[0055] Fig. 5 is a timing chart showing an example of an operation when acquiring flicker detection image data. The horizontal axis indicates time. In Fig. 5, flicker detection image data 301 is shown as an example of the acquired flicker detection image data. The upper part shows the visible light flicker period Tba (10 ms; 100 Hz in the first embodiment), and the lower part shows the shutter timing of a total of 10 frames of exposure time Tsba (Tsba1 to Tsba10) of the flicker detection image data 301.

[0056] In this embodiment, exposure is performed 10 times over a period of 20 ms. All of the exposure times Tsba (Tsba1 to Tsba10) have the same value. If the invisible light flicker period is known, the exposure time Tsba is set to a value equal to the known invisible light flicker period. If the invisible light flicker period is not known, a predetermined value is set to the exposure time Tsba. In this embodiment, the invisible light flicker period is known (invisible light flicker period Tir=2 ms) as a result of the invisible light flicker processing 302 being executed first. Therefore, one exposure time Tsba is set to (Tir=2 ms). That is, this value of 2 ms is a numerical value calculated by the invisible light flicker processing 302 (described later in FIG. 6).

[0057] In this way, if the invisible light flicker period is known, an image from which invisible light flicker has been removed can be obtained by setting the exposure time to this value, thereby improving the accuracy of detecting visible light flicker.

[0058] At the same time as reading out the flicker detection image data 301, an image for LV may be read out from the sensor using lines (thinned out vertical lines) that were not used for flicker detection, and used for LV display.

[0059] Fig. 6 is a flowchart of invisible light flicker processing. This processing is realized by the CPU of control unit 110 (control means) expanding a program stored in non-volatile memory 112 into a RAM of control unit 110 and executing it. This processing is started in response to the start of the processing shown in Fig. 3. When the processing shown in Fig. 3 is executed multiple times, the second and subsequent invisible light flicker processing (Fig. 6) is started after the visible light flicker processing (Fig. 7) is completed.

[0060] The control unit 110 first acquires flicker detection image data 300 (FIG. 3). Images read out from the image sensor 100 multiple times are separated by the non-visible light pixel separation unit 101 to obtain non-visible light image data, and this non-visible light image data is processed by the non-visible light image correction unit 102 to obtain the flicker detection image data 300.

[0061] In step S400, the control unit 110 detects the invisible light flicker cycle by executing a flicker detection process (FIG. 8) described later using the flicker detection image data 300, and the process proceeds to step S401.

[0062] In step S401, control unit 110 determines whether or not invisible light flicker is occurring based on the invisible light flicker cycle detected in the immediately preceding step S400. Here, if the detected invisible light flicker cycle is 0, it is determined that invisible light flicker is not occurring, and if the detected invisible light flicker cycle is other than 0, it is determined that invisible light flicker is occurring.

[0063] If it is determined that invisible light flicker is not occurring, control unit 110 ends the invisible light flicker process (FIG. 6). If it is determined that invisible light flicker is occurring, control unit 110 proceeds to step S402.

[0064] In step S402, control unit 110 controls image sensor 100 based on the invisible light flicker cycle detected in step S400 so that invisible light flicker does not occur in the captured image, and ends the invisible light flicker process (FIG. 6). For example, control unit 110 controls exposure with an electronic shutter or a mechanical shutter (not shown) of image sensor 100 so as to avoid the brightening and fading cycle of the invisible light source. Specifically, control unit 110 may set the shutter speed to the same value as the invisible light flicker cycle.

[0065] 7 is a flowchart of the visible light flicker processing. This processing is executed by the same entity as the invisible light flicker processing (FIG. 6). This processing is executed after the invisible light flicker processing (FIG. 6) is completed.

[0066] The control unit 110 first acquires flicker detection image data 301 (FIG. 3). At that time, if it is determined in FIG. 6 that non-visible light flicker is occurring, the flicker detection image data 301 is acquired with the processing of step S402 applied (with non-visible light flicker suppressed). With the processing of step S402 applied, an image read out multiple times from the image sensor 100 is separated by the non-visible light pixel separation unit 101 to obtain visible light image data. This visible light image data is processed by the visible light image correction unit 103 to obtain the flicker detection image data 301.

[0067] In step S500, the control unit 110 detects the visible light flicker cycle by executing a flicker detection process (FIG. 8) described later using the flicker detection image data 301, and the process proceeds to step S501.

[0068] In step S501, control unit 110 determines whether or not visible light flicker is occurring based on the visible light flicker cycle detected in the immediately preceding step S500. Here, if the detected visible light flicker cycle is 0, it is determined that visible light flicker is not occurring, and if the detected visible light flicker cycle is other than 0, it is determined that visible light flicker is occurring.

[0069] If it is determined that no visible light flicker is occurring, control unit 110 ends the visible light flicker processing (FIG. 7). If it is determined that visible light flicker is occurring, control unit 110 proceeds to step S502.

[0070] In step S502, the control unit 110 controls the image sensor 100 so that the captured image does not include invisible light flicker and visible light flicker, based on the invisible light flicker cycle detected in step S400 and the visible light flicker cycle detected in step S500. Thereafter, the control unit 110 ends the visible light flicker process (FIG. 7). For example, the control unit 110 controls exposure with an electronic shutter or a mechanical shutter (not shown) of the image sensor 100 based on a common multiple of the invisible light flicker cycle and the visible light flicker cycle, so as to avoid the brightening cycle of the invisible light source and the visible light source. Specifically, the control unit 110 may set the shutter speed to a value equal to the common multiple (for example, the least common multiple) of the invisible light flicker cycle and the visible light flicker cycle.

[0071] 8 is a flowchart of the flicker detection process executed in step S400 or step S500. This process is realized by the control unit 110, or by cooperation between the control unit 110 and the flicker detection unit .

[0072] In step S600, the flicker detection unit 106 generates inter-frame absolute difference image data by taking the absolute difference in pixel units using frames before and after the flicker detection image data (300 or 301). This makes it possible to remove the effect of the subject, and since a change appears in the signal of the absolute difference if a flicker exists, it is possible to determine whether or not there is primarily an in-screen flicker.

[0073] Fig. 9 is a diagram showing a plurality of frames of image data for flicker detection. In Figs. 9 to 11, image data for flicker detection 300 is exemplified as image data for flicker detection. In this embodiment, the image data for flicker detection 300 is the subject in step S600 of step S400. In step S600 of step S500, the image data for flicker detection 301 is the subject. In a second embodiment described later, the opposite is true.

[0074] 9, image data 800 to 805 are flicker detection image data 300, and are image data captured at exposure times Tsir1 to Tsir6 (FIG. 4). Image data 806 to 810 are inter-frame difference absolute value image data. For example, image data 806 is inter-frame difference absolute value image data between image data 800 and image data 801 acquired thereafter. Similarly, image data 807 is inter-frame difference absolute value image data between image data 801 and 802. Similarly, image data 808, 809, and 810 are inter-frame difference absolute value image data between image data 802 and 803, between image data 803 and 804, and between image data 804 and 805, respectively.

[0075] In step S601, the flicker detection unit 106 calculates a flicker cycle from the inter-frame difference absolute value image data generated in step S600. This calculation of the flicker cycle will be described with reference to Figs.

[0076] 10 is a conceptual diagram showing how inter-frame difference absolute value image data is divided. The flicker detection unit 106 divides the inter-frame difference absolute value image data into a plurality of divided image data 900-905. The number of divisions is not important. Furthermore, the flicker detection unit 106 calculates a luminance integral value for each of the divided image data 900-905.

[0077] When the image data for flicker detection is a visible light image (image data for flicker detection 301), the flicker detection unit 106 integrates each color, and calculates the luminance Y from the R, G, and B values ​​in the positional relationship shown in Figure 2 using equation (8), and sets this as the luminance integrated value. Y=0.299*R+0.587*G+0.114*B...Equation (8) Furthermore, when the flicker detection image data is a non-visible light image (flicker detection image data 300), the flicker detection unit 106 determines the Ir integral value obtained by integrating all pixels as the luminance integral value.

[0078] FIG. 11 is a diagram showing the signal intensity versus time of inter-frame difference absolute value image data. In FIG. 11, inter-frame difference absolute value image data 806 is shown as a representative example. The signal intensity may be the integral value of the divided image data 900 to 905, or may be the integral value multiplied by a coefficient. The time on the horizontal axis is calculated from the number of lines in the divided image data, with the exposure time divided by the number of lines being one line time. From FIG. 11, the signal change time Tfir is detected as the flicker cycle. The change time Tfir coincides with twice the flicker cycle Tir shown in FIG. 4.

[0079] 11, for convenience of explanation, a graph is used for explanation. However, in reality, the control unit 110 calculates the flicker period by finding the period from maximum value to maximum value or from minimum value to minimum value using the signal intensities of the divided image data 900 to 905 calculated by the flicker detection unit 106.

[0080] Fig. 12 is a diagram showing a plurality of frames of flicker detection image data, in which flicker detection image data 301 is shown as an example of the flicker detection image data.

[0081] 12, image data 1200 to 1204 are flicker detection image data 301, and image data 1205 to 1208 are inter-frame difference absolute value image data. For example, image data 1205 is inter-frame difference absolute value image data between image data 1200 and image data 1201 acquired thereafter. Similarly, image data 1206, 1207, and 1208 are inter-frame difference absolute value image data between image data 1201 and 1202, between image data 1202 and 1203, and between image data 1203 and 1204, respectively.

[0082] In step S602, the control unit 110 determines whether an in-screen flicker (horizontal stripe flicker within one frame) is occurring. Here, if the flicker period is found (calculated) in step S601, it is determined that an in-screen flicker is present. If an in-screen flicker is occurring, the control unit 110 proceeds to step S603, and if an in-screen flicker is not occurring, the control unit 110 proceeds to step S604.

[0083] In the example shown in Fig. 9, light, dark, light and dark, light and dark occur within one screen in each of the image data 806 to 810, so it is determined that in-screen flicker is occurring. On the other hand, in the example shown in Fig. 12, image data 1205, 1206, 1207, and 1208 do not have light, dark, light and dark, light and dark within one screen, so it is determined that in-screen flicker is not occurring.

[0084] In step S603, the control unit 110 stores the flicker cycle calculated in step S601 or step S605 in the memory 109, and ends the flicker detection process (FIG. 8).

[0085] In step S604, the flicker detection unit 106 calculates the signal intensity from the image data for flicker detection. When the image data for flicker detection is a visible light image, the flicker detection unit 106 integrates the entire image for each color, and calculates the luminance Y from the R, G, and B values ​​in the positional relationship shown in FIG. 2 using equation (8), for example, to obtain the luminance integral value. When the image data is non-visible light image data, the flicker detection unit 106 integrates all pixels to obtain the Ir integral value, and calculates this integral value as the signal intensity, or calculates the integral value multiplied by a coefficient as the signal intensity. After step S604, the control unit 110 proceeds to step S605.

[0086] In step S605, the control unit 110 calculates the flicker period from the signal intensity obtained in step S604, and the process proceeds to step S606. This flicker period calculation will be described with reference to FIGS.

[0087] Fig. 13 is a diagram showing the signal strength of image data versus time. In Fig. 13, the vertical axis represents signal strength and the horizontal axis represents time. In Fig. 13, the signal strengths of image data 1200 to 1204 shown in Fig. 12 are plotted at times Tsba1*1 to Tsba1*5, respectively. Note that the signal strengths of image data (not shown) exposed for exposure times Tsba6 to Tsba10 shown in Fig. 5 are plotted at times Tsba1*6 to Tsba1*10.

[0088] 13, the signal change time Tfba is detected as the flicker period. This change time Tfba coincides with the flicker period Tba shown in FIG.

[0089] For convenience of explanation, a graph is used in Fig. 13. However, in reality, the control unit 110 calculates the flicker period by finding the period from maximum value to maximum value or the period from minimum value to minimum value using the signal intensity calculated by the flicker detection unit 106.

[0090] In step S606, the control unit 110 determines whether flicker is occurring. Here, if the flicker cycle was found (calculated) in step S605, it is determined that flicker is occurring. If flicker is occurring, the control unit 110 proceeds to step S603, and if flicker is not occurring, the control unit 110 proceeds to step S607.

[0091] In step S607, the control unit 110 stores information indicating that no flicker is occurring in the memory 109, and ends the flicker detection process (FIG. 8). The information indicating that no flicker is occurring is, for example, information indicating that the flicker cycle is 0.

[0092] When moving to actual shooting after the flicker suppression process (FIG. 3) has been repeatedly executed multiple times, imaging is executed in a state in which the exposure control adopted in the final step S502 in FIG. 7 is applied.

[0093] According to this embodiment, the control unit 110 executes the invisible light flicker process 302 (first process) and then executes the visible light flicker process 303 (second process). In the invisible light flicker process 302, the control unit 110 causes the flicker detection unit 106 to detect the invisible light flicker cycle from the flicker detection image data 300 (invisible light image) (S400). The control unit 110 controls the image sensor 100 based on the detected invisible light flicker cycle so that invisible light flicker does not occur in the image output from the image sensor 100 (S402). Next, in the visible light flicker process 303, the control unit 110 acquires the separated flicker detection image data 301 (visible light image) in a state where the image sensor 100 is controlled so that invisible light flicker does not occur in the image output from the image sensor 100. In visible light flicker processing 303, control unit 110 causes flicker detection unit 106 to detect the visible light flicker cycle from flicker detection image data 301 (S500). Control unit 110 then controls image sensor 100 based on the invisible light flicker cycle and the visible light flicker cycle so that invisible light flicker and visible light flicker do not occur in the image output from image sensor 100 (S502).

[0094] This makes it possible to obtain an image in which the effects of invisible or visible light flicker are suppressed. In addition, since an infrared cut filter that is generally inserted when detecting flicker is not required, costs are reduced. Therefore, even in a situation in which invisible or visible light flicker occurs, the effects of flicker can be suppressed at low cost.

[0095] Furthermore, by repeatedly executing the invisible light flicker processing 302 and the visible light flicker processing 303 multiple times, the accuracy of detecting the flicker cycle can be improved, and the effects of flicker can be suppressed more accurately.

[0096] Furthermore, in step S502, control unit 110 controls image sensor 100 so that the exposure time becomes a common multiple of the visible light flicker cycle and the invisible light flicker cycle, so that the effects of flicker can be suppressed more accurately.

[0097] (Modification of the first embodiment) Next, a modification of the first embodiment will be described.

[0098] In the first embodiment, the shutter speed is set to the common multiple in step S502, which restricts the shutter speed. Therefore, in this modification, in step S402 in Fig. 6, instead of exposure control, "flicker correction" is applied when the non-visible light pixel separator 101 separates the image output by the image sensor 100. Flicker correction is a process for digitally removing flicker.

[0099] Specifically, in the flicker correction, the control unit 110 switches the coefficients αr, αg, and αb in the above formulas (1), (2), and (3) applied by the non-visible light pixel separation unit 101 on a line-by-line basis or on a multiple-line basis. Then, the control unit 110 removes the non-visible light flicker by setting the coefficients of the lines that become dark lines from the non-visible light flicker cycle so that the lines have the same brightness as the bright lines.

[0100] Furthermore, in obtaining flicker detection image data 301 used in visible light flicker processing (FIG. 7), flicker correction is applied at the stage where the output image is separated by the non-visible light pixel separation unit 101. Therefore, flicker detection image data 301 in which non-visible light flicker is reduced is obtained by flicker correction.

[0101] In this case, in step S502, the control unit 110 may perform exposure control based on the visible light flicker cycle so as to avoid the brightening and fading cycle of the visible light source. This is because the non-visible light flicker has been removed at the stage when the flicker detection image data 301 is acquired, so it is only necessary to focus on removing the visible light flicker. Therefore, for example, the shutter speed may be set to the same value as the visible light flicker cycle. Note that if the visible light flicker is an in-plane flicker, flicker correction may be applied instead of exposure control.

[0102] When moving to actual shooting after the flicker suppression process (FIG. 3) has been repeatedly executed multiple times, the exposure control (or flicker correction) adopted in the final step S502 in FIG. 7 is applied.

[0103] In this way, even in a situation where flicker of invisible light or visible light occurs, the modified example can also achieve the same effect as the first embodiment in terms of suppressing the influence of flicker at low cost.

[0104] Second Embodiment In the second embodiment of the present invention, the order of execution of the invisible light flicker process and the visible light flicker process in the flicker suppression process is reversed from that of the first embodiment. Therefore, in this embodiment, the flicker suppression process (FIG. 14) is applied instead of the flicker suppression process (FIG. 3) of the first embodiment. Also, the invisible light flicker process (FIG. 16) and the visible light flicker process (FIG. 15) are applied instead of the invisible light flicker process (FIG. 6) and the visible light flicker process (FIG. 7). The other configurations are the same as those of the first embodiment.

[0105] Fig. 14 is a timing chart of the flicker suppression processing. Flicker detection image data 1400 is an image for detecting visible light flicker, and flicker detection image data 1401 is an image for detecting invisible light flicker. Therefore, the flicker detection image data 1400, 1401 correspond to the flicker detection image data 301, 300 shown in Fig. 3. The invisible light flicker processing 1402 and the visible light flicker processing 1403 correspond to the invisible light flicker processing 302 and the visible light flicker processing 303 shown in Fig. 3, respectively. The flicker suppression processing is executed at least once.

[0106] In this embodiment, the control unit 110 executes the visible light flicker processing 1403 (second processing) and then executes the non-visible light flicker processing 1402 (first processing).

[0107] The outline of the flow is as follows. Flicker detection image data 1400 is acquired, and visible light flicker processing 1403 is executed using the flicker detection image data 1400 (described later in FIG. 15), thereby detecting the visible light flicker cycle. Then, based on the detected visible light flicker cycle, flicker detection image data 1401 is acquired under conditions where visible light flicker is suppressed. Then, non-visible light flicker processing 1402 is executed using the flicker detection image data 1401 (described later in FIG. 16), thereby detecting the non-visible light flicker cycle. Then, image sensor 100 is controlled based on the non-visible light flicker cycle and the visible light flicker cycle, thereby obtaining an image in which both non-visible light flicker and visible light flicker are suppressed.

[0108] Fig. 15 is a flowchart of the visible light flicker processing. The execution body and execution conditions of this processing are the same as those of the visible light flicker processing in Fig. 7. The processing in steps S500 and S501 is the same as that described in Fig. 7.

[0109] When acquiring the flicker detection image data 1400 used in step S500, exposure is performed 12 times in 20 ms, and each exposure time is set to 1.67 ms (=20 ms / 12). Note that addition in the horizontal direction within the sensor and thinning in the vertical direction, and using the thinned vertical lines for LV display may be applied in the same manner as in the first embodiment.

[0110] If it is determined in step S501 that no visible light flicker is occurring, control unit 110 ends the visible light flicker process (FIG. 15). If it is determined that visible light flicker is occurring, control unit 110 proceeds to step S1500.

[0111] In step S1500, control unit 110 controls image sensor 100 based on the visible light flicker cycle detected in step S500 so that visible light flicker does not occur in the captured image, and ends the visible light flicker processing (FIG. 15). For example, control unit 110 controls exposure with an electronic shutter or a mechanical shutter (not shown) of image sensor 100 so as to avoid the brightening and fading cycle of the visible light source. Specifically, control unit 110 may set the shutter speed to the same value as the visible light flicker cycle.

[0112] Fig. 16 is a flowchart of the invisible light flicker processing. The executing entity and the execution conditions of this processing are the same as those of the invisible light flicker processing of Fig. 6. The processing of steps S400 and S401 is the same as that described in Fig. 6.

[0113] If it is determined in step S401 that invisible light flicker is not occurring, control unit 110 ends the invisible light flicker process (FIG. 16). If it is determined that invisible light flicker is occurring, control unit 110 proceeds to step S1600.

[0114] In step S1600, the control unit 110 controls the image sensor 100 based on the invisible light flicker cycle detected in step S400 and the visible light flicker cycle detected in step S500. That is, the control unit 110 controls the image sensor 100 so that invisible light flicker and visible light flicker do not occur in the captured image. The invisible light flicker process (FIG. 16) ends. For example, the control unit 110 controls exposure with an electronic shutter or a mechanical shutter (not shown) of the image sensor 100 based on a common multiple of the invisible light flicker cycle and the visible light flicker cycle so as to avoid the brightening cycle of the invisible light source and the visible light source. Specifically, the control unit 110 may set the shutter speed to a value equal to a common multiple (for example, the least common multiple) of the invisible light flicker cycle and the visible light flicker cycle.

[0115] According to this embodiment, even under circumstances where flicker of invisible light or visible light occurs, it is possible to achieve the same effect as the first embodiment in terms of suppressing the influence of flicker at low cost.

[0116] (Modification of the second embodiment) Next, a modification of the second embodiment will be described.

[0117] In the second embodiment, the shutter speed is restricted by setting the shutter speed to the common multiple in step S1600. Therefore, in this modification, the above-mentioned flicker correction is applied instead of exposure control in step S1500 in FIG.

[0118] Furthermore, in obtaining flicker detection image data 1401 used in the non-visible light flicker processing (FIG. 16), flicker correction is applied at the stage where the output image is separated by the non-visible light pixel separation unit 101. Therefore, flicker detection image data 1401 in which visible light flicker is reduced is obtained by the flicker correction.

[0119] In this case, in step S1600, the control unit 110 may perform exposure control based on the invisible light flicker cycle so as to avoid the brightening and fading cycle of the invisible light source. This is because the visible light flicker has been removed at the stage when the flicker detection image data 1401 is acquired, so it is only necessary to focus on removing the invisible light flicker. Therefore, for example, the shutter speed may be set to the same value as the invisible light flicker cycle.

[0120] When moving to actual shooting after the flicker suppression process (FIG. 14) has been repeatedly executed multiple times, the exposure control (or flicker correction) adopted in the final step S1600 in FIG. 16 is applied.

[0121] In this way, even in a situation where flicker of invisible light or visible light occurs, the modified example can also achieve the same effect as the second embodiment in terms of suppressing the influence of flicker at low cost.

[0122] In the first embodiment, the non-visible light flicker processing 302 (FIG. 6) corresponds to the first processing in the present invention, and the visible light flicker processing 303 (FIG. 7) corresponds to the second processing in the present invention. However, the combination of obtaining the flicker detection image data 300 and the non-visible light flicker processing 302 (FIG. 6) may be the first processing in the present invention. Also, the combination of obtaining the flicker detection image data 301 and the visible light flicker processing 303 (FIG. 7) may be the second processing in the present invention.

[0123] Similarly, in the second embodiment, a combination of obtaining flicker detection image data 1401 and non-visible light flicker processing 1402 (FIG. 16) may be regarded as a first process of the present invention. Also, a combination of obtaining flicker detection image data 1400 and visible light flicker processing 1403 (FIG. 15) may be regarded as a second process of the present invention.

[0124] In the above embodiments, the invisible light flicker is an in-plane flicker, and the visible light flicker is a flicker, but the invisible light flicker may be a flicker, or the visible light flicker may be an in-plane flicker. Alternatively, the present invention can be applied to a case where the invisible light flicker and the visible light flicker each have both types of flicker.

[0125] It should be noted that the above-described method for detecting the presence or absence of flicker and the flicker period is merely an example, and other methods may be adopted.

[0126] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0127] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-mentioned embodiments to a system or device via a network or a non-transitory storage medium, and having one or more processors of a computer in the system or device read and execute the program. The above program and the storage medium that stores the above program constitute the present invention. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more of the functions.

[0128] The disclosure of this embodiment includes the following configurations (and methods). (Configuration 1) An imaging means for outputting an image including visible light and invisible light; a separation means for separating the image output by the imaging means into a visible light image and a non-visible light image; A detection means for detecting flicker; A control means for executing a second process after executing the first process, The control means, in the first processing, causes the detection means to detect a flicker period of invisible light from the invisible light image, and controls the imaging means based on the detected flicker period of invisible light so that invisible light flicker does not occur in the image, and, in the second processing, causes the detection means to detect a flicker period of visible light from the visible light image separated in a state in which the imaging means is controlled so that invisible light flicker does not occur in the image, and controls the imaging means based on the flicker period of invisible light and the detected flicker period of visible light so that invisible light flicker and visible light flicker do not occur in the image. (Configuration 2) An imaging means for outputting an image including visible light and invisible light; a separation means for separating the image output by the imaging means into a visible light image and a non-visible light image; A detection means for detecting flicker; A control means for executing the first process after executing the second process, The control means, in the second processing, causes the detection means to detect a flicker period of visible light from the visible light image, and controls the imaging means based on the detected flicker period of visible light so that visible light flicker does not occur in the image, and, in the first processing, causes the detection means to detect a flicker period of invisible light from the non-visible light image separated in a state in which the imaging means is controlled so that visible light flicker does not occur in the image, and controls the imaging means based on the flicker period of visible light and the detected flicker period of non-visible light so that non-visible light flicker and visible light flicker do not occur in the image. (Configuration 3) An imaging means for outputting an image including visible light and invisible light; a separation means for separating the image output by the imaging means into a visible light image and a non-visible light image; A detection means for detecting flicker; A control means for executing a second process after executing the first process, The control means, in the first processing, causes the detection means to detect a flicker period of invisible light from the invisible light image, and then, when separating the image output by the imaging means, applies flicker correction to reduce flicker of invisible light based on the detected flicker period of the invisible light, and, in the second processing, causes the detection means to detect a flicker period of visible light from the visible light image separated after applying the flicker correction, and controls the imaging means based on the detected flicker period of visible light so that visible light flicker does not occur in the image. (Configuration 4) An imaging means for outputting an image including visible light and invisible light; a separation means for separating the image output by the imaging means into a visible light image and a non-visible light image; A detection means for detecting flicker; A control means for executing the first process after executing the second process, The control means, in the second processing, causes the detection means to detect a flicker period of visible light from the visible light image, and then, when separating the image output by the imaging means, applies flicker correction to reduce visible light flicker based on the detected flicker period of the visible light, and, in the first processing, causes the detection means to detect a flicker period of invisible light from the non-visible light image separated after applying the flicker correction, and controls the imaging means based on the detected flicker period of the non-visible light so that non-visible light flicker does not occur in the image. (Configuration 5) The imaging device described in any one of configurations 1 to 4, characterized in that when separating the visible light image and the non-visible light image, the separation means corrects pixel values ​​of the visible light image using the non-visible light image. (Configuration 6) The imaging device described in any one of configurations 1 to 4, characterized in that when separating the visible light image and the non-visible light image, the separation means corrects pixel values ​​of the non-visible light image using the visible light image. (Configuration 7) The imaging device according to any one of configurations 1 to 4, wherein the control means repeatedly executes the first process and the second process a plurality of times. (Configuration 8) The imaging device described in Configuration 1, characterized in that, in the second processing, when controlling the imaging means so that non-visible light flicker and visible light flicker do not occur in the image, the control means controls the imaging means so that the exposure time is a common multiple of the flicker period of the visible light and the flicker period of the non-visible light. (Configuration 9) The imaging device described in Configuration 2, wherein, in the first processing, when controlling the imaging means so that invisible light flicker and visible light flicker do not occur in the image, the control means controls the imaging means so that the exposure time is a common multiple of the visible light flicker period and the invisible light flicker period. (Method 1) A method for controlling an imaging device having an imaging means for outputting an image including visible light and invisible light, separating the image output by the imaging means into a visible light image and an invisible light image, and detecting flicker, comprising the steps of: After the first process is executed, the second process is executed, a flicker period of invisible light detected from the invisible light image, and based on the detected flicker period of invisible light, the imaging means is controlled so that invisible light flicker does not occur in the image; and a flicker period of visible light detected from the visible light image separated while the imaging means is controlled so that invisible light flicker does not occur in the image, and based on the flicker period of invisible light and the detected flicker period of visible light, the imaging means is controlled so that invisible light flicker and visible light flicker do not occur in the image. (Method 2) A method for controlling an imaging device having an imaging means for outputting an image including visible light and invisible light, separating the image output by the imaging means into a visible light image and an invisible light image, and detecting flicker, comprising the steps of: execute the second process, then execute the first process, a flicker period of visible light detected from the visible light image, and the imaging means controlled based on the detected flicker period of visible light so that visible light flicker does not occur in the image; and a flicker period of invisible light detected from the non-visible light image separated in a state in which the imaging means is controlled so that visible light flicker does not occur in the image, and the imaging means controlled based on the flicker period of visible light and the detected flicker period of invisible light so that invisible light flicker and visible light flicker do not occur in the image. (Method 3) A method for controlling an imaging device having an imaging means for outputting an image including visible light and invisible light, separating the image output by the imaging means into a visible light image and an invisible light image, and detecting flicker, comprising the steps of: After the first process is executed, the second process is executed, a flicker period of invisible light is detected from the invisible light image, and then, when separating the image output by the imaging means, flicker correction is applied to reduce the flicker of invisible light based on the detected flicker period of invisible light; and a flicker period of visible light is detected from the visible light image separated by applying the flicker correction, and the imaging means is controlled based on the detected flicker period of visible light so that visible light flicker does not occur in the image. (Method 4) A method for controlling an imaging device having an imaging means for outputting an image including visible light and invisible light, separating the image output by the imaging means into a visible light image and an invisible light image, and detecting flicker, comprising the steps of: execute the second process, then execute the first process, a flicker period of visible light is detected from the visible light image, and then, when separating the image output by the imaging means, flicker correction is applied to reduce the flicker of visible light based on the detected flicker period of the visible light; and a flicker period of invisible light is detected from the non-visible light image separated by applying the flicker correction, and the imaging means is controlled based on the detected flicker period of the non-visible light so that flicker of invisible light does not occur in the image. [Explanation of symbols]

[0129] 100 Image sensor 101 Invisible light pixel separation section 106 Flicker detection unit 110 Control section 300, 301, 1400, 1401 Flicker detection image data 302, 1402 Non-visible light flicker processing 303, 1403 Visible light flicker processing

Claims

1. an imaging means for outputting an image including visible light and invisible light; a separation means for separating the image output by the imaging means into a visible light image and a non-visible light image; A detection means for detecting flicker; A control means for executing a second process after executing the first process, The control means, in the first processing, causes the detection means to detect a flicker period of invisible light from the invisible light image, and controls the imaging means based on the detected flicker period of invisible light so that invisible light flicker does not occur in the image, and, in the second processing, causes the detection means to detect a flicker period of visible light from the visible light image separated in a state in which the imaging means is controlled so that invisible light flicker does not occur in the image, and controls the imaging means based on the flicker period of invisible light and the detected flicker period of visible light so that invisible light flicker and visible light flicker do not occur in the image.

2. an imaging means for outputting an image including visible light and invisible light; a separation means for separating the image output by the imaging means into a visible light image and a non-visible light image; A detection means for detecting flicker; A control means for executing the first process after executing the second process, The control means, in the second processing, causes the detection means to detect a flicker period of visible light from the visible light image, and controls the imaging means based on the detected flicker period of visible light so that visible light flicker does not occur in the image, and, in the first processing, causes the detection means to detect a flicker period of invisible light from the non-visible light image separated in a state in which the imaging means is controlled so that visible light flicker does not occur in the image, and controls the imaging means based on the flicker period of visible light and the detected flicker period of non-visible light so that non-visible light flicker and visible light flicker do not occur in the image.

3. an imaging means for outputting an image including visible light and invisible light; a separation means for separating the image output by the imaging means into a visible light image and a non-visible light image; A detection means for detecting flicker; A control means for executing a second process after executing the first process, The control means, in the first processing, causes the detection means to detect a flicker period of invisible light from the invisible light image, and then, when separating the image output by the imaging means, applies flicker correction to reduce flicker of invisible light based on the detected flicker period of the invisible light, and, in the second processing, causes the detection means to detect a flicker period of visible light from the visible light image separated after applying the flicker correction, and controls the imaging means based on the detected flicker period of visible light so that visible light flicker does not occur in the image.

4. an imaging means for outputting an image including visible light and invisible light; a separation means for separating the image output by the imaging means into a visible light image and a non-visible light image; A detection means for detecting flicker; A control means for executing the first process after executing the second process, The control means, in the second processing, causes the detection means to detect a flicker period of visible light from the visible light image, and then, when separating the image output by the imaging means, applies flicker correction to reduce visible light flicker based on the detected flicker period of the visible light, and, in the first processing, causes the detection means to detect a flicker period of invisible light from the non-visible light image separated after applying the flicker correction, and controls the imaging means based on the detected flicker period of the non-visible light so that non-visible light flicker does not occur in the image.

5. 5. The imaging device according to claim 1, wherein the separation means corrects pixel values ​​of the visible light image by using the non-visible light image when separating the visible light image from the non-visible light image.

6. 5. The imaging device according to claim 1, wherein the separation means corrects pixel values ​​of the non-visible light image by using the visible light image when separating the visible light image from the non-visible light image.

7. 5. The imaging apparatus according to claim 1, wherein the control unit repeatedly executes the first process and the second process a plurality of times.

8. The imaging device according to claim 1, characterized in that, when controlling the imaging means in the second processing so that invisible light flicker and visible light flicker do not occur in the image, the control means controls the imaging means so that an exposure time is a common multiple of the visible light flicker period and the invisible light flicker period.

9. The imaging device according to claim 2, characterized in that, when controlling the imaging means in the first processing so that invisible light flicker and visible light flicker do not occur in the image, the control means controls the imaging means so that an exposure time is a common multiple of the visible light flicker period and the invisible light flicker period.

10. 1. A method for controlling an imaging device having an imaging means for outputting an image including visible light and invisible light, separating the image output by the imaging means into a visible light image and an invisible light image, and detecting flicker, comprising: execute the first process and then execute the second process; a control method for an imaging device, characterized in that in the first processing, a flicker period of invisible light is detected from the invisible light image, and the imaging means is controlled based on the detected flicker period of invisible light so that invisible light flicker does not occur in the image, and in the second processing, a flicker period of visible light is detected from the visible light image separated in a state in which the imaging means is controlled so that invisible light flicker does not occur in the image, and the imaging means is controlled based on the flicker period of invisible light and the detected flicker period of visible light so that invisible light flicker and visible light flicker do not occur in the image.

11. 1. A method for controlling an imaging device having an imaging means for outputting an image including visible light and invisible light, separating the image output by the imaging means into a visible light image and an invisible light image, and detecting flicker, comprising: execute the first process after executing the second process; a flicker period of visible light detected from the visible light image, and the imaging means controlled based on the detected flicker period of visible light so that visible light flicker does not occur in the image; and a flicker period of invisible light detected from the non-visible light image separated in a state in which the imaging means is controlled so that visible light flicker does not occur in the image, and the imaging means controlled based on the flicker period of visible light and the detected flicker period of invisible light so that invisible light flicker and visible light flicker do not occur in the image.

12. 1. A method for controlling an imaging device having an imaging means for outputting an image including visible light and invisible light, separating the image output by the imaging means into a visible light image and an invisible light image, and detecting flicker, comprising: execute the first process and then execute the second process; A method for controlling an imaging device, characterized in that in the first processing, a flicker period of invisible light is detected from the invisible light image, and then, when separating the image output by the imaging means, flicker correction is applied to reduce the flicker of the invisible light based on the detected flicker period of the invisible light, and in the second processing, a flicker period of visible light is detected from the visible light image separated after the flicker correction is applied, and the imaging means is controlled based on the detected flicker period of the visible light so that visible light flicker does not occur in the image.

13. 1. A method for controlling an imaging device having an imaging means for outputting an image including visible light and invisible light, separating the image output by the imaging means into a visible light image and an invisible light image, and detecting flicker, comprising: execute the first process after executing the second process; a flicker period of visible light is detected from the visible light image, and then, when separating the image output by the imaging means, flicker correction is applied to reduce the flicker of visible light based on the detected flicker period of the visible light; and a flicker period of invisible light is detected from the non-visible light image separated by applying the flicker correction, and the imaging means is controlled based on the detected flicker period of the non-visible light so that flicker of invisible light does not occur in the image.

Images