Imaging device and control method for the imaging device

Abstract

This technology suppresses count loss in photon-counting image sensors, improving image brightness and color reproduction. [Solution] The imaging device includes an image sensor having a light receiving unit that outputs a pulse signal in response to the incidence of photons and a counting unit that counts the pulse signal, a control means for controlling the imaging frame rate and the image storage time of the image sensor, and a detection means for detecting the lighting period of a blinking light source based on the change in brightness of the image obtained by imaging with the storage time changed by the control means.

Description

Technical Field

[0001] The present invention relates to an imaging device and a control method for the imaging device.

Background Art

[0002] There has been proposed a photon counting type imaging device using a photodiode array in which avalanche photodiodes (APDs) to which a reverse bias voltage greater than the breakdown voltage is applied are arranged in an array to detect the number of photons incident in a certain range. Further, when shooting a moving image under a blinking light source such as a fluorescent lamp, periodic light and dark called flicker may appear in the obtained image. In order to eliminate flicker, a technique of generally performing detection called flicker detection, detecting the on-off cycle of the blinking light source, and adjusting the accumulation period (exposure period) of the imaging device accordingly to suppress the luminance change of the image is widely known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a photon counting type imaging device, an avalanche multiplication phenomenon generated when photons (photons) are incident is converted into a pulse signal, and a pixel signal is generated by counting this. The number of photons (photons) that can be counted per unit time is determined by a so-called dead time that is rate-limited by the device structure of the APD. When a plurality of photons are incident in a period shorter than the dead time, the number of counted pulses becomes one, so in the case of a blinking light source, photons are concentratedly incident in a short time and counting omission increases, and there is a problem that the actual brightness and color are not reproduced in the image.

Means for Solving the Problems

[0005] The imaging device according to the present invention is characterized by comprising an image sensor having a light receiving unit that outputs a pulse signal in response to the incidence of photons and a counting unit that counts the pulse signal, a control means for controlling the imaging frame rate and the image storage time of the image sensor, and a detection means for detecting the illumination period of a blinking light source based on the change in brightness of an image obtained by imaging with the storage time changed by the control means. [Effects of the Invention]

[0006] According to the present invention, it becomes possible to detect the illumination period of a flashing light source, thereby suppressing count loss in a photon counting type image sensor and improving the brightness and color reproduction of the image. [Brief explanation of the drawing]

[0007] [Figure 1] This figure shows an example of the configuration of an imaging device. [Figure 2] This figure shows an example of the relationship between the flashing period of the light source and the imaging frame rate. [Figure 3] This is a diagram illustrating the operation of the image sensor. [Figure 4] This is a timing chart illustrating the method for detecting the illumination period of the light source in the first embodiment. [Figure 5] This figure illustrates a method for detecting the illumination period of a light source in the first embodiment. [Figure 6] This is a timing chart illustrating the method for detecting the illumination period of the light source in the second embodiment. [Figure 7] This figure illustrates a method for detecting the illumination period of a light source in the second embodiment. [Figure 8] This figure illustrates an example of a modification to the imaging drive in the third embodiment. [Figure 9] This figure illustrates an example of a modification to the imaging drive in the third embodiment. [Figure 10]This figure illustrates an example of correcting the brightness of an image in response to a change in the drive related to imaging in the fourth embodiment. [Figure 11] This diagram illustrates a user interface for detecting the illumination period of a flashing light source and for changing the drive related to image capture. [Modes for carrying out the invention]

[0008] Embodiments of the present invention will be described below with reference to the drawings. Note that the following embodiments do not limit the invention as defined in the claims. While multiple features are described in the embodiments, not all of these features are necessarily essential to the invention, and the features may be combined in any way. Furthermore, in the accompanying drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0009] <First Embodiment> The first embodiment will be described. In this embodiment, an example of detecting the illumination period of a blinking light source (for example, a pulse light source such as a PWM (pulse width modulation) controlled LED) will be described. Figure 1 is a block diagram showing an example configuration of the imaging device 100 in this embodiment. For example, the imaging device 100 may include imaging devices such as so-called digital still cameras and digital camcorders. However, it is not limited to these imaging devices, and each embodiment described below, including this embodiment, is applicable to devices having a photon counting type image sensor.

[0010] The imaging device 100 includes a lens 101, an image sensor 102, an image control unit 103, an accumulation control unit 104, a system control unit 105, a frame rate control unit 106, a display unit 107, a recording unit 108, an operation unit 109, a gain control unit 110, and an offset control unit 111. The imaging device 100 generates an image by capturing incident light transmitted through the lens 101 with the image sensor 102. The image sensor 102 is a photon counting type image sensor and has an image unit which has a light receiving unit that outputs a pulse signal in response to the incidence of photons and a counting unit which counts the pulse signal. Furthermore, the imaging device 100 generates a developed image by adjusting the color, brightness, etc. of the generated image using the image control unit 103, and then displays the developed image using the display unit 107 or records the developed image on a recording medium using the recording unit 108.

[0011] The system control unit 105 controls the entire imaging device 100. The system control unit 105 is an example of control and detection means. The frame rate control unit 106 controls the frame rates for imaging, display, and recording, respectively. The frame rate control unit 106 is an example of control means. In this embodiment, the frame rates for imaging, display, and recording can be set individually. Based on instructions from the frame rate control unit 106, the system control unit 105 controls the drive of the image sensor 102 and the image control unit 103 related to the imaging frame rate. The system control unit 105 also controls the drive of the display unit 107 related to the display frame rate and the drive of the recording unit 108 related to the recording frame rate, based on instructions from the frame rate control unit 106.

[0012] The accumulation control unit 104 controls the accumulation start timing and the accumulation end timing (readout timing) of the imaging device 102 based on the instruction regarding the accumulation time received from the system control unit 105. The accumulation control unit 104 is an example of control means. The gain control unit 110 can correct an image by performing gain processing on the image displayed and recorded by the display unit 107 and the recording unit 108. The offset control unit 111 can correct an image by performing offset processing on the image displayed and recorded by the display unit 107 and the recording unit 108.

[0013] The display unit 107 displays the image obtained by imaging, the operation screen, etc. The recording unit 108 records the image obtained by imaging, etc. on a recording medium. The operation unit 109 has operation members such as switches and push buttons, and sends instructions corresponding to the operations on the operation members, for example, the state of the power switch of the imaging device 100, the instruction for displaying the image before shooting, various shooting instructions, etc. to the system control unit 105. Further, the operation unit 109 sends menu operations, etc. for preliminarily instructing the display of the captured image, the operation of the imaging device, for example, the shutter speed for changing the accumulation time of the image, etc. to the system control unit 105. When the operation unit 109 receives an operation instruction for executing flicker detection, for example, it notifies the system control unit 105. The system control unit 105 that has received the notification controls the entire imaging device to perform flicker detection based on the instruction from the operation unit 109.

[0014] Note that the operation unit 109 may have a display member such as an LCD or a photodiode, and display the state of the imaging device, etc. by a control signal from the system control unit 105 using the display member of the operation unit 109 or the display unit 107. Further, a touch panel may be attached to the display unit 107, and operations on the screen may be performed using the touch panel.

[0015] The operation of the imaging device 100 for detecting the lighting period of the blinking light source will be described. To detect the illumination period of the flashing light source, the system control unit 105 of the imaging device 100 first controls each part of the imaging device 100 to detect the flashing period (blinking period) of the light source using flicker detection. The detection of the flashing period of the light source using flicker detection can be performed by applying any known flicker detection method. For example, the flashing period of the light source can be detected using a flicker detection method such as that described in International Publication No. 2016 / 132615. This allows the flashing period of the light source to be detected.

[0016] The system control unit 105 controls the imaging frame rate to match the blinking period of the detected light source, as shown in Figure 2, based on instructions from the frame rate control unit 106. Figure 2 is a timing chart showing an example of the relationship between the blinking period of the light source and the imaging frame rate. In Figure 2, 201 represents the blinking period of the light source, 202 represents the imaging frame rate, and 203 schematically represents the image accumulation time output according to the imaging frame rate.

[0017] The blinking period 201 of the light source indicates the on and off of a PWM light source such as an LED controlled by PWM (pulse width modulation). In FIG. 2, for easier explanation, the high-level period is represented as the lighting period and the low-level period is represented as the non-lighting period in a rectangular wave. Since the blinking period 201 of the light source can be obtained by flicker detection, the system control unit 105 performs control to match the imaging frame rate 202 with the obtained blinking period, as shown in an example in FIG. 2(a). When the blinking period of the light source is fast and the imaging frame rate 202 cannot be matched for each cycle of the blinking period, the system control unit 105 performs control to match the imaging frame rate 202 with a multiple of the obtained blinking period, as shown in an example in FIG. 2(b). FIG. 2(b) shows an example where the imaging frame rate 202 is matched with a period twice the obtained blinking period. As shown by the accumulation time 203 of the images read in FIGS. 2(a) and 2(b), one image is obtained in one section of the imaging frame rate 202. FIGS. 2(a) and 2(b) show the case of the rolling shutter method as an example, and the time shift between the accumulation time and the reading in image acquisition is schematically represented by a parallelogram.

[0018] [[ID=३]] FIG. 3(a) shows a timing diagram related to imaging when driving in the rolling shutter method. The upper left point 303 of the parallelogram corresponding to one image 302 obtained according to the imaging frame rate 301 is the accumulation start timing of the first row of the image, and the upper right point 304 is the accumulation end timing of the first row of the image. The period between point 303 and point 304 is the accumulation time 305 of the first row of the image, and reading is performed after the accumulation time 305 has elapsed. Similarly, since accumulation and reading are sequentially performed from the second row to the last row of the image, the time from the accumulation end timing of the first row of the image to the lower right point of the parallelogram, which is the accumulation end timing of the last row of the image, is the reading time 306 of one image. There are image sensors that read one row at a time, image sensors that read multiple rows at a time, and image sensors that can select to read one row or multiple rows at a time.

[0019] Figure 3(b) shows a timing diagram for imaging when driven by a global shutter system. The top-left point 308 of the rectangle corresponding to one image 307 acquired according to the imaging frame rate 301 represents the start timing of accumulation of the first row of the image, and the top-right point 309 represents the end timing of accumulation of the first row of the image. The period between points 308 and 309 is the accumulation time 310 of the first row of the image, and reading is performed after the accumulation time 305 has elapsed. In the global shutter system, all rows of the image are read out at once, so there is no need to stagger the reading for each row, and therefore no reading time occurs as in the rolling shutter system.

[0020] Next, we will describe the operation after detecting the flashing period of the light source and controlling the imaging frame rate to match the flashing period of the light source, as described above. The imaging device 100 performs imaging by sequentially changing the storage time and detects the lighting period of the flashing light source based on the change in brightness of the image obtained from the imaging. In the first embodiment, the imaging device 100 detects the lighting period of the flashing light source by controlling the storage start / end timing to be staggered, that is, by staggering the storage time. The method by which the imaging device 100 detects the lighting period of the flashing light source by controlling the storage start / end timing to be staggered will be described below with reference to Figure 4.

[0021] Figure 4 is a timing chart illustrating the method for detecting the illumination period of the light source in the first embodiment. In Figures 4(a) and 4(b), 401 indicates the blinking period of the light source, 402 indicates the imaging frame rate, and 403 indicates the image accumulation time acquired in one interval of the imaging frame rate. The accumulation time 403-i is the image accumulation time acquired in the i-th interval of the imaging frame rate. Note that i is a subscript, and i = 1, 2, ..., n (n is arbitrary; in the example shown in Figure 4(a), n = 11).

[0022] Figure 4(b) shows the first and second intervals of the imaging frame rate. The accumulation timing 404 is shown with equally spaced markings for illustrative purposes, but the number and spacing of these markings, and whether they are equally spaced, depend on the specifications of the image sensor. 405-1 is the start timing of accumulation for the first row of the image acquired in the first interval of the imaging frame rate, and 406-1 is the end timing of accumulation for the first row. Similarly, 405-2 is the start timing of accumulation for the first row of the image acquired in the second interval of the imaging frame rate, and 406-2 is the end timing of accumulation for the first row. Comparing the start timing of accumulation for the first row of the image acquired in the first interval (405-1) and the start timing of accumulation for the first row of the image acquired in the second interval (405-2), there is a shift of one mark on the accumulation timing 404. Similarly, the end timing of accumulation for the first row of the image acquired in the first interval (406-1) and the end timing of accumulation for the first row of the image acquired in the second interval (406-2) are also shifted by one mark on the accumulation timing 404.

[0023] The system control unit 105 controls the accumulation time for each image to be constant, and shifts the accumulation time for each interval of the imaging frame rate, as shown in Figure 4(a) for accumulation times 403-1 to 403-11. In other words, the system control unit 105 keeps the accumulation time for each image constant and shifts the start timing and end timing of accumulation, for example, from the start to the end of the imaging frame rate interval. Here, a shorter accumulation time for an image is preferable from the viewpoint of detection accuracy of the illumination period of the light source. For example, it is preferable to set the accumulation time for an image to the smallest possible accumulation time. Note that the example shown in Figure 3 is just one example of the unit for shifting the accumulation time, and various patterns can be taken. Also, the position for shifting the accumulation time does not have to be from the start to the end of the imaging frame rate interval. For example, if the accumulation time is shifted from the start of the imaging frame rate interval and the brightness of the acquired image falls below a predetermined threshold, the process may be stopped thereafter.

[0024] In this way, the brightness value of the image acquired by driving the image sensor 102 changes depending on whether or not the light source is on. The brightness of the image here may be the sum of the count values ​​obtained by photon counting, or a mechanism for acquiring an evaluation value of the overall brightness of the image provided in the imaging device may be used. When the accumulation time 403-4 and 403-5 in the 4th and 5th intervals of the imaging frame rate shown in Figure 4(a) spans the period when the light source is on, the rows that are not exposed become dark, so the overall brightness of the image becomes darker.

[0025] Figure 5 shows an example of the change in image brightness when detecting the on-time of a light source, along with the flashing period of the light source. In Figure 5, 501 shows the flashing period of the light source according to the accumulation timing 404, and 502 shows the change in image brightness when the image accumulation time is shifted, as shown in Figure 4(a), according to the accumulation timing 404. As shown in Figure 5, the image brightness 502 is generally constant during the on-time (shown at a high level) and off-time (shown at a low level) of the flashing period 501 of the light source, and the image brightness changes during the period when the light source switches from on to off. Note that the flashing period 501 of the light source is schematically shown as a square wave, but in reality it is often not a clear square wave as shown. Also, the image brightness 502 is actually a waveform that contains errors due to noise, etc., but the average value is thought to be as shown. Furthermore, the image brightness 502 is thought to be shown as a dashed line because the brightness from sunlight or steady light other than the flashing light source is added. Figure 5 shows the case where the light source is switched on twice for ease of explanation, but it can also be once or three or more times.

[0026] Based on the relationship between the blinking period of the light source and the change in image brightness, it is possible to detect that the blinking light source is on during periods 511-513 in which the image brightness 502 is below a certain amount of change (approximately constant), and that the periods 511 and 513 in which the image brightness is high are the periods when the blinking light source is on. It is possible that the image brightness 502 may not be as shown due to noise, etc., but it is presumed that if the value is acquired multiple times and the average value is obtained, a trend like the image brightness 502 shown in Figure 5 can be obtained. Furthermore, in cases where the readout time is slow due to the rolling shutter drive and the entire image does not fit into one interval of the shooting frame rate, the period when the light source is on can be detected by comparing the brightness of the first row of the image for each image, rather than comparing each image.

[0027] According to this embodiment, it is possible to detect the illumination period of a flashing light source. In a photon counting image sensor, if the maximum count value for counting photons is set to a constant value regardless of the illumination period, the ratio between the actual count value corresponding to the number of incident photons and the maximum count value (pixel saturation value) changes when the light source is a flashing light source. This is because, when the light source is a flashing light source, photons are incident only during the illumination period, and no photons are incident even if the photon counting operation is performed outside of the illumination period. On the other hand, if the illumination period of the flashing light source is known, the maximum count value of photons can be determined, and the pixel saturation value can be set accordingly, thereby improving the reproduction of brightness and color.

[0028] <Second Embodiment> A second embodiment will now be described. In the first embodiment described above, as shown in Figure 4(a), the detection of the blinking light source's illumination period is performed by controlling the system to shorten the accumulation time and gradually shift it backward from the start of the imaging frame rate. In the second embodiment, the accumulation time for the first section of the imaging frame rate is set to the same length as the first section of the imaging frame rate, a so-called full accumulation state, and from the second section onward, the accumulation time is controlled to gradually shorten to detect the illumination period of the blinking light source. In the second embodiment described below, the descriptions of the same configuration and operation as in the first embodiment described above will be omitted, and the differences from the first embodiment will be described.

[0029] The operation in the second embodiment, after detecting the blinking period of the light source and controlling the imaging frame rate to match the blinking period of the light source, will be described below. In the second embodiment as well, after controlling the imaging frame rate to match the blinking period of the light source, the imaging device 100 sequentially changes the storage time and performs imaging, and detects the lighting period of the blinking light source based on the change in brightness of the image obtained from the imaging.

[0030] Figure 6 is a timing chart illustrating the method for detecting the illumination period of the light source in the second embodiment. In Figures 6(a) and 6(b), 601 indicates the blinking period of the light source, 602 indicates the imaging frame rate, and 603 indicates the image accumulation time acquired in one interval of the imaging frame rate. The accumulation time 603-i is the image accumulation time acquired in the i-th interval of the imaging frame rate. Note that i is a subscript, and i = 1, 2, ..., n (n is arbitrary; in the example shown in Figure 6(a), n = 11).

[0031] In the example shown in Figure 6(a), the imaging device 100 starts with the image storage time for each frame rate interval being the same length as the frame rate interval, a so-called full storage. For the second and subsequent frames rates, the imaging device 100 controls the storage end timing to gradually shift forward while keeping the storage start timing fixed to the start timing of the imaging frame rate 602, thereby detecting the illumination period of the blinking light source. That is, the image storage time 603-1 for the first frame rate interval is controlled to be the same length as the frame rate interval. Furthermore, for the image storage times 603-2 to 603-11 for the second frame rate interval, the storage end timing is gradually shifted forward, and the image storage time is controlled to decrease with each frame rate interval.

[0032] Refer to Figure 6(b) for explanation. 605-1 is the start timing for accumulating the first row of the image acquired in the first interval of the imaging frame rate, and 606-1 is the end timing for accumulating the first row. Also, 605-2 and 605-3 are the start timings for accumulating the first row of the image acquired in the second and third intervals of the imaging frame rate, respectively, and 606-2 and 606-3 are the end timings for accumulating the first row. Furthermore, 605-m and 605-(m+1) are the start timings for accumulating the first row of the image acquired in the m interval and (m+1) interval of the imaging frame rate, respectively, and 606-m and 606-(m+1) are the end timings for accumulating the first row. The accumulation start timings 605-1 to 605-3, ..., 605-m, 605-(m+1), ... are fixed for the imaging frame rate 602. On the other hand, the accumulation end timings 606-1 to 606-3, ..., 606-m, 606-(m+1), ... shift the accumulation time from the full accumulation state in the first interval of the imaging frame rate to the start of the imaging frame rate 602 by one increment of the accumulation timing 604 scale. By controlling the accumulation time of the acquired images in this way, the brightness of the acquired images changes according to the illumination period of the light source included in the accumulation times 603-1 to 603-11, as shown in the accumulation times 603-1 to 603-11 in Figure 6(a).

[0033] Figure 7 shows an example of the change in image brightness when detecting the lighting period of a light source, along with the flashing period of the light source. In Figure 7, 701 shows the flashing period of the light source according to the accumulation timing 604, and 702 shows the change in image brightness according to the accumulation timing 604 when the image accumulation time is shortened, as shown in Figure 6(a). As shown in periods 711, 713, and 715 in Figure 7, even if the accumulation end timing is shifted during the off period, which is shown at a low level in the flashing period of the light source 701, the exposure period does not change, so the image brightness 702 remains approximately constant. For example, in the accumulation times 603-1 to 603-7 shown in Figure 6, the exposure period does not change, so the image brightness remains approximately constant. On the other hand, as shown in periods 712 and 714 in Figure 7, if the accumulation end timing is shifted during the lighting period, which is shown at a high level in the flashing period of the light source 701, the exposure period within the lighting period becomes shorter, so the image brightness 702 gradually becomes darker as shown. For example, in the accumulation times 603-8 to 603-11 shown in Figure 6, the exposure period shortens, and the brightness of the image decreases. Note that the flashing period 701 of the light source is schematically shown as a square wave, but in reality, it is often not a clear square wave as shown. Also, the image brightness 702 is actually a waveform that contains errors due to noise, etc., but the average value is thought to be as shown. Furthermore, if there is sunlight or steady light other than the flashing light source, the brightness from that is added, so the image brightness 702 changes according to the accumulation time, but the change is smaller compared to the flashing light source.

[0034] Based on the relationship between the blinking period of the light source and the change in image brightness, it is possible to detect that the blinking light source is on during periods 712 and 714 when the image brightness 702 exceeds an arbitrary amount of change (the amount of change is greater than when it is approximately constant). It is possible that the image brightness 702 may not be as shown due to noise, etc., but it is presumed that if the value is acquired multiple times and the average value is obtained, a trend similar to the image brightness 702 shown in Figure 7 can be obtained.

[0035] According to this embodiment, similar to the first embodiment, the illumination period of the flashing light source can be detected, and the saturation value of the pixels can be set accordingly, thereby improving the reproduction of brightness and color. Furthermore, in this embodiment, only the accumulation termination timing is shifted compared to the first embodiment, making the design easier.

[0036] In the explanation above, the timing of the end of accumulation is shifted. However, it is possible to detect the illumination period of the flashing light source by sequentially changing either the start or end of the accumulation time while keeping the other unchanged. Therefore, it is also possible to keep the end of accumulation timing fixed and shift the start of accumulation timing.

[0037] <Third Embodiment> A third embodiment will now be described. In the third embodiment, an example will be described in which the drive related to imaging is changed according to the illumination period of the flashing light source detected in the first or second embodiment described above. In the third embodiment described below, the same configurations and operations as in the first and second embodiments described above will be omitted, and the differences from the first and second embodiments will be described.

[0038] Figure 8 illustrates an example of changing the drive related to imaging according to the illumination period of a blinking light source. Referring to Figure 8(a), an example of changing the drive according to the illumination period of the light source in a rolling shutter drive will be explained. In Figure 8(a), 801 indicates the blinking period of the light source, and 804 is the illumination period of the light source. Also, 805 is the storage time of the image sensor, and 806 is the readout time. In the case of a rolling shutter, since readout is performed row by row or multiple rows at a time, a time difference occurs from the end of storage of the first row of the image to the end of storage of the last row. This time becomes the readout time 806, and the time required to complete storage and readout for one image is the sum of the storage time 805 and the readout time 806. In addition, in a photon counting image sensor, the photon counting operation is performed within the image storage time.

[0039] The imaging device 100 is configured such that (accumulation time 805 + readout time 806) is included in the illumination period 804 of the light source, as shown in Figure 8(a), so that the accumulation of the entire image is completed in accordance with the illumination period of the light source detected as in the first or second embodiment. By configuring it in this way, the image sensor counts photons during the accumulation time, making it possible to concentrate the counting of photons during the illumination period. Furthermore, by limiting the period for counting photons in the image sensor to only the illumination period of the light source, and not performing counting operations during the period when the light source is off and no photons are incident, power consumption can be reduced. Although it has been explained that (accumulation time 805 + readout time 806) should be set to fall within the illumination period, (accumulation time 805 + readout time 806) may coincide with the illumination period, or it may be shorter or longer than the illumination period. The settings for the accumulation time 805 and readout time 806 should be set within a range that can be set according to the setting resolution of the accumulation time and other specifications of the image sensor and imaging device.

[0040] Referring to Figure 8(b), an example of changing the drive according to the illumination period of the light source in the global shutter drive will be described. In Figure 8(b), 807 indicates the blinking period of the light source, and 808 is the illumination period of the light source. Also, 809 is the storage time of the image sensor. In the global shutter method, the entire image is read out at once, so there is no time difference in reading out for each row, and therefore there is no readout time 806 as shown in Figure 8(a). Accordingly, by setting the storage time 809 to be included in the illumination period 808 of the light source detected as in the first or second embodiment, it becomes possible to concentrate the counting of photons during the illumination period of the light source.

[0041] Figure 8(c) is a timing chart showing an example of driving the image sensor in the rolling shutter method, where the (accumulation time 805 + readout time 806) is set to be included in the illumination period 804 of the light source, as shown in Figure 8(a). In Figure 8(c), 803-1 and 803-2 show the accumulation time of the image acquired in one interval of the imaging frame rate 802. This shows an example where the blinking period of the blinking light source is fast, and the imaging frame rate 802 cannot be matched to one blinking period of the light source, but is instead matched to multiple periods, in this example to six periods. It is also possible that the imaging frame rate matches one blinking period of the light source, or to multiple periods other than six. In the example shown in Figure 8(c), the image sensor is driven by setting it so that the accumulation time and readout time for one image fit within one illumination period. In the case of the global shutter method, it is sufficient to drive the sensor so that the accumulation time for one image fits within one illumination period.

[0042] However, because flashing light sources such as LEDs have a high flashing frequency, it may not be possible to ensure that the image accumulation time fits within a single flashing period. In such cases, for example, as shown in Figures 8(d) and 8(e), the image sensor is driven by setting the settings so that the accumulation time and readout time for one image fit within a flashing period that is a multiple of the number of flashes, from the start to the end of flashing. In Figures 8(d) and 8(e), 803-3 and 803-4 show the accumulation time of images acquired in one interval of the imaging frame rate 802. Figure 8(d) shows an example of matching two flashing periods (two flashing cycles), where the image sensor is driven by setting the settings so that the accumulation time and readout time for one image fit within the period from the start of the first flash to the end of the second flash. Figure 8(e) shows an example of matching three flashing periods (three flashing cycles), where the image sensor is driven by setting the settings so that the accumulation time and end time for one image fit within the period from the start of the first flash to the end of the third flash. The same applies to the global shutter system; the system should be driven by setting the accumulation time to fall within the aforementioned period. Note that Figures 8(c) to 8(e) illustrate a case where one interval of the imaging frame rate includes six illumination periods, but this is just one example, and the number of times the light source is illuminated within one interval of the imaging frame rate is not limited to six.

[0043] Here, when the image sensor is driven in accordance with a multiple of the illumination period, as shown in Figures 8(d) and 8(e), the rolling shutter drive is used to drive the image sensor in such a way that no rows are left unexposed. An example of this will be explained with reference to Figures 9(a) to 9(c). In Figures 9(a) to 9(c), 901 indicates the blinking period of the light source. Also, 902 is the storage time of the image sensor, and 903 is the readout time of the image sensor.

[0044] Figure 9(a) shows an example where the readout time 903 is slow (long) when using a rolling shutter drive, in which the accumulation time 902 and readout time 903 for one image are included in two lighting periods of the light source. 904-1 to 904-7 indicate the accumulation time for each row of the acquired image, and in this example, it is schematically represented by 7 rows. In reality, the number of rows is set in various ways depending on the number of light-receiving elements of the image sensor and the readout method. Furthermore, for accumulation times 904-1 to 904-7, the parts exposed during the lighting period, which is indicated at a high level in the blinking cycle 901 of the light source, are shown in white, and the parts not exposed during the off period, which is indicated at a low level in the blinking cycle 901 of the light source, are shown in shaded areas.

[0045] In the example shown in Figure 9(a), the exposure times 904-1 to 904-3 for rows 1 to 3, and 904-5 to 904-7 for rows 5 to 7, include areas (white) that are exposed during the light source's on-times 911 and 913. However, some rows, such as row 4 with exposure time 904-4, which falls within the light source's off-time 912, may not be exposed at all (black). If an area is exposed even for a short period, it can be corrected by the subsequent image control unit 103 or gain control unit 110 according to the exposure time. However, if an area is not exposed at all, it is not possible to obtain image information through exposure time-based correction, resulting in rows with missing image information.

[0046] To prevent this, when driving the image sensor in accordance with multiples of the illumination period, the setting should be configured so that the accumulation time of the first row includes at least two illumination periods, as shown in Figures 9(b) and 9(c). Note that the accumulation time does not need to be equal to two illumination periods; the accumulation time may be slightly longer or shorter than two illumination periods (for example, 1.8 times, 1.9 times, 2.1 times, 2.2 times, etc.).

[0047] In Figure 9(b), 905-1 to 905-7 represent the accumulation time for each row of the acquired image, schematically represented by seven rows. As shown by the accumulation times 905-1 to 905-3 for rows 1 to 3, rows 1 to 3 of the image are exposed, including the first illumination period 921 and the second illumination period 922. Also, as shown by the accumulation times 905-4 to 905-6 for rows 4 to 6, rows 4 to 6 of the image are exposed because, although the first illumination period 921 has ended at the start of accumulation, the second illumination period 922 is always included in the accumulation time. Furthermore, as shown by the accumulation time 905-7 for row 7, row 7 of the image is also always exposed because, although the first illumination period 921 is not included, the second illumination period 922 and the third illumination period 923 are included.

[0048] Figure 9(c) shows an example where the image sensor readout time 903 is even longer. In Figure 9(c), 906-1 to 906-7 represent the accumulation time of each row of the acquired image, schematically represented by seven rows. As shown in accumulation times 906-1 to 906-7, if the exposure time 906-1 of the first row includes two illumination periods, then the accumulation times 906-2 to 906-7 of the second row and subsequent rows will include at least one illumination period and be exposed. For example, even if the exposure time does not include the first illumination period 931, no rows will be left unexposed because it includes the subsequent second illumination period 932 and the third illumination period 933, or the third illumination period 933 and the fourth illumination period 934.

[0049] Therefore, by driving the image sensor as shown in Figures 9(b) and 9(c), it is possible to prevent rows from being lost due to the lack of image information. In the image obtained in this way, the exposure amount differs from row to row, and the brightness changes from row to row. However, the exposure time for each row can be obtained, for example, from the illumination period detected in the first embodiment and the image sensor readout time, and by applying a gain that takes this into account to each row, brightness correction can be performed, making it possible to generate an image with suppressed changes in brightness from row to row.

[0050] In this embodiment, the drive is changed by adjusting the accumulation time to coincide with the end of the lighting period so that accumulation does not occur during the period when the light source is off. Even if the user sets and changes the accumulation period (generally called shutter speed) so that it ends during the period when the light source is off, it is conceivable that the brightness of the acquired image will not change because the change is made within the period when the light source is off. To prevent this phenomenon from occurring, the accumulation period (shutter speed) that the user can set may be limited. It would be sufficient to allow setting only shutter speeds that can be matched from the start to the end of lighting, which are multiples of the number of times the image sensor's accumulation time and readout time fit within.

[0051] Furthermore, if there is a uniform light source that emits light uniformly, such as sunlight or constant light, in addition to the flashing light source, the brightness of the image will also change if the accumulation period (shutter speed), which is set to end when the flashing light source is off, changes. In this embodiment, if there is a uniform light source in addition to the flashing light source, there is a possibility that the brightness of the image will not change even if the user changes the shutter speed setting. To prevent this phenomenon, it is possible to correct the image so that the brightness of the image changes according to the set shutter speed by applying gain processing and offset processing in the gain control unit 110 and offset control unit 111 according to the brightness of the uniform light source.

[0052] According to this embodiment, by setting the image accumulation time in accordance with the illumination period of the detected blinking light source and driving the image sensor 102, photon counting is performed only during the illumination period, and counting is not performed during the period when the light is not illuminated, thereby reducing power consumption.

[0053] <Fourth Embodiment> A fourth embodiment will now be described. In the fourth embodiment, descriptions of the configuration and operation, etc., which are the same as those described in the first to third embodiments, will be omitted. In the fourth embodiment, an example of correcting the brightness of an image in a method in which the user adjusts the drive related to imaging in accordance with the lighting period of the blinking light source described in the third embodiment will be described with reference to Figures 10(a) to 10(c). In Figures 10(a) to 10(c), 1001 indicates the blinking period of the light source, 1002 indicates the imaging frame rate, and 1003-1, 1003-2, and 1003-3 indicate the accumulation time of the image acquired in one interval of the imaging frame rate. Figure 10(a) shows an example in which the accumulation time and readout time for one image are included within one lighting period. Figures 10(b) and 10(c) show examples in which the accumulation time and readout time for one image are included from the start to the end of lighting, which is a multiple of the number of lighting cycles.

[0054] In the example shown in Figure 10(a), the image acquired at storage time 1003-1 records the exposure amount for one lighting period, and is not exposed during other lighting periods 1004-1. For example, in the so-called full storage setting, where the storage period is set to one interval of the imaging frame rate, if the storage time is not changed to match the lighting period of the light source, the acquired image will have the exposure amount for six lighting periods. Therefore, there is a concern that the image read out will be darker than the image brightness at the value set by the user as the storage period (shutter speed). To resolve this, the image can be corrected as follows, for example. Since the length of the lighting period and the blinking period of the light source are known by detection in the first or second embodiment, it is possible to acquire how many times the lighting period of the blinking light source is included within the set storage period. If the number of lighting times included in the storage period set by the user is known, it is possible to correct the acquired image to an appropriate brightness by applying gain processing according to the number of lighting times in the gain control unit 110.

[0055] As shown in Figure 10(a), the user-set storage period includes six lighting periods, but the image sensor is driven to match one lighting period. In this case, since we want to correct the number of lighting periods to six, which should be included in the user-set storage period, the gain control unit 110 can apply a gain processing of 6 times to the acquired image to achieve this correction. Similarly, correction is possible when the image sensor is driven as shown in Figures 10(b) and 10(c). In the case of the drive shown in Figure 10(b), since exposure does not occur during four lighting periods 1004-2, the gain control unit 110 applies a gain processing of 3 times to the image exposed during an exposure time 1003-2 which includes two lighting periods. In the case of the drive shown in Figure 10(c), since exposure does not occur during three lighting periods 1004-3, the gain control unit 110 applies a gain processing of 2 times to the image exposed during an exposure time 1003-3 which includes three lighting periods. This allows the exposure amount to be corrected to match the six lighting periods that should be included in the user-set accumulation period (shutter speed).

[0056] According to this embodiment, even if the lighting period of the flashing light source described in the first to third embodiments is detected, and the drive is changed to match the detected lighting period, it is possible to obtain an image of brightness that matches the actual lighting period.

[0057] <Fifth Embodiment> This embodiment describes a method by which the user causes the imaging device 100 to perform the detection of the illumination period of the blinking light source and to change the drive related to imaging, as described in each of the embodiments described above. Figures 11(a) to 11(d) illustrate the user interface (UI) for causing the imaging device 100 to perform the detection of the illumination period of the blinking light source and to change the drive related to imaging.

[0058] Figure 11(a) shows an example of a menu image 1101 used in the operation unit 109 for detecting the illumination period of the flashing light source in this embodiment. As shown in Figure 11(a), the menu image 1101 includes a menu 1002 for performing automatic flicker detection and a menu 1003 for performing detection of the illumination period of the flashing light source. Menus 1102 and 1103 in the menu image 1101 shown in Figure 11(a) are just examples, and various other menu items may be displayed.

[0059] When the user selects the illumination period detection menu 1003, a message indicating "Detecting" or a message such as "Do not move the camera" 1112 will appear on the screen, as shown in image 1111 in Figure 11(b), and the detection of the illumination period of the light source will begin. The message 1112 shown in Figure 11(b) is just an example, and other messages may be displayed. In addition, the image being captured during detection may be displayed as shown in image 1111, or a black screen may be displayed.

[0060] Then, once the detection of the light source's illumination period is complete, a message 1121 indicating "Detection complete" is displayed on the screen, as shown in Figure 11(c). Furthermore, as shown in Figure 11(c), if it is possible to change the drive related to imaging in accordance with the illumination period, in addition to detecting the illumination period, a message asking "Do you want to change the drive?" may also be displayed. If the user does not want to change the drive related to imaging, the message regarding the drive change does not need to be displayed, and if the user wants to change the drive related to imaging but does not want the message to be displayed, it does not need to be displayed. In addition, a button 1112 may be displayed for the user to confirm the change in the drive related to imaging, allowing the user to choose whether or not to change the drive.

[0061] Upon receiving a user instruction to change the drive related to imaging, and once the change to the drive related to imaging is completed, a message 1133 indicating "Drive has been changed" may be displayed, as shown in Figure 11(d).

[0062] As described above, by having an operating unit like that of this embodiment, if the user notices something unusual about the image under a flashing light source, it becomes possible to detect the duration of the flashing light source's illumination and make changes such as adjusting the saturation or drive. This makes it possible to suppress changes in the brightness and color of the image under a flashing light source.

[0063] (Other embodiments of the present invention) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0064] It should be noted that the embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0065] The disclosure of this embodiment includes the following configurations and methods, etc. (Composition 1) An image sensor having a light-receiving unit that outputs a pulse signal in response to the incidence of photons, and a counting unit that counts the pulse signals, Control means for controlling the imaging frame rate and image storage time of the image sensor, An imaging device characterized by having a detection means for detecting the illumination period of a flashing light source based on a change in the brightness of an image obtained by imaging with the accumulation time changed by the control means. (Configuration 2) The imaging apparatus according to Configuration 1, characterized in that, when the illumination period of the flashing light source is detected, the control means adjusts the imaging frame rate to match the flashing period of the flashing light source and sequentially changes at least one of the start timing and end timing of the accumulation time. (Composition 3) The imaging apparatus according to configuration 2, characterized in that the control means keeps the length of the storage time constant and sequentially changes the start timing and end timing of the storage time. (Composition 4) The imaging apparatus according to configuration 3, characterized in that the detection means sets the period during which the brightness is greater among the periods in which the amount of change in brightness of the image obtained by imaging with varying storage time is less than a predetermined amount as the lighting period of the flashing light source. (Composition 5) The imaging apparatus according to configuration 3 or 4, characterized in that the control means sets the length of the storage time to the minimum possible time. (Composition 6) The imaging apparatus according to any one of configurations 3 to 5, characterized in that the control means sequentially changes the start timing and end timing of the storage time from the start to the end of one interval of the imaging frame rate. (Composition 7) The imaging apparatus according to configuration 2, characterized in that the control means sequentially changes the start timing or the end timing of the storage time without changing the other of the two. (Composition 8) The imaging apparatus according to configuration 7, characterized in that the detection means sets the period during which the change in brightness of an image obtained by imaging with varying storage time exceeds a predetermined change to be the lighting period of the flashing light source. (Composition 9) The imaging apparatus according to configuration 7 or 8, characterized in that the control means sequentially changes the storage time so that it becomes shorter. (Composition 10) The imaging apparatus according to any one of configurations 7 to 9, characterized in that the control means sequentially changes the start timing or the other of the end timing of the accumulation time from the start to the end of one interval of the imaging frame rate. (Composition 11) The imaging device according to any one of configurations 1 to 10, characterized in that the control means controls the storage time according to the illumination period of the detected flashing light source. (Composition 12) The imaging apparatus according to configuration 11, characterized in that, when the image sensor is a rolling shutter type image sensor, the control means controls the storage time so that the storage time and readout time for one image are included within the illumination period of the detected flashing light source. (Composition 13) The imaging apparatus according to configuration 12, characterized in that the control means controls the storage time such that the storage time of the first row of the image includes at least two periods of illumination of the flashing light source. (Composition 14) The imaging device according to configuration 12 or 13, characterized in that it can set only shutter speeds that can be matched to the start and end of illumination, which is a multiple of the number of illuminations that accommodate the storage time and readout time for one image. (Composition 15) The imaging apparatus according to configuration 11, characterized in that, when the image sensor is a global shutter type image sensor, the control means controls the storage time so that the storage time for one image is included within the illumination period of the detected flashing light source. (Composition 16) The imaging device according to configuration 15, characterized in that it can only set a shutter speed that matches the start and end of illumination to a multiple of the number of illuminations that contain the accumulation time for one image. (Composition 17) The imaging apparatus according to any one of configurations 1 to 16, characterized in that it has gain control means that performs gain processing on an image captured according to the number of lighting periods of the blinking light source included in one interval of the imaging frame rate. (Composition 18) The imaging device according to any one of configurations 1 to 17, characterized by having an operating means for instructing the execution of detecting the illumination period of the flashing light source. (Method 1) A control method for an imaging device having an image sensor having a light receiving unit that outputs a pulse signal in response to the incidence of photons, and a counting unit that counts the pulse signal, A control step for controlling the imaging frame rate and image storage time in the image sensor, A control method for an imaging device, characterized by comprising: a detection step of detecting the lighting period of a blinking light source based on a change in the brightness of an image obtained by imaging with the storage time changed in the control step. [Explanation of symbols]

[0066] 100: Imaging device 101: Lens 102: Image sensor 103: Image control unit 104: Accumulation control unit 105: System control unit 106: Frame rate control unit 107: Display unit 108: Recording unit 109: Operation unit 110: Gain control unit 111: Offset control unit

Claims

1. An image sensor having a light-receiving unit that outputs a pulse signal in response to the incidence of photons, and a counting unit that counts the pulse signals, Control means for controlling the imaging frame rate and image storage time of the image sensor, An imaging device characterized by having a detection means for detecting the illumination period of a flashing light source based on a change in the brightness of an image obtained by imaging with the accumulation time changed by the control means.

2. The imaging apparatus according to claim 1, characterized in that, when the illumination period of the flashing light source is detected, the control means adjusts the imaging frame rate to match the flashing period of the flashing light source and sequentially changes at least one of the start timing and end timing of the accumulation time.

3. The imaging apparatus according to claim 2, characterized in that the control means keeps the length of the storage time constant and sequentially changes the start timing and end timing of the storage time.

4. The imaging apparatus according to claim 3, characterized in that the detection means sets the period during which the brightness is greater among the periods in which the amount of change in brightness of the image obtained by imaging with varying storage time is less than a predetermined amount as the lighting period of the flashing light source.

5. The imaging apparatus according to claim 3, characterized in that the control means sets the length of the storage time to the minimum possible time.

6. The imaging apparatus according to claim 3, characterized in that the control means sequentially changes the start timing and end timing of the accumulation time from the start to the end of one interval of the imaging frame rate.

7. The imaging apparatus according to claim 2, characterized in that the control means sequentially changes the start timing or the end timing of the storage time without changing the other of the two.

8. The imaging apparatus according to claim 7, characterized in that the detection means sets the period during which the amount of change in brightness of an image obtained by imaging with varying storage time exceeds a predetermined amount of change to be the lighting period of the flashing light source.

9. The imaging apparatus according to claim 7, characterized in that the control means sequentially changes the storage time so that it becomes shorter.

10. The imaging apparatus according to claim 7, characterized in that the control means sequentially changes the start timing or the other of the end timing of the storage time from the start to the end of one interval of the imaging frame rate.

11. The imaging device according to claim 1, characterized in that the control means controls the storage time according to the illumination period of the detected flashing light source.

12. The imaging apparatus according to claim 11, characterized in that, when the image sensor is a rolling shutter type image sensor, the control means controls the storage time such that the storage time and readout time for one image are included within the illumination period of the detected flashing light source.

13. The imaging apparatus according to claim 12, characterized in that the control means controls the storage time such that the storage time of the first row of the image includes at least two periods of illumination of the flashing light source.

14. The imaging device according to claim 12, characterized in that it can set only shutter speeds that can be matched to the start and end of illumination, which is a multiple of the number of illuminations that accommodate the storage time and readout time for one image.

15. The imaging apparatus according to claim 11, characterized in that, when the image sensor is a global shutter type image sensor, the control means controls the storage time such that the storage time for one image is included within the illumination period of the detected blinking light source.

16. The imaging device according to claim 15, characterized in that it can set only shutter speeds that can be matched to the start and end of illumination, which is a multiple of the number of illuminations that contain the accumulation time for one image.

17. The imaging apparatus according to claim 1, further comprising gain control means for performing gain processing on an image captured according to the number of lighting periods of the blinking light source included in one interval of the imaging frame rate.

18. The imaging device according to claim 1, further comprising an operating means for instructing the execution of detecting the illumination period of the flashing light source.

19. A control method for an imaging device having an image sensor having a light receiving unit that outputs a pulse signal in response to the incidence of photons, and a counting unit that counts the pulse signal, A control step for controlling the imaging frame rate and image storage time in the image sensor, A control method for an imaging device, characterized by comprising: a detection step of detecting the lighting period of a blinking light source based on a change in the brightness of an image obtained by imaging with the storage time changed in the control step.

Images