Imaging device, imaging method

Abstract

To provide an imaging apparatus capable of simultaneously obtaining an image with excellent dark part visibility while obtaining an image of a target distance area by range gate control.SOLUTION: A plurality of pixel units each including a color filter that transmits light of a specific wavelength, a light emitting unit that emits light a plurality of times within one frame period, a sensor unit that emits a pulse according to a light reception frequency of a photon, a counter that counts the number of pulses, and a memory that stores a value of the count; The count enable generation section generates either a first count enable signal which is asynchronous with a light emission timing of the light emitting section and has one enable period within one frame period, or a second count enable signal which is synchronous with the light emission timing of the light emitting section and has a plurality of enable periods within one frame period.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present invention relates to an imaging device and an imaging method that generate different count enable signals according to a color filter to obtain an RGB image and a range gate image.

Background Art

[0002] There is a photographing method using a camera called a range gate camera. This is a technique that emits pulsed light in front of the camera at a predetermined cycle, and the image sensor inside the camera is exposed at a predetermined timing according to the target distance, enabling only the subject at the target distance to be clearly imaged. Hereinafter, this technology is referred to as range gate control. With this range gate control, for example, even in bad weather, a subject (object) at a predetermined distance can be clearly imaged.

[0003] For example, Patent Document 1 describes a technique for adjusting the timing of light emission by pulsed light and exposure by the camera by a timing controller to prevent imaging of an unnecessary target distance region. Also, in this patent document, a configuration is described that enables obtaining images of a plurality of target distance regions having different distances from the camera by changing the timing (delay time) of light emission by pulsed light and exposure by the camera for each frame.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the technology described in Patent Document 1 changes the target distance range for each frame, and there was a problem that it required the time of multiple frames to capture multiple distance ranges from near the camera to far away. Therefore, when used in applications that require immediacy, such as collision avoidance, there was a possibility that the recognition of objects ahead (vehicles, etc.) using images would be delayed, and collision avoidance measures (automatic braking, etc.) would be delayed.

[0006] Furthermore, because the image sensor inside the camera exposes the image at a predetermined timing corresponding to the target distance, the exposure time is limited to the time corresponding to the target distance. As a result, the exposure time within the predetermined time tended to be shorter compared to the exposure time without range gate control. In this case, even if repeated exposures are performed, the total amount of reflected light accumulated from the object within the predetermined time will be less. Therefore, during dark times such as at night, when the amount of reflected light per unit time is low, it may be difficult to recognize objects in front of the camera from the image.

[0007] Therefore, one of the objectives of the present invention is to provide an imaging device that can obtain an image of the target distance region by range gate control while simultaneously obtaining an image with good visibility of dark areas. [Means for solving the problem]

[0008] One aspect of the present invention is an imaging device comprising: a color filter that transmits light of a specific wavelength; a light-emitting unit that emits light of a wavelength corresponding to the frequency characteristics of the color filter multiple times within one frame period; a sensor unit that emits pulses according to the frequency of reception of photons of light transmitted through the color filter; a counter that counts the number of pulses; a memory that stores the count value of the counter; and a count enable generation unit that generates a count enable signal to control the count period of the counter, wherein the count enable generation unit generates either a first count enable signal asynchronous with the light emission timing of the light-emitting unit and having one enable period within one frame period, or a second count enable signal synchronized with the light emission timing of the light-emitting unit and having multiple enable periods within one frame period. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an imaging device that can clearly image a subject at a predetermined distance even in bad weather, and furthermore, can simultaneously capture images with good visibility of dark areas from near to far without requiring multiple frames. [Brief explanation of the drawing]

[0010] [Figure 1] This is a diagram showing an example configuration of the photoelectric conversion element 100 according to the embodiment. [Figure 2] This figure shows an example configuration of the sensor board 11. [Figure 3] This figure shows an example of the configuration of the color filter 30 in the first embodiment. [Figure 4] This figure shows an example of the configuration of the circuit board 21. [Figure 5] Figure 2 shows the equivalent circuit of pixel 101 and the signal processing circuit 103 corresponding to pixel 101. [Figure 6] This diagram schematically illustrates the relationship between the operation of the APD201 and its output signals. [Figure 7]This is a functional block diagram of the IR emitter 500, camera 600, and mobile body 700 according to the embodiment. [Figure 8] This figure shows the relationship between the propagation of reflected light and the exposure timing according to the embodiment. [Figure 9] This figure shows the light emission and exposure control operation for one frame time in the first embodiment. [Figure 10] This figure shows an example of a display image on the display unit 703 in the embodiment. [Figure 11] This flowchart shows the details of an example of operation in the embodiment. [Figure 12] This figure shows an example of the configuration of the color filter 40 in the second embodiment. [Figure 13] This figure shows an example of the configuration of the visible light emitter 900 in the second embodiment. [Figure 14] This figure shows the light emission and exposure control operation for one frame time in the second embodiment. [Modes for carrying out the invention]

[0011] <First Embodiment> Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to the following embodiments. In each drawing, the same reference numeral is used for the same member or element, and redundant explanations are omitted or simplified.

[0012] Figure 1 shows an example of the configuration of a photoelectric conversion element according to an embodiment of the present invention. In the following description, a photoelectric conversion device having a so-called stacked structure, in which the photoelectric conversion element 100 is composed of two substrates, a sensor substrate 11 and a circuit substrate 21, stacked and electrically connected, will be used as an example. However, a so-called non-stacked structure may also be used, in which the components included in the sensor substrate and the components included in the circuit substrate are arranged on a common semiconductor layer. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes the signal detected in the pixel region 12.

[0013] FIG. 2 is a diagram showing a configuration example of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes pixels 101 (pixel portions) two-dimensionally arranged in a plurality of rows and columns. The pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, APD).

[0014] Here, the photoelectric conversion unit 102 functions as a sensor unit that emits pulses at a frequency corresponding to the light reception frequency of photons. Note that the number of rows and columns of the pixel array forming the pixel region 12 is not particularly limited.

[0015] FIG. 3 is a diagram showing a configuration example of the color filter 30 included in the pixel 101. Each of the pixels 101 inside the pixel region 12 has one of the colors of the color filter 30. That is, the color filter has any frequency characteristics. The color filter 30 is roughly divided into the following two types of filters. First, an R filter, a G filter, and a B filter (collectively, an RGB filter 31) that transmit light of wavelengths of red (R), blue (B), and green (G), respectively. Second, an IR filter 32 that is a filter in the infrared region that transmits light of the wavelength of infrared light (IR). The color filter having this configuration is called an RGB-IR filter.

[0016] Here, one pixel 101 corresponds to any one of the color filters of the R filter, the G filter, the B filter, and the IR filter. As shown in FIG. 3, the arrangement in the present embodiment is an arrangement in which a column in which the B filter and the G filter are alternately arranged and a column in which the IR filter and the R filter are alternately arranged are combined. Note that the arrangement combining the RGB filter 31 and the IR filter 32 is not limited to this.

[0017] FIG. 4(a) is a diagram showing a configuration example of the circuit board 21. The circuit board 21 includes a signal processing circuit 103 that processes charges photoelectrically converted by each of the photoelectric conversion units 102 in FIG. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, vertical signal lines 113, a vertical scanning circuit 110, and an output circuit 114.

[0018] The vertical scanning circuit 110 receives control pulses supplied from the control pulse generation unit 115 and sequentially supplies control pulses to multiple pixels arranged in the row direction. Logic circuits such as a shift register and an address decoder are used in the vertical scanning circuit 110.

[0019] The signals output from the photoelectric conversion unit 102 of each pixel are processed by the respective signal processing circuit 103. The signal processing circuit 103 is equipped with a counter and memory, and digital values ​​are stored in the memory. The horizontal scanning circuit 111 inputs control pulses to the signal processing circuit 103 to sequentially select each column in order to read the signal from the memory of each pixel in which the digital signal is stored.

[0020] Signals are output to the vertical signal line 113 from the signal processing circuit 103 of the pixels in the row selected by the vertical scanning circuit 110. The signals output to the vertical signal line 113 are then output to the outside of the photoelectric conversion element 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has multiple buffers built in that are connected to the vertical signal line 113.

[0021] As shown in Figures 2 and 4(a), multiple signal processing circuits 103 are arranged in the region that overlaps with the pixel region 12 in a plan view. Then, in a plan view, a vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generation unit 115 are arranged so as to overlap between the edge of the sensor substrate 11 and the edge of the pixel region 12.

[0022] In other words, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generation unit 115 are arranged in the region that overlaps with the non-pixel region in a plan view.

[0023] Furthermore, the arrangement of the vertical signal line 113, the readout circuit 112, and the output circuit 114 is not limited to the example shown in Figure 4(a). For example, the vertical signal line 113 may be arranged extending in the row direction, and the readout circuit 112 may be placed at the end of the vertical signal line 113. Also, the signal processing circuit 103 does not necessarily need to be provided for each photoelectric conversion unit; a single signal processing unit may be shared by multiple photoelectric conversion units, and sequential signal processing may be performed.

[0024] Figure 4(b) shows an example of the configuration of the count enable generation unit 104, which generates signals to be supplied to each signal processing circuit 103 as described in Figure 4(a). In Figure 4(b), the signal processing circuit 103 that handles the light transmitted through the R filter of the RGB filter 31 (the R signal after photoelectric conversion) is the R signal processing circuit 103r. Similarly, the signal processing circuit that handles the G signal is the G signal processing circuit 103g, and the signal processing circuit that handles the B signal is the B signal processing circuit 103b. In addition, the signal processing circuit that handles the light transmitted through the IR filter 32 (the IR signal after photoelectric conversion) is the IR signal processing circuit 103IR.

[0025] The count enable generation unit 104 generates a count enable signal to be supplied to the counter inside the signal processing circuit 103. The count enable signal is a signal used to control the enable and disable of the counter inside the signal processing circuit 103. This count enable signal is configured to generate different signals for each of the signal processing circuits 103 for the R signal, G signal, B signal, and IR signal. That is, the count enable generation unit 104r for R signals generates the count enable signal to be supplied to the multiple R signal processing circuits 103r. Similarly, the count enable generation units 104g for G signals and 104b for B signals generate the count enable signals for the G signal processing circuit 103g and B signal processing circuit 103b, respectively. The count enable generation unit 104IR for IR signals generates the count enable signal to be supplied to the multiple IR signal processing circuits 103IR. It is also possible to generate the count enable signals for R, G, B, and IR at independent timings. Note that in Figure 4(b), the connection between the signal processing circuit 103 and the count enable generation unit 104 is omitted.

[0026] Figure 5 shows the equivalent circuit of the pixel 101 in Figures 2 and 4(a) and the signal processing circuit 103 corresponding to the pixel 101.

[0027] The APD201 included in the photoelectric conversion unit 102 generates charge pairs corresponding to incident light through photoelectric conversion. One of the two nodes of the APD201 is connected to a power line to which a drive voltage VL (first voltage) is supplied. The other of the two nodes of the APD201 is connected to a power line to which a drive voltage VH (second voltage), which is higher than voltage VL, is supplied.

[0028] In Figure 5, one node of the APD201 is the anode, and the other node of the APD is the cathode. A reverse bias voltage is supplied to the anode and cathode of the APD201 so that the APD201 performs avalanche multiplication. By supplying such a voltage, the charge generated by the incident light undergoes avalanche multiplication, and an avalanche current is generated.

[0029] Furthermore, when a reverse bias voltage is supplied, there are two modes of operation: Geiger mode, where the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, where the voltage difference between the anode and cathode is near or below the breakdown voltage. An APD operating in Geiger mode is called a SPAD. In the case of a SPAD, for example, the voltage VL (first voltage) is -30V and the voltage VH (second voltage) is 1V.

[0030] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212. The quench element 202 is connected to a power line to which a drive voltage VH is supplied and to one of the nodes, either the anode or the cathode, of the APD 201.

[0031] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, suppressing the voltage supplied to the APD201 and thereby suppressing avalanche multiplication (quench operation). In addition, the quench element 202 also works to restore the voltage supplied to the APD201 to the drive voltage VH by flowing the current that compensates for the voltage drop caused by the quench operation (recharge operation).

[0032] Figure 5 shows an example in which the signal processing circuit 103 includes a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212 in addition to the quench element 202.

[0033] The waveform shaping unit 210 shapes the cathode voltage change of the APD201 obtained during photon detection and outputs a pulse signal. For example, an inverter circuit can be used as the waveform shaping unit 210. Figure 5 shows an example in which one inverter is used as the waveform shaping unit 210, but a circuit in which multiple inverters are connected in series may be used, or other circuits that have a waveform shaping effect may be used.

[0034] The counter circuit 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. When the control pulse RES is supplied via the drive line 213, the signal held by the counter circuit 211 is reset. Here, the counter circuit 211 generates a signal based on the difference between the count values ​​at the start and end of the accumulation period.

[0035] The counter circuit 211 is supplied with a count enable signal from the count enable generation unit 104, as described in Figure 4(b). During the High interval (counting period) of this count enable signal, the counter circuit 211 counts the number of pulses output from the waveform shaping unit 210, and during the Low interval (counting period), it does not count the number of pulses but holds the count value. As a result, if the count enable generation unit 104 is a circuit operating at a clock frequency of 100 MHz, the count enable signal can be controlled to High or Low (enable / disable control) in units of 10 nsec of the clock period. Here, since the number of pulses output by the photoelectric conversion unit 102 according to the photon reception frequency is counted only when count enable is enabled, the count enable period for the number of pulses can be rephrased as the exposure period of the pixel 101. Therefore, in this embodiment, the switching control of exposure and non-exposure can be controlled in units of 10 nsec.

[0036] Furthermore, as shown in Figure 4(b), the count enable signal can be generated by the count enable generation unit 104 as different signals by the R signal processing circuit 103r, the G signal processing circuit 103g, the B signal processing circuit 103b, and the IR signal processing circuit 103IR.

[0037] In this embodiment, the R signal processing circuit 103r, the G signal processing circuit 103g, the B signal processing circuit 103b, and the IR signal processing circuit 103IR generate different count enable signals. Details of the generated signals will be described later using a timing chart.

[0038] The memory circuit 212 receives control pulses SEL from the vertical scanning circuit 110 in Figure 4(a) via the drive line 214 (not shown in Figure 4(a)) in Figure 5, which switches the electrical connection between the counter circuit 211 and the vertical signal line 113. The memory circuit 212 functions as a memory that temporarily stores the counter's count value and outputs the output signal from the pixel's counter circuit 211 to the vertical signal line 113.

[0039] Furthermore, switches such as transistors may be placed between the quench element 202 and the APD201, or between the photoelectric conversion unit 102 and the signal processing circuit 103, to switch the electrical connections. Similarly, the supply of voltage VH or voltage VL to the photoelectric conversion unit 102 may be electrically switched using switches such as transistors.

[0040] Figure 6 schematically shows the relationship between the operation of APD201 and the output signal. The input side of the waveform shaping unit 210 is nodeA, and the output side is nodeB. Between time t0 and time t1, a potential difference of VH-VL is applied to APD201. When a photon is incident on APD201 at time t1, avalanche multiplication occurs in APD201, an avalanche multiplication current flows through the quench element 202, and the voltage at nodeA drops.

[0041] As the voltage drop increases further and the potential difference applied to APD201 decreases, the avalanche multiplication of APD201 stops, as at time t2, and the voltage level at nodeA no longer drops below a certain value. Subsequently, between time t2 and time t3, a current flows through nodeA to compensate for the voltage drop from voltage VL, and at time t3, nodeA settles to its original potential level. At this time, any portion of the output waveform at nodeA that exceeds a certain threshold is shaped by the waveform shaping unit 210 and output as a pulse signal at nodeB.

[0042] Next, the imaging device of the embodiment, consisting of an IR emitter 500, a camera 600, and a mobile unit 700, will be described. Figure 7 is a functional block diagram of the IR emitter 500, camera 600, and mobile unit 700 according to the embodiment. Note that some of the functional blocks shown in Figure 7 are realized by having a computer (not shown) included in the IR emitter 500, camera 600, and mobile unit 700 execute a computer program stored in a memory (not shown) which is a storage medium.

[0043] However, some or all of these may be implemented in hardware. Hardware options include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs). Furthermore, each functional block shown in Figure 7 does not necessarily have to be housed in the same enclosure; they may be composed of separate devices connected to each other via signal paths.

[0044] The camera 600 includes a photoelectric conversion element 100, an imaging optical system 601, an image processing unit 603, a recognition unit 604, a camera control unit 605, a memory unit 606, a communication unit 607, etc., as described in Figures 1 to 6. The photoelectric conversion element 100 is composed of an avalanche photodiode, as described in Figures 1 to 6, for photoelectric conversion of an optical image.

[0045] The imaging device (camera 600, IR emitter 500) of this embodiment is mounted on a mobile body 700, and the camera unit, which consists of an imaging optical system 601 and a photoelectric conversion element 100, is configured to capture images in at least one direction, such as the front, rear, or side of the mobile body. Multiple camera units may be provided on the mobile body 700.

[0046] The image processing unit 603 performs image processing on the image signal acquired by the photoelectric conversion element 100, such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaicing, and data compression, to generate the final image signal.

[0047] The image signals output by the photoelectric conversion element 100 are generated from light transmitted through the RGB filter 31 and IR filter 32 as described in Figure 3, and are therefore R, G, B, and IR signals, respectively. The image processing unit 603 uses the R, G, and B signals to perform demosaicing and other processes to generate a color image (RGB image, first image data). In other words, it generates image data from RGB pixel signals. At this time, the image processing unit 603 may also perform processes such as white balance correction and color conversion. Simultaneously, it uses the IR signal to generate an IR image (monochrome image, second image data). In other words, it generates image data from IR pixel signals. Note that different image processing may be performed for color image generation and IR image generation.

[0048] Furthermore, the output of the image processing unit 603 is supplied to the recognition unit 604, the ECU (Electric Control Unit) 701 of the mobile unit 700, and the camera control unit 605. The recognition unit 604 (recognition processing unit) performs image recognition based on the image signal to recognize objects such as people and vehicles in the surroundings. Deep learning is used for this recognition process. For example, it is preferable to use YOLO (You Only Look Once) as the deep learning method because it is easy to learn and has fast detection. Alternatively, SSD (Single Shot MultiBox Detector) may be used as another deep learning method. Or, Faster R-CNN (Regional Convolution Neural Network), Fast R-CNN, R-CNN, etc. may be used.

[0049] In this embodiment, the recognition unit 604 calculates the distance to the recognized object. One method for measuring the distance is to estimate the distance using deep learning. For example, one method is to calculate the distance value by analyzing information such as the blur of the image of the detected object using deep learning. Another method is to use a stereo camera as the imaging device and measure the distance using the principle of triangulation. This recognition process, including distance estimation, is performed on the color image and IR image input from the image processing unit 603, and the recognition result is output to the subsequent ECU 701.

[0050] In this embodiment, the mobile body 700 is described using the example of an automobile, but the mobile body can be any mobile object such as an aircraft, train, ship, drone, AGV, or robot.

[0051] The camera control unit 605 has a built-in CPU and memory that stores computer programs, and the CPU controls various parts of the camera 600 by executing the computer programs stored in the memory.

[0052] Furthermore, the camera control unit 605 functions as a control means, and for example, via the count enable generation unit 104 for the photoelectric conversion element 100, it controls the length of the exposure period for each frame of the photoelectric conversion element 100 and the timing of the control signal.

[0053] Specifically, the camera control unit 605 transmits a reference signal to the count enable generation unit 104, which is repeatedly output at predetermined intervals. The count enable generation unit 104 uses the reference signal as a timing reference and generates a signal that repeatedly enables and disables at predetermined timings. Here, the count enable generation unit 104 can set the period from the reference signal until the count is enabled, the enable width, the disable width, and the repetition period and number of repetitions of enable and disable. By setting predetermined values ​​for these via a control signal, the camera control unit 605 inputs the count enable signal to the counter circuit 211 at predetermined timings based on the reference signal, and the exposure period of the pixels is controlled. As explained in Figure 4(b), this count enable signal can generate different signals for the R signal, G signal, B signal, and IR signal signal processing circuits 103 using the count enable generation unit 104.

[0054] Furthermore, the camera control unit 605 also transmits the same reference signal to the IR emitter 500 via the communication unit 607. By transmitting the same reference signal to the IR emitter 500 as transmitted to the photoelectric conversion element 100, the IR emitter 500 can perform light emission control based on the reference signal. This makes it possible to synchronize the exposure timing inside the photoelectric conversion element 100 with the light emission timing of the IR emitter 500.

[0055] The storage unit 606 includes, for example, a recording medium such as a memory card or hard disk, and can store and read image signals. The communication unit 607 is equipped with wireless and wired interfaces and outputs the generated image signals to the outside of the camera 600 and receives various signals from the outside. In this embodiment, the communication unit 607 is also connected to the communication unit 503 of the IR emitter 500 and is responsible for transmitting the aforementioned reference signal and control commands from the camera control unit 605 to the IR emitter 500.

[0056] The IR emitter 500 includes an IR light-emitting unit 501, a light-emitting control unit 502, and a communication unit 503.

[0057] The IR light-emitting unit 501 is, for example, a near-infrared LED positioned in front of the moving body 700, and consists of a lens and a light-emitting unit. The light-emitting unit outputs pulsed light for a predetermined emission time in response to a pulse signal output from the light-emitting control unit 502. In other words, it emits pulsed light multiple times within one frame period.

[0058] The light emission control unit 502 receives a reference signal transmitted by the camera control unit 605 of the camera 600 via the communication unit 503, generates a pulse signal at a predetermined timing based on that reference signal, and outputs it to the IR light emission unit 501. Here, the light emission control unit 502 can set the period from the reference signal to the output of the pulse, the pulse output width, the pulse non-output width, and the repetition period and number of repetitions from one pulse output to the next. By setting predetermined values ​​to the light emission control unit 502 via the communication unit 607 and the communication unit 503, the pulse signal is output to the IR light emission unit 501 at a predetermined timing based on the reference signal, and the light emission period of the IR light emitter 500 is controlled. In this way, the light emission control unit 502 controls the light emission based on the same signal as the reference signal input to the photoelectric conversion element 100.

[0059] The communication unit 503 communicates with the communication unit 607 of the camera 600, receives setting information and reference signals from the camera control unit 605 to the light emission control unit 502, and transmits them to the light emission control unit 502.

[0060] The ECU701 has a built-in CPU and memory that stores computer programs, and the CPU executes the computer programs stored in the memory to control various parts of the mobile unit 700.

[0061] The output of the ECU 701 is supplied to the vehicle control unit 702 and the display unit 703. The vehicle control unit 702 functions as a motion control means that controls the driving, stopping, and direction of the vehicle as a moving body based on the output of the ECU 701. The display unit 703 functions as a display means and includes a display element such as a liquid crystal device or an organic EL, and is mounted on the moving body 700.

[0062] In this embodiment, the ECU 701 receives recognition result information from the recognition unit 604 and can perform vehicle stopping control (such as automatic braking) according to the content of the recognition result. The ECU 701 also receives a color image and an IR image from the image processing unit 603 and transmits them to the display unit 703 along with the recognition result.

[0063] Based on the output of the ECU 701, the display unit 703 displays various information to the driver of the mobile unit 700, such as images acquired by the photoelectric conversion element 100, recognition results from the recognition unit 604, and the vehicle's driving status, for example, using a GUI.

[0064] Furthermore, the image processing unit 603, recognition unit 604, etc., shown in Figure 7 do not necessarily have to be mounted on the mobile body 700. For example, they may be installed on an external terminal, etc., provided separately from the mobile body 700, for remotely controlling the mobile body 700 or for monitoring the movement of the mobile body.

[0065] Figure 8 shows the relationship between the propagation of synchrotron radiation from the IR emitter 500 and its reflected light, and the exposure timing of the camera 600. Figure 8 explains a method for acquiring an image of the target distance (range gate image) by synchronizing the emission timing and exposure timing according to the target distance (range gate control). A camera that acquires a target distance image using this range gate control is called a range gate camera. In Figure 8, the horizontal axis represents distance and the vertical axis represents time.

[0066] First, let's explain the horizontal axis. Fog 810 is present between distance x1 and distance x2, and vehicle 820 is present at distance x3. Also, in Figure 8, range gate control is used, starting at position D, and range gate images are acquired within the range width R. In this case, the range width R is the target distance range to be imaged. At this time, vehicle 820 is located within the range width R.

[0067] Next, let's explain the vertical axis. Time 0 is defined as the start time of emission from the IR emitter 500, and time tf is defined as the end time of emission. In this case, the emission period is tf. Furthermore, when acquiring a range gate image within a range width R, starting from position D, the exposure start time is defined as time t1 and the exposure end time as time t2. Time t1 is the timing when the synchrotron radiation emitted from the IR emitter 500 at time 0 returns to the camera 600 as reflected light from distance D. Time t2 is the timing when the synchrotron radiation emitted from the IR emitter 500 at time tf returns to the camera 600 as reflected light from a point that has traveled a range width R from distance D. In addition, time t3 is defined as the timing when the first reflected light from the fog 810 returns to the camera 600. Finally, time t4 is defined as the timing when the last reflected light from the fog 810 returns to the camera 600.

[0068] In range gate control, exposure is not performed during the period from time t3 to time t4 when the reflected light from the fog 810 reaches the camera 600, and exposure is performed only during the period from time t1 to time t2 when the reflected light from distance D reaches the camera with a range width R. This makes it possible to remove the fog 810 while clearly acquiring an image of the vehicle 820.

[0069] Here, we will explain the time it takes for reflected light from an object at distance x to return to camera 600. Let time tr be the timing when the synchrotron radiation, which began to be emitted at time 0, strikes an object at distance x and returns to the imaging unit as reflected light. At this time, the relationship between the timing time tr of the reflected light returning and the distance x from the object to be imaged is given by the following equation.

[0070] Time tr=2x / speed of light c (approximately 3×10^8m / s)...Equation (1) As shown in Figure 8, when the imaging range is defined as the range width R from the distance D, the exposure timing time t1 at the start of the range can be calculated by substituting the distance D into the distance x in equation (1) above, using the following formula.

[0071] Time t1=2D / speed of light c...Equation (2) Furthermore, the exposure timing time t2 at the end of the range can be calculated by substituting the distance D + range width R into the distance x in equation (1) above and adding the time tf, resulting in the following formula.

[0072] Time t2=tf+2(D+R) / speed of light c...Equation (3) By controlling the time tr from light emission to exposure according to the desired imaging distance x (target distance), range gate control is achieved, enabling clear imaging of subjects at the target distance even if there is fog or other obstructions between the camera and the target distance.

[0073] Figure 9 is a timing chart illustrating the control operations for obtaining a color image and a range-gate image per frame time (within a frame period). In this embodiment, the range-gate image is obtained by generating the aforementioned IR image by exposure synchronized with the emission from the IR emitter 500.

[0074] In Figure 9, the vertical synchronization signal indicates the frame period of imaging, with the period between one low pulse and the next low pulse being one frame time. Next, the RGB count enable waveform indicates the start and end timing of the photon count of the count enable signal output by the R count enable generation unit 104r, the G count enable generation unit 104g, and the B count enable generation unit 104b. The RGB count enable waveform is asynchronous with the IR light emission control and is enabled only once within one frame period (first count enable signal). Also, the timing and length of the enable interval within one frame time varies depending on the brightness of the ambient visible light. The RGB counter value indicates the increase or decrease in the photon count of the pixel counter circuit 211 on the RGB filter 31 side. The IR light emission control indicates the emission timing at the IR emitter 500, and the IR count enable indicates the start and end timing of the photon count of the count enable signal output by the IR count enable generation unit 104IR. Since IR light emission control is performed multiple times during one frame period, the IR count enable waveform is enabled multiple times during one frame period (second count enable signal). The IR counter value indicates the increase or decrease in the photon count of the counter circuit 211 on the pixel side of the IR filter 32. The RES signal is a control pulse supplied to the counter circuit 211 via the drive line 213, and the pulse resets the count value that was being held.

[0075] First, let's explain the RGB control for obtaining a color image. In this control, the camera is constantly receiving reflected visible light such as sunlight, so the RGB counter value gradually increases from 0 during the period from when the RGB count enable starts until when it ends. The period from when the RGB count enable starts until when it ends is the exposure time. After the RGB count enable ends, the information of the RGB counter value is sent from the counter circuit 211 to the memory circuit 212, and the RGB counter value is reset by the RES signal. The exposure time from when the RGB count enable starts until when it ends is within one frame time.

[0076] Next, we will explain the range gate control for obtaining a range gate image. In this control, the emission period of IR light is pulsed by the light emission control unit 502, and the number of photons is counted only for the reflected IR light from a specific range.

[0077] Let tf be the emission period from the start to the end of emission, t1 be the time from the start of emission to the start of photon counting, and t2 be the time from the start of emission to the end of photon counting. In this case, t1 represents the period from the start of emission until the light reaches a specific range and the reflected light returns to camera 600. The time from t1 to t2 is the period during which the number of photons of the reflected light in the specific range is counted, and this is the period from the start to the end of IR count enablement.

[0078] During the period from when the IR count enablement starts until when it ends, the IR counter value increases according to the number of photons.

[0079] For range gate control to function correctly, it is necessary to synchronize the timing of light emission start and exposure start according to a predetermined target distance range. In this embodiment, the camera control unit 605 synchronizes the timing by transmitting the same reference signal to the count enable generation unit 104 and the light emission control unit 502.

[0080] The period from the start of one IR light emission to the start of the next, as shown in the timing chart's IR light emission control, constitutes the range gate operation cycle. The IR counter value counted in one range gate operation cycle is retained, and the IR counter value is incremented in the next range gate operation cycle. The period from one emission to the next is set based on the time it takes for the reflected light to sufficiently attenuate and no longer return to camera 600.

[0081] As shown in the diagram, the range gate operation cycle is performed a predetermined number of times within one frame time, and the information of the last added IR counter value within one frame time is sent from the counter circuit 211 to the memory circuit 212. After that, the IR counter value is reset by the RES signal.

[0082] In RGB control, the exposure period is longer than in range gate control, and photons can be accumulated more easily, resulting in good color images even during dark times such as nighttime when the amount of reflected light per unit time is low. Furthermore, because exposure control is not synchronized with light emission as in range gate control, reflected light can be exposed regardless of distance, making it possible to obtain color images of subjects at various distances.

[0083] On the other hand, in range gate control, the exposure period is synchronized with the emission from the IR emitter 500, making it possible to obtain a clear IR image for the target range even in adverse weather conditions such as fog.

[0084] In this embodiment, count enable generation is performed separately for pixels on the RGB filter 31 side and pixels on the IR filter 32 side. This makes it possible to obtain both a color image that is not range-gated and a range-gated IR image simultaneously.

[0085] Next, Figure 10 shows the preferred effects obtained by using the color image and IR image acquired from the photoelectric conversion element 100 in this embodiment. The IR image is acquired by exposure with the IR signal processing circuit 103IR while synchronizing with the light emission timing of the IR light emitter 500. In this embodiment, the camera 600 is mounted in front of the mobile body 700 and is imaging the area in front of the mobile body 700 in the direction of travel.

[0086] Figure 10(a) shows an example of a color image in this embodiment. In Figure 10(a), a pedestrian 830, fog 810, and a vehicle 820 are captured in the image. The color image is then processed by the recognition unit 604, which detects the pedestrian 830, and a pedestrian detection frame 831 is displayed in the image. The recognition unit 604 also performs distance estimation, so a numerical value indicating the distance to the detected object is also displayed in the pedestrian detection frame 831; in this example, the distance from the camera 600 to the pedestrian 830 is 5m.

[0087] On the other hand, although vehicle 820 is beyond fog 810, vehicle 820 is unclear in the color image due to fog 810. Therefore, vehicle 820 is not detected by the recognition processing of the recognition unit 604. In this way, because the color image is captured without range gate control, vehicle 820 and pedestrian 830 at different distances can be captured in a single frame. Furthermore, the image can be obtained as a color image suitable for display and notification, but on the other hand, it becomes an unclear image in bad weather such as fog.

[0088] Next, Figure 10(b) shows an example of an IR image in this embodiment. This image is captured by the range gate control described in Figures 8 and 9 above. In Figure 10(b), the fog 810 and the vehicle 820 are captured. In this embodiment, the target distance captured by the range gate control is set to approximately 40m away from the camera 600. This target distance is set, for example, to a distance at which the moving object 700 can stop safely if it applies the brakes in an emergency. Therefore, it may be adaptively changed depending on the current speed of the moving object 700. By setting it in this way, the moving object 700 can stop safely without colliding with the vehicle 820 after detecting it. In addition, it is important that the camera 600 always keeps an eye on 40m ahead in order to stop safely. In this embodiment, since it is possible to monitor the vicinity of the moving object 700, which is different from 40m ahead, using the aforementioned color image, the IR image can be operated with a setting that always keeps an eye on 40m ahead.

[0089] This IR image is subjected to recognition processing by the recognition unit 604, which detects the vehicle 820, and the vehicle detection frame 821 is displayed in the image. In addition, the recognition unit 604 performs distance estimation, so a numerical value indicating the distance to the detected object is also displayed in the vehicle detection frame 821, and in this example, the distance from camera 600 to vehicle 820 is 40m. Since the IR image is captured by range gate control, the fog 810 is thinned, and vehicle 820 can be captured as a clear image, and vehicle 820 can be detected by the recognition processing of the recognition unit 604.

[0090] Next, Figure 10(c) shows an example of a warning image displayed to the driver of the mobile unit 700 using the display unit 703. This image is generated by the ECU 701 based on Figures 10(a) and 10(b). In Figure 10(c), the base image is first the same as that of Figure 10(a). Therefore, the image in Figure 10(c) is displayed as a color image. In Figure 10(c), a pedestrian detection frame 831, which is the result of recognition processing using the color image, and a vehicle detection frame 821, which is the result of recognition processing using the IR image, are superimposed on it. As a result, the driver who views the image can understand that a vehicle 820 is present beyond the fog 810 by looking at the vehicle detection frame 821. Furthermore, if the ECU 701 determines from the results of this recognition processing that emergency braking is necessary for the mobile unit 700, it instructs the vehicle control unit 702 to apply the brakes and displays a notification 840 to the driver of the mobile unit 700. As a result, the driver can correctly understand that automatic braking is about to be performed.

[0091] In this embodiment, a warning image is generated by superimposing the object detection results obtained by recognizing the color image and the object detection results obtained by recognizing the IR image onto a color image. However, the invention is not limited to this; for example, the image of the detection frame region of the IR image may be extracted and composited onto the color image to generate a warning image. This makes it possible to display the vehicle 820 as a clear image with the fog 810 thinned out on the warning image.

[0092] In this embodiment, a single camera acquires both a color image and an IR image, and these two images have the same field of view. Therefore, it is possible to easily generate a composite image and integrate the detection results of the color image and the IR image onto a single image, as shown in Figure 10(c), without complex calculations.

[0093] Figure 11 is a flowchart detailing the operation in the embodiment. In this flowchart, each step from S101 to S107 is executed sequentially by the CPU, etc., acting as a computer in the camera control unit 605 executing a computer program stored in memory. Similarly, each step from S201 to S206 is executed sequentially by the CPU, etc., acting as a computer in the ECU 701 executing a computer program stored in memory.

[0094] In step S101 of Figure 11, the camera control unit 605 configures the IR emitter 500. Specifically, it configures the light emission control unit 502 inside the IR emitter 500 to generate pulse signals at predetermined timings, setting parameters such as pulse output width, output duration, repetition period, and number of repetitions. As shown in Figure 8, these settings are determined according to the target distance to be imaged in range gate control.

[0095] Next, in step S102, the camera control unit 605 configures the photoelectric conversion element 100. Specifically, it configures various settings on the circuit board 21 inside the photoelectric conversion element 100 for photoelectric conversion from the optical image from the imaging optical system 601 to generate an image signal. In this embodiment, the settings include the period for the count enable generation unit 104 to generate the count enable signal, the enable width, the repetition period, and the number of repetitions. These settings differ for pixels on the RGB filter 31 side and pixels on the IR filter 32 side, and in particular, for pixels on the IR filter 32 side, the IR count enable generation unit 104IR configures the settings to be synchronized with the emission period of the IR light emitter 500. As a result, the resulting IR pixels are range-gate controlled. In this step, the parameter settings for the image processing unit 603 and the recognition unit 604 are also completed.

[0096] Next, in step S103, the camera control unit 605 controls the camera 600 to start imaging. This involves issuing a light emission start command to the IR emitter 500, causing it to begin emitting light. Furthermore, it instructs the photoelectric conversion element 100 to output a vertical synchronization signal, causing exposure and image signal generation to begin. As mentioned above, the light emission from the IR emitter 500 and the exposure (count enable generation) from the photoelectric conversion element 100 are synchronized and controlled based on the reference signal from the camera control unit 605, enabling range gate control.

[0097] Next, in step S104, the camera control unit 605 controls the image processing unit 603 to perform various image processing operations on the image signal output from the photoelectric conversion element 100 to generate the final image signal. Here, a color image is generated using the R signal (R pixel signal), G signal (G pixel signal), and B signal (B pixel signal) output from the photoelectric conversion element 100, and an IR image is generated using the IR signal.

[0098] Next, in step S105, recognition processing is performed on the color image and IR image acquired in step S104 using the recognition unit 604. Through the recognition processing, objects such as people and vehicles in the image are detected, and the distance to the detected objects is estimated. In this embodiment, since both the color image and the IR image acquired by range gate control are obtained simultaneously, it is possible to simultaneously detect objects in adverse weather conditions such as fog and detect objects around the moving body 700 in a single frame.

[0099] Next, in step S106, the color image and IR image acquired in step S104, as well as the recognition results from step S105, are transmitted to the ECU 701 inside the mobile unit 700. The recognition results transmitted include, for example, the name of the detected object, the position and size information of the detection frame, and the distance information of the detected object.

[0100] In step S107, the camera control unit 605 determines whether there is another frame to process. If there is another frame to process, it returns to step S104 and continues processing; if there is no next frame to process, the flowchart on the camera control unit 605 side ends.

[0101] Next, we will explain the processes of steps S201 to S206 executed by the CPU inside the ECU701. In step S201, the ECU701 determines whether or not it has received the image and recognition result. This is the data reception process for the data transmitted in step S106 mentioned above. If the image and recognition result have been received, the process proceeds to step S202.

[0102] Next, in step S202, the ECU 701 determines whether or not an object exists within a predetermined range in front of the camera 600. The predetermined range is, for example, a distance shorter than the range in which the moving body 700 can stop safely without colliding with an object if it performs an emergency braking action. If it is determined that there is no object within the predetermined range, the process proceeds to step S204; if it is determined that there is an object within the predetermined range, the process proceeds to step S203.

[0103] Next, in step S203, the ECU 701 controls the vehicle control unit 702 to perform stopping control of the moving body 700. This avoids collisions between the moving body 700 and objects detected within a predetermined range.

[0104] Next, in step S204, the ECU 701 generates an image to be displayed on the display unit 703. An example of this image is shown in Figure 10(c), which is generated by superimposing the object detection results obtained by recognizing the color image and the object detection results obtained by recognizing the IR image onto the acquired color image.

[0105] Next, in step S205, the image generated in step S204 is displayed on the display unit 703 to notify the driver of the mobile body 700. This allows the driver to understand the object detection results and the execution of the automatic brake.

[0106] In step S206, ECU701 determines whether there is another frame to process. If there is another frame to process, it returns to step S201 and continues processing; if there is no next frame to process, the flowchart on the ECU701 side ends.

[0107] With this embodiment, it is possible to simultaneously obtain, in a single frame, a color image with good visibility of dark areas, acquired with sufficient exposure time from near to far, and an IR image that can clearly capture subjects at a predetermined distance even in bad weather, thanks to range gate control.

[0108] Furthermore, there are SPADs that use a so-called clock recharging method, where a reverse voltage is applied to the photodiode inside the pixel such that avalanche multiplication occurs, and then another avalanche multiplication occurs after an external clock is input. In this method, the avalanche multiplication and counting (exposure) by the counter circuit can be stopped by stopping the supplied clock (so-called clock gating).

[0109] In this embodiment, counting (exposure) was stopped by a count enable signal. However, if a SPAD uses a clock recharging method, the same effect as in this embodiment may be achieved by the clock gating described above. In that case, avalanche multiplication does not occur when counting is stopped, and power consumption can be reduced.

[0110] <Second Embodiment> A second embodiment of the present invention will be described below.

[0111] In the first embodiment, a range gate control method using a camera that exposes with IR light in synchronization with IR emission was described. In the second embodiment, a case in which range gate control is performed using visible light instead of IR light will be described. The functional block diagram in the second embodiment has the same configuration as Figure 7 of the first embodiment, with only the light emitter and the color filter section inside the camera differing from the first embodiment.

[0112] Figure 12 shows an example of the configuration of a color filter 40 in a pixel 101 according to a second embodiment of the present invention. Each pixel 101 within the pixel region 12 has a color filter 40, which is an R filter, a G filter, and a B filter that transmit light of red (R), blue (B), and green (G) wavelengths, respectively. The G filter is divided into a G1 filter and a G2 filter as shown in the figure. A color filter with this configuration is called an R-G1-G2-B filter.

[0113] Here, each pixel 101 corresponds to one of the following color filters: R filter, G1 filter, G2 filter, or B filter. The arrangement in this embodiment is a Bayer-type array, as shown in Figure 12, but the combination of arrangements is not limited to this.

[0114] In this embodiment, the signal processing circuit that handles the light transmitted through the G1 filter of the color filter 40 (the G1 signal after photoelectric conversion) and the signal processing circuit that handles the light transmitted through the G2 filter (the G2 signal after photoelectric conversion) are different. The count enable signals supplied to them are also generated as different signals. This is the same configuration as described in Figure 4(b) of the first embodiment, as each of the signal processing circuits for the R signal, G1 signal, G2 signal, and B signal has a different count enable generation unit, and different count enable signals are connected to them. As a result, in this embodiment, different exposure times can be set using the count enable signals in each of the signal processing circuits for the R signal, G1 signal, G2 signal, and B signal.

[0115] Figure 13 shows an example configuration of the visible light emitter 900, camera 600, and mobile body 700 in the second embodiment. The camera 600 and mobile body 700 have the same configuration as in the first embodiment, except for the color filter 40 mentioned above.

[0116] The visible light emitter 900 includes a visible light emitting unit 901, a light emission control unit 902, and a communication unit 903.

[0117] In a preferred embodiment, the visible light emitting unit 901 is a headlight module having one or more solid-state light-emitting devices, light-emitting diodes (LEDs), or organic LEDs (OLEDs). Therefore, the visible light emitting unit 901 has the function of emitting visible light that can be seen by a person in order for the driver of the moving vehicle 700 to acquire visual information in dark environments such as at night or in tunnels.

[0118] The light emission control unit 902 receives a reference signal transmitted by the camera control unit 605 of the camera 600 via the communication unit 903, generates a pulse signal at a predetermined timing based on that reference signal, and outputs it to the visible light emission unit 901. Here, the light emission control unit 902 can set the period from the reference signal to the output of the pulse, the pulse output width, the pulse non-output width, and the repetition period and number of repetitions from one pulse output to the next. By setting predetermined values ​​to the light emission control unit 902 via the communication unit 607 and the communication unit 903, the pulse signal is output to the visible light emission unit 901 at a predetermined timing based on the reference signal, and the light emission period of the visible light emitter 900 is controlled. In this way, the light emission control unit 902 controls the light emission based on the same signal as the reference signal input to the photoelectric conversion element 100.

[0119] The communication unit 503 communicates with the communication unit 607 of the camera 600, receives setting information and reference signals from the camera control unit 605 to the light emission control unit 902, and transmits them to the light emission control unit 902.

[0120] Figure 14 is a timing chart illustrating the control operation for obtaining a color image and a range-gate image using visible light per frame time. In this embodiment, the range-gate image is obtained by generating the image through exposure synchronized with the emission from the visible light emitter 900.

[0121] In Figure 14, the vertical synchronization signal indicates the frame period of imaging, with the period between one low pulse and the next low pulse being 1 frame time. Next, the R-G1-B count enable waveform indicates the start and end timing of the photon count of the count enable signal output by the count enable generation unit 104 to the signal processing circuits for the R signal, G1 signal, and B signal, respectively. The R-G1-B counter value indicates the increase or decrease in the photon count of the counter circuit 211 inside the signal processing circuits for the R signal, G1 signal, and B signal, respectively. Note that the R-G1-B counter value tends to be higher than in the first embodiment because it includes reflected light from the visible light emitter 900. Visible light emission control indicates the emission timing of the visible light emitter 900, and the G2 count enable indicates the start and end timing of the photon count of the count enable signal output by the count enable generation unit 104 to the signal processing circuit for the G2 signal. The G2 counter value indicates the increase or decrease in the photon count of the counter circuit 211 inside the signal processing circuit for the G2 signal.

[0122] First, let's explain the R-G1-B control for obtaining a color image. In this control, the R-G1-B counter value gradually increases from 0 during the period from when the R-G1-B count enable is started until when it is finished. The period from when the R-G1-B count enable is started until when it is finished is the exposure time. After the R-G1-B count enable is finished, the information of the R-G1-B counter value is sent from the counter circuit 211 to the memory circuit 212, and the R-G1-B counter value is reset by the RES signal. The exposure time from when the R-G1-B count enable is started until when it is finished is within one frame time. The R signal, G1 signal, and B signal generated in this way are demosaiced by the image processing unit 603 to generate a color image (RGB image, first image data).

[0123] Next, in the second embodiment, range gate control for obtaining a range gate image using visible light will be described. In this control, the visible light emission period is pulsed by the light emission control unit 902, and the number of photons is counted only for the reflected visible light from a specific range. Therefore, this embodiment is effective in environments with little ambient light, such as at night.

[0124] Furthermore, the timing control of light emission and exposure (G2 counter enable) for range gate control is the same as the timing control in the first embodiment, and the timing of the start of visible light emission and the start of exposure are synchronized to match a predetermined target distance range.

[0125] In the second embodiment, the visible light emitter 900 is assumed to be a headlight module. In this case, since it not only controls the range gate but also assists in the visibility of the driver of the mobile unit 700, the visible light emission control continues regardless of whether or not there is exposure, i.e., whether or not the G2 counter enable is operating.

[0126] The temporal resolution of the human eye is approximately 50 msec to 100 msec, and flashing lights shorter than this time are perceived as continuous illumination. Therefore, even if pulse emission on the order of nsec, as in this embodiment, is repeated, it will not appear to flicker to the human eye and will not interfere with the driver's visibility. Furthermore, by having an average light output equivalent to that of a continuous light source, it is possible to achieve a light source level equivalent to that of a continuous light source. Thus, range gate control becomes possible in environments with low ambient light without interfering with the driver's visibility.

[0127] As shown in the diagram, the range gate operation cycle is performed a predetermined number of times within one frame time. The information of the last added G2 counter value within one frame time is sent from the counter circuit 211 to the memory circuit 212, and then the G2 counter value is reset by the RES signal. The G2 signal generated in this way is processed separately from the R signal, G1 signal, and B signal by the image processing unit 603, and a monochrome image (second image data) using only the G2 signal is generated.

[0128] With the above control, R-G1-B control allows for a longer exposure period compared to range gate control, and photons can be accumulated more easily, resulting in good color images even during dark times such as nighttime when the amount of reflected light per unit time is low. Furthermore, since exposure control is not synchronized with the emission as in range gate control, reflected light can be exposed regardless of distance, making it possible to obtain color images of subjects at various distances.

[0129] On the other hand, in range gate control, the exposure period is synchronized with the emission from the visible light emitter 900, making it possible to obtain a clear G2 image (monochrome image) for the target range even under adverse weather conditions such as fog.

[0130] In this embodiment, count enable generation is performed individually for the pixels on the R-G1-B side and the pixels on the G2 side. This makes it possible to obtain both a color image that is not range-gate controlled and a range-gate controlled G2 image (monochrome image) simultaneously.

[0131] When this camera is mounted on a vehicle, the installation of an IR emitter becomes unnecessary. Furthermore, by controlling the LED headlight module already installed on the vehicle as a visible light emitter 900, range gate control can be achieved, and the cost of installing an IR emitter can be reduced compared to the first embodiment.

[0132] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the above embodiments may be combined as appropriate. [Explanation of symbols]

[0133] 11 Sensor board 12-pixel area 21 Circuit board 22 Circuit area 30 Color Filters 31 RGB filters 32 IR filters 100 Photoelectric conversion elements 101 pixels 102 Photoelectric conversion unit 103 Signal Processing Circuit 104 Count Enable Generation Unit 210 Waveform shaping section 211 Counter Circuit 212 Memory Circuit 213 Drive Line 500 IR light emitters 501 IR light-emitting section 502 IR Light Emission Control Unit 503 Communications Department 600 Camera 601 Imaging Optical System 603 Signal Processing Unit 604 Recognition part 605 Camera Control Unit

Claims

1. A color filter that transmits light of a specific wavelength, A light-emitting unit that emits IR light or visible light of a wavelength corresponding to the frequency characteristics of the color filter multiple times within one frame period, A sensor unit that emits pulses according to the reception frequency of photons of light transmitted through the color filter, A plurality of pixel units each having a counter for counting the number of pulses and a memory for storing the count value of the counter, A count enable generation unit generates a count enable signal that controls the count period of the counter, An image processing unit generates first image data from pixel signals generated by a count enable signal that is not synchronized with the light emission timing of the light-emitting unit, and generates second image data from pixel signals generated by a count enable signal that is synchronized with the light emission timing of the light-emitting unit. A recognition processing unit that performs recognition processing on the first image data and the second image data, A display unit that displays the recognition result of the first image data and the recognition result of the second image data superimposed on the first image data by the recognition processing unit, Equipped with, The count enable generation unit generates both a first count enable signal, which is asynchronous with the light emission timing of the light-emitting unit and has one enable period within one frame period, and a second count enable signal, which is synchronized with the light emission timing of the light-emitting unit and has multiple enable periods within one frame period. An imaging device characterized by the following features.

2. The count enable generation unit generates the count enable signal based on the light emission timing of the light emission unit and the target distance range of the image target. The imaging apparatus according to feature 1.

3. The count enable generation unit generates the count enable signal as a signal with a different timing for each pixel corresponding to the color filter, and further generates the count enable signal for some of the pixel corresponding to the color filter as a signal synchronized with the light emission timing of the light emission unit. The imaging apparatus according to feature 1.

4. The light-emitting unit emits the IR light, The aforementioned color filter is composed of an IR filter as part of the color filter. The count enable generation unit generates a count enable signal as a signal synchronized with the light emission timing of the light emission unit and supplies it to the pixel unit corresponding to the IR filter. The imaging device according to feature 3.

5. The aforementioned color filter is an RGB-IR filter. The imaging apparatus according to feature 4.

6. The first image data is a color image, and the second image data is a monochrome image. The imaging apparatus according to feature 1.

7. Based on the recognition result of the first image data and the recognition result of the second image data by the recognition processing unit, a composite image of the first image data and the second image data is generated. The imaging apparatus according to feature 1.

8. The light-emitting unit emits visible light, The count enable generation unit generates a count enable signal as a signal synchronized with the light emission timing of the light emission unit and supplies it to some of the pixel units corresponding to the color filter. The imaging device according to feature 3.

9. The aforementioned color filter is an R-G1-G2-B filter. The count enable generation unit supplies a count enable signal to the pixel unit corresponding to either the G1 or G2 color filter, as a signal synchronized with the light emission timing of the light emission unit. The imaging apparatus according to feature 8.

10. The image processing unit generates first image data from either a G1 or G2 pixel signal, an R pixel signal, and a B pixel signal, which are generated by a count enable signal that is not synchronized with the light emission timing of the light-emitting unit, and generates second image data from the other pixel signal, either G1 or G2, which are generated by a count enable signal that is synchronized with the light emission timing of the light-emitting unit. The imaging apparatus according to feature 9.

11. The first image data is a color image, and the second image data is a monochrome image. The imaging apparatus according to feature 10.

12. The system further includes a recognition processing unit that performs recognition processing on the first image data and the second image data. The imaging apparatus according to feature 10.

13. The recognition result of the first image data by the recognition processing unit and the recognition result of the second image data are superimposed on the first image data. The imaging apparatus according to feature 12.

14. Based on the recognition result of the first image data and the recognition result of the second image data by the recognition processing unit, a composite image of the first image data and the second image data is generated. The imaging apparatus according to feature 12.

15. A color filter that transmits light of a specific wavelength, A light-emitting unit that emits IR light or visible light of a wavelength corresponding to the frequency characteristics of the color filter multiple times within one frame period, A sensor unit that emits pulses according to the reception frequency of photons of light transmitted through the color filter, A plurality of pixel units each having a counter for counting the number of pulses and a memory for storing the count value of the counter, A count enable generation unit generates a count enable signal that controls the count period of the counter, An image processing unit generates first image data from pixel signals generated by a count enable signal that is not synchronized with the light emission timing of the light-emitting unit, and generates second image data from pixel signals generated by a count enable signal that is synchronized with the light emission timing of the light-emitting unit. A recognition processing unit that performs recognition processing on the first image data and the second image data, A display unit that displays the recognition result of the first image data and the recognition result of the second image data superimposed on the first image data by the recognition processing unit, An imaging method comprising, The imaging method is characterized in that the count enable generation unit generates both a first count enable signal, which is asynchronous with the light emission timing of the light emission unit and has one enable period within one frame period, and a second count enable signal, which is synchronized with the light emission timing of the light emission unit and has multiple enable periods within one frame period.

Images