Imaging apparatus and imaging method

Abstract

To provide an imaging apparatus capable of simultaneously obtaining an image with excellent dark part visibility while obtaining an image of a target distance area by range gate control.SOLUTION: A plurality of pixel units each including a color filter that transmits light of a specific wavelength, a light emitting unit that emits light a plurality of times within one frame period, a sensor unit that emits a pulse according to a light reception frequency of a photon, a counter that counts the number of pulses, and a memory that stores a value of the count; The count enable generation section generates either a first count enable signal which is asynchronous with a light emission timing of the light emitting section and has one enable period within one frame period, or a second count enable signal which is synchronous with the light emission timing of the light emitting section and has a plurality of enable periods within one frame period.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to an imaging device and imaging method for generating different count enable signals according to color filters to acquire an RGB image and a range gate image. [Background technology]

[0002] There is a photography method using a camera called a range gate camera. This is a technology that emits pulsed light at a predetermined cycle in front of the camera and exposes the image sensor inside the camera at a predetermined timing according to the target distance, making it possible to clearly capture only subjects at the target distance. Hereinafter, this technology will be referred to as range gate control. This range gate control makes it possible to clearly capture subjects (objects) at a predetermined distance even in bad weather, for example.

[0003] For example, Patent Document 1 describes a technology that uses a timing controller to adjust the timing of pulsed light emission and camera exposure to prevent capturing images of unnecessary target distance areas. This patent document also describes a configuration that makes it possible to obtain images of multiple target distance areas at different distances from the camera by changing the timing (delay time) of pulsed light emission and camera exposure for each frame. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] WO17 / 110417 Public Relations Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology described in Patent Document 1 has the problem that the target distance area is changed for each frame, and it takes the time of multiple frames to capture multiple distance areas from near to far. Therefore, when used in an application that requires immediacy, such as collision prevention, there is a possibility that the recognition of an object (vehicle, etc.) ahead using the image will be delayed, which may delay the collision prevention measures (automatic braking, etc.).

[0006] Furthermore, because the image sensor inside the camera is exposed at a predetermined timing according to the target distance, the exposure time is limited to a time according to the target distance. Therefore, the exposure time within a predetermined time tends to be shorter than the exposure time without range gate control. In this case, even if repeated exposures are performed, the overall amount of reflected light from the object accumulated within a predetermined time is small. Therefore, it may be difficult to recognize an object ahead from the image during dark times such as at night when the amount of reflected light per unit time is small.

[0007] Therefore, an object of the present invention is to provide an imaging device that can obtain an image of a target distance region by range gate control, while also obtaining an image with good visibility in dark areas. [Means for solving the problem]

[0008] An imaging device according to one aspect of the present invention comprises a plurality of pixel units each including a color filter that transmits light of a specific wavelength, a light-emitting unit that emits light of a wavelength corresponding to the frequency characteristics of the color filter multiple times within one frame period, a sensor unit that emits pulses in accordance with the frequency of receiving photons of light that has passed through the color filter, a counter that counts the number of pulses, and a memory that stores the count value of the counter; and a count enable generation unit that generates a count enable signal that controls the count period of the counter, wherein the count enable generation unit generates either a first count enable signal that is asynchronous with the light-emitting timing of the light-emitting unit and has one enable period within one frame period, or a second count enable signal that is synchronized with the light-emitting timing of the light-emitting unit and has multiple enable periods within one frame period. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an imaging device that can clearly capture an image of a subject at a predetermined distance even in bad weather, and that can also simultaneously capture images with good visibility in dark areas from close to far away without requiring multiple frames. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of the configuration of a photoelectric conversion element 100 according to an embodiment. [Figure 2] 2 is a diagram showing an example of the configuration of a sensor substrate 11. FIG. [Figure 3] 3 is a diagram showing an example of the configuration of a color filter 30 in the first embodiment. FIG. [Figure 4] 2 is a diagram showing an example of the configuration of a circuit board 21. FIG. [Figure 5] 3 is a diagram showing an equivalent circuit of a pixel 101 in FIG. 2 and a signal processing circuit 103 corresponding to the pixel 101. FIG. [Figure 6] 2 is a diagram schematically illustrating the relationship between the operation of an APD 201 and an output signal. FIG. [Figure 7]1 is a functional block diagram of an IR emitter 500, a camera 600, and a moving object 700 according to an embodiment. [Figure 8] 5A and 5B are diagrams illustrating the relationship between the travel of reflected light and exposure timing according to the embodiment. [Figure 9] 5A and 5B are diagrams illustrating light emission and exposure control operations for one frame time in the first embodiment. [Figure 10] 7A and 7B are diagrams showing an example of a display image on a display unit 703 in the embodiment. [Figure 11] 10 is a flowchart showing details of an operation example in the embodiment. [Figure 12] FIG. 10 is a diagram showing an example of the configuration of a color filter 40 in a second embodiment. [Figure 13] FIG. 10 is a diagram illustrating an example of the configuration of a visible light emitter 900 according to a second embodiment. [Figure 14] 10A and 10B are diagrams illustrating light emission and exposure control operations for one frame time in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] First Embodiment Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions will be omitted or simplified.

[0012] FIG. 1 is a diagram showing an example of the configuration of a photoelectric conversion element according to an embodiment of the present invention. The following description will be given taking as an example a photoelectric conversion device having a so-called stacked structure, in which the photoelectric conversion element 100 is configured by stacking and electrically connecting two substrates, a sensor substrate 11 and a circuit substrate 21. However, the photoelectric conversion element 100 may have a so-called non-stacked structure, in which the components included in the sensor substrate and the components included in the circuit substrate are arranged on a common semiconductor layer. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes signals detected in the pixel region 12.

[0013] 2 is a diagram showing an example of the configuration of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 (pixel units) arranged two-dimensionally across multiple rows and columns. Each pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, referred to as APD).

[0014] Here, the photoelectric conversion unit 102 functions as a sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons. The number of rows and columns of the pixel array that forms the pixel region 12 is not particularly limited.

[0015] FIG. 3 is a diagram showing an example of the configuration of a color filter 30 included in a pixel 101. Each pixel 101 in the pixel region 12 has one of the colors of the color filter 30. In other words, the color filter has one of the frequency characteristics. The color filter 30 is roughly divided into the following two types of filters. The first type is an R filter, a G filter, and a B filter (collectively referred to as an RGB filter 31) that transmit light of red (R), blue (B), and green (G) wavelengths, respectively. The second type is an IR filter 32, which is an infrared region filter that transmits light of infrared (IR) wavelengths. A color filter with this configuration is called an RGB-IR filter.

[0016] Here, one pixel 101 corresponds to one of the color filters, R filter, G filter, B filter, and IR filter. As shown in Fig. 3, the arrangement in this embodiment is a combination of a row in which B filters and G filters are alternately arranged and a row in which IR filters and R filters are alternately arranged. Note that the arrangement in which the RGB filters 31 and IR filters 32 are combined is not limited to this.

[0017] 4(a) is a diagram showing an example of the configuration of the circuit board 21. The circuit board 21 has a signal processing circuit 103 that processes charges photoelectrically converted by each photoelectric conversion unit 102 in FIG. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114.

[0018] The vertical scanning circuit 110 receives control pulses supplied from the control pulse generating unit 115 and sequentially supplies the control pulses to a plurality of pixels arranged in the row direction. The vertical scanning circuit 110 uses logic circuits such as a shift register and an address decoder.

[0019] The signal output from the photoelectric conversion unit 102 of each pixel is processed by each signal processing circuit 103. The signal processing circuit 103 is provided with a counter, memory, etc., and digital values ​​are stored in the memory. In order to read the signal from the memory of each pixel where the digital signal is stored, the horizontal scanning circuit 111 inputs a control pulse that sequentially selects each column to the signal processing circuit 103.

[0020] A signal is output to the vertical signal line 113 from the signal processing circuit 103 of the pixel of the row selected by the vertical scanning circuit 110. The signal output to the vertical signal line 113 is output to the outside of the photoelectric conversion element 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has a plurality of buffers built in and connected to the vertical signal line 113.

[0021] 2 and 4(a), a plurality of signal processing circuits 103 are arranged in an area overlapping the pixel area 12 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged so as to overlap between an end of the sensor substrate 11 and an end of the pixel area 12 in a plan view.

[0022] In other words, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in a region overlapping the non-pixel region in a plan view.

[0023] The arrangement of the vertical signal lines 113, the readout circuits 112, and the output circuits 114 is not limited to the example shown in Fig. 4(a). For example, the vertical signal lines 113 may be arranged extending in the row direction, and the readout circuits 112 may be arranged at the ends of the vertical signal lines 113. Furthermore, it is not necessary to provide one signal processing circuit 103 for each photoelectric conversion unit, and a configuration may be adopted in which one signal processing unit is shared by multiple photoelectric conversion units and performs signal processing sequentially.

[0024] Fig. 4(b) is a diagram showing an example of the configuration of the count enable generation unit 104 that generates signals to be supplied to each signal processing circuit 103 described in Fig. 4(a). In Fig. 4(b), the signal processing circuit 103 that processes light that has passed through the R filter of the RGB filter 31 (R signal after photoelectric conversion) is the R signal processing circuit 103r. Similarly, the signal processing circuit that processes G signals is the G signal processing circuit 103g, and the signal processing circuit that processes B signals is the B signal processing circuit 103b. Furthermore, the signal processing circuit that processes light that has passed through the IR filter 32 (IR signal after photoelectric conversion) is the IR signal processing circuit 103IR.

[0025] The count enable generation unit 104 generates a count enable signal to be supplied to a counter within the signal processing circuit 103. The count enable signal is a signal for controlling the enable / disable state of the counter within the signal processing circuit 103. This count enable signal is configured so that different signals can be generated for the signal processing circuits 103 for R, G, B, and IR signals. That is, an R count enable generation unit 104r generates a count enable signal to be supplied to the multiple R signal processing circuits 103r. Similarly, a G count enable generation unit 104g and a B count enable generation unit 104b generate count enable signals for the G signal processing circuit 103g and the B signal processing circuit 103b, respectively. An IR count enable generation unit 104IR generates count enable signals to be supplied to the multiple IR signal processing circuits 103IR. It is also possible to generate the R count enable, G count enable, B count enable, and IR count enable signals at independent timings. In FIG. 4(b), the connection between the signal processing circuit 103 and the count enable generation unit 104 is omitted.

[0026] FIG. 5 is a diagram showing an equivalent circuit of the pixel 101 in FIGS. 2 and 4(a) and the signal processing circuit 103 corresponding to the pixel 101. In FIG.

[0027] The APD 201 included in the photoelectric conversion unit 102 generates charge pairs in response to incident light through photoelectric conversion. One of the two nodes of the APD 201 is connected to a power supply line that supplies a drive voltage VL (first voltage). The other of the two nodes of the APD 201 is connected to a power supply line that supplies a drive voltage VH (second voltage) that is higher than the voltage VL.

[0028] In Figure 5, one node of the APD 201 is the anode, and the other node of the APD is the cathode. A reverse bias voltage is supplied to the anode and cathode of the APD 201 so that the APD 201 performs avalanche multiplication. With this voltage supplied, the charge generated by incident light undergoes avalanche multiplication, generating an avalanche current.

[0029] When a reverse bias voltage is supplied, there are two modes: Geiger mode, in which the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, in which the voltage difference between the anode and cathode is close to or less than the breakdown voltage. An APD operating in Geiger mode is called a SPAD. In the case of a SPAD, for example, the voltage VL (first voltage) is -30V and the voltage VH (second voltage) is 1V.

[0030] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212. The quench element 202 is connected to a power supply line to which a drive voltage VH is supplied and one of the anode and cathode nodes of the APD 201.

[0031] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, suppressing the voltage supplied to the APD 201 and suppressing avalanche multiplication (quench operation).The quench element 202 also functions to return the voltage supplied to the APD 201 to the drive voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0032] 5 shows an example in which the signal processing circuit 103 includes a waveform shaping section 210, a counter circuit 211, and a memory circuit 212 in addition to the quench element 202. In FIG.

[0033] The waveform shaping unit 210 shapes the voltage change at the cathode of the APD 201 obtained when photons are detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. While Fig. 5 shows an example in which one inverter is used as the waveform shaping unit 210, a circuit in which multiple inverters are connected in series, or another circuit with a waveform shaping effect, may also be used.

[0034] Counter circuit 211 counts the number of pulses output from waveform shaping unit 210 and holds the count value. When control pulse RES is supplied via drive line 213, the signal held in counter circuit 211 is reset. Here, counter circuit 211 generates a signal based on the difference between the count values ​​at the start and end of the accumulation period.

[0035] The counter circuit 211 receives a count enable signal from the count enable generation unit 104 described in FIG. 4B. When this count enable signal is high (count period), the counter circuit 211 counts the number of pulses output from the waveform shaping unit 210. When the count enable signal is low, the counter circuit 211 retains the count value without counting the number of pulses. For example, if the count enable generation unit 104 is a circuit that operates at a clock frequency of 100 MHz, the count enable signal can be controlled to be high or low (enable or disable) in 10-nsec increments of the clock cycle. Here, the number of pulses output by the photoelectric conversion unit 102 in accordance with the frequency of photon reception is counted only when count is enabled, so the count enable period for the number of pulses can be rephrased as the exposure period of the pixel 101. Therefore, in this embodiment, switching between exposure and non-exposure can be controlled in 10-nsec increments.

[0036] Also, as shown in Figure 4(b), the count enable signal can be generated by the count enable generation unit 104 as different signals for the R signal processing circuit 103r, the G signal processing circuit 103g, the B signal processing circuit 103b, and the IR signal processing circuit 103IR.

[0037] In this embodiment, different count enable signals are generated by the R signal processing circuit 103r, the G signal processing circuit 103g, the B signal processing circuit 103b, and the IR signal processing circuit 103IR. Details of the generated signals will be described later using timing charts.

[0038] A control pulse SEL is supplied to the memory circuit 212 from the vertical scanning circuit 110 in FIG. 4(a) via a drive line 214 (not shown in FIG. 4(a)) in FIG. 5, and switches between electrical connection and disconnection between the counter circuit 211 and the vertical signal line 113. The memory circuit 212 functions as a memory that temporarily stores the count value of the counter, and outputs an output signal from the counter circuit 211 of the pixel to the vertical signal line 113.

[0039] Note that electrical connections may be switched by disposing a switch such as a transistor between the quench element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103. Similarly, the supply of the voltage VH or the voltage VL to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0040] 6 is a diagram schematically illustrating the relationship between the operation of the APD 201 and the output signal. The input side of the waveform shaping unit 210 is designated node A, and the output side is designated node B. Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage at node A drops.

[0041] As the voltage drop increases further and the potential difference applied to APD 201 decreases, avalanche multiplication in APD 201 stops, as shown at time t2, and the voltage level at node A no longer drops below a certain value. Between time t2 and time t3, a current flows from voltage VL to node A to compensate for the voltage drop, and at time t3, node A stabilizes to its original potential level. At this time, the portion of the output waveform at node A that exceeds a certain threshold is shaped by waveform shaping unit 210 and output as a pulse signal at node B.

[0042] Next, an IR emitter 500, a camera 600, and a mobile object 700, which are image capture devices according to an embodiment, will be described. Fig. 7 is a functional block diagram of the IR emitter 500, the camera 600, and the mobile object 700 according to an embodiment. Note that some of the functional blocks shown in Fig. 7 are realized by causing a computer (not shown) included in the IR emitter 500, the camera 600, and the mobile object 700 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0043] However, some or all of these functions may be implemented by hardware. Examples of hardware that can be used include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs). Furthermore, the functional blocks shown in Fig. 7 do not have to be built into the same housing, and may be configured as separate devices connected to each other via signal paths.

[0044] The camera 600 includes the photoelectric conversion element 100 described in Figures 1 to 6, an imaging optical system 601, an image processing unit 603, a recognition unit 604, a camera control unit 605, a storage unit 606, and a communication unit 607. The photoelectric conversion element 100 is configured with an avalanche photodiode described in Figures 1 to 6 for photoelectrically converting an optical image.

[0045] The imaging device (camera 600, IR emitter 500) of the embodiment is mounted on a moving body 700, and the camera unit, which is a set of an imaging optical system 601 and a photoelectric conversion element 100, is configured to capture images in at least one direction, for example, in front, behind, or to the side of the moving body. Note that a plurality of camera units may be provided on the moving body 700.

[0046] The image processing unit 603 performs image processing such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaic processing, and data compression on the image signal acquired by the photoelectric conversion element 100, and generates a final image signal.

[0047] The image signals output by the photoelectric conversion element 100 are generated from light transmitted through the RGB filter 31 and IR filter 32 described in FIG. 3, and are therefore R, G, B, and IR signals, respectively. The image processing unit 603 uses the R, G, and B signals to perform demosaic processing and the like to generate a color image (RGB image, first image data). That is, image data is generated from the RGB pixel signals. At this time, the image processing unit 603 may also perform processing such as white balance correction and color conversion. At the same time, the IR signal is used to generate an IR image (monochrome image, second image data). That is, image data is generated from the IR pixel signal. Note that different image processing may be performed for generating the color image and the IR image.

[0048] The output of the image processing unit 603 is supplied to a recognition unit 604, an ECU (Electric Control Unit) 701 of the moving object 700, and a camera control unit 605. The recognition unit 604 (recognition processing unit) performs image recognition based on the image signal to recognize objects such as people and vehicles in the vicinity. Deep learning is used for this recognition processing. For example, it is preferable to use YOLO (You Only Look Once) as deep learning, which is easy to learn and has fast detection speed. Alternatively, SSD (Single Shot MultiBox Detector) may be used as another deep learning method. Alternatively, Faster R-CNN (Regional Convolution Neural Network), Fast R-CNN, R-CNN, etc. may be used.

[0049] In this embodiment, the recognition unit 604 calculates the distance to the recognized object. One distance measurement method is, for example, a method of estimating distance using deep learning. One example is a method of calculating a distance value by analyzing information such as blur in an image of a detected object using deep learning. Another method is a method of measuring distance using a stereo camera as the imaging device and the principle of triangulation. This recognition process, including distance estimation, is performed on each of the color image and IR image input from the image processing unit 603, and the recognition result is output to the downstream ECU 701.

[0050] In this embodiment, the mobile body 700 is described using an example of an automobile, but the mobile body may be any mobile body such as an airplane, train, ship, drone, AGV, or robot.

[0051] The camera control unit 605 has a built-in CPU as a computer and memory that stores a computer program, and controls each unit of the camera 600 by the CPU executing the computer program stored in the memory.

[0052] The camera control unit 605 functions as a control means, and controls the length of the exposure period of each frame of the photoelectric conversion element 100 and the timing of the control signal, for example, via the count enable generation unit 104 of the photoelectric conversion element 100.

[0053] Specifically, the camera control unit 605 transmits a reference signal to the count enable generation unit 104, which is repeatedly output at predetermined intervals. The count enable generation unit 104 uses the reference signal as a timing reference and generates a signal that repeatedly enables and disables at predetermined timing. Here, the count enable generation unit 104 can set the period from the reference signal until the count is enabled, the enable width, the disable width, the enable and disable repetition period, and the number of repetitions. By setting predetermined values ​​for these via control signals, the camera control unit 605 inputs the count enable signal to the counter circuit 211 at predetermined timing based on the reference signal, thereby controlling the exposure period of the pixel. Note that, as described in FIG. 4(b), different count enable signals can be generated by the count enable generation unit 104 for the signal processing circuits 103 for the R signal, G signal, B signal, and IR signal, respectively.

[0054] The camera control unit 605 also transmits the same signal as the reference signal described above to the IR emitter 500 via the communication unit 607. By transmitting the same reference signal as that transmitted to the photoelectric conversion element 100 to the IR emitter 500 in this way, the IR emitter 500 can perform light emission control based on the reference signal. This makes it possible to synchronize the exposure timing inside the photoelectric conversion element 100 and the light emission timing by the IR emitter 500.

[0055] The storage unit 606 includes a recording medium such as a memory card or a hard disk, and can store and read image signals. The communication unit 607 has a wireless or wired interface, and outputs the generated image signal to the outside of the camera 600 and receives various signals from the outside. In this embodiment, the communication unit 607 is connected to the communication unit 503 of the IR emitter 500, and also plays a role in transmitting the above-mentioned reference signal and control commands from the camera control unit 605 to the IR emitter 500.

[0056] The IR emitter 500 includes an IR emitter 501 , an emission control unit 502 , and a communication unit 503 .

[0057] The IR light emitter 501 is, for example, a near-infrared LED placed in front of the moving object 700, and is composed of a lens and a light emitter. The light emitter outputs pulsed light for a predetermined light emission time in response to a pulse signal output from the light emission control unit 502. In other words, the light emitter emits pulsed light multiple times within one frame period.

[0058] The light emission control unit 502 receives a reference signal transmitted by the camera control unit 605 of the camera 600 via the communication unit 503, generates a pulse signal at a predetermined timing based on the reference signal, and outputs the pulse signal to the IR emitter 501. The light emission control unit 502 can set the period from the reference signal to the pulse output, the pulse output width, the pulse non-output width, the repetition cycle and the number of repetitions from one pulse output to the next, and other parameters. The camera control unit 605 sets predetermined values ​​for the light emission control unit 502 via the communication units 607 and 503, so that a pulse signal is output to the IR emitter 501 at a predetermined timing based on the reference signal, thereby controlling the light emission period of the IR emitter 500. In this way, the light emission control unit 502 controls light emission based on the same signal as the reference signal input to the photoelectric conversion element 100.

[0059] The communication unit 503 communicates with the communication unit 607 of the camera 600 , receives setting information and a reference signal from the camera control unit 605 to the light emission control unit 502 , and transmits them to the light emission control unit 502 .

[0060] The ECU 701 incorporates a CPU as a computer and a memory that stores a computer program, and controls each part of the moving object 700 by the CPU executing the computer program stored in the memory.

[0061] The output of the ECU 701 is supplied to a vehicle control unit 702 and a display unit 703. The vehicle control unit 702 functions as a movement control means that drives, stops, controls the direction, etc. of the vehicle as a moving object based on the output of the ECU 701. The display unit 703 functions as a display means, and includes display elements such as a liquid crystal device or an organic EL, and is mounted on the moving object 700.

[0062] In this embodiment, the ECU 701 receives information on the recognition result from the recognition unit 604, and can execute vehicle stop control (automatic braking, etc.) depending on the content of the recognition result. The ECU 701 also receives color images and IR images from the image processing unit 603, and transmits them to the display unit 703 together with the recognition result.

[0063] The display unit 703 displays, for example, using a GUI based on the output of the ECU 701, to the driver of the vehicle 700 images acquired by the photoelectric conversion element 100, the recognition results by the recognition unit 604, and various information related to the vehicle's driving state, etc.

[0064] 7 may not be mounted on the moving body 700. For example, they may be provided in an external terminal or the like that is provided separately from the moving body 700 and that is used to remotely control the moving body 700 or monitor the traveling of the moving body.

[0065] Figure 8 is a diagram showing the relationship between the propagation of emitted light from IR emitter 500 and its reflected light and the exposure timing of camera 600. Figure 8 explains a method for acquiring an image (range gate image) of a target distance by performing control (range gate control) that synchronizes the emission timing and exposure timing according to the target distance. A camera that acquires a target distance image using range gate control in this way is called a range gate camera. In Figure 8, the horizontal axis represents distance and the vertical axis represents time.

[0066] First, let's explain the horizontal axis. Fog 810 exists between distance x1 and distance x2, and vehicle 820 exists at distance x3. In addition, in FIG. 8, range gate control sets the position of distance D as the starting point, and acquires a range gate image within a range width R from there. In this case, range width R is the target distance range to be imaged. At this time, vehicle 820 exists within range width R.

[0067] Next, the vertical axis will be described. Time 0 is the timing when the IR emitter 500 starts emitting light, and time tf is the timing when the light emission ends. At this time, the light emission period is tf. Furthermore, when a range gate image is acquired from the distance D as the starting point within the range width R, the exposure start time is time t1 and the exposure end time is time t2. Time t1 is the timing when the light emitted from the IR emitter 500 at time 0 returns to the camera 600 as reflected light from the distance D. Furthermore, time t2 is the timing when the light emitted from the IR emitter 500 at time tf returns to the camera 600 as reflected light from a point traveling the range width R from the distance D. Furthermore, time t3 is the timing when the first reflected light from the fog 810 returns to the camera 600. Finally, time t4 is the timing when the last reflected light from the fog 810 returns to the camera 600.

[0068] In range gate control, exposure is not performed during the period from time t3 to time t4 when reflected light from fog 810 reaches camera 600, but exposure is performed only during the period from time t1 to time t2 when reflected light of range width R reaches camera 600 from distance D. This makes it possible to remove fog 810 while still obtaining a clear image of vehicle 820.

[0069] Here, we will explain the time it takes for reflected light from an object located at distance x to return to camera 600. The timing at which radiated light, which starts to be emitted at time 0, hits an object located at distance x, and returns to the imaging unit as reflected light is defined as time tr. In this case, the relationship between the timing tr at which the reflected light returns and the distance x to the imaging object is expressed by the following equation:

[0070] Time tr=2x / speed of light c (approximately 3×10^8m / s)...Equation (1) As shown in FIG. 8, when the imaging range is from the distance D to the range width R, the exposure timing time t1 at the start of the range can be calculated by substituting the distance D for the distance x in the above equation (1) using the following equation.

[0071] Time t1=2D / speed of light c...Equation (2) Furthermore, the exposure timing time t2 at the end of the range can be calculated by substituting the distance D+range width R for the distance x in the above formula (1) and adding the time tf, as shown below.

[0072] Time t2=tf+2(D+R) / speed of light c...Equation (3) In this way, by controlling the time tr from light emission to exposure according to the distance x (target distance) to be imaged, range gate control is realized that can clearly image a subject at the target distance even if there is fog or the like between the camera and the target distance.

[0073] 9 is a timing chart illustrating the control operation for obtaining a color image and a range gate image per frame time (within one frame period). In this embodiment, the range gate image is obtained by generating the IR image described above by exposure synchronized with light emission from the IR emitter 500.

[0074] In Figure 9, the vertical synchronization signal indicates the frame period of imaging, and the period between one low pulse and the next low pulse is one frame time. Next, the RGB count enable waveform indicates the timing of the start and end of photon counting for the count enable signal output by the R count enable generation unit 104r, the G count enable generation unit 104g, and the B count enable generation unit 104b. The RGB count enable waveform is asynchronous with the IR light emission control and is enabled only once per frame period (first count enable signal). Furthermore, the timing and length of the enable section within one frame time vary depending on the brightness of the ambient visible light. The RGB counter value indicates the increase or decrease in the photon count of the counter circuit 211 of the pixel on the RGB filter 31 side. The IR light emission control indicates the emission timing of the IR emitter 500, and the IR count enable indicates the timing of the start and end of photon counting for the count enable signal output by the IR count enable generation unit 104IR. Since IR light emission control is performed multiple times per frame period, the IR count enable waveform is enabled multiple times per frame period (second count enable signal). The IR counter value indicates the increase or decrease in the number of photons counted by the counter circuit 211 of the pixel on the IR filter 32 side. The RES signal is a control pulse supplied to the counter circuit 211 via the drive line 213, and the count value held by the pulse is reset.

[0075] First, we will explain the RGB control for obtaining a color image. In this control, the camera is constantly exposed to reflected visible light such as sunlight, so the RGB counter value gradually increases from 0 during the period from the start to the end of the RGB count enable. The period from the start to the end of the RGB count enable is the exposure time. After the RGB count enable ends, information about the RGB counter value is sent from the counter circuit 211 to the memory circuit 212, and the RGB counter value is reset by the RES signal. The exposure time from the start to the end of the RGB count enable is performed within one frame time.

[0076] Next, we will explain range gate control for obtaining a range gate image. In this control, the IR light emission period is controlled in a pulsed manner by the light emission control unit 502, and the number of photons is counted only for the IR light reflected from a specific range.

[0077] The light emission period from the start to the end of light emission is defined as tf, the time from the start of light emission to the start of photon counting is defined as t1, and the time from the start of light emission to the end of photon counting is defined as t2. In this case, t1 indicates the period from the start of light emission until light reaches a specific range and the reflected light returns to the camera 600. The period from t1 to t2 is the period during which the number of photons of reflected light in a specific range is counted, and is the period from the start to the end of IR count enable.

[0078] During the period from the start to the end of the IR count enable, the IR counter value increases according to the number of photons.

[0079] In order to perform range gate control correctly, it is necessary to synchronize the timing of the start of light emission and the start of exposure according to the range of a predetermined target distance. In this embodiment, the camera control unit 605 synchronizes the timing by sending the same reference signal to the count enable generation unit 104 and the light emission control unit 502.

[0080] As shown in the timing chart for IR light emission control, the period from the start of light emission to the start of the next light emission is a range gate operation cycle. The IR counter value counted in one range gate operation cycle is maintained, and the IR counter value is incremented in the next range gate operation cycle. The period from one light emission to the next light emission is set based on the time it takes for the reflected light to sufficiently attenuate and no longer return to the camera 600.

[0081] As shown in the figure, the range gate operation cycle is performed a predetermined number of times within one frame time, and information on the IR counter value last added within one frame time is sent from the counter circuit 211 to the memory circuit 212. Thereafter, the IR counter value is reset by the RES signal.

[0082] With RGB control, the exposure period is longer than with range gate control, and photons are more easily accumulated, so good color images can be obtained even in dark times such as at night when the amount of reflected light per unit time is low. Also, because exposure control synchronized with light emission is not performed as with range gate control, it is possible to expose reflected light regardless of distance, making it possible to obtain color images of subjects at various distances.

[0083] On the other hand, in range gate control, the exposure period is synchronized with the light emitted by the IR emitter 500, so it is possible to obtain a clear IR image at the targeted range even in bad weather such as fog.

[0084] In this manner, in this embodiment, count enable signals are generated separately for the pixels on the RGB filter 31 side and the pixels on the IR filter 32 side. This makes it possible to simultaneously obtain a color image that is not range gate controlled and an IR image that is range gate controlled.

[0085] 10 is a diagram showing a preferable effect obtained by using the color image and the IR image acquired from the photoelectric conversion element 100 in this embodiment. The IR image is acquired by exposing the IR signal processing circuit 103IR in synchronization with the light emission timing of the IR emitter 500. In this embodiment, the camera 600 is attached to the front of the mobile object 700 and captures an image of the area ahead in the traveling direction of the mobile object 700.

[0086] Fig. 10(a) shows an example of a color image in this embodiment. In Fig. 10(a), a pedestrian 830, fog 810, and a vehicle 820 are captured. This color image is then subjected to recognition processing by the recognition unit 604, which detects the pedestrian 830 and displays a pedestrian detection frame 831 in the image. Since the recognition unit 604 also performs distance estimation, the pedestrian detection frame 831 also displays a numerical value indicating the distance to the detected object; in this example, the distance from the camera 600 to the pedestrian 830 is 5 m.

[0087] On the other hand, although there is a vehicle 820 beyond the fog 810, the vehicle 820 is unclear in the color image due to the fog 810. As a result, the vehicle 820 cannot be detected by the recognition process of the recognition unit 604. In this way, because the color image is captured without range gate control, the vehicle 820 and the pedestrian 830, which are at different distances, can be captured in one frame. Furthermore, although the image can be obtained as a color image suitable for display or notification, the image becomes unclear in bad weather such as fog.

[0088] Next, FIG. 10(b) shows an example of an IR image in this embodiment. This image was captured using the range gate control described above with reference to FIGS. 8 and 9. In FIG. 10(b), fog 810 and a vehicle 820 are captured. In this embodiment, the target distance captured using range gate control is set to approximately 40 m from the camera 600. This target distance is set, for example, as a distance at which the moving body 700 can safely stop if it brakes urgently. Therefore, the target distance may be adaptively changed depending on the current speed of the moving body 700, etc. By setting the target distance in this manner, the moving body 700 can safely stop after detecting the vehicle 820 without colliding with the vehicle 820. Furthermore, it is important for the camera 600 to always focus 40 m ahead to safely stop. In this embodiment, the color image described above can be used to monitor the surroundings of the moving body 700, which are different from the surroundings of the moving body 700, so the IR image can be set to always focus 40 m ahead.

[0089] This IR image is subjected to recognition processing by the recognition unit 604, which detects a vehicle 820 and displays a vehicle detection frame 821 in the image. In addition, since the recognition unit 604 performs distance estimation, a numerical value indicating the distance to the detected object is also written in the vehicle detection frame 821, which in this example is the distance of 40 m from the camera 600 to the vehicle 820. Because the IR image is captured using range gate control, the fog 810 thins and the vehicle 820 can be captured as a clear image, and the recognition processing by the recognition unit 604 is also able to detect the vehicle 820.

[0090] Next, FIG. 10(c) shows an example of a warning image displayed to the driver of the moving object 700 using the display unit 703. This image is generated by the ECU 701 based on FIGS. 10(a) and 10(b). In FIG. 10(c), the entire base image is firstly FIG. 10(a). Therefore, the image in FIG. 10(c) is displayed as a color image. In FIG. 10(c), a pedestrian detection frame 831, which is the result of recognition processing using the color image, and a vehicle detection frame 821, which is the detection result of recognition processing using the IR image, are superimposed on the image. This allows the driver viewing the image to recognize the presence of a vehicle 820 beyond the fog 810 through the vehicle detection frame 821. Furthermore, if the ECU 701 determines from the results of this recognition processing that emergency braking is necessary for the moving object 700, it instructs the vehicle control unit 702 to brake and displays a notification 840 to the driver of the moving object 700. This allows the driver to correctly understand that automatic braking will be performed.

[0091] In this embodiment, the warning image is generated by superimposing the object detection results obtained by recognizing the color image and the object detection results obtained by recognizing the IR image onto the color image. However, this is not limiting, and the warning image may be generated by, for example, cutting out an image of the detection frame area of ​​the IR image and combining it with the color image. This makes it possible to display the vehicle 820 as a clear image in the warning image, with the fog 810 thinning.

[0092] In this embodiment, a single camera captures both a color image and an IR image, and these two images have the same angle of view, making it possible to easily generate a composite image and integrate the detection results of the color image and the IR image onto a single image, as shown in Figure 10(c), without complex calculations.

[0093] 11 is a flowchart showing details of the operation in this embodiment. In this flowchart, steps S101 to S107 are sequentially executed by a CPU or the like serving as a computer in camera control unit 605 executing a computer program stored in memory. Also, steps S201 to S206 are sequentially executed by a CPU or the like serving as a computer in ECU 701 executing a computer program stored in memory.

[0094] In step S101 of Figure 11, the camera control unit 605 configures the IR emitter 500. Specifically, the camera control unit 605 configures the pulse output width, output period, repetition cycle, and number of repetitions for generating a pulse signal at a predetermined timing for the light emission control unit 502 inside the IR emitter 500. This is configured according to the target distance to be imaged in range gate control, as shown in Figure 8.

[0095] Next, in step S102, the camera control unit 605 configures the photoelectric conversion element 100. Specifically, various settings are configured for the circuit board 21 inside the photoelectric conversion element 100 to photoelectrically convert the optical image from the imaging optical system 601 and generate an image signal. In particular, in this embodiment, the configuration includes the period, enable width, repetition period, and number of repetitions required for the count enable generation unit 104 to generate a count enable signal. These settings are different for pixels on the RGB filter 31 side and pixels on the IR filter 32 side. In particular, for pixels on the IR filter 32 side, the IR count enable generation unit 104IR configures the IR count enable generation unit 104IR to be synchronized with the emission period of the IR emitter 500. The resulting IR pixels are range-gate controlled. This step also completes parameter configuration for the image processing unit 603 and the recognition unit 604.

[0096] Next, in step S103, the camera control unit 605 controls the camera 600 to start capturing images. This instructs the IR emitter 500 to start emitting light, causing it to start emitting light. It also instructs the photoelectric conversion element 100 to output a vertical synchronization signal, causing it to start exposure and image signal generation. As mentioned above, the light emission on the IR emitter 500 side and the exposure (count enable generation) on the photoelectric conversion element 100 side are synchronously controlled based on the reference signal of the camera control unit 605, making range gate control possible.

[0097] Next, in step S104, the camera control unit 605 controls the image processing unit 603 to perform various image processing on the image signal output from the photoelectric conversion element 100 and generate a final image signal. Here, a color image is generated using the R signal (R pixel signal), G signal (G pixel signal), and B signal (B pixel signal) output from the photoelectric conversion element 100, and an IR image is generated using the IR signal.

[0098] Next, in step S105, the recognition unit 604 performs recognition processing on the color image and IR image acquired in step S104. Through the recognition processing, objects such as people and vehicles in the images are detected, and the distance to the detected objects is estimated. In this embodiment, since both the color image and the IR image acquired by range gate control are simultaneously obtained, it is possible to simultaneously detect objects in bad weather such as fog and objects around the moving body 700 in one frame.

[0099] Next, in step S106, the color image and IR image acquired in step S104 and the recognition result in step S105 are transmitted to ECU 701 inside moving body 700. The transmitted recognition result may include, for example, the name of the detected object, the position and size information of the detection frame, and distance information of the detected object.

[0100] In step S107, the camera control unit 605 determines whether there is a next frame to be processed. If there is a next frame to be processed, the process returns to step S104 to continue, and if there is no next frame to be processed, the flowchart on the camera control unit 605 side ends.

[0101] Next, the processing of steps S201 to S206 executed by the CPU inside ECU 701 will be described. In step S201, ECU 701 determines whether or not it has received an image and a recognition result. This is the reception processing of the data transmitted in step S106 described above. If it has received an image and a recognition result, it proceeds to step S202.

[0102] Next, in step S202, ECU 701 determines whether or not an object exists within a predetermined range in front of camera 600. The predetermined range is, for example, a distance range closer than the range in which mobile object 700 can safely stop without colliding with an object if it brakes suddenly. If it is determined that no object exists within the predetermined range, the process proceeds to step S204, and if it is determined that an object exists within the predetermined range, the process proceeds to step S203.

[0103] Next, in step S203, the ECU 701 controls the vehicle control unit 702 to execute stop control of the moving body 700. This prevents the moving body 700 from colliding with an object detected within a predetermined range.

[0104] Next, in step S204, the ECU 701 generates an image to be displayed on the display unit 703. An example of this image is the image shown in Fig. 10(c), which is generated by superimposing, on the acquired color image, the object detection results obtained by performing recognition processing on the color image and the object detection results obtained by performing recognition processing on the IR image.

[0105] Next, in step S205, the image generated in step S204 is displayed on the display unit 703 to notify the driver of the moving body 700. This allows the driver to understand the object detection result and the execution of automatic braking.

[0106] In step S206, the ECU 701 determines whether there is a next frame to be processed. If there is a next frame to be processed, the process returns to step S201 to continue, and if there is no next frame to be processed, the flowchart on the ECU 701 side ends.

[0107] With the present embodiment described above, it is possible to simultaneously obtain, in one frame, a color image with good visibility in dark areas, acquired with sufficient exposure time from close to far objects, and an IR image that can clearly capture a subject at a specified distance even in bad weather by using range gate control.

[0108] There is also a SPAD with a so-called clock recharging method, in which after avalanche multiplication occurs in the voltage across the photodiode inside the pixel, a reverse voltage is applied so that avalanche multiplication occurs again when an external clock is input. With this method, avalanche multiplication and counting (exposure) by the counter circuit can be stopped by stopping the supplied clock (so-called clock gating).

[0109] In this embodiment, the count (exposure) is stopped by the count enable signal, but if the SPAD is a clock recharging type, the same effect as this embodiment can be obtained by the above-mentioned clock gating. In this case, avalanche multiplication does not occur in the count stopped state, making it possible to reduce power consumption.

[0110] Second Embodiment A second embodiment of the present invention will now be described.

[0111] In the first embodiment, a range gate control method using a camera that is synchronized with IR light emission and exposes to IR light was described. In the second embodiment, a case where range gate control is performed using visible light without using IR light will be described. Note that the functional block diagram in the second embodiment has the same configuration as that in FIG. 7 of the first embodiment, and only the light emitter and the color filter section inside the camera are different from those in the first embodiment.

[0112] 12 is a diagram showing an example of the configuration of a color filter 40 included in a pixel 101 according to the second embodiment of the present invention. Each pixel 101 in the pixel region 12 has a color filter 40, which is an R filter, a G filter, and a B filter that transmit light of red (R), blue (B), and green (G) wavelengths, respectively. As shown in the figure, the G filter is divided into a G1 filter and a G2 filter. A color filter with this configuration is called an R-G1-G2-B filter.

[0113] Here, one pixel 101 corresponds to one of the color filters, R filter, G1 filter, G2 filter, and B filter. The arrangement in this embodiment is a Bayer type arrangement as shown in Fig. 12, but the combined arrangement is not limited to this.

[0114] In this embodiment, the signal processing circuit that processes the light that has passed through the G1 filter of the color filter 40 (the G1 signal after photoelectric conversion) is different from the signal processing circuit that processes the light that has passed through the G2 filter (the G2 signal after photoelectric conversion). The count enable signals that are supplied to these circuits are also generated as different signals. This is because, similar to the configuration described in FIG. 4(b) of the first embodiment, different count enable generation units are provided for the signal processing circuits for the R signal, G1 signal, G2 signal, and B signal, and different count enable signals are connected to them. As a result, in this embodiment, different exposure times can be set in the signal processing circuits for the R signal, G1 signal, G2 signal, and B signal using the count enable signals.

[0115] 13 is a diagram showing an example of the configuration of a visible light emitter 900, a camera 600, and a moving object 700 in the second embodiment. The camera 600 and the moving object 700 have the same configuration as in the first embodiment, except for the color filter 40 described above.

[0116] The visible light emitter 900 includes a visible light emitter 901 , a light emission controller 902 , and a communication unit 903 .

[0117] In a preferred embodiment, the visible light emitting unit 901 is a headlight module having one or more solid-state light emitting devices, light emitting diodes (LEDs), or organic LEDs (OLEDs). Thus, the visible light emitting unit 901 has the function of emitting visible light that can be seen by humans to enable the driver of the vehicle 700 to obtain visual information in dark environments such as at night or in a tunnel.

[0118] The light emission control unit 902 receives a reference signal transmitted by the camera control unit 605 of the camera 600 via the communication unit 903, generates a pulse signal at a predetermined timing based on the reference signal, and outputs the pulse signal to the visible light emitter 901. The light emission control unit 902 can set the period from the reference signal to the output of a pulse, the pulse output width, the pulse non-output width, the repetition cycle and the number of repetitions from one pulse output to the next, and other parameters. The camera control unit 605 sets predetermined values ​​for the light emission control unit 902 via the communication units 607 and 903, so that a pulse signal is output to the visible light emitter 901 at a predetermined timing based on the reference signal, thereby controlling the light emission period of the visible light emitter 900. In this way, the light emission control unit 902 controls light emission based on the same signal as the reference signal input to the photoelectric conversion element 100.

[0119] The communication unit 503 communicates with the communication unit 607 of the camera 600 , receives setting information and a reference signal from the camera control unit 605 to the light emission control unit 902 , and transmits them to the light emission control unit 902 .

[0120] 14 is a timing chart illustrating the control operation for obtaining a color image per frame time and a range gate image using visible light. In this embodiment, the range gate image is obtained by generating an image through exposure synchronized with light emission from the visible light emitter 900.

[0121] In FIG. 14 , the vertical synchronization signal indicates the frame period of the image capture, and the period between one low pulse and the next low pulse is one frame time. Next, the waveform of the R-G1-B count enable indicates the timing of the start and end of photon counting for the count enable signal output by the count enable generation unit 104 to the signal processing circuits for the R, G1, and B signals, respectively. The R-G1-B counter value indicates the increase or decrease in the photon count of the counter circuit 211 within the signal processing circuit for the R, G1, and B signals, respectively. Note that the R-G1-B counter value also includes reflected light from the visible light emitter 900, so the count number tends to be higher than in the first embodiment. The visible light emission control indicates the emission timing of the visible light emitter 900, and the G2 count enable indicates the timing of the start and end of photon counting for the count enable signal output by the count enable generation unit 104 to the signal processing circuit for the G2 signal. The G2 counter value indicates the increase or decrease in the photon count of the counter circuit 211 within the signal processing circuit for the G2 signal.

[0122] First, the R-G1-B control for obtaining a color image will be described. In this control, the R-G1-B counter value gradually increases from 0 during the period from the start to the end of the R-G1-B count enable. The period from the start to the end of the R-G1-B count enable is the exposure time. After the R-G1-B count enable ends, information on the R-G1-B counter value is sent from the counter circuit 211 to the memory circuit 212, and the R-G1-B counter value is reset by the RES signal. The exposure time from the start to the end of the R-G1-B count enable is performed within one frame time. The R, G1, and B signals generated in this manner are demosaic processed by the image processing unit 603 to generate a color image (RGB image, first image data).

[0123] Next, range gate control for obtaining a range gate image using visible light in the second embodiment will be described. In this control, the emission period of visible light is controlled in a pulsed manner by the light emission control unit 902, and the number of photons is counted only for reflected visible light from a specific range. Therefore, this embodiment is effective in environments with little external light, such as at night.

[0124] In addition, the timing control of light emission and exposure (counter enable for G2) for range gate control is the same as the timing control in the first embodiment, and the timing of the start of visible light emission and the start of exposure is synchronized to match the range of a specified target distance.

[0125] It should be noted that a headlight module is assumed for visible light emitter 900, which is the emitter of the second embodiment. In this case, since the visible light emitter not only controls the range gate but also plays a role in assisting the visibility of the driver of vehicle 700, the visible light emission control continues to emit light regardless of whether exposure is present or not, i.e., whether the G2 counter enable is operating or not.

[0126] The time resolution of the human eye is approximately 50 msec to 100 msec, and flashing of light shorter than this time is perceived as continuous lighting. Therefore, repeated pulsed light emission on the order of nanoseconds as in this embodiment does not appear to flicker to the human eye and does not impair the driver's visibility. Furthermore, by providing an average light output equivalent to that of a continuous light source, it is possible to achieve a light source level equivalent to that of a continuous light source. Therefore, range gate control is possible in environments with little external light without impairing the driver's visibility.

[0127] As shown in the figure, the range gate operation cycle is performed a predetermined number of times within one frame time. Information on the G2 counter value last added within one frame time is sent from the counter circuit 211 to the memory circuit 212, and then the G2 counter value is reset by the RES signal. The G2 signal generated in this manner is processed by the image processing unit 603 separately from the R, G1, and B signals, and a monochrome image (second image data) using only the G2 signal is generated.

[0128] As a result of the above control, the exposure period in R-G1-B control is longer than in range gate control, and photons are more easily accumulated, so good color images can be obtained even in dark times such as at night when the amount of reflected light per unit time is low. Also, because exposure control synchronized with light emission is not performed as in range gate control, it is possible to expose reflected light regardless of distance, making it possible to obtain color images of subjects at various distances.

[0129] On the other hand, in range gate control, the exposure period is synchronized with the light emitted by the visible light emitter 900, so it is possible to obtain a clear G2 image (monochrome image) for the targeted range even in bad weather such as fog.

[0130] In this embodiment, count enable signals are generated separately for the R-G1-B pixels and the G2 pixels, making it possible to simultaneously obtain a color image that is not range gate controlled and a G2 image (monochrome image) that is range gate controlled.

[0131] When this camera is mounted on a vehicle, it is not necessary to mount an IR emitter. By controlling the LED headlight module already mounted on the vehicle as the visible light emitter 900, range gate control can be realized, and the cost of mounting the IR emitter can be reduced compared to the first embodiment.

[0132] While the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate. [Explanation of symbols]

[0133] 11 Sensor board 12 pixel area 21 Circuit Board 22 Circuit area 30 Color Filters 31 RGB filters 32 IR filters 100 Photoelectric conversion element 101 pixels 102 Photoelectric conversion unit 103 Signal Processing Circuit 104 Count enable generation unit 210 Waveform shaping section 211 Counter Circuit 212 Memory Circuit 213 Drive Wire 500 IR light emitter 501 IR light emitter 502 IR light emission control unit 503 Communications Department 600 Camera 601 Imaging Optical System 603 Signal Processing Unit 604 Recognition part 605 Camera control unit

Claims

1. A color filter that transmits light of a specific wavelength; a light emitting unit that emits light having a wavelength corresponding to the frequency characteristics of the color filter a plurality of times within one frame period; a sensor unit that emits a pulse in accordance with the frequency of receiving photons of light that has passed through the color filter; a plurality of pixel units each including a counter that counts the number of pulses and a memory that stores the count value of the counter; a count enable generating unit that generates a count enable signal that controls a count period of the counter; Equipped with an imaging device characterized in that the count enable generation unit generates either a first count enable signal that is asynchronous with the light emission timing of the light emitting unit and has one enable period within one frame period, or a second count enable signal that is synchronized with the light emission timing of the light emitting unit and has multiple enable periods within one frame period.

2. the count enable generation unit generates the count enable signal based on a light emission timing of the light emission unit and a target distance range of an image capture target.

2. The imaging device according to claim 1.

3. the count enable generation unit generates the count enable signal as a different signal for each pixel unit corresponding to the color filter, and further generates the count enable signal for pixel units corresponding to some of the color filters as a signal synchronized with the light emission timing of the light emitting unit.

2. The imaging device according to claim 1.

4. The light emitting unit emits IR light, a part of the color filter is formed of an IR filter, the count enable generation unit generates a count enable signal as a signal synchronized with light emission timing of the light emission unit, and supplies the count enable signal to a pixel unit corresponding to the IR filter; 4. The imaging device according to claim 3.

5. The color filter is an RGB-IR filter.

5. The imaging device according to claim 4.

6. an image processing unit that generates first image data from pixel signals generated by a count enable signal that is not synchronized with the light emission timing of the light emitting unit, and generates second image data from pixel signals generated by a count enable signal that is synchronized with the light emission timing of the light emitting unit; 2. The imaging device according to claim 1.

7. The first image data is a color image, and the second image data is a monochrome image.

7. The imaging device according to claim 6.

8. further comprising a recognition processing unit that executes a recognition process on the first image data and the second image data; 7. The imaging device according to claim 6.

9. superimposing the recognition result of the first image data by the recognition processing unit and the recognition result of the second image data on the first image data; 9. The imaging device according to claim 8.

10. generating a composite image of the first image data and the second image data based on a recognition result of the first image data and a recognition result of the second image data by the recognition processing unit; 9. The imaging device according to claim 8.

11. the light-emitting unit emits visible light, the count enable generation unit generates a count enable signal as a signal synchronized with the light emission timing of the light emitting unit, and supplies the count enable signal to a part of pixel units corresponding to the color filter; 4. The imaging device according to claim 3.

12. the color filter is an R-G1-G2-B filter, the count enable generation unit supplies a count enable signal, which is a signal synchronized with the light emission timing of the light emitting unit, to a pixel unit corresponding to either a G1 or G2 color filter; 12. The imaging device according to claim 11.

13. an image processing unit that generates first image data from either a G1 or G2 pixel signal, an R pixel signal, and a B pixel signal, which are generated by a count enable signal that is not synchronized with the light emission timing of the light emitting unit, and generates second image data from either the G1 or G2 pixel signal that is generated by a count enable signal that is synchronized with the light emission timing of the light emitting unit; 13. The imaging device according to claim 12.

14. The first image data is a color image, and the second image data is a monochrome image.

14. The imaging device according to claim 13.

15. further comprising a recognition processing unit that executes a recognition process on the first image data and the second image data; 14. The imaging device according to claim 13.

16. superimposing a recognition result of the first image data by the recognition processing unit and a recognition result of the second image data on the first image data; 16. The imaging device according to claim 15.

17. generating a composite image of the first image data and the second image data based on a recognition result of the first image data and a recognition result of the second image data by the recognition processing unit; 16. The imaging device according to claim 15.

18. A color filter that transmits light of a specific wavelength; a light emitting unit that emits light having a wavelength corresponding to the frequency characteristics of the color filter a plurality of times within one frame period; a sensor unit that emits a pulse in accordance with the frequency of receiving photons of light that has passed through the color filter; a plurality of pixel units each including a counter that counts the number of pulses and a memory that stores the count value of the counter; a count enable generating unit that generates a count enable signal that controls a count period of the counter; An imaging method comprising: an imaging method characterized in that the count enable generation unit generates either a first count enable signal that is asynchronous with the light emission timing of the light emitting unit and has one enable period within one frame period, or a second count enable signal that is synchronized with the light emission timing of the light emitting unit and has multiple enable periods within one frame period.

Images