Imaging device, imaging method, and computer program

Abstract

An imaging device is provided that can obtain images with good visibility in a plurality of distance ranges with a single light emission. [Solution] In an imaging device, the photoelectric conversion element has a plurality of pixels, each of which has a sensor unit that emits pulses in response to incident photons, a first counter and a second counter that count the number of pulses, a first memory that stores the count value of the first counter, a second memory that stores the count value of the second counter, and a switch circuit that switches the connection between the sensor unit and the first counter or the connection between the sensor unit and the second counter; a light-emitting unit that illuminates a subject; and a control unit that controls the switch circuit so that, after a single light emission from the light-emitting unit, the first counter counts the light reflected from the subject within a first distance range, and the second counter counts the light reflected from the subject within a second distance range.

Description

[Technical Field]

[0001] The present invention relates to an imaging device, an imaging method, a computer program, and the like that are capable of capturing a clearer image of a subject within a target distance range. [Background technology]

[0002] A camera called a range gate camera is known, which emits pulsed light at a predetermined frequency in front of the camera and exposes the image sensor inside the camera at a predetermined timing according to the target distance range, thereby capturing clear images of only subjects within the target distance range. Hereinafter, this technique will be referred to as range gate control. By using this range gate control, it is possible to capture a clear image of a subject (object) at a predetermined distance even in bad weather, for example.

[0003] Patent Document 1 also describes a technology that uses a timing controller to adjust the timing of pulsed light emission and camera exposure to prevent capturing images of unnecessary distance ranges. Patent Document 1 also describes a configuration in which multiple memories are installed in each pixel of an image sensor, and multiple exposure operations (accumulation operations) are performed by the camera in response to a single light pulse emission, thereby enabling the acquisition of images of multiple target distance ranges with different distances. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2017-195573 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the configuration of Patent Document 1, in order to capture multiple target distance ranges, it is necessary to place the same number of memories as the number of target distance ranges in all pixels. For example, if two memories can be installed, the target distance range will be divided into two areas.

[0006] Therefore, to capture images of three or more target distance ranges, pulsed light must be emitted again to capture the image, which increases the number of pulsed light emissions and exposures required to capture the same amount of light, resulting in issues such as increased power consumption and increased readout processing volume.

[0007] Furthermore, while conventional range gate cameras typically use CMOS image sensors as the imaging element, noise increases between exposure and readout, making noise particularly noticeable when shooting in dark places.

[0008] Therefore, an object of the present invention is to provide an imaging device that can obtain images with good visibility in a plurality of distance regions with a single light emission. [Means for solving the problem]

[0009] An imaging device according to one aspect of the present invention comprises: A photoelectric conversion element having a plurality of pixels, wherein the pixels are: a sensor unit that emits a pulse in response to incident photons; a first counter and a second counter that count the number of pulses; a first memory that stores the count value of the first counter; a second memory that stores the count value of the second counter; a photoelectric conversion element including a switch circuit for switching the connection between the sensor unit and the first counter or the connection between the sensor unit and the second counter; a light emitting unit for illuminating a subject; a control unit that controls the switch circuit so that the first counter counts the reflected light from the subject within a first distance range and the second counter counts the reflected light from the subject within a second distance range, in response to a single light emission from the light emitting unit; The present invention is characterized by having the following. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide an imaging device that can obtain images with good visibility in a plurality of distance regions with a single light emission. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram showing an example of the configuration of a photoelectric conversion element 100 according to a first embodiment. [Figure 2] 1 is a diagram showing an example of the configuration of a sensor substrate 11 of a photoelectric conversion element 100 according to a first embodiment. [Figure 3] 3A is a diagram showing an example of the configuration of the circuit board 21 of the photoelectric conversion element 100 according to embodiment 1, and FIG. 3B is a diagram showing an example of the configuration of the counter selection control generation unit 104 that generates signals to be supplied to each signal processing circuit 103 described in FIG. 3A. [Figure 4] 3A is a diagram showing an example of an equivalent circuit of the photoelectric conversion unit 102 of the pixel 101 in FIG. 2 and FIG. 3A and the signal processing circuit 103 corresponding to the photoelectric conversion unit 102. FIG. [Figure 5] 2 is a diagram schematically illustrating the relationship between the operation of an APD 201 and an output signal. FIG. [Figure 6] 1 is a functional block diagram showing an example of the configuration of a light emitter 500, a camera 600, and a moving object 700 according to the first embodiment. [Figure 7] 10 is a diagram showing the relationship between the travel of light emitted from the light emitter 500 and its reflected light, and the exposure timing of the camera 600, according to the first embodiment. FIG. [Figure 8] 4 is a timing chart illustrating a control operation for obtaining a range gate image in one frame period according to the first embodiment. [Figure 9]4 is a flowchart showing details of an operation example in the first embodiment. [Figure 10] FIG. 6 is a diagram showing an example in which the camera 600 acquires images of two target ranges at all pixels with one light emission. [Figure 11] FIG. 10 is a diagram showing an example in which images of four target ranges are acquired with one light emission in the camera 600 according to the second embodiment. [Figure 12] 10A and 10B are diagrams showing a modification in which the pixel and counter used areas of the second embodiment are shown as four pixels. [Figure 13] 12(B) is a diagram showing an example of the configuration of the counter selection control generation unit 104 corresponding to the example of FIG. [Figure 14] 10A and 10B are diagrams showing an example in which the camera 600 in the third embodiment acquires images of three target ranges with one light emission. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each drawing, the same members or elements are designated by the same reference numerals, and duplicate descriptions will be omitted or simplified.

[0013] <Embodiment 1> 1 is a diagram showing an example of the configuration of a photoelectric conversion element 100 according to a first embodiment of the present invention. The photoelectric conversion element 100 of this embodiment includes two substrates: a sensor substrate 11 and a circuit substrate 21. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes signals detected in the pixel region 12.

[0014] In this embodiment, the sensor substrate 11 and the circuit substrate 21 are stacked and electrically connected to each other, forming a so-called stacked structure. However, the pixel region 12 included in the sensor substrate 11 and the circuit region 22 included in the circuit substrate 21 may be arranged on a common semiconductor layer, forming a so-called non-stacked structure.

[0015] 2 is a diagram showing an example of the configuration of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally across a plurality of rows and columns. As described above, the photoelectric conversion element 100 of this embodiment has a plurality of pixels 101. Each pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, referred to as APD).

[0016] Here, the photoelectric conversion unit 102 functions as a sensor unit that emits a pulse in response to incident photons. The number of rows and columns of the pixel array that constitutes the pixel region 12 is not particularly limited.

[0017] Figure 3(A) is a diagram showing an example configuration of the circuit board 21 of the photoelectric conversion element 100 of embodiment 1, and Figure 3(B) is a diagram showing an example configuration of the counter selection control generation unit 104 that generates signals to be supplied to each signal processing circuit 103 described in Figure 3(A).

[0018] The circuit board 21 has a signal processing circuit 103 that processes the charges photoelectrically converted by each photoelectric conversion unit 102 in Fig. 2. The circuit board 21 also has a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, vertical signal lines 113, a vertical scanning circuit 110, and an output circuit 114.

[0019] The vertical scanning circuit 110 receives control pulses supplied from the control pulse generating unit 115 and supplies the control pulses sequentially, row by row, to a plurality of pixels arranged in the row direction. The vertical scanning circuit 110 uses logic circuits such as a shift register and an address decoder.

[0020] The photoelectric conversion signals generated by the photoelectric conversion unit 102 of each pixel are processed by each signal processing circuit 103. The signal processing circuit 103 is provided with a counter, memory, etc., and digital values ​​are stored in the memory. In order to read out the digital signals from the memory of each pixel, the horizontal scanning circuit 111 inputs control pulses to the signal processing circuit 103 to sequentially select each column.

[0021] A signal is output to the vertical signal line 113 from the signal processing circuit 103 of the pixel of the row selected by the vertical scanning circuit 110. The signal output to the vertical signal line 113 is output to the outside of the photoelectric conversion element 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has a plurality of buffers built in and connected to the vertical signal line 113.

[0022] 2 and 3A, a plurality of signal processing circuits 103 are arranged in an area overlapping the pixel area 12 in a plan view. The vertical scanning circuit 110, horizontal scanning circuit 111, readout circuit 112, output circuit 114, and control pulse generating unit 115 of the circuit board 21 are arranged so as to overlap with the outer peripheral portion of the pixel area 12 of the sensor substrate 11 in a plan view.

[0023] The arrangement of the vertical signal lines 113, the readout circuits 112, and the output circuits 114 is not limited to the example shown in Fig. 3(A). For example, the vertical signal lines 113 may be arranged to extend in the row direction, and the readout circuits 112 may be arranged at the ends of the vertical signal lines 113.

[0024] Furthermore, it is not necessary for one signal processing circuit 103 to be provided for each photoelectric conversion unit, but one signal processing unit may be shared by multiple photoelectric conversion units and perform signal processing sequentially.

[0025] 3B generates a count control signal to be supplied to a switch circuit 215 (described later) inside the signal processing circuit 103. The count control signal is a signal for controlling the switch circuit 215 inside the signal processing circuit 103.

[0026] 3B, the count control signal can be sent for each row. That is, signals can be sent simultaneously to multiple pixels arranged in the row direction. That is, by sequentially supplying count control signals to multiple pixels row by row, exposure control of multiple pixels row by row can be performed.

[0027] FIG. 4 is a diagram showing an example of an equivalent circuit of the photoelectric conversion unit 102 of the pixel 101 in FIGS. 2 and 3(A) and the signal processing circuit 103 corresponding to the photoelectric conversion unit 102. In FIG.

[0028] The APD 201 included in the photoelectric conversion unit 102 generates charge pairs in response to incident photons. A voltage VHVL (first voltage) is supplied to the anode of the APD 201. A voltage VH (second voltage) higher than the voltage VL is supplied to the cathode of the APD 201 via a quench element 202.

[0029] A reverse bias voltage is applied to the anode and cathode of the APD 201 so that the APD 201 performs avalanche multiplication. With this voltage applied, the charge generated by the incident photons undergoes avalanche multiplication, generating an avalanche current.

[0030] When a reverse bias voltage is supplied, there are two modes: a Geiger mode in which the anode and cathode operate at a voltage difference greater than the breakdown voltage, and a linear mode in which the anode and cathode operate at a voltage difference close to or less than the breakdown voltage.

[0031] An APD operated in Geiger mode is called a SPAD. In the case of a SPAD, for example, the voltage VL (first voltage) is set to −30V, and the voltage VH (second voltage) is set to 1V.

[0032] The signal processing circuit 103 includes a quenching element 202, a waveform shaping unit 210, a switch circuit 215, a counter circuit 211a, a counter circuit 211b, a memory circuit 212a, and a memory circuit 212b. The quenching element 202 is connected between the voltage VH and the cathode of the APD 201.

[0033] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, suppressing the voltage supplied to the APD 201 and suppressing avalanche multiplication (quench operation).The quench element 202 also functions to return the voltage supplied to the APD 201 to voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0034] The waveform shaping unit 210 shapes the voltage change of the cathode of the APD 201 obtained when a photon is detected, and outputs a pulse signal. As the waveform shaping unit 210, for example, an inverter circuit is used.

[0035] Although FIG. 4 shows an example in which one inverter is used as the waveform shaping unit 210, a circuit in which a plurality of inverters are connected in series may be used, or other circuits that have a waveform shaping effect may be used.

[0036] Counter circuit 211a and counter circuit 211b are connected to the nodes of waveform shaping section 210 via switch circuit 215, count the number of pulses output from waveform shaping section 210, and hold the count value.

[0037] Here, the counter circuit 211a and the counter circuit 211b function as a first counter and a second counter, respectively, that count the number of pulses. Also, the memory circuit 212a functions as a first memory that stores the count value of the first counter, and the memory circuit 212b functions as a second memory that stores the count value of the second counter.

[0038] Furthermore, when a control pulse RES is supplied via a RES signal line 213, the count values ​​held in the counter circuits 211a and 211b are stored in the memory circuits 212a and 212b, respectively, and the count values ​​are reset.

[0039] The counter circuits 211a and 211b generate the difference between the count values ​​at the start and end of the accumulation period as the count value.

[0040] The switch circuit 215 receives a count control signal from the counter selection control generation unit 104 described in Fig. 3(B) via a counter selection signal line 216. The switch circuit 215 is used to switch between the connection between the sensor unit and the first counter and the connection between the sensor unit and the second counter.

[0041] When contact 1 of switch circuit 215 is selected, counter circuit 211a and waveform shaping section 210 are connected, and counter circuit 211a counts the number of pulses.

[0042] Next, when contact 2 of switch circuit 215 is selected, counter circuit 211b and waveform shaping section 210 are connected, and counter circuit 211b counts the number of pulses.

[0043] Thereafter, when the contact point of the switch circuit 215 is selected to be off, the connection between the counter circuit 211b and the waveform shaping unit 210 is released, and the counter circuit 211b stops counting the number of pulses.

[0044] The count values ​​of the counter circuits 211a and 211b are held even when the contact of the switch circuit 215 is set to off and the connection with the waveform shaping unit 210 is released. For example, if the counter selection control generation unit 104 is a circuit that operates at a frequency of 100 MHz, the count control signal can be switched in units of 10 nsec.

[0045] Here, the number of pulses output by the photoelectric conversion unit 102 in response to the reception of photons is counted only while the switch circuit 215 is connected to contact 1 and contact 2. Therefore, the count period for the number of pulses can be rephrased as the exposure period (accumulation period, count period) of the pixel 101.

[0046] In this embodiment, the exposure period (accumulation period, count period) can be switched in units of 10 nsec. In the following description, "exposure" means counting the APD output by a counter circuit.

[0047] A control pulse SEL is supplied to the memory circuits 212a and 212b from the vertical scanning circuit 110 via a control pulse SEL line 214, and electrical connection and disconnection between the counter circuits 211a and 211b and the vertical signal line 113 are switched.

[0048] The memory circuits 212a and 212b function as memories that temporarily store the count values ​​of the counter circuits 211a and 211b, respectively. In addition, the memory circuits 212a and 212b sequentially output the output signals (count values) from the counter circuits 211a and 211b to the vertical signal line 113.

[0049] Note that the electrical connection may be switched by disposing a switch such as a transistor between the quench element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103. Similarly, the electrical connection of the voltage VH or the voltage VL supplied to the photoelectric conversion unit 102 may be switched using a switch such as a transistor.

[0050] 5 is a diagram showing a schematic diagram of the relationship between the operation and output signal of the APD 201. The input side of the waveform shaping section 210 is designated as nodeA, and the output side is designated as nodeB.

[0051] Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage at node A drops.

[0052] When the voltage drop amount further increases and the potential difference applied to the APD 201 decreases, the avalanche multiplication of the APD 201 stops as at time t2, and the voltage level of node A does not drop below a certain value.

[0053] After that, between time t2 and time t3, a current flows through node A to compensate for the voltage drop from voltage VL, and at time t3, node A settles to its original potential level. At this time, the portion of the output waveform at node A that falls below a predetermined determination threshold is shaped by waveform shaping unit 210 and output as a pulse signal at node B.

[0054] 6 is a functional block diagram showing an example of the configuration of the light emitter 500, camera 600, and moving body 700 according to embodiment 1. In this embodiment, the light emitter 500 and camera 600 configure an imaging device.

[0055] Some of the functional blocks shown in FIG. 6 are realized by causing a computer (not shown) included in each of the light emitter 500, the camera 600, and the mobile object 700 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0056] However, some or all of these functions may be implemented by hardware. Examples of hardware that can be used include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs). Furthermore, the functional blocks shown in Fig. 6 do not have to be built into the same housing, and may be configured as separate devices connected to each other via signal paths.

[0057] 1 to 5, an imaging optical system 601, an image processing unit 603, a recognition unit 604, a camera control unit 605, a storage unit 606, a communication unit 607, etc. The photoelectric conversion element 100 is configured by an avalanche photodiode for photoelectrically converting an optical image, as described in FIGS.

[0058] The imaging device (camera 600 and light emitter 500) of this embodiment is mounted on a moving body 700, and the imaging unit consisting of a set of an imaging optical system 601 and a photoelectric conversion element 100 is configured to capture images in at least one direction, for example, in front, behind, or to the side of the moving body. Note that a plurality of imaging units may be provided on the moving body 700. Alternatively, a plurality of imaging devices may be provided on the moving body 700.

[0059] The image processing unit 603 performs image processing such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaic processing, and data compression on the image signal acquired by the photoelectric conversion element 100, and generates a final image signal.

[0060] The output of the image processing unit 603 is supplied to a recognition unit 604 and a camera control unit 605, as well as to an ECU (Electric Control Unit) 701 of the moving object 700. The recognition unit 604 performs image recognition based on the image signal to recognize objects such as people and vehicles in the vicinity. Deep learning is used for this recognition processing.

[0061] For example, YOLO (You Only Look Once), which is easy to learn and has fast detection, may be used as a deep learning method. Other deep learning methods that may be used include SSD (Single Shot Multi Box Detector), Faster R-CNN (Regional Convolution Neural Network), Fast R-CNN, and R-CNN.

[0062] In this embodiment, the recognition unit 604 calculates the distance to the recognized object. That is, the recognition unit 604 calculates a first distance range and a second distance range by recognizing the subject. Note that, as a distance measurement method, distance estimation may be performed using deep learning, for example. That is, for example, the distance value may be calculated by analyzing information such as blur in the image of the detected object using deep learning.

[0063] Alternatively, the imaging device may be a stereo camera, and distance measurement may be performed using the principle of triangulation. Alternatively, the photoelectric conversion element may be a phase difference detection type imaging element, and distance measurement may be performed using a phase difference signal from the photoelectric conversion element. Recognition processing including distance estimation is performed on the image input from the image processing unit 603, and the recognition result is output to the downstream ECU 701.

[0064] In this embodiment, the mobile body 700 is described using an example of an automobile, but the mobile body may be any mobile body such as an airplane, train, ship, drone, AGV, or robot.

[0065] The camera control unit 605 has a built-in CPU as a computer and memory that stores a computer program, and controls each unit of the camera 600 by the CPU executing the computer program stored in the memory.

[0066] In addition, the camera control unit 605 functions as a control means, and controls, for example, the length of the exposure period (accumulation period) of each frame of the photoelectric conversion element 100 and the timing of the control signal via the counter selection control generation unit 104 of the photoelectric conversion element 100.

[0067] Specifically, the camera control unit 605 transmits a reference signal that is repeatedly output at predetermined intervals to the counter selection control generation unit 104. The counter selection control generation unit 104 uses the reference signal as a timing reference and generates a signal that switches the switch circuit 215 at a predetermined timing.

[0068] Here, the counter selection control generation unit 104 can arbitrarily set the timing of switching the switch, the switch connection time, the repetition period, the number of repetitions, etc. The camera control unit 605 sets predetermined values ​​for these via control signals, and the count control signal is input to the switch circuit 215 at the predetermined timing.

[0069] As a result, the output of the waveform shaping unit 210 is connected to the counter circuit 211a and the counter circuit 211b for a predetermined time at a predetermined timing, thereby controlling the exposure period of the pixel.

[0070] The camera control unit 605 also transmits the same signal as the reference signal described above to the light emitter 500 via the communication unit 607. In this way, the same reference signal transmitted to the photoelectric conversion element 100 is also transmitted to the light emitter 500, and the light emitter 500 performs light emission control based on the reference signal, thereby synchronously controlling the exposure timing inside the photoelectric conversion element 100 and the light emission timing by the light emitter 500.

[0071] The storage unit 606 includes a recording medium such as a memory card or a hard disk, and can store and read image signals. The communication unit 607 includes a wireless or wired interface, and outputs the generated image signals to the outside of the camera 600 and receives various signals from the outside.

[0072] In this embodiment, the communication unit 607 is connected to the communication unit 503 of the light emitter 500 and also plays a role in transmitting the reference signal and control commands from the camera control unit 605 to the light emitter 500 .

[0073] Light emitter 500 has light emitter 501, light emission control unit 502, and communication unit 503. Light emitter 501 includes, for example, a near-infrared LED for illuminating a subject in front of moving body 700, and is capable of emitting beam light in combination with a lens. Furthermore, the light emitter outputs near-infrared pulsed light for a predetermined light emission time in response to a pulse signal output from light emission control unit 502.

[0074] The light emission control unit 502 receives a reference signal transmitted by the camera control unit 605 of the camera 600 via the communication unit 503, generates a pulse signal at a predetermined timing based on the reference signal, and outputs it to the light emission unit 501.

[0075] Here, the light emission control unit 502 can set the period from the reference signal to the output of a pulse, the pulse output width, the pulse non-output width, the repetition period from one pulse output to the next, the number of repetitions, and the like.

[0076] The camera control unit 605 sets a predetermined value to the light emission control unit 502 via the communication unit 607 and the communication unit 503, so that a pulse signal is output to the light emission unit 501 at a predetermined timing based on the reference signal, thereby controlling the light emission period of the light emitter 500.

[0077] In this way, the light emission control unit 502 controls light emission using the same signal as the reference signal input to the photoelectric conversion element 100 as a reference.

[0078] The communication unit 503 communicates with the communication unit 607 of the camera 600 , receives setting information and a reference signal from the camera control unit 605 to the light emission control unit 502 , and transmits them to the light emission unit 501 .

[0079] The ECU 701 incorporates a CPU as a computer and a memory that stores a computer program, and controls each part of the moving object 700 by the CPU executing the computer program stored in the memory.

[0080] The output of the ECU 701 is supplied to a vehicle control unit 702 and a display unit 703. The vehicle control unit 702 functions as a movement control means that drives, stops, controls the direction, etc. of the vehicle as a moving object based on the output of the ECU 701. The display unit 703 functions as a display means, and includes display elements such as a liquid crystal device or an organic EL, and is mounted on the moving object 700.

[0081] In this embodiment, the ECU 701 receives information on the recognition result from the recognition unit 604 and is able to execute vehicle stop control (automatic braking, etc.) according to the content of the recognition result. The ECU 701 also receives an image from the image processing unit 603 and transmits it to the display unit 703 together with the recognition result.

[0082] The display unit 703 displays, for example, using a GUI based on the output of the ECU 701, to the driver of the vehicle 700 images acquired by the photoelectric conversion element 100, the recognition results by the recognition unit 604, and various information related to the vehicle's driving state, etc.

[0083] It should be noted that the image processing unit 603, the recognition unit 604, etc. in Figure 6 do not have to be mounted on the moving body 700, and may be provided, for example, in an external terminal provided separately from the moving body 700, for remotely controlling the moving body 700 or for monitoring the movement of the moving body.

[0084] 7 is a diagram showing the relationship between the travel of the emitted light from the light emitter 500 and its reflected light and the exposure timing of the camera 600 according to the first embodiment. In Fig. 7, the horizontal axis represents distance and the vertical axis represents time.

[0085] As shown in FIG. 7, in this embodiment, a control (range gate control) is performed to synchronize the light emission timing and the exposure timing according to the target distance range, thereby obtaining an image (range gate image) capturing the target distance range.

[0086] In this embodiment, a camera that acquires a target distance image by range gate control in this manner is called a range gate camera.

[0087] First, the horizontal axis will be explained. In the example shown in Fig. 7, fog 810 exists between distance x1 and distance x2, and vehicle 820 exists at distance x3. Also, in Fig. 7, range gate control sets the position at distance D to the target vehicle 820 as the starting point, and acquires a range gate image within target distance range R from there.

[0088] In this case, the target distance range R is the target distance range to be imaged. At this time, the vehicle 820 is present within the target distance range R.

[0089] Next, the vertical axis will be described. Time 0 is the timing when light emission starts in light emitter 500, and time tf is the timing when light emission ends. At this time, the light emission period is tf. Furthermore, when the position of distance D is the starting point and a range gate image is acquired from there within target distance range R, the exposure start time is time t1 and the exposure end time is time t2.

[0090] Time t1 is the timing at which the light emitted from the light emitter 500 at time 0 returns to the camera 600 as reflected light from an object at distance D. Time t2 is the timing at which the light emitted from the light emitter 500 at time tf returns to the camera 600 as reflected light from a point that has traveled the target distance range R from the object at distance D.

[0091] Furthermore, the timing at which the first reflected light from the fog 810 returns to the camera 600 is set to time t3, and the timing at which the last reflected light from the fog 810 returns to the camera 600 is set to time t4.

[0092] In range gate control, exposure is not performed during the period from time t3 to time t4 when reflected light from fog 810 reaches camera 600. By performing exposure only during the period from time t1 to time t2 when reflected light from distance D corresponding to the target distance range R reaches camera 600, it is possible to remove fog 810 while still obtaining a clear image of vehicle 820.

[0093] Here, we will explain the time it takes for reflected light from an object located at distance x to return to camera 600. The timing at which radiated light, which starts to be emitted at time 0, hits an object located at distance x, and returns to the imaging unit as reflected light is taken as time tr. In this case, the relationship between the timing tr at which the reflected light returns and the distance x to the imaging object is given by the following equation (1).

[0094] Time tr=2x / speed of light c (approximately 3×10^8m / s)...Equation (1)

[0095] As shown in Figure 7, when the imaging range is from distance D to target distance range R, the exposure timing time t1 at the start of the time range corresponding to target distance range R can be calculated using the following equation (2) by substituting distance D for distance x in the above equation (1).

[0096] Time t1=2D / speed of light c...Equation (2)

[0097] Furthermore, the exposure timing time t2 at the end of the range can be calculated by substituting the distance D+target distance range R for the distance x in the above equation (1) and adding the time tf, as shown in the following equation (3).

[0098] Time t2=tf+2(D+R) / speed of light c...Equation (3)

[0099] In this way, the time tf from the start of light emission to the end of light emission, the time t1 from the start of light emission to the start of exposure, and the time t2 from the end of exposure are controlled according to the distance x (target distance range R) to be imaged. This realizes range gate control that can clearly capture images of subjects within the target distance range even if there is fog or the like between the camera and the target distance range.

[0100] 8 is a timing chart illustrating a control operation for obtaining a range gate image during one frame period according to embodiment 1. In this embodiment, the range gate image is generated as described above by exposure synchronized with light emission by the light emitter 500.

[0101] 8, an example will be described in which one counter circuit and one memory circuit are mounted in the circuit diagram explained in FIG.

[0102] 8, "vertical synchronization signal" indicates the frame period of imaging, and the period from one low pulse to the next low pulse is one frame period. "Light emission control" indicates the light emission timing of the light emitter 500, and light is emitted by the light emitter 500 while it is at a high level. "Exposure control" indicates the counting period of the counter circuit, and while it is at a high level, the waveform shaping unit 210 and the counter circuit are connected, and the counter circuit counts the number of photons.

[0103] The "counter value" indicates the increase or decrease in the number of photons counted by the counter circuit. The "RES signal" indicates a control pulse supplied to the counter circuit via the RES signal line 213, and the count value held in the counter circuit is reset by the pulse.

[0104] Next, we will explain range gate control for obtaining a range gate image. In this embodiment, the light emission period is controlled in a pulsed manner by the light emission control unit 502, and the number of photons is counted only for light reflected from a predetermined target distance range.

[0105] In this way, the light emission period from the start to the end of light emission is tf, the time from the start of light emission to the start of photon counting is t1, and the time from the start of light emission to the end of photon counting is t2. In this case, t1 indicates the period from the start of light emission until the light reaches the target distance range and the reflected light returns to camera 600.

[0106] Furthermore, the period from t1 to t2 is the period during which the number of photons of reflected light within the target distance range is counted, and is the period from the start to the end of the connection between the waveform shaping unit 210 and the counter circuit, and during this period the counter value increases according to the number of photons.

[0107] In order to perform range gate control correctly, it is necessary to synchronize the timing of the start of light emission and the start of exposure to match a predetermined target distance range. In this embodiment, the camera control unit 605 transmits the same reference signal to the counter selection control generation unit 104 and the light emission control unit 502, thereby synchronizing the operation timing of both.

[0108] 7, the period from the start of light emission to the start of the next light emission constitutes one range gate operation cycle. The counter value counted in one range gate operation cycle is maintained, and the counter value is incremented in the next range gate operation cycle. The period from one light emission to the next light emission is set based on the time it takes for the reflected light to attenuate sufficiently and no longer return to the camera 600.

[0109] As shown in Figure 8, a predetermined number of range gate operation cycles are performed within one frame period, and the count value of the counter circuit is incremented. Then, information on the counter value last incremented within one frame period is sent from the counter circuit to the memory circuit by the RES signal, and the counter value is then reset.

[0110] Since the exposure period is synchronized with the light emitted by the light emitter 500, it is possible to obtain a clear image at the target range even in bad weather such as fog.

[0111] 9 is a flowchart showing details of an operation example in embodiment 1. In this flowchart, steps S101 to S107 are sequentially executed by a CPU or the like serving as a computer in the camera control unit 605 executing a computer program stored in memory.

[0112] Furthermore, steps S201 to S206 are sequentially executed by a CPU or the like serving as a computer within the ECU 701 executing a computer program stored in a memory.

[0113] 9, the camera control unit 605 performs settings on the light emitter 500. Specifically, the camera control unit 605 performs settings on the light emission control unit 502 inside the light emitter 500, such as the pulse output width, output period, repetition cycle, and number of repetitions, for generating a pulse signal at a predetermined timing. These are set according to the target distance range to be imaged in range gate control, as shown in FIG.

[0114] Next, in step S102, the photoelectric conversion element 100 is set. That is, the camera control unit 605 sets the photoelectric conversion element 100. Specifically, various settings are made to the circuit board 21 inside the photoelectric conversion element 100 to perform photoelectric conversion on the optical image from the imaging optical system 601 and generate an image signal.

[0115] In this embodiment, the above settings of the photoelectric conversion element 100 include the timing at which the counter selection control generation unit 104 switches the switch 215 circuit with the count control signal, the switch connection time, the repetition period, and the number of repetitions. In this step, parameter setting for the image processing unit 603 and the recognition unit 604 is completed.

[0116] Next, in step S103, imaging is started. That is, the camera control unit 605 controls the camera 600 to start imaging. That is, it issues an instruction to start light emission to the light emitter 500 to start light emission, and also instructs the photoelectric conversion element 100 to output a vertical synchronization signal, to start an exposure operation (a count operation of the counter circuit by generating a count control signal) and generation of an image signal.

[0117] As described above, the light emission on the light emitter 500 side and the exposure operation on the photoelectric conversion element 100 side are synchronously controlled based on the reference signal of the camera control unit 605 .

[0118] Next, in step S104, an image is acquired. That is, the camera control unit 605 controls the image processing unit 603 to perform various image processing on the image signal output from the photoelectric conversion element 100, and generates a final image signal. Here, an image is generated using the signal output from the photoelectric conversion element 100.

[0119] Next, in step S105, recognition processing is performed. That is, the recognition processing is performed on the image acquired in step S104 using the recognition unit 604. Through the recognition processing, objects such as people and vehicles in the image are detected, and the distance to the detected objects is estimated. In this embodiment, it is possible to detect objects even in bad weather such as fog.

[0120] Here, step S105 functions as a recognition step (recognition unit) for calculating at least the first distance range and the second distance range by recognizing the subject.

[0121] Next, in step S106, the image and the recognition result are transmitted. That is, the image acquired in step S104 and the recognition result in step S105 are transmitted to ECU 701 inside moving body 700. The transmitted recognition result may include, for example, the name of the detected object, the position and size information of the detection frame, and distance information of the detected object.

[0122] In step S107, the camera control unit 605 determines whether there is a next frame to be processed. If there is a next frame to be processed, the process returns to step S104 to continue, and if there is no next frame to be processed, the flowchart on the camera control unit 605 side ends.

[0123] Next, the processing of steps S201 to S206 executed by the CPU inside ECU 701 will be described. In step S201, ECU 701 determines whether or not it has received an image and a recognition result. This is the reception processing of the data transmitted in step S106 described above. If it has received an image and a recognition result, it proceeds to step S202, and if it has not received, it returns to step S201.

[0124] Next, in step S202, ECU 701 determines whether or not an object exists within a predetermined range ahead of camera 600. The predetermined range is, for example, a range that is a predetermined distance longer than the range in which moving body 700 can safely stop without colliding with an object when emergency braking is performed.

[0125] If it is determined that there is no object within the predetermined range, the process proceeds to step S204, and if it is determined that there is an object within the predetermined range, the process proceeds to step S203.

[0126] In step S203, the ECU 701 executes automatic braking, that is, controls the vehicle control unit 702 to execute stop control of the moving body 700. This prevents collision between the moving body 700 and an object detected within a predetermined range.

[0127] Next, in step S204, ECU 701 generates a display image to be displayed on display unit 703. The display image includes information indicating the object detection result and the execution of automatic braking. Next, in step S205, the display image generated in step S204 is displayed on display unit 703 to notify the driver of moving body 700. This allows the driver to understand the object detection result and the execution of automatic braking.

[0128] In step S206, the ECU 701 determines whether there is a next frame to be processed. If there is a next frame to be processed, the process returns to step S201 to continue, and if there is no next frame to be processed, the flowchart on the ECU 701 side ends.

[0129] Fig. 10 is a diagram showing an example in which camera 600 captures images of two target ranges at all pixels with a single flash of light. In Fig. 10, the "driving image" shows a state in which target distance ranges from camera 600 mounted on a vehicle 820 are designated range A and range B in order of proximity, with subjects present in each range and fog 810 present in all ranges.

[0130] 10, the horizontal axis represents the time axis, and the exposure time corresponding to each range after emitting pulsed light and the photon count status of the counter circuit. Furthermore, the "used pixel / counter image" indicates two adjacent pixels in the pixel region 12, for example, in the column direction, as pixel 1 and pixel 2, and shows an example of which of the two counter circuits mounted in pixel 1 and pixel 2 is used.

[0131] In Fig. 10, the first counter circuit is shown as counter 1, and the second counter circuit is shown as counter 2. The pixels and counter circuits used are shown in gray. Note that Fig. 10 shows an example of a camera 600 with two counter circuits per pixel, but it is also possible to have three or more counter circuits per pixel.

[0132] The relationship between the distance range, exposure time, photon counting status, pixels used, and counter circuit when the camera 600 performs two exposures per light emission will be described below.

[0133] After one emission, when the reflected light of range A returns to camera 600, the range of range A is exposed (APD output is counted by the counter circuit). As shown in the "Operational Image," exposure time A corresponding to range A is shown as "Range A exposure," and the number of photons is counted by counter 1 of pixel 1 and counter 1 of pixel 2 according to exposure time A. As mentioned above, in the "Image of pixels and counters used," the pixels and counters used are shown in gray for ease of understanding.

[0134] Next, when the reflected light from range B returns to camera 600, the range of range B is exposed (APD output is counted by the counter circuit). As shown in the "Operational Image", exposure time B corresponding to range B is shown in "Range B Exposure". Then, the number of photons is counted by counter 2 of pixel 1 and counter 2 of pixel 2 according to exposure time B.

[0135] As described above, in order to perform two exposures (APD output is counted by a counter circuit) using reflected light from one emission, two counter circuits are installed in each pixel, and during the first exposure, the number of photons from reflected light in range A is counted and held by counter 1. During the second exposure, the number of photons from reflected light in range B is counted and held by counter 2.

[0136] That is, a control step is executed to control the switch circuit 215 so that the first counter counts the light reflected from the subject in range A due to one emission of light from the light emitting unit, and the second counter counts the light reflected from the subject in range B. In this embodiment, range A functions as the first distance range, and range B functions as the second distance range.

[0137] In this embodiment, the light reflected from a subject in range A, which is a first distance range, due to a single emission of light from the light-emitting unit is counted by the first counters in pixels 1 and 2. The light reflected from a subject in range B, which is a second distance range, is counted by the second counters in pixels 1 and 2.

[0138] In this way, the photon count information that forms the basis of images for two target distance ranges can be obtained with a single flash, allowing for the capture of subjects within the target distance ranges. Furthermore, by reducing the number of flashes, power consumption can be reduced, and the time required to obtain the same amount of light can be shortened.

[0139] As described above, in this embodiment, by installing two counter circuits in each pixel, it is possible to clearly capture an image of a subject at a specified distance even in bad weather using range gate control, and it is possible to simultaneously capture images with good visibility in dark areas from close to far away without requiring multiple frames.

[0140] Furthermore, the image processing unit 603 of this embodiment reads out the count value of counter 1 obtained during exposure time A to generate a first image, reads out the count value of counter 2 obtained during exposure time B to generate a second image, and synthesizes the first image and the second image. As a result, a composite image can be obtained by synthesizing the image in range A and the image in range B.

[0141] At this time, the image processing unit 603 functions as a synthesis unit that synthesizes an image obtained based on the count value of the first counter and an image obtained based on the count value of the second counter.

[0142] There is also a so-called clock recharging type SPAD, in which after avalanche multiplication occurs in the voltage across the photodiode inside the pixel, a reverse voltage is applied so that avalanche multiplication occurs again after an external clock is input.

[0143] In this method, the avalanche multiplication and counting (exposure) by the counter circuit can be stopped by stopping the supplied clock (so-called clock gating).

[0144] In this embodiment, the count (exposure) is stopped by the count control signal, but if the SPAD is a clock recharging type, the same effect as this embodiment can be obtained by the above-mentioned clock gating. In this case, avalanche multiplication does not occur in the count stopped state, making it possible to reduce power consumption.

[0145] <Embodiment 2> Hereinafter, a description will be given of a second embodiment of the present invention. In the first embodiment, two counter circuits are provided in each pixel, and a method for acquiring images of two target ranges with one light emission has been described.

[0146] In the second embodiment, a method is described in which two counter circuits are installed in each pixel and images of four target ranges are acquired with one light emission. Note that the configuration of the functional blocks of the imaging device in the second embodiment is basically the same as that in the first embodiment, and the following description will focus on the differences between the two, and will omit a description of similar parts. Furthermore, parts corresponding to those in the first embodiment will be described with the same reference numerals.

[0147] Fig. 11 is a diagram showing an example in which images of four target ranges are acquired with one light emission in the camera 600 according to embodiment 2. In Fig. 11, the "driving image" shows a state in which the target distance ranges at a certain distance from the camera 600 mounted on a vehicle 820 are, in order of proximity, range A, range B, range C, and range D, with a subject in each range and fog 810 in all ranges.

[0148] The "operation image" shows, with the horizontal axis as the time axis, the exposure time of the reflected light corresponding to each range after emitting pulsed light, and the photon count status of the counter circuit. Also, the "used pixel / counter image" illustrates, for example, two adjacent pixels in the column direction in the pixel area 12 as pixel 1 and pixel 2, and further shows the connection with the two counter circuits mounted on each pixel.

[0149] 11 also shows an example of a camera 600 having two counter circuits per pixel, with the first counter circuit shown as counter 1 and the second counter circuit shown as counter 2. The pixels and counter circuits used are shown in gray.

[0150] The relationship between the distance range, exposure time, photon counting status, pixels used, and counter circuit when the camera 600 performs four exposures per light emission will be described below.

[0151] After emission, when the reflected light of range A returns to camera 600, the range of range A is exposed (APD output is counted by the counter circuit). As shown in the "Operational Image", exposure time A corresponding to range A is shown as "Range A exposure", and the number of photons is counted by counter 1 of pixel 1 according to exposure time A. In Figure 11, the used pixel / counter image shows the used pixels and counters in gray.

[0152] Next, when the reflected light from range B returns to camera 600, the range of range B is exposed (APD output is counted by the counter circuit). As shown in the "Operational Image," exposure time B corresponding to range B is shown as "Range B exposure," and the number of photons is counted by counter 2 of pixel 1 according to exposure time B.

[0153] Next, when the reflected light from range C returns to the camera 600, the range of range C is exposed (the APD output is counted by the counter circuit). As shown in the "Operational Image", the exposure time C corresponding to range C is shown as "Range C exposure", and the number of photons is counted by counter 1 of pixel 2 according to the exposure time C.

[0154] Furthermore, when the reflected light of range D returns to the camera 600, the range of range B is exposed (the APD output is counted by the counter circuit). As shown in the "Operational Image", the exposure time D corresponding to range B is shown as "Range B exposure", and the number of photons is counted by counter 2 of pixel 2 according to the exposure time D.

[0155] In this way, in order to perform four exposures using the reflected light from one emission, two counter circuits are installed in each pixel, and in the first exposure, the number of photons of the reflected light in range A is counted and stored using counter 1 of pixel 1.

[0156] In the second exposure, the number of photons of reflected light in range B is counted and stored using pixel 1 counter 2, and in the third exposure, the number of photons of reflected light in range C is counted and stored using pixel 2 counter 1. In addition, in the fourth exposure, the number of photons of reflected light in range D is counted and stored using pixel 2 counter 2.

[0157] That is, in this embodiment, the switch circuit is controlled so that reflected light from range A is counted by the first counter of the pixels in a predetermined row, and reflected light from range C or range D is counted by the first counter or second counter of the pixels in a row adjacent to the predetermined row. In this embodiment, range A functions as the first distance range, and ranges C and D function as the second distance range.

[0158] With the above configuration, photon count information that forms the basis of images for four target distance ranges can be obtained with a single flash, allowing for the capture of subjects within the target distance ranges. Furthermore, reducing the number of flashes reduces power consumption.

[0159] In the second embodiment, the resolution in the column direction (vertical scanning direction) is halved, and there are pixels for which information on the number of photons cannot be obtained. Therefore, when synthesizing images, it is necessary to interpolate using information from adjacent pixels that contain information on the number of photons.

[0160] 12A and 12B are diagrams showing a modification in which the pixel and counter used areas of the second embodiment are shown as four pixels.

[0161] As in Figure 11, the target distance ranges that are gradually farther away from the camera 600 are designated Range A, Range B, Range C, and Range D in order of proximity, and the pixels and counters used during exposure in each range are shown in gray.

[0162] 12 shows a predetermined 2×2 array of four pixels extracted from the pixel region 12, with pixel 1 and pixel 2 from the top left downwards, and pixel 3 and pixel 4 from the top right downwards. The first counter circuit is labeled C1 and the second counter circuit is labeled C2, both of which are written inside the pixels.

[0163] In the example shown in Figure 12(A), in range A, exposure information is stored in counter 1 for pixel 1 and counter 1 for pixel 3, and in range B, exposure information is stored in counter 1 for pixel 2 and counter 1 for pixel 4. In range C, exposure information is stored in counter 2 for pixel 1 and counter 2 for pixel 3, and in range D, exposure information is stored in counter 2 for pixel 2 and counter 2 for pixel 4.

[0164] In Fig. 12(A), the stored information is in the row direction, so the count control signal line can be connected to the signal processing circuits 103 arranged in a horizontal row as in Fig. 3(B) with a single wiring, allowing for a simple layout of the count control signal line.

[0165] 12B, in range A, exposure information is stored in counter 1 for pixel 1 and counter 1 for pixel 4, and in range B, exposure information is stored in counter 1 for pixel 2 and counter 1 for pixel 3. In range C, exposure information is stored in counter 2 for pixel 1 and counter 2 for pixel 4, and in range D, exposure information is stored in counter 2 for pixel 2 and counter 2 for pixel 3.

[0166] That is, in this embodiment, the switch circuit is controlled so that the reflected light from the first distance range is counted by the first counter of the pixel in the odd-numbered column of a predetermined row and the first counter of the pixel in the even-numbered column of a row adjacent to the predetermined row. Note that at this time, the switch circuit may be controlled so that the reflected light from the first distance range is counted by the second counter of the pixel in the even-numbered column of a row adjacent to the predetermined row.

[0167] 12(B), the stored information is arranged in a staggered manner, and the data for each range is acquired from the signal processing circuits 103 located in a staggered manner. Therefore, when synthesizing images, interpolation can be performed by referring to the information of the four adjacent pixels above, below, left, and right, so that the resolution can be improved compared to FIG. 12(A), in which information is acquired in a straight line.

[0168] Fig. 13 is a diagram showing an example of the configuration of the counter selection control generation unit 104 corresponding to the example of Fig. 12(B). In the explanation of Fig. 12(B), it was explained that the resolution can be improved by acquiring information from the signal processing circuits 103 located in a staggered pattern and interpolating it, but in that case the wiring becomes complicated as shown in Fig. 13.

[0169] In FIG. 13, in order to send a count control signal from the counter selection control generation unit 104 to the signal processing circuits 103 arranged in a staggered pattern relative to the signal processing circuits 103 arranged in the row direction, wiring is connected to every other signal processing circuit 103.

[0170] For example, in the top row of signal processing circuits 103, the count control signal from the counter selection control generation unit 104 is supplied to the first and third odd-numbered signal processing circuits 103 via a common wiring. Also, it is supplied to the second and fourth even-numbered signal processing circuits 103 via another common wiring. Therefore, two wirings are required for the signal processing circuits 103 lined up in the row direction, which increases the difficulty of circuit formation.

[0171] As described above, in order to perform four exposures using the reflected light from a single emission, two counter circuits are installed in each pixel, and the four counters installed in two pixels count and store information on the number of photons in four target ranges, thereby improving resolution.

[0172] Furthermore, the photon count information that forms the basis of images for the four target ranges can be obtained with a single flash, allowing for the capture of subjects within the target ranges. In this way, even when capturing images for the four target ranges, power consumption can be reduced by reducing the number of flashes.

[0173] <Embodiment 3> Hereinafter, a third embodiment of the present invention will be described. In the first embodiment, two counter circuits are provided in each pixel, and a method for acquiring images of two target distance ranges with one light emission is described. In the second embodiment, two counter circuits are provided in each pixel, and a method for acquiring images of four target distance ranges with one light emission is described.

[0174] In the third embodiment, a method for acquiring images of three target distance ranges with one light emission will be described, which combines the first and second embodiments. The imaging device in the third embodiment is basically the same as that in the first embodiment. Therefore, the following description will focus on the differences between the two, and a description of similar parts will be omitted. Parts corresponding to those in the first embodiment will be described with the same reference numerals.

[0175] Fig. 14 is a diagram showing an example in which camera 600 in embodiment 3 acquires images of three target ranges with one light emission. In Fig. 14, "driving image" shows a state in which target distance ranges that are gradually farther away from camera 600 mounted on a vehicle 820 are designated as range A, range B, and range C in order of proximity, with a subject present in each range and fog 810 present in all ranges.

[0176] The "Operation Image" shows the exposure time of the reflected light corresponding to each range after emitting pulsed light and the photon count status of the counter circuit, with the horizontal axis being the time axis.

[0177] In addition, the "used pixel / counter image" indicates that two adjacent pixels in the pixel region 12, for example in the column direction, are pixel 1 and pixel 2, and indicates the connection with two counter circuits mounted on pixel 1 and pixel 2, respectively.

[0178] In Figure 14, the first counter circuit is shown as counter 1 and the second counter circuit as counter 2. The pixels and counter circuits used are shown in gray. Note that Figure 14 shows an example of a camera 600 with two counter circuits per pixel.

[0179] The relationship between the distance range, exposure time, photon counting status, pixels used, and counter circuit when the camera 600 performs three exposures per light emission will be described.

[0180] The exposure for range A and range B (APD output is counted by a counter circuit) is the same as in the example of Figure 12(A). Of the four counter circuits for two pixels, one counter circuit is used for either range A or range B.

[0181] 14, counter 1 for pixel 1 is used for information on range A, and counter 1 for pixel 2 is used for information on range B. For exposure in range C, counter 2 for pixel 1 and counter 2 for pixel 2 are used, as in FIG.

[0182] In this embodiment, range A functions as a first distance range, range B functions as a second distance range, and range C functions as a third distance range, where the third distance range is farther from the imaging device than the first distance range.

[0183] Furthermore, the switch circuit is controlled so that the reflected light from the third distance range is counted by the second counter of pixel 1 in a predetermined row and also by the second counter of pixel 2 in a row adjacent to the predetermined row. However, in this case, the switch circuit may also be controlled so that the first counter of pixel 2 is used for counting.

[0184] By exposing in this manner, an image of range C, which is farther away from the camera 600, can be acquired with high resolution, which is advantageous for recognizing objects and characters.

[0185] On the other hand, ranges A and B are close to camera 600, so even if the resolution is lowered, it is relatively easy to recognize objects and characters. In this way, in this embodiment, it is possible to increase the number of target distance ranges in a way that is effective in practical use.

[0186] 10 to 14, the combination may be changed depending on the situation. For example, all pixels may be exposed in range A or range B. Furthermore, in the above embodiment, an example in which the resolution in the column direction (vertical scanning direction) is halved has been described, but a combination in which the resolution in the column direction (vertical scanning direction) is 1 / 3 or 1 / 4 may also be used.

[0187] Although the present invention has been described in detail above based on preferred embodiments, the present invention is not limited to these specific embodiments, and various modifications and combinations of the above embodiments are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. In addition, parts of the above embodiments may be combined as appropriate.

[0188] The present invention also includes those that realize the functions of the above embodiments using, for example, at least one processor such as a CPU, memory, or circuit (for example, ASIC). Also, multiple processors may be used to perform distributed processing.

[0189] In order to realize some or all of the controls in the above-described embodiments, a computer program that realizes the functions of the above-described embodiments may be supplied to an imaging device or the like via a network or various storage media.

[0190] The computer (or CPU, MPU, etc.) in the imaging device or the like may then read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. The present invention also includes the following combinations.

[0191] (Configuration 1) An imaging device comprising: a photoelectric conversion element having a plurality of pixels, each of which includes a sensor unit that emits pulses in response to incident photons, a first counter and a second counter that count the number of pulses, a first memory that stores the count value of the first counter, a second memory that stores the count value of the second counter, and a switch circuit that switches the connection between the sensor unit and the first counter or the connection between the sensor unit and the second counter; a light emitting unit that illuminates a subject; and a control unit that controls the switch circuit so that, after a single light emission from the light emitting unit, the first counter counts the reflected light from the subject within a first distance range and the second counter counts the reflected light from the subject within a second distance range.

[0192] (Configuration 2) The imaging device described in Configuration 1, characterized in that the plurality of pixels are arranged two-dimensionally across multiple rows and columns, and the control unit controls exposure of the pixels row by row by sequentially supplying count control signals to the plurality of pixels row by row.

[0193] (Configuration 3) The imaging device described in configuration 1 or 2, characterized in that the control unit controls the switch circuit to count reflected light from the first distance range using the first counter of the pixel in a specified row, and to count reflected light from the second distance range using the first counter or the second counter of the pixel in a row adjacent to the specified row.

[0194] (Configuration 4) The imaging device described in Configuration 3, characterized in that the control unit counts reflected light from a third distance range using the second counter of the pixel in the specified row, and controls the switch circuit to count using the first counter or the second counter of the pixel in a row adjacent to the specified row.

[0195] (Configuration 5) The imaging device according to configuration 4, wherein the third distance range is a distance range farther than the first distance range.

[0196] (Configuration 6) The imaging device described in any one of configurations 1 to 5, characterized in that the control unit controls the switch circuit to count reflected light from the first distance range with the first counter of the pixel in an odd column of a predetermined row, and to count with the first counter of the pixel in an even column of a row adjacent to the predetermined row.

[0197] (Configuration 7) The imaging device described in any one of configurations 1 to 6, characterized in that the control unit controls the switch circuit to count reflected light from the first distance range with the first counter of the pixel in an odd column of a predetermined row, and to count with the first counter of the pixel in an even column of a row adjacent to the predetermined row.

[0198] (Configuration 8) The imaging device according to any one of configurations 1 to 7, further comprising a recognition unit for calculating the first distance range and the second distance range by recognizing the subject.

[0199] (Method) An imaging method using a light-emitting unit for illuminating a subject and a photoelectric conversion element having a plurality of pixels, wherein the pixels each include a sensor unit that emits pulses in response to incident photons, a first counter and a second counter that count the number of the pulses, a first memory that stores the count value of the first counter, a second memory that stores the count value of the second counter, and a switch circuit that switches the connection between the sensor unit and the first counter or the connection between the sensor unit and the second counter, and the imaging method includes a control step of controlling the switch circuit so that the first counter counts reflected light from a first distance range due to a single emission of light from the light-emitting unit and the second counter counts reflected light from a second distance range.

[0200] (Program) A computer program for controlling each unit of the imaging device according to any one of configurations 1 to 8 by a computer. [Explanation of symbols]

[0201] 11: Sensor board 12: Pixel area 21: Circuit board 22: Circuit area 100: Photoelectric conversion element 101: Pixel 102: Photoelectric conversion unit 103: Signal processing circuit 104: Counter selection control generation unit 110: Vertical scanning circuit 111: Horizontal scanning circuit 112: Readout circuit 113: Vertical signal line 114: Output circuit 115: Control pulse generation unit 201:APD 202: Quench element 210: Waveform shaping section 211a: Counter circuit 211b: Counter circuit 212a: memory circuit 212b: memory circuit 213:RES signal line 214: Control pulse SEL line 215: Switch circuit 216: Counter selection signal line 600: Camera 601: Imaging optical system 603: Image processing unit 604: Recognition part 605: Camera control unit 606: Storage section 607: Communications Department 700: Mobile 701:ECU 702: Vehicle control unit 703: Display section 500: Light emitter 501: Light-emitting part 502: Light emission control unit 503: Communications Department 810: Fog 820: Vehicle

Claims

1. A photoelectric conversion element having a plurality of pixels, wherein the pixels are: a sensor unit that emits a pulse in response to incident photons; a first counter and a second counter for counting the number of pulses; a first memory for storing the count value of the first counter; a second memory that stores the count value of the second counter; a photoelectric conversion element including a switch circuit for switching the connection between the sensor unit and the first counter or the connection between the sensor unit and the second counter; a light emitting unit for illuminating a subject; a control unit that controls the switch circuit so that the first counter counts the amount of light reflected from the subject within a first distance range and the second counter counts the amount of light reflected from the subject within a second distance range, resulting from a single light emission from the light emitting unit; An imaging device comprising:

2. the plurality of pixels are two-dimensionally arranged across a plurality of rows and columns, 2. The imaging device according to claim 1, wherein the control unit controls exposure of the pixels row by row by sequentially supplying count control signals to the plurality of pixels row by row.

3. 2. The imaging device according to claim 1, wherein the control unit controls the switch circuit so that reflected light from the first distance range is counted by the first counter of the pixel in a predetermined row, and reflected light from the second distance range is counted by the first counter or the second counter of the pixel in a row adjacent to the predetermined row.

4. 4. The imaging device according to claim 3, wherein the control unit controls the switch circuit so that reflected light from a third distance range is counted by the second counter of the pixel in the specified row and also by the first counter or the second counter of the pixel in a row adjacent to the specified row.

5. 5. The imaging device according to claim 4, wherein the third distance range is a distance range farther than the first distance range.

6. 2. The imaging device according to claim 1, wherein the control unit controls the switch circuit so that reflected light from the first distance range is counted by the first counter of the pixel in an odd-numbered column of a predetermined row, and is also counted by the first counter of the pixel in an even-numbered column of a row adjacent to the predetermined row.

7. 2. The imaging device according to claim 1, further comprising a synthesis unit that synthesizes an image obtained based on the count value of the first counter and an image obtained based on the count value of the second counter.

8. The imaging device according to claim 1 , further comprising a recognition unit for calculating the first distance range and the second distance range by recognizing the subject.

9. An imaging method using a light-emitting unit for illuminating a subject and a photoelectric conversion element having a plurality of pixels, The pixel is a sensor unit that emits a pulse in response to incident photons; a first counter and a second counter for counting the number of pulses; a first memory for storing the count value of the first counter; a second memory that stores the count value of the second counter; a switch circuit for switching the connection between the sensor unit and the first counter or the connection between the sensor unit and the second counter, a control step of controlling the switch circuit so that the first counter counts reflected light from a first distance range and the second counter counts reflected light from a second distance range due to one light emission from the light emitting unit; An imaging method comprising:

10. A computer program for controlling each unit of the imaging device according to any one of claims 1 to 8 by a computer.

Images