Photoelectric conversion device, movable body, photoelectric conversion method, and computer program

Abstract

To provide a photoelectric conversion device that can read out a result of accumulation in a short time during an accumulation period.SOLUTION: A photoelectric conversion device has: a plurality of pixels each including a sensor unit that generates pulses at a frequency in accordance with the frequency of photon reception, and a counter that counts the number of the pulses; and control means that performs control to: generate signals on the basis of the difference between a count value of the counter at the start of an accumulation period and a count value of the counter at the end of the accumulation period, has a first accumulation period and a second accumulation period in a full frame, the first accumulation period being shorter than the second accumulation period; and output the signals generated in the first accumulation period in a period from the end of the first accumulation period until the end of the second accumulation period.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a photoelectric conversion device, a moving body, a photoelectric conversion method, a computer program, and the like. [Background technology]

[0002] In recent years, photoelectric conversion devices have been developed that digitally count the number of photons incident on an avalanche photodiode (APD) and output the counted value from a pixel as a photoelectrically converted digital signal.

[0003] Furthermore, for example, Patent Document 1 describes a configuration in which a photoelectric conversion device having an APD can output multiple images whose accumulation periods overlap each other, thereby enabling continuous shooting even under low illumination. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7223070 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when considering the image sensor of an in-vehicle camera installed on a moving object, for example, normal sensor driving performs recognition processing on a frame-by-frame basis, so in the case of 30 fps, for example, recognition processing can only be performed every 33.3 ms. Therefore, even if an object appears immediately after a frame change in an in-vehicle camera, recognition processing cannot be performed until the end of the frame.

[0006] Furthermore, in order to suppress flicker caused by LED traffic lights, in-vehicle cameras often have an accumulation period that is longer than the LED blinking cycle (for example, 10 msec for 100 Hz), and particularly in low light conditions, a longer accumulation period is set to capture brighter images. However, a longer accumulation period causes subject blurring for fast-moving subjects, and even if the subject is stationary, blurring occurs if the camera is moving at high speed, resulting in a lower recognition rate.

[0007] Therefore, an object of the present invention is to provide a photoelectric conversion device that can read out the results of short-term accumulation midway. [Means for solving the problem]

[0008] A photoelectric conversion device according to one aspect of the present invention comprises: a sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons; a plurality of pixels each including a counter that counts the number of pulses; generating a signal based on a difference between the count values ​​of the counter at the start and end of an accumulation period; The image sensor is characterized by having a first accumulation period and a second accumulation period within a full frame, the first accumulation period being shorter than the second accumulation period, and having a control means that controls so that a signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a photoelectric conversion device that is capable of reading out the results of short-term accumulation midway. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating a configuration example of a photoelectric conversion element according to an embodiment of the present invention. [Figure 2] 2 is a diagram showing an example of the configuration of a sensor substrate 11. FIG. [Figure 3] 2 is a diagram showing an example of the configuration of a circuit board 21. FIG. [Figure 4] 2 and 3, and shows an example of an equivalent circuit of the pixel 101 and the signal processing circuit 103 corresponding to the pixel 101. FIG. [Figure 5] 2 is a diagram schematically illustrating the relationship between the operation of an APD 201 and an output signal. FIG. [Figure 6] 1 is a functional block diagram of a photoelectric conversion device 600 and a moving object 700 according to an embodiment. [Figure 7] 10A and 10B are diagrams for explaining an example of a photoelectric conversion method by a camera control unit 605 according to an embodiment. [Figure 8] FIG. 10 is a diagram for explaining an example of output via a buffer according to the embodiment. [Figure 9] FIG. 10 is a diagram showing an example of an image of a plurality of divided frames. [Figure 10] FIG. 10 is a diagram illustrating an example of a relationship between a counter circuit and a buffer according to an embodiment. [Figure 11] 10 is a flowchart showing details of an example of driving a photoelectric conversion element in the embodiment. [Figure 12] 12 is a flowchart continuing from FIG. 11. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions will be omitted or simplified.

[0012] 1 is a diagram showing an example of the configuration of a photoelectric conversion element according to an embodiment of the present invention. In the following, the photoelectric conversion element 100 will be described as an example of a photoelectric conversion device having a so-called stacked structure, which is configured by stacking and electrically connecting two substrates, a sensor substrate 11 and a circuit substrate 21.

[0013] However, a so-called non-laminated structure may be used in which the components included in the sensor substrate and the components included in the circuit substrate are arranged on a common semiconductor layer. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes signals detected in the pixel region 12.

[0014] 2 is a diagram showing an example of the configuration of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally in rows and columns. Each pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter referred to as an APD).

[0015] Here, the photoelectric conversion unit 102 functions as a sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons. The number of rows and columns of the pixel array that forms the pixel region 12 is not particularly limited.

[0016] 3 is a diagram showing an example of the configuration of the circuit board 21. The circuit board 21 has a signal processing circuit 103 that processes charges photoelectrically converted by each photoelectric conversion unit 102 in FIG. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114.

[0017] The vertical scanning circuit 110 receives control pulses supplied from the control pulse generating unit 115 and supplies the control pulses to a plurality of pixels arranged in the row direction, row by row, in sequence. The vertical scanning circuit 110 uses logic circuits such as a shift register and an address decoder.

[0018] The signal output from the photoelectric conversion unit 102 of each pixel is processed by each signal processing circuit 103. The signal processing circuit 103 is provided with a counter, memory, etc., and digital values ​​are stored in the memory. In order to read the signal from the memory of each pixel where the digital signal is stored, the horizontal scanning circuit 111 inputs a control pulse that sequentially selects each column to the signal processing circuit 103.

[0019] A signal is output to the vertical signal line 113 from the signal processing circuit 103 of the pixel of the row selected by the vertical scanning circuit 110. The signal output to the vertical signal line 113 is output to the outside of the photoelectric conversion element 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has a plurality of buffers built in, which are connected to each vertical signal line 113.

[0020] 2 and 3, a plurality of signal processing circuits 103 are arranged in an area overlapping the pixel area 12 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged so as to overlap between an end of the sensor substrate 11 and an end of the pixel area 12 in a plan view.

[0021] In other words, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in a region overlapping the non-pixel region in a plan view.

[0022] The arrangement of the vertical signal lines 113, the readout circuits 112, and the output circuits 114 is not limited to the example shown in Fig. 3. For example, the vertical signal lines 113 may be arranged extending in the row direction, and the readout circuits 112 may be arranged at the ends of the vertical signal lines 113. Furthermore, it is not necessary to provide one signal processing circuit 103 for each photoelectric conversion unit, and one signal processing circuit may be shared by multiple photoelectric conversion units to perform signal processing sequentially.

[0023] 2 and 3 and an example of an equivalent circuit of the signal processing circuit 103 corresponding to the pixel 101. As shown in Fig. 4, each pixel 101 includes a photoelectric conversion unit 102 as a sensor unit, a counter circuit 211, etc.

[0024] The APD 201 included in the photoelectric conversion unit 102 generates charge pairs in response to incident light through photoelectric conversion. One of the two nodes of the APD 201 (anode) is connected to a power supply line that supplies a voltage VL (first voltage). The other of the two nodes of the APD 201 (cathode) is connected to a power supply line that supplies a voltage VH (second voltage) that is higher than the voltage VL.

[0025] A reverse bias voltage is applied to the anode and cathode of the APD 201 so that the APD 201 performs avalanche multiplication. With this voltage applied, the charge generated by the incident light undergoes avalanche multiplication, generating an avalanche current.

[0026] When a reverse bias voltage is supplied, there are two modes: Geiger mode, in which the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, in which the voltage difference between the anode and cathode is close to or less than the breakdown voltage. An APD operating in Geiger mode is called a SPAD. In the case of a SPAD, the voltage VL (first voltage) is, for example, -30 V, and the voltage VH (second voltage) is, for example, 1 V.

[0027] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, and a counter circuit 211. The quench element 202 is connected to a power supply line to which a drive voltage VH is supplied and one of the anode and cathode nodes of the APD 201.

[0028] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, suppressing the voltage supplied to the APD 201 and suppressing avalanche multiplication (quench operation).The quench element 202 also functions to return the voltage supplied to the APD 201 to voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0029] FIG. 4 shows an example in which the signal processing circuit 103 includes a waveform shaping section 210 and a counter circuit 211 in addition to the quench element 202 .

[0030] The waveform shaping unit 210 shapes the voltage change at the cathode of the APD 201 obtained when photons are detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. While Fig. 4 shows an example in which one inverter is used as the waveform shaping unit 210, a circuit in which multiple inverters are connected in series, or another circuit that has a waveform shaping effect, may also be used.

[0031] The counter circuit 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. When a control pulse RES is supplied via a drive line (RES) 212, the signal held in the counter circuit 211 is reset. Here, the counter circuit 211 generates a signal based on the difference between the count values ​​at the start and end of the accumulation period.

[0032] Furthermore, when a control pulse SEL is supplied to the counter circuit 211 from the vertical scanning circuit 110 in FIG. 3 via a drive line (SEL) 213 (not shown in FIG. 3) in FIG. 4, the counter circuit 211 stops counting the number of pulses and outputs the count value to the vertical signal line 113.

[0033] Note that electrical connections may be switched by disposing a switch such as a transistor between the quench element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103. Similarly, the supply of the voltage VH or the voltage VL to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0034] 5 is a diagram schematically illustrating the relationship between the operation of the APD 201 and the output signal. The input side of the waveform shaping unit 210 is designated node A, and the output side is designated node B. Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage at node A drops.

[0035] When the voltage drop amount further increases and the potential difference applied to the APD 201 decreases, the avalanche multiplication of the APD 201 stops as at time t2, and the voltage level of node A does not drop below a certain value.

[0036] After that, between time t2 and time t3, a current flows through node A to compensate for the voltage drop from voltage VL, and at time t3, node A settles to its original potential level. At this time, the part of the output waveform at node A that exceeds a certain threshold is shaped by waveform shaping unit 210 and output as a pulse signal at node B.

[0037] Next, a photoelectric conversion device 600 and a mobile object 700 according to an embodiment will be described. Fig. 6 is a functional block diagram of the photoelectric conversion device 600 and the mobile object 700 according to the embodiment. Note that some of the functional blocks shown in Fig. 6 are realized by causing a computer (not shown) included in the photoelectric conversion device 600 and the mobile object 700 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0038] However, some or all of these functions may be implemented by hardware. Examples of hardware that can be used include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs). Furthermore, the functional blocks shown in Fig. 6 do not have to be built into the same housing, and may be configured as separate devices connected to each other via signal paths.

[0039] 1 to 5, an imaging optical system 601, an image processing unit 603, a recognition unit 604, a camera control unit 605, a storage unit 606, a communication unit 607, etc. The photoelectric conversion element 100 is configured by an avalanche photodiode for photoelectrically converting an optical image, as described in FIGS.

[0040] The photoelectric conversion device of this embodiment is mounted on a moving body 700, and a camera unit consisting of a set of an imaging optical system 601 and a photoelectric conversion element 100 is disposed so as to capture an image in at least one direction, for example, in front, behind, or to the side of the moving body. Note that a plurality of camera units may be provided on the moving body 700.

[0041] The image processing unit 603 performs image processing such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaic processing, and data compression on the image signal acquired by the photoelectric conversion element 100 to generate a final image signal. Note that if each pixel of the photoelectric conversion element 100 has an on-chip color filter such as RGB, the image processing unit 603 also performs processing such as white balance correction and color conversion.

[0042] Furthermore, when the read image is to be used for multiple purposes, such as for visual confirmation and recognition, it is desirable to store different image processing parameters in advance for each purpose for which the image is to be used, and to switch the image processing parameters as the purpose for which the image is to be used is switched.

[0043] The output of the image processing unit 603 is supplied to a recognition unit 604, an ECU (Electric Control Unit) 701 of the moving object 700, and a camera control unit 605. The recognition unit 604 functions as a recognition means that recognizes subjects by performing image recognition based on the image signal, and recognizes subjects such as people, animals, vehicles, traffic lights, and signs around the moving object 700 and identifies their types.

[0044] The recognition result acquired by the recognition unit 604 is output to the ECU 701. In this embodiment, the mobile object 700 will be described using an example of an automobile, but the mobile object may be any object that is movable, such as an airplane, a train, a ship, a drone, an AGV, or a robot.

[0045] The camera control unit 605 has a built-in CPU as a computer and memory that stores a computer program, and controls each unit of the photoelectric conversion device 600 by the CPU executing the computer program stored in the memory.

[0046] The camera control unit 605 functions as a control means, and controls the length of the accumulation period (photoelectric conversion period) of each frame of the photoelectric conversion element 100 and the timing of the control signal CLK, for example, via a control pulse generation unit of the photoelectric conversion element 100. The camera control unit 605 also has a function to set various image processing parameters in the image processing unit 603, a function to control the recognition unit 604, and a function to obtain the recognition results.

[0047] The storage unit 606 includes a recording medium such as a memory card or a hard disk, and can store and read image signals. The communication unit 607 includes a wireless or wired interface, and outputs the generated image signals to the outside of the photoelectric conversion device 600 and receives various signals from the outside.

[0048] The communication unit 607 may perform wired communication in accordance with standards such as SPI or I2C as an interface, or may perform communication using a wireless LAN system such as Wi-Fi or Bluetooth (registered trademark).

[0049] The ECU 701 incorporates a CPU as a computer and a memory that stores a computer program, and controls each part of the moving body 700 by the CPU executing the computer program stored in the memory.

[0050] The output of the ECU 701 is supplied to a vehicle control unit 702 and a display unit 703. The vehicle control unit 702 functions as a movement control means that controls the movement (driving, stopping, direction control, etc.) of the vehicle as a moving body based on the output of the ECU 701. Furthermore, the vehicle control unit 702 as a movement control means controls the moving body based on the recognition result output from the recognition unit.

[0051] The display unit 703 functions as a display means, includes a display element such as a liquid crystal device or an organic EL, and is mounted on the mobile object 700. Based on the output of the ECU 701, the display unit 703 displays to the driver of the mobile object 700, for example, using a GUI, images acquired by the photoelectric conversion element 100, recognition results acquired by the recognition unit 604, various information related to the vehicle's running state, etc.

[0052] It should be noted that the image processing unit 603, the recognition unit 604, etc. in Figure 6 do not have to be mounted on the moving body 700, and may be provided, for example, in an external terminal provided separately from the moving body 700, for remotely controlling the moving body 700 or for monitoring the movement of the moving body.

[0053] 7 is a diagram illustrating an example of a photoelectric conversion method by the camera control unit 605 according to the embodiment. In this embodiment, photoelectric conversion is periodically driven at, for example, 30 full frames per second. Furthermore, a frame corresponding to one vertical period having a length of 33.3 ms is called a full frame, and each of the four divisions of a full frame is called a frame.

[0054] That is, as shown in FIG. 7, a full frame 1 from time T0 to time T12 is divided into frames 1_1, 1_2, 1_3, and 1_4, each having an equal period (8.33 ms).

[0055] Assuming that the signal processing circuits 103 in FIG. 3 are arranged in N rows and M columns, the first row of frame 1_1 has an accumulation period (photoelectric conversion period) from time T1 to time T3, and the Nth row of frame 1_1 has an accumulation period from time T3 to time T5.

[0056] From the second row to the (N-1)th row of frame 1_1, the accumulation start time is sequentially delayed by the readout time of one row, and each row has an accumulation period of the same length. Also, the first row of frame 1_2 has a total accumulation period from time T1 to time T3 and from time T4 to time T6, and the Nth row of frame 1_2 has a total accumulation period from time T3 to time T5 and from time T6 to time T8.

[0057] From the second row to the N-1th row of frame 1_2, the accumulation start time is delayed by the readout time of one row, and there is a pause once along the way for the readout time of one row, with each row having an accumulation period of the same length.

[0058] Furthermore, the first row of frame 1_3 has a total accumulation period from time T1 to time T3, time T4 to time T6, and time T7 to time T9, and the Nth row of frame 1_3 has a total accumulation period from time T3 to time T5, time T6 to time T8, and time T9 to time T11. From the second row to the (N-1)th row of frame 1_3, the accumulation start time is delayed by the readout time of one row, and each row has an accumulation period of the same length, with two pauses along the way for the readout time of one row in sequence.

[0059] Furthermore, the first row of frame 1_4 has a total accumulation period of time T1 to time T3, time T4 to time T6, time T7 to time T9, and time T10 to time T12. Furthermore, the Nth row of frame 1_4 has a total accumulation period of time T3 to time T5, time T6 to time T8, time T9 to time T11, and time T12 to time T14.

[0060] From the second row to the N-1th row of frame 1_4, the accumulation start time is delayed by the readout time of one row, and there are three stops along the way, each for the readout time of one row, with each row having an accumulation period of the same length.

[0061] The counter circuit 211 of the first row is reset at time T1. Also, the control pulse SEL is supplied to the counter circuit 211 of the first row at times T3 to T4, T6 to T7, T9 to T10, and T12 to T13, and the counting stops. As a result, the count values ​​C1_1_1, C1_2_1, C1_3_1, and C1_4_1 are held, respectively.

[0062] The held signals for one row are sequentially output from the photoelectric conversion elements via the buffer of the readout circuit 112 during the holding period. Thereafter, the counter circuit 211 for the first row is reset at time T13. The reset operation is performed at time T1 in full frame 1 and time T13 in full frame 2. That is, the counter circuit 211 is reset every full frame period.

[0063] The counter circuit 211 in the Nth row is reset at time T3. Furthermore, the counter circuit 211 in the Nth row is supplied with a control pulse SEL at times T5 to T6, T8 to T9, T11 to T12, and T14 to T15, and stops counting.

[0064] As a result, count values ​​C1_1_N, C1_2_N, C1_3_N, and C1_4_N are held, respectively. The held signals for one row are sequentially output from the photoelectric conversion elements via a buffer in the readout circuit 112 during the holding period, as shown in Fig. 8. Note that Fig. 8 is a diagram for explaining an example of output via a buffer according to the embodiment.

[0065] Although not shown, the counter circuits 211 in the second to (N-1)th rows operate at timings delayed by the readout time of one row compared to the counter circuit 211 in the first row.

[0066] As described above, according to this embodiment, the image signal of frame 1_1 is read out from time T3 to T6 and is promptly processed by the recognition unit 604. Therefore, image recognition can be performed promptly. Similarly, the signals of frames 1_2, 1_3, and 1_4 are sequentially read out from time T6 to T9, T9 to T12, and T12 to T15, respectively, and image recognition can be repeatedly performed at the cycle of each frame.

[0067] FIG. 9 is a diagram showing an example of images of multiple divided frames. As shown in FIG. 9, the image of frame 1_1 is dark because the accumulation time is short, but there is little subject blur of a person who suddenly appears in front of the camera. On the other hand, the images read out when frames 1_2, 1_3, and 1_4 are read out are more likely to have subject blur because the accumulation times are longer in that order. Note that stopped vehicles and white lines are less likely to have blur, and the longer the accumulation time, the more likely the contrast is to improve.

[0068] As described above, in this embodiment, one full frame has a first accumulation period (for example, time T1 to time T3) and a second accumulation period (for example, the sum of time T1 to time T3, time T4 to time T6, time T7 to time T9, and time T10 to time T12). Furthermore, the first accumulation period is shorter than the second accumulation period, and the signal generated during the first accumulation period is controlled to be output between the end of the first accumulation period and the end of the second accumulation period.

[0069] In this embodiment, the first and second accumulation periods overlap, and start simultaneously. Furthermore, the end of the second accumulation period for the first row coincides with the end of a full-frame period (one vertical period) (time T12). That is, the end of the second accumulation period coincides with the end of a full frame. Furthermore, the second accumulation period is an integral multiple of the first accumulation period (four times in the example of FIG. 7).

[0070] However, the second accumulation period does not need to be an integer multiple of the first accumulation period; the second accumulation period only needs to be longer than the first accumulation period and the end of the second accumulation period for the same row must be after the end of the first accumulation period.

[0071] That is, an image with a short accumulation period and an image with a long accumulation period are created, and the timing at which the short accumulation period ends is set earlier than the timing at which the long accumulation period ends. As soon as the short accumulation period ends, the image is output and sent to a downstream recognition unit. The subject is recognized based on signals generated during at least the first accumulation period. The recognition unit 604 as a recognition means recognizes the subject based on signals generated during at least the first accumulation period.

[0072] Therefore, while image recognition was only possible in a full-frame period in the prior art, in this embodiment, image recognition can be performed in every 1 / 4 full-frame period, and for example, when a moving object is moving at high speed, obstacles and the like can be quickly recognized. Furthermore, if the accumulation period is further shortened, image recognition can be performed more frequently.

[0073] This allows for the driver to brake quickly or avoid obstacles early. Furthermore, image recognition can be performed every 1 / 4 full frame period, so obstacles that appear during that time can be recognized in a timely manner.

[0074] Furthermore, the display device of this embodiment displays, as an image, a signal generated during at least the second accumulation period. Note that an image generated during the second accumulation period, which has a longer accumulation period, can improve contrast and is therefore suitable as a display image. That is, an image generated during the first accumulation period, which has a shorter accumulation period, is suitable for rapid subject recognition, and an image generated during the second accumulation period, which has a longer accumulation period, is suitable as a display image.

[0075] In this embodiment, since an APD is used, unlike a CMOS sensor, the accumulated charge does not deteriorate due to the readout operation, so the accumulation periods can be overlapped. Also, since there is no readout noise, the original signal does not deteriorate no matter how many times it is read out from a single accumulation.

[0076] Fig. 10 is a diagram showing an example of the relationship between counter circuits and buffers in an embodiment. Fig. 10 shows a state in which the counter circuits 211 in the signal processing circuit 103 in Fig. 3 are arranged in N rows and M columns, and each counter circuit is represented as counter 1-1 to counter NM. Also, buffer 1 to buffer M in Fig. 10 represent buffers included in the readout circuit 112 in Fig. 3. Output circuit 114 in Fig. 10 corresponds to output circuit 114 in Fig. 3.

[0077] Fig. 11 is a flowchart showing details of an example of driving a photoelectric conversion element in an embodiment, and Fig. 12 is a flowchart continuing from Fig. 11. Note that the operations of the steps in the flowcharts of Fig. 11 and Fig. 12 are performed sequentially by a CPU or the like serving as a computer in the camera control unit 605 executing a computer program stored in memory.

[0078] 12 and 13, i corresponds to the frame number in frame 1_i in FIGS. 7 and 8, and takes on values ​​from 1 to 4. j represents the number of rows in FIG. 10, and takes on values ​​from 1 to N. k represents the number of columns in FIG. 10, and takes on values ​​from 1 to M. Also, the count value of counter circuit jk in the jth row and kth column in frame 1_i is denoted as Count(jki). Below, we will explain the driving when reading out full frame 1.

[0079] In step S101, i is set to 1, and then in step S102, j is set to 1. That is, the first row of the first frame is targeted.

[0080] Next, in step S103, counting Count(j-1-i) to Count(jMi) is stopped. At this time, counting is stopped simultaneously for columns 1 to M. This operation corresponds to the count stopping operation at time T0, T3, T6, T9, or T12 for the first row in FIG. 7. Also, it corresponds to the count stopping operation at time T2, T5, T8, T11, or T14 for the Nth row.

[0081] Next, in step S104, the count values Count(j - 1 - i) to Count(j - M - i) are output to buffers 1 to M respectively. At this time, the output to the buffers is performed for columns 1 to M simultaneously.

[0082] Next, in step S105, k is set to 1, and in step S106, the count value Count(j - k - i) of buffer k is output to the output circuit 114. This operation corresponds to the operation of reading the signal of the buffer in the leftmost column of FIG. 10 from the output circuit.

[0083] Next, in step S107, it is determined whether k < M. If Yes, then in step S108, k is incremented by 1 as k = k + 1, and the process returns to step S106 to perform the operation of step S106. This operation corresponds to the operation of reading the signal of the buffer in the second column from the left in FIG. 10 from the output circuit.

[0084] When it becomes No in step S107, that is, when k = M, it means that the signal of the buffer in the Mth column of FIG. 10 has been read out from the output circuit.

[0085] Next, in step S109, it is determined whether i < 4. If it becomes No, that is, when i = 4, the count values Count(j - 1 - i) to Count(j - M - i) are reset. At this time, the reset is performed for columns 1 to M simultaneously. This operation corresponds to the operation at time T13 for the first row and time T15 for the Nth row in FIG. 7. If Yes in step S109, the process proceeds to step S111.

[0086] Next, in step S111, the count stop of the count values Count(j - 1 - i) to Count(j - M - i) is released. At this time, the count stop is released for columns 1 to M simultaneously. This operation corresponds to the operations at times T1, T4, T7, T10, or T13 for the first row and times T3, T6, T9, T12, or T15 for the Nth row in FIG. 7. <​Next, in step S112, it is determined whether j < N. If Yes, then in step S113, j = j + 1, incrementing j by 1, and return to step S103. This corresponds to the operation for starting the reading of the next line.

[0088] If it is determined as No in step S112, it means that the reading of all lines has been completed. So, proceed to step S114 and determine whether i < 4. If it is determined as Yes in step S114, then proceed to step S115, set i = i + 1 to increment i by 1, and return to step S102. This operation corresponds to the operation for starting the reading of the next frame 1_2.

[0089] If it becomes No in step S114, that is, when i = 4, it means that the reading of the full frame 1 has been completed, so end the process. As described above, the signals accumulated in the photoelectric conversion element 100 can be sequentially read out.

[0090] As described above, the present invention has been described in detail based on its preferred embodiments. However, the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention and are not excluded from the scope of the present invention.

[0091] For example, in the above embodiment, the accumulation is performed for a minimum of 1 / 4 full frame period. However, depending on the recognition accuracy in the recognition unit 604, the length of the shortest accumulation period may be changed to, for example, 1 / 5 full frame period or 1 / 3 full frame period. Or, depending on the brightness of the subject, the length of the shortest accumulation period may be changed.

[0092] Furthermore, even when the readout period is 1 / 4 full frame period, depending on the brightness of the subject or the image recognition accuracy, etc., the counter circuit may be reset during the accumulation period of frame 1_1 in FIG. 7. Thereby, the substantial accumulation period may be made shorter than 1 / 4 full frame period.

[0093] Alternatively, the counter circuit of the first row may be reset once at time T4 in Fig. 7. This may adjust the count values ​​of frames 1_2, 1_3, and 1_4, which start to be read out at times T6, T9, and T12, respectively.

[0094] In addition, the accumulation period of the sensor may be set to an integer multiple of the readout time for one row, but it may also be possible to add the number of rows at which counting stops to the set number of rows and start accumulation at a timing earlier by that total time. By doing this, even if counting stops midway, it is possible to achieve an accumulation period for the set number of rows accurately.

[0095] In this embodiment, when reading out the count value of one row, the counting of that row is stopped, but if the time required for reading is short compared to the count-up interval and the timing can be controlled, it is not necessarily necessary to stop the counting.

[0096] The present embodiment includes the following combinations.

[0097] (Configuration 1) A photoelectric conversion device characterized by having a plurality of pixels, each pixel having a sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons and a counter that counts the number of said pulses, and a control means that generates a signal based on the difference between the count values ​​of said counter at the start and end of an accumulation period, has a first accumulation period and a second accumulation period within a full frame, said first accumulation period is shorter than said second accumulation period, and controls so that the signal generated during said first accumulation period is output between the end of said first accumulation period and the end of said second accumulation period.

[0098] (Configuration 2) The photoelectric conversion device according to configuration 1, wherein the control means controls the counter so that the counting in the counter stops when the count value of the counter is read out.

[0099] (Configuration 3) The photoelectric conversion device according to configuration 1 or 2, wherein the first accumulation period and the second accumulation period overlap.

[0100] (Configuration 4) The photoelectric conversion device according to any one of configurations 1 to 3, wherein the first accumulation period and the second accumulation period start simultaneously.

[0101] (Configuration 5) The photoelectric conversion device according to any one of configurations 1 to 4, wherein the end of the second accumulation period coincides with the end of a full frame.

[0102] (Configuration 6) The photoelectric conversion device according to any one of configurations 1 to 5, further comprising a recognition means for recognizing a subject based on a signal generated during at least the first accumulation period.

[0103] (Configuration 7) The photoelectric conversion device according to configuration 6, wherein the recognition means further recognizes the subject based on a signal generated during the second accumulation period.

[0104] (Configuration 8) The photoelectric conversion device according to any one of configurations 1 to 7, further comprising a display means for displaying, as an image, at least the signal generated during the second accumulation period.

[0105] (Configuration 9) The photoelectric conversion device according to any one of configurations 1 to 8, wherein the sensor section includes an avalanche photodiode.

[0106] (Configuration 10) The photoelectric conversion device according to any one of Configurations 1 to 9; and a movement control means for controlling the movement of the moving body.

[0107] (Method) A photoelectric conversion method for performing photoelectric conversion using a plurality of pixels, each of which has a sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons, and a counter that counts the number of said pulses, characterized in that a signal is generated based on the difference in the count values ​​of said counter at the start and end of an accumulation period, and the photoelectric conversion method has a first accumulation period and a second accumulation period within a full frame, the first accumulation period is shorter than the second accumulation period, and the signal generated during the first accumulation period is controlled to be output between the end of the first accumulation period and the end of the second accumulation period.

[0108] (Program) A computer program for controlling the photoelectric conversion device according to any one of configurations 1 to 9 or each means of the moving body according to configuration 10 by a computer.

[0109] Note that a computer program that realizes part or all of the control in this embodiment and the functions of the above-described embodiment may be supplied to the photoelectric conversion device via a network or various storage media. Then, a computer (or a CPU, MPU, etc.) in the photoelectric conversion device may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0110] 11: Sensor board 12: Pixel area 21: Circuit board 22: Circuit area 100: Photoelectric conversion element 101: Pixel 102: Photoelectric conversion unit 103: Signal processing circuit 110: Vertical scanning circuit 111: Horizontal scanning circuit 112: Readout circuit 113: Vertical signal line 114: Output circuit 115: Control pulse generation unit 201: Avalanche photodiode 202: Quench element 210: Waveform shaping section 211: Counter circuit 212: Drive line (RES) 213: Drive line (SEL) 600: Photoelectric conversion device 601: Imaging optical system 603: Image processing unit 604: Recognition part 605: Camera control unit 606: Storage section 607: Communications Department 700: Mobile 701:ECU 702: Vehicle control unit 703: Display section

Claims

1. a sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons; a plurality of pixels each including a counter that counts the number of pulses; generating a signal based on a difference between the count values ​​of the counter at the start and end of an accumulation period; a control means for controlling a photoelectric conversion device to have a first accumulation period and a second accumulation period within a full frame, the first accumulation period being shorter than the second accumulation period, and to output a signal generated during the first accumulation period between the end of the first accumulation period and the end of the second accumulation period.

2. 2. The photoelectric conversion device according to claim 1, wherein the control means controls the counter so that the counting in the counter stops when the count value of the counter is read out.

3. 2. The photoelectric conversion device according to claim 1, wherein the first accumulation period and the second accumulation period overlap each other.

4. 2. The photoelectric conversion device according to claim 1, wherein the first accumulation period and the second accumulation period start simultaneously.

5. 2. The photoelectric conversion device according to claim 1, wherein the end of the second accumulation period coincides with the end of a full frame.

6. 2. The photoelectric conversion device according to claim 1, further comprising recognition means for recognizing a subject based on a signal generated during at least the first accumulation period.

7. 7. The photoelectric conversion device according to claim 6, wherein the recognition means further recognizes the subject based on a signal generated during the second accumulation period.

8. 2. The photoelectric conversion device according to claim 1, further comprising a display means for displaying, as an image, at least the signal generated during the second accumulation period.

9. 2. The photoelectric conversion device according to claim 1, wherein the sensor unit includes an avalanche photodiode.

10. The photoelectric conversion device according to any one of claims 1 to 9, and a movement control means for controlling the movement of the moving body.

11. a sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons; a counter that counts the number of pulses; A photoelectric conversion method for performing photoelectric conversion using a plurality of pixels, each of which includes: generating a signal based on a difference between the count values ​​of the counter at the start and end of an accumulation period; having a first integration period and a second integration period within a full frame, the first accumulation period is shorter than the second accumulation period; A photoelectric conversion method comprising controlling the signal generated during the first accumulation period to be output between the end of the first accumulation period and the end of the second accumulation period.

12. A computer program for controlling the photoelectric conversion device according to any one of claims 1 to 9 or each means of the moving body according to claim 10 by a computer.

Images