Photoelectric conversion device, mobile, processing method, and computer program

Abstract

To provide a photoelectric conversion device which suppresses reduction of recognition accuracy of an image, a processing method therefor and a program.SOLUTION: In a photoelectric conversion device, each of a plurality of pixels includes: a first counter and a second counter for counting the number of pulses from a sensor section; and a first memory and a second memory for storing count values. In an operation for reading count values stored during a plurality of different storage periods within a full frame period out of the first counter and the second counter, a camera control section 605 controls the plurality of pixels in such a manner that the count values for a first storage period are read out until the end of a second storage period which is longer than the first storage period, and performs control in such a manner that timing of reading-out for each storage period is different between the first counter and the second counter. A differential image generation section 608 generates a differential image based on a differential of signals corresponding to the count values outputted from the first memory and the second memory.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present disclosure relates to techniques for photoelectric conversion and image processing in a photoelectric conversion device. [Background technology]

[0002] In recent years, photoelectric conversion devices have been developed that digitally count the number of photons incident on an avalanche photodiode (APD) and output the counted value from a pixel as a photoelectrically converted digital signal. For example, Patent Document 1 describes a configuration in which a photoelectric conversion device having an APD can output multiple images whose accumulation periods overlap with each other, thereby enabling continuous shooting even in low illuminance.

[0003] Meanwhile, in-vehicle camera systems aimed at autonomous driving and collision avoidance prevent accidents by performing image recognition processing as fast as possible and controlling the vehicle based on the recognition results. In the image sensor of an in-vehicle camera, image recognition processing is performed on a frame-by-frame basis when the sensor is normally driven. For example, when the frame rate is 30 fps, recognition processing is performed every 33.3 ms. If the frame rate is low, the recognition processing speed is also slower accordingly. For example, the faster a subject that jumps into the vehicle is, the slower the recognition processing will be, and the lower the recognition accuracy will be. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7223070 Summary of the Invention [Problem to be solved by the invention]

[0005] In order to suppress flicker caused by traffic lights, in-vehicle cameras often use a certain period (for example, 11 ms) or more as the accumulation period for pixel signals, and in particular in low-light conditions, the accumulation period is extended to capture brighter images. However, a long accumulation period can easily cause blurring of fast-moving subjects, which can lead to reduced recognition accuracy. One way to suppress such subject blurring is to increase the frame rate, but normal recognition processing requires a certain amount of processing time, so there is a possibility that the processing will not be completed in time.

[0006] The present disclosure provides a technique capable of suppressing a decrease in image recognition accuracy. [Means for solving the problem]

[0007] A photoelectric conversion device according to an embodiment of the present disclosure includes a plurality of pixels, a control unit, and a generation unit. The plurality of pixels each include a sensor unit that emits a pulse at a frequency corresponding to a frequency of receiving photons, a first counter and a second counter that count the number of the pulses, and a first memory and a second memory that store the count values ​​of the first counter and the second counter. The control unit controls the plurality of pixels to read out the count value of the first accumulation period by the end of a second accumulation period that is longer than the first accumulation period in an operation of reading out the count values ​​accumulated in a plurality of different accumulation periods within a full frame period from each of the first counter and the second counter. The control unit also controls the first counter and the second counter so that the readout timing for each accumulation period is different. The generation unit generates a difference image based on the difference between signals corresponding to the count values ​​output from each of the first memory and the second memory. Effect of the Invention

[0008] According to the present disclosure, it is possible to suppress a decrease in image recognition accuracy. [Brief description of the drawings]

[0009] [Figure 1]1 is a diagram illustrating a configuration example of a photoelectric conversion element according to an embodiment. [Diagram 2] FIG. 4 is a diagram illustrating a configuration example of a sensor substrate. [Diagram 3] FIG. 2 is a diagram illustrating an example of the configuration of a circuit board. [Figure 4] 4 is a diagram showing an equivalent circuit of a pixel and a signal processing circuit corresponding to the pixel in FIG. 2 and FIG. 3. FIG. [Diagram 5] 4 is a diagram showing a relationship between a memory circuit and a buffer according to the embodiment; [Figure 6] 1 is a diagram illustrating a relationship between the operation of an APD and an output signal. [Figure 7] 1 is a functional block diagram of a photoelectric conversion device and a moving object according to an embodiment. [Figure 8] FIG. 2 is a block diagram showing an example of a functional configuration of an image processing unit according to the embodiment. [Figure 9] 11A to 11C are diagrams for explaining a photoelectric conversion method by a camera control unit according to an embodiment. [Figure 10] FIG. 13 is a diagram showing an example of two frame images generated via two counter circuits and a difference image between them. [Figure 11] 5 is a flowchart showing an example of driving a photoelectric conversion element according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. In the drawings, elements having the same configuration or function are given the same reference numerals, and repeated description thereof will be omitted. The configurations shown in the following embodiments are merely examples, and the present disclosure is not limited to the configurations shown in the drawings.

[0011] 1 is a diagram showing a configuration example of a photoelectric conversion element according to the first embodiment. In the following, a photoelectric conversion device having a so-called stacked structure in which a photoelectric conversion element 100 is configured by stacking and electrically connecting two substrates, a sensor substrate 11 and a circuit substrate 21, will be described as an example.

[0012] However, a so-called non-laminated structure in which the components included in the sensor substrate and the components included in the circuit substrate are disposed on a common semiconductor layer may be used. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes signals detected in the pixel region 12.

[0013] 2 is a diagram showing a configuration example of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 201 arranged two-dimensionally across a plurality of rows and columns. Each pixel 201 includes a photoelectric conversion unit 202 including an avalanche photodiode (hereinafter, APD).

[0014] Here, the photoelectric conversion unit 202 functions as a sensor unit that emits a pulse at a frequency corresponding to the frequency of receiving photons. The number of rows and columns of the pixel array that constitutes the pixel region 12 is not particularly limited.

[0015] Fig. 3 is a diagram showing an example of the configuration of the circuit board 21. The circuit board 21 has a signal processing circuit 303 that processes charges photoelectrically converted by each photoelectric conversion unit 202 in Fig. 2, a readout circuit 312, a control pulse generation unit 315, a horizontal scanning circuit 311, a vertical signal line 313, a vertical scanning circuit 310, and an output circuit 314.

[0016] The vertical scanning circuit 310 receives a control pulse supplied from a control pulse generating unit 315, and sequentially supplies the control pulse to a plurality of pixels arranged in the row direction. The vertical scanning circuit 310 uses logic circuits such as a shift register and an address decoder.

[0017] The signals output from the photoelectric conversion unit 202 of each pixel are processed by each signal processing circuit 303. As described below, the signal processing circuit 303 is provided with counters (counter circuits 411 and 412 in FIG. 4) and memories (memory circuits 413 and 414 in FIG. 4), and digital values ​​are stored in the memories. The horizontal scanning circuit 311 inputs a control pulse to the signal processing circuit 303 to sequentially select each column in order to read out the signals from the memories of each pixel in which the digital signals are stored.

[0018] A signal is output to the vertical signal line 313 from the signal processing circuit 303 of the pixel of the row selected by the vertical scanning circuit 310. The signal output to the vertical signal line 313 is output to the outside of the photoelectric conversion element 100 via the readout circuit 312 and the output circuit 314. The readout circuit 312 has a plurality of buffers built therein and connected to the vertical signal line 313.

[0019] 2 and 3, a plurality of signal processing circuits 303 are arranged in a region overlapping the pixel region 12 in a plan view. A vertical scanning circuit 310, a horizontal scanning circuit 311, a readout circuit 312, an output circuit 314, and a control pulse generating unit 315 are arranged so as to overlap between an end of the sensor substrate 11 and an end of the pixel region 12 in a plan view.

[0020] That is, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 310, a horizontal scanning circuit 311, a readout circuit 312, an output circuit 314, and a control pulse generating unit 315 are arranged in a region overlapping the non-pixel region in a plan view.

[0021] The arrangement of the vertical signal lines 313, the readout circuits 312, and the output circuits 314 are not limited to the example shown in Fig. 3. For example, the vertical signal lines 313 may be arranged extending in the row direction, and the readout circuits 312 may be arranged at the ends of the vertical signal lines 313. Also, it is not necessary to provide one signal processing circuit 303 for each photoelectric conversion unit, and one signal processing circuit may be shared by a plurality of photoelectric conversion units to perform signal processing sequentially.

[0022] FIG. 4 is a diagram showing an equivalent circuit of the pixel 201 in FIG. 2 and FIG.

[0023] The APD 401 included in the photoelectric conversion unit 202 generates a pair of electric charges according to incident light by photoelectric conversion. One of the two nodes of the APD 401 is connected to a power supply line to which a driving voltage VL (first voltage) is supplied. The other of the two nodes of the APD 401 is connected to a power supply line to which a driving voltage VH (second voltage) higher than the voltage VL is supplied.

[0024] In Fig. 4, one node of the APD 401 is the anode, and the other node of the APD is the cathode. A reverse bias voltage is supplied to the anode and cathode of the APD 401 so that the APD 401 performs avalanche multiplication. By supplying such a voltage, the charge generated by the incident light undergoes avalanche multiplication, generating an avalanche current.

[0025] When a reverse bias voltage is supplied, there are two modes: Geiger mode, in which the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, in which the voltage difference between the anode and cathode is close to or less than the breakdown voltage. An APD that operates in Geiger mode is called a SPAD. In the case of a SPAD, for example, the drive voltage VL (first voltage) is -30V, and the drive voltage VH (second voltage) is 1V.

[0026] The signal processing circuit 303 includes a quench element 402, a waveform shaping unit 410, counter circuits 411 and 412, and memory circuits 413 and 414. The quench element 402 is connected to a power supply line through which a driving voltage VH is supplied and one of the anode and cathode nodes of the APD 401.

[0027] The quench element 402 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, and has a function of suppressing avalanche multiplication by suppressing the voltage supplied to the APD 401 (quench operation).The quench element 402 also has a function of returning the voltage supplied to the APD 401 to the drive voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0028] 4 shows an example in which the signal processing circuit 303 has a waveform shaping section 410, counter circuits 411 and 412, and memory circuits 413 and 414 in addition to the quench element 402. The counter circuit 411 functions as, for example, a first counter, and the counter circuit 412 functions as, for example, a second counter. The counter circuits 411 and 412 function collectively as a counter section. The memory circuit 413 functions as a first memory, and the memory circuit 414 functions as a second memory. The memory circuits 413 and 414 function collectively as a memory section.

[0029] The waveform shaping unit 410 shapes the voltage change of the cathode of the APD 401 obtained when a photon is detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 410. Although an example in which one inverter is used as the waveform shaping unit 410 is shown in Fig. 4, a circuit in which a plurality of inverters are connected in series may be used, or another circuit having a waveform shaping effect may be used.

[0030] The counter circuit 411 counts the number of pulses output from the waveform shaping unit 410 and holds the count value. When a control pulse RES1 is supplied via a drive line 415, the signal held in the counter circuit 411 is reset. Here, the counter circuit 411 generates a signal based on the difference between the count values ​​at the start and end of the signal accumulation period. Similarly, the counter circuit 412 counts the number of pulses output from the waveform shaping unit 410 and holds the count value. Furthermore, when a control pulse RES2 is supplied via a drive line 416, the signal held in the counter circuit 412 is reset. Here, the counter circuit 412 generates a signal based on the difference between the count values ​​at the start and end of the signal accumulation period.

[0031] 3 to the memory circuit 413 via drive lines 417 (not shown in FIG. 3), respectively, to switch between electrical connection and disconnection between the counter circuit 411 and the vertical signal line 313. The memory circuit 413 functions as a memory that temporarily stores the count value of the counter circuit 411, and outputs an output signal from the counter circuit 411 of the pixel to the vertical signal line 313. 3 to the memory circuit 414 via the drive lines 418 (not shown in FIG. 3), respectively, and switches between electrical connection and disconnection between the counter circuit 412 and the vertical signal line 313. The memory circuit 414 functions as a memory that temporarily stores the count value of the counter circuit 412, and outputs an output signal from the counter circuit 412 of the pixel to the vertical signal line 313.

[0032] Note that electrical connections may be switched by disposing a switch such as a transistor between the quench element 402 and the APD 401 or between the photoelectric conversion unit 202 and the signal processing circuit 303. Similarly, the supply of the driving voltage VH or the driving voltage VL to the photoelectric conversion unit 202 may be electrically switched using a switch such as a transistor.

[0033] Fig. 5 is a diagram showing the relationship between the memory circuits and buffers according to the embodiment. Fig. 5 shows a state in which memory circuits 413 and 414 in signal processing circuit 303 in Fig. 3 are combined into one and arranged in N rows and M columns, with each memory circuit represented as memory 1-1-1 to memory NMh. Memory circuit 413 has a value of h=1, and memory circuit 414 has a value of h=2. Buffers 1 to M in Fig. 5 indicate buffers included in read circuit 312 in Fig. 3. Output circuit 314 in Fig. 5 corresponds to output circuit 314 in Fig. 3.

[0034] 6 is a diagram showing a schematic diagram of the relationship between the operation of the APD 401 and the output signal. The input side of the waveform shaping unit 410 is nodeA, and the output side is nodeB. Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 401. When a photon is incident on the APD 401 at time t1, avalanche multiplication occurs in the APD 401, an avalanche multiplication current flows through the quench element 402, and the voltage at nodeA drops.

[0035] When the amount of voltage drop further increases and the potential difference applied to the APD 401 decreases, the avalanche multiplication of the APD 401 stops as shown at time t2, and the voltage level of nodeA does not drop below a certain value.

[0036] After that, between time t2 and time t3, a current flows through nodeA to compensate for the voltage drop from voltage VL, and at time t3, nodeA is stabilized to its original potential level. At this time, the portion of the output waveform at nodeA that exceeds a certain threshold is shaped by waveform shaping section 410 and is output as a pulse signal at nodeB.

[0037] Next, a photoelectric conversion device 600 and a moving object 700 according to an embodiment will be described. Fig. 7 is a functional block diagram of the photoelectric conversion device 600 and the moving object 700 according to the embodiment. Some of the functional blocks shown in Fig. 7 are realized by causing a computer (not shown) included in the photoelectric conversion device 600 and the moving object 700 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0038] However, some or all of these may be realized by hardware. A dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP) may be used as the hardware. Furthermore, the functional blocks shown in Fig. 7 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths. The above explanation regarding Fig. 7 also applies to Fig. 8 described later.

[0039] 1 to 6, an imaging optical system 601, an image processing unit 603, a recognition unit 604, a camera control unit 605, a storage unit 606, a communication unit 607, etc. The photoelectric conversion element 100 is composed of an avalanche photodiode for photoelectrically converting an optical image, as described in FIGS.

[0040] The photoelectric conversion device 600 of the embodiment is mounted on a moving body 700, and a camera unit consisting of a set of an imaging optical system 601 and a photoelectric conversion element 100 is configured to capture an image in at least one direction, for example, in front, behind, or to the side of the moving body. Note that a plurality of camera units may be provided on the moving body 700.

[0041] 8 is a block diagram mainly showing the functional configuration of the image processing unit 603. The image processing unit 603 has a difference image generating unit 608. The difference image generating unit 608 functions as a generating means for generating a difference image based on the image output from the photoelectric conversion element 100.

[0042] As other processing, the image processing unit 603 performs image processing such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaic processing, data compression, etc. on the image signal acquired by the photoelectric conversion element 100 to generate a final image signal. Note that, if the photoelectric conversion element 100 has an on-chip color filter such as RGB, it is desirable for the image processing unit 603 to perform processing such as white balance correction and color conversion.

[0043] Moreover, the output of the image processing unit 603 is supplied to a recognition unit 604, an ECU (Electric Control Unit) 701 of the moving body 700, and a camera control unit 605. The recognition unit 604 recognizes people, vehicles, etc. in the vicinity by performing image recognition based on the image signal, and issues a warning, etc. as necessary.

[0044] In this embodiment, the moving body 700 is described using an example of an automobile, but the moving body may be any moving body, such as an airplane, a train, a ship, a drone, an AGV, or a robot.

[0045] The camera control unit 605 has a built-in CPU as a computer and a memory that stores a computer program, and controls each unit of the photoelectric conversion device 600 by the CPU executing the computer program stored in the memory.

[0046] In addition, the camera control unit 605 functions as a control means, and for example, controls the length of the accumulation period of each frame of the photoelectric conversion element 100, the number of accumulations, accumulation, and / or the timing of the control signal CLK via the control pulse generation unit of the photoelectric conversion element 100.

[0047] The storage unit 606 includes a recording medium such as a memory card, a hard disk, etc., and can store and read image signals. The communication unit 607 includes a wireless or wired interface, and outputs the generated image signal to the outside of the photoelectric conversion device 600 and receives various signals from the outside.

[0048] The ECU 701 incorporates a CPU as a computer and a memory that stores a computer program, and controls each part of the moving body 700 by the CPU executing the computer program stored in the memory.

[0049] The output of the ECU 701 is supplied to a vehicle control unit 702 and a display unit 703. The vehicle control unit 702 functions as a movement control means for driving, stopping, controlling the direction, etc. of the vehicle as a moving body based on the output of the ECU 701. The display unit 703 functions as a display means, and includes a display element such as a liquid crystal device or an organic EL, and is mounted on the moving body 700. The vehicle control unit 702 as a movement control means controls the operation of the moving body based in particular on the result of a recognition process using the output signal of the photoelectric conversion element 100.

[0050] The display unit 703 displays images acquired by the photoelectric conversion element 100 and various information related to the vehicle's running state and the like to the driver of the vehicle 700 using, for example, a GUI based on the output of the ECU 701 .

[0051] In addition, the image processing unit 603, the recognition unit 604, etc. in Figure 7 do not have to be mounted on the moving body 700, and may be provided, for example, in an external terminal provided separately from the moving body 700 for remotely controlling the moving body 700 or for monitoring the movement of the moving body.

[0052] 9 is a diagram for explaining a photoelectric conversion method by the camera control unit 605 according to the embodiment. In particular, the operation of reading out signals (count values) accumulated in a plurality of different accumulation periods within a full frame period from each of the counter circuits 411 and 412 will be explained here.

[0053] In this embodiment, photoelectric conversion is performed periodically, for example, at 30 full frames / second (one frame has a time length of 33.3 ms), and one frame having a length of 33.3 ms is divided into four. That is, as shown in Fig. 9, frame 1 (full frame 1) is divided into frames 1_1, 1_2, 1_3, and 1_4 of equal duration (8.33 ms). In this embodiment, one frame having a length of 33.3 ms is called a full frame, and a full frame divided into four is called a frame. In the following description, "frame" with a number is written as "F", for example, "frame 1_1" is written as "F1_1".

[0054] In this embodiment, F1_1 is composed of two frames F1_1_1 and F1_1_2 whose accumulation periods overlap. Similarly, F1_2, F1_3, and F1_4 are composed of two frames F1_2_1 and F1_2_2, F1_3_1 and F1_3_2, and F1_4_1 and F1_4_2 whose accumulation periods overlap. F1_1_1, F1_2_1, F1_3_1, and F1_4_1 are frames generated via the counter circuit 411. On the other hand, F1_1_2, F1_2_2, F1_3_2, and F1_4_2 are frames generated via the counter circuit 412.

[0055] F1_1_1 has an accumulation period from start time T0_1 of F1 to time T1_1, and F1_1_2 has an accumulation period from T0_2 to time T1_2, and the lengths of the respective accumulation periods are equal. F1_2_1 has an accumulation period from time T0_1 to time T2_1, F1_2_2 has an accumulation period from time T0_2 to time T2_2, and the lengths of the respective accumulation periods are equal. F1_3_1 has an accumulation period from time T0_1 to time T3_1, F1_3_2 has an accumulation period from time T0_2 to time T3_2, and the lengths of the respective accumulation periods are equal. Moreover, F1_4_1 has an accumulation period from time T0_1 to time T4_1, and F1_4_2 has an accumulation period from time T0_2 to time T4_2, and the lengths of the respective accumulation periods are equal.

[0056] The start times of F1_1_1, F1_2_1, F1_3_1, and F1_4_1 are the same, that is, time T0_1. Also, the start times of F1_1_2, F1_2_2, F1_3_2, and F1_4_2 are the same, that is, time T0_2.

[0057] The above-mentioned control pulse RES1 resets the counter circuit 411 at time T0_1 (T4_1), which is the end of the full frame period. The time T4_1 is the start time T0_1 of F2_1_1 of the next full frame 2. Furthermore, the counter circuit 412 is reset at time T0_2 (T4_2), which is the end of the full frame period, by the control pulse RES2. The time T4_2 is the start time T0_2 of F2_1_2 of the next full frame 2.

[0058] At the end times of F1_1, F1_2, F1_3, and F1_4, the count values ​​are read out from the counter circuits 411 and 412, respectively. Specifically, count values ​​C1_1_1, C1_2_1, C1_3_1, and C1_4_1 are read out from the counter circuit 411 at times T1_1, T2_1, T3_1, and T4_1, respectively. Moreover, count values ​​C1_1_2, C1_2_2, C1_3_2, and C1_4_2 are read out from the counter circuit 412 at times T1_2, T2_2, T3_2, and T4_2, respectively.

[0059] That is, each accumulation period ends at the timing of reading out each counter value from each of the counter circuits 411 and 412. The timing of reading out each accumulation period differs between the counter circuits 411 and 412. For example, for F1_1, the time T1_2 which is the readout timing (second timing) of the count value is delayed from the time T1_1 which is the readout timing (first timing) by, for example, a time shorter than the shortest accumulation period (for F1_1). The same is true for the relationship between the time T2_2 and the time T2_1, the relationship between the time T3_2 and the time T3_1, and the relationship between the time T4_2 (T0_2) and the time T4_1 (T0_2).

[0060] The read count values ​​C1_1_1, C1_2_1, C1_3_1, and C1_4_1 are temporarily stored in the memory circuit 413. In addition, the read count values ​​C1_1_2, C1_2_2, C1_3_2, and C1_4_2 are temporarily stored in the memory circuit 414. Then, the signals for one row temporarily stored in each of the memory circuits 413 and 414 are sequentially output from the photoelectric conversion element 100 via the buffer of the readout circuit 312 and input to the image processing unit 603 .

[0061] Thus, according to this embodiment, the count value C1_1_1 corresponding to the signal accumulated during the period F1_1_1 is read out at the time T1_1 to T2_1. Also, the count value C1_1_2 corresponding to the signal accumulated during the period F1_1_2 is read out at the time T1_2 to T2_2. That is, the signals accumulated during the periods F1_1_1 and F1_1_2 are output by the end of the longer accumulation periods F1_2_1 and F1_2_2. Thereafter, in the image processing unit 603, the difference image generating unit 608 subtracts the pixel value of F1_1_1 from the pixel value of F1_1_2, and converts the calculated value into a ternary value of the minimum 2 bits, positive, negative, or 0. The ternary image is output as a difference image, and the difference image is processed by the recognition unit 604. Therefore, image recognition can be performed quickly.

[0062] Similarly, count values ​​C1_2_1 and C1_2_2 corresponding to signals accumulated during periods F1_2_1 and F1_2_2 are read out at times T2_1 to T3_1 and T2_2 to T3_2, respectively. Similarly, count values ​​corresponding to signals accumulated during subsequent periods F1_3_1, F1_3_2, F1_4_1, and F1_4_2 are output by the end of the longer accumulation periods. Then, similarly to the above, a differential image is generated by the differential image generating unit 608 based on the read image signals, and image recognition is performed by the recognition unit 604.

[0063] 10 is a diagram showing an example of two frame images generated via two counter circuits 411 and 412, and a difference image between them. The difference image generating unit 608 can reduce subject blur by generating a difference image generated from two frames that have the same length of accumulation period (exposure time) and whose accumulation periods overlap, such as F1_1_1 and F1_1_2. Therefore, the contour of a subject moving at high speed can be clearly captured regardless of the length of the exposure time.

[0064] From now on, attention will be focused on the accumulation operation performed via one count circuit 411 (or counter circuit 412), and the operation and effect thereof will be described. The photoelectric conversion element 100 according to this embodiment performs accumulation operations for a first accumulation period and a second accumulation period within one frame. The first accumulation period is shorter than the second accumulation period, and the camera control unit 605 controls so that the signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period.

[0065] In this embodiment, the first and second accumulation periods overlap, and the first and second accumulation periods start at the same time. Furthermore, if the second accumulation period is the accumulation period of F1_4_1 (or F1_4_2), for example, the end of the second accumulation period is the boundary between full frames, and the second accumulation period is an integer multiple of the first accumulation period. That is, the interval between the readout timing of the count value for each accumulation period is constant, and the end of the second accumulation period coincides with the end of the full frame.

[0066] However, the second accumulation period does not need to be an integer multiple of the first accumulation period. In other words, the interval between readout timings for each accumulation period does not need to be constant. The second accumulation period only needs to be longer than the first accumulation period and end after the first accumulation period.

[0067] That is, an image with a short accumulation period and an image with a long accumulation period are created, the timing at which the short accumulation period ends is set earlier than the timing at which the long accumulation period ends, and as soon as the short accumulation period ends, that image is output and sent to the subsequent recognition unit 604. Then, the subject is recognized based on a signal generated during at least the first accumulation period. The recognition unit 604 as a recognition means recognizes the subject based on a signal generated during at least the first accumulation period.

[0068] Therefore, while conventionally image recognition was not possible until one frame (one full frame) had elapsed, in this embodiment, image recognition can be performed after as little as a 1 / 4 frame period, and for example, when a moving body is moving at high speed, obstacles and the like can be quickly recognized. Therefore, the moving body can apply the brakes quickly, or the moving body can avoid obstacles at an early stage.

[0069] In addition, an image with a long accumulation period can improve the contrast, and can be used as a display image. That is, an image with a short accumulation period is suitable for quick subject recognition, and an image with a long accumulation period is suitable as a display image. In this way, the display device of this embodiment displays the signal generated during at least the second accumulation period as an image.

[0070] In addition, because this embodiment uses an APD, unlike a CMOS sensor, the accumulated charge does not deteriorate due to readout. The accumulation periods can be overlapped. Also, since there is no readout noise, the original signal does not deteriorate even if it is read out many times after one accumulation. Up to this point, the operation and effects of the accumulation operation performed via one count circuit 411 (or counter circuit 412) have been described.

[0071] Fig. 11 is a flowchart showing an example of driving the photoelectric conversion element 100 in the embodiment. The operation of each step in the flowchart in Fig. 11 is performed sequentially by a CPU or the like as a computer in the camera control unit 605 executing a computer program stored in a memory.

[0072] In step S101, the CPU sets i to 1. In this embodiment, the possible values ​​of i are 1 to 4, each of which corresponds to the number of frames divided into four of one full frame.

[0073] Next, in step S102, the CPU sets h = 1. Possible values ​​of h are the numbers 1 and 2, which correspond to the counter circuits 411 and 412, respectively, as will be described below.

[0074] In step S103, the CPU outputs the count value Count(Ti_h) of the counter circuit h at time Ti_h to the memory (jkh). This operation corresponds to the operations at times T1_1 and T1_2 in FIG.

[0075] The counter circuit h indicates the counter circuit 411 when h=1, and indicates the counter circuit 412 when h=2. Moreover, the memory (jkh) indicates the memory circuit 413 when h=1, and indicates the memory circuit 414 when h=2. That is, in step S103, the count value is output from the counter circuit 411 to the memory circuit 413 when h=1, and the count value is output from the counter circuit 412 to the memory circuit 414 when h=2.

[0076] In step S104, the CPU sets j=1 (first row of a plurality of pixels). Then, in step S105, the CPU outputs the count value Count(jkhi) in the memory (jkh) in FIG. 5 to the buffer. At this time, the count values ​​Count(jkhi) are output to the buffer simultaneously for columns 1 to M. This operation corresponds to the operation of simultaneously taking the count values ​​in the first row in FIG. 5 (the count values ​​stored in memories 1-1-1, 1-1-2, 1-2, 1-1-2, . . . 1-M-1, 1_M_2) into the buffer.

[0077] In step S106, the CPU sets k=1 (first column of multiple pixels). Then, in step S107, the CPU outputs the count value Count(jkhi) of buffer k to the output circuit 314. This operation corresponds to the operation of reading out the signal of buffer 1 of the leftmost column in the read circuit 312 in FIG. 5 (the count values ​​stored in memories 1-1-1, 1-1-2, 2-1-1, 2-1-2, . . . N-1-1, N-1-2) to the output circuit 314.

[0078] In step S108, the CPU determines whether k < M. If Yes, in step S109, k is incremented by 1 as k = k + 1, and the process returns to step S107. This operation corresponds to the operation of reading the signal of buffer 2 in the second column from the left in FIG. 9 from the output circuit.

[0079] If it is determined as No in step S108, that is, when k = M, it means that the signal of buffer M in column M in the read circuit 312 has been completely read from the output circuit. In that case, in step S110, the CPU determines whether j < N. If Yes in step S110, the CPU increments j by 1 as j = j + 1 in step S111, and the process returns to step S105. This corresponds to the operation for starting to read the next line.

[0080] If it is determined as No in step S110, it means that the reading of all lines has been completed. Therefore, in step S112, the CPU determines whether h < 2. If it is determined as Yes in step S112, the CPU increments h by 1 as h = h + 1, and the process returns to step S103. If it is determined as No in step S112, it means that the reading of the count values at time Ti-1 and time Ti-2 has been completed. Therefore, in step S114, the CPU determines whether i < 4. If it is determined as Yes in step S114, in step S115, the CPU increments i by 1 as i = i + 1, and the process returns to step S102.

[0081] If it is determined No in step S114, it means that the readout at time T4_1 (T0_1) and T4_2 (T0_2) is completed. That is, in step S116, the counter circuit 411 is reset by a reset signal (control pulse RES1) at time T4_1, and the counter circuit 412 is reset by a reset signal (control pulse RES2) at time T4_2. This operation corresponds to the reset operation of the counter circuit 411 at time T4_1 in FIG. 9, and corresponds to the reset operation of the counter circuit 412 at time T4_2. In this manner, the signals accumulated in the photoelectric conversion element 100 can be read out sequentially.

[0082] As described above, according to this embodiment, the accumulation results for a period shorter than one full frame period are read out in the middle of one full frame period, and the recognition process is performed for each of the accumulation results obtained. This makes it possible to achieve low delay in the recognition process.

[0083] In addition, in this embodiment, the read timing of the count value for each accumulation period is different between the counter circuit 411 and the counter circuit 412, so that the memory circuits 413 and 414 store count values ​​having a signal of the timing difference between them. The difference image generating unit 608 generates a difference image based on the difference of the signals, so that it is possible to reduce the subject blurring that is likely to occur when a moving object moves at high speed. The recognition unit 604 can perform recognition processing on the ternary difference image in which the subject blurring is reduced in this way. That is, since the recognition processing is performed on an image that requires fewer bits than a color image, the processing load of image recognition is reduced by reducing the amount of data, and the processing speed is improved. This makes it possible to recognize a subject that moves at high speed, and the recognition accuracy can be maintained even when the subject moves at high speed, that is, the deterioration of the recognition accuracy can be suppressed. Moreover, in this embodiment, the accumulation result for a period shorter than one full frame period is used for the recognition processing, so that the effect of reducing the subject blurring can be improved.

[0084] Furthermore, the recognition process according to this embodiment does not require convolution processing or bounding box-based moving object detection, and therefore has the advantage of being faster in recognition speed than pattern matching or AI. Even if pattern matching or AI is used in the recognition process, the processing load for recognition is reduced because the number of bits of the difference image is reduced.

[0085] Other embodiments will be described below. For example, in the above embodiment, the shortest accumulation period (accumulation period of frame 1_1) is ¼ frame period as shown in FIG. 9, but it may be changed to, for example, ⅕ frame period or ⅓ frame period.

[0086] Furthermore, even if the readout cycle is set to every 1 / 4 frame period, the counter circuits 411 and 412 may be reset midway through the accumulation period of frame 1_1 in Fig. 9 depending on the brightness of the subject, the image recognition accuracy, etc. As a result, the substantial accumulation period may be made shorter than the 1 / 4 frame period.

[0087] 9, the counter circuits 411 and 412 may be reset once, for example, at time T1, thereby adjusting the count value read out at time T4.

[0088] The photoelectric conversion device 600 of the above embodiment generates a difference image and performs recognition processing on the generated difference image. However, the photoelectric conversion device may have a function (switching means) to switch between a mode (first mode) in which a difference image is generated and recognition processing is performed, and a mode (second mode) in which recognition processing is performed without outputting a difference image, in response to predetermined information. The predetermined information may be, for example, control information for the moving object 700 and / or information on the movement of the subject. It is assumed that the photoelectric conversion device selects the first mode as the control information of the moving object 700 when the moving object 700 is moving at a certain speed or more, and selects the second mode when the moving object 700 is moving at a speed less than the certain speed. In the second mode, the photoelectric conversion element may output a signal accumulated in the counter circuit 411 or 412. As a method of detecting the speed of the moving object 700, it is possible to read each piece of control information such as the accelerator, brake, steering wheel, and engine (or motor) speed of the own vehicle, or to read a value that quantifies the magnitude of the subject blur. The information on the movement of the subject includes the position, speed, acceleration, direction of movement, and magnitude of blur of the subject.

[0089] A computer program that realizes part or all of the control in this embodiment and the functions of the above-mentioned embodiment may be supplied to the photoelectric conversion device via a network or various storage media. Then, a computer (or a CPU, MPU, etc.) in the photoelectric conversion device may read and execute the program. In this case, the program and the storage medium storing the program constitute the present disclosure.

[0090] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a plurality of pixels each including a sensor unit that emits a pulse at a frequency corresponding to a frequency of receiving a photon, a first counter and a second counter that count the number of the pulses, and a first memory and a second memory that store the count values ​​of the first counter and the second counter, a control means for controlling the plurality of pixels so that, in an operation of reading out the count values ​​accumulated in a plurality of different accumulation periods within a full frame period from each of the first counter and the second counter, the count value of a first accumulation period is read out by the end of a second accumulation period that is longer than the first accumulation period, and for controlling the first counter and the second counter so that the readout timing for each accumulation period is different; a generating means for generating a differential image based on a difference between signals corresponding to the count values ​​output from the first memory and the second memory, A photoelectric conversion device comprising: (Configuration 2) The control means performs control so as to read out the count value of the first counter at a first timing and to read out the counter value of the second counter at a second timing that is delayed from the first timing by a time shorter than the first accumulation period. 2. The photoelectric conversion device according to configuration 1. (Configuration 3) The interval between the read timings of the first counter and the second counter is constant. 3. The photoelectric conversion device according to configuration 1 or 2. (Configuration 4) In each of the first counter and the second counter, the first accumulation period and the second accumulation period overlap. 4. The photoelectric conversion device according to any one of configurations 1 to 3. (Configuration 5) In each of the first counter and the second counter, the first accumulation period and the second accumulation period start simultaneously. 5. The photoelectric conversion device according to any one of configurations 1 to 4. (Configuration 6) In each of the first counter and the second counter, the end of the second accumulation period coincides with the end of the full frame period. 6. The photoelectric conversion device according to any one of configurations 1 to 5, (Configuration 7) The image sensor further includes a display unit for displaying, as an image, a signal generated during at least the second accumulation period. 7. The photoelectric conversion device according to any one of configurations 1 to 6, (Configuration 8) The apparatus further includes a recognition unit that performs a subject recognition process based on the image signals output from the plurality of pixels. 8. The photoelectric conversion device according to any one of configurations 1 to 7, (Configuration 9) The image processing apparatus further includes a switching means for switching between a first mode in which the recognition processing is performed on the difference image and a second mode in which the recognition processing is performed on the image signals output from the plurality of pixels without going through an output processing of the difference image. 9. The photoelectric conversion device according to configuration 8. (Configuration 10) The switching means switches between the first mode and the second mode based on at least one of control information of a moving object and information on the movement of the subject. 10. The photoelectric conversion device according to configuration 9. (Configuration 11) The sensor portion includes an avalanche photodiode. 11. The photoelectric conversion device according to any one of configurations 1 to 10, (Configuration 12) A moving body having a photoelectric conversion device and a movement control means for controlling the movement of the moving body based on an output of the photoelectric conversion device, The photoelectric conversion device includes: a plurality of pixels each including a sensor unit that emits a pulse at a frequency corresponding to a frequency of receiving a photon, a first counter and a second counter that count the number of the pulses, and a first memory and a second memory that store the count values ​​of the first counter and the second counter, a control means for controlling the plurality of pixels so that, in an operation of reading out the count values ​​accumulated in a plurality of different accumulation periods within a full frame period from each of the first counter and the second counter, the count value of a first accumulation period is read out by the end of a second accumulation period that is longer than the first accumulation period, and for controlling the first counter and the second counter so that the readout timing for each accumulation period is different; a generating means for generating a differential image based on a difference between signals corresponding to the count values ​​output from the first memory and the second memory, A moving object characterized by: (method) A processing method for a photoelectric conversion device having a plurality of pixels, each of which includes a sensor unit that emits a pulse at a frequency corresponding to a frequency of receiving photons, a counter unit that counts the number of the pulses, and a memory unit that stores a count value of the counter unit, comprising: the counter unit includes a first counter and a second counter, the memory unit includes a first memory and a second memory for storing the count values ​​of the first counter and the second counter, a step of controlling the plurality of pixels so that, in an operation of reading out the count values ​​accumulated in a plurality of different accumulation periods within a full frame period from each of the first counter and the second counter, the count value of a first accumulation period is read out by the end of a second accumulation period that is longer than the first accumulation period; performing control so that the readout timing for each accumulation period differs between the first counter and the second counter; generating a difference image based on a difference between signals corresponding to the count values ​​output from the first memory and the second memory, A processing method comprising: (program) A computer program for causing a photoelectric conversion device having a plurality of pixels, each of which includes a sensor unit that emits a pulse at a frequency corresponding to a frequency of receiving photons, a counter unit that counts the number of the pulses, and a memory unit that stores a count value of the counter unit, to execute a process, the counter unit includes a first counter and a second counter, the memory unit includes a first memory and a second memory for storing the count values ​​of the first counter and the second counter, The process comprises: a step of controlling the plurality of pixels so that, in an operation of reading out the count values ​​accumulated in a plurality of different accumulation periods within a full frame period from each of the first counter and the second counter, the count value of a first accumulation period is read out by the end of a second accumulation period that is longer than the first accumulation period; performing control so that the readout timing for each accumulation period differs between the first counter and the second counter; generating a difference image based on a difference between signals corresponding to the count values ​​output from the first memory and the second memory, A computer program comprising:

[0091] The present disclosure has been described above in detail based on preferred embodiments thereof, but the present disclosure is not limited to the above embodiments, and various modifications are possible based on the gist of the present disclosure, and these modifications are not excluded from the scope of the present disclosure. [Explanation of symbols]

[0092] 100: Photoelectric conversion element 201: Pixel 202: Photoelectric conversion unit 314: Output circuit 315: Control pulse generation unit 411, 412: First counter circuit 413, 414: Memory circuit 600: Photoelectric conversion device 604: Recognition part 605: Camera control unit 608: Differential image generation unit 703: Display section

Claims

1. a plurality of pixels each including a sensor unit that emits a pulse at a frequency corresponding to a frequency of receiving a photon, a first counter and a second counter that count the number of the pulses, and a first memory and a second memory that store the count values ​​of the first counter and the second counter, a control means for controlling the plurality of pixels so that, in an operation of reading out the count values ​​accumulated in a plurality of different accumulation periods within a full frame period from each of the first counter and the second counter, the count value of a first accumulation period is read out by the end of a second accumulation period that is longer than the first accumulation period, and for controlling the first counter and the second counter so that the readout timing for each accumulation period is different; a generating means for generating a differential image based on a difference between signals corresponding to the count values ​​output from the first memory and the second memory, A photoelectric conversion device comprising:

2. The control means performs control so as to read out the count value of the first counter at a first timing and to read out the count value of the second counter at a second timing that is delayed from the first timing by a time shorter than the first accumulation period.

2. The photoelectric conversion device according to claim 1.

3. The interval between the read timings of the first counter and the second counter is constant.

2. The photoelectric conversion device according to claim 1.

4. In each of the first counter and the second counter, the first accumulation period and the second accumulation period overlap.

2. The photoelectric conversion device according to claim 1.

5. In each of the first counter and the second counter, the first accumulation period and the second accumulation period start simultaneously.

2. The photoelectric conversion device according to claim 1.

6. In each of the first counter and the second counter, the end of the second accumulation period coincides with the end of the full frame period.

2. The photoelectric conversion device according to claim 1.

7. The image sensor further includes a display unit for displaying, as an image, a signal generated during at least the second accumulation period.

2. The photoelectric conversion device according to claim 1.

8. The apparatus further includes a recognition unit that performs a subject recognition process based on the image signals output from the plurality of pixels.

2. The photoelectric conversion device according to claim 1.

9. The image processing apparatus further includes a switching means for switching between a first mode in which the recognition processing is performed on the difference image and a second mode in which the recognition processing is performed on the image signals output from the plurality of pixels without going through an output processing of the difference image.

9. The photoelectric conversion device according to claim 8.

10. The switching means switches between the first mode and the second mode based on at least one of control information of a moving object and information on the movement of the subject.

10. The photoelectric conversion device according to claim 9.

11. The sensor portion includes an avalanche photodiode.

2. The photoelectric conversion device according to claim 1.

12. A moving body having a photoelectric conversion device and a movement control means for controlling the movement of the moving body based on an output of the photoelectric conversion device, The photoelectric conversion device includes: a plurality of pixels each including a sensor unit that emits a pulse at a frequency corresponding to a frequency of receiving a photon, a first counter and a second counter that count the number of the pulses, and a first memory and a second memory that store the count values ​​of the first counter and the second counter, a control means for controlling the plurality of pixels so that, in an operation of reading out the count values ​​accumulated in a plurality of different accumulation periods within a full frame period from each of the first counter and the second counter, the count value of a first accumulation period is read out by the end of a second accumulation period that is longer than the first accumulation period, and for controlling the first counter and the second counter so that the readout timing for each accumulation period is different; a generating means for generating a differential image based on a difference between signals corresponding to the count values ​​output from the first memory and the second memory, A moving object characterized by:

13. A processing method for a photoelectric conversion device having a plurality of pixels, each of which includes a sensor unit that emits a pulse at a frequency corresponding to a frequency of receiving photons, a counter unit that counts the number of the pulses, and a memory unit that stores a count value of the counter unit, comprising: the counter unit includes a first counter and a second counter, the memory unit includes a first memory and a second memory for storing the count values ​​of the first counter and the second counter, a step of controlling the plurality of pixels so that, in an operation of reading out the count values ​​accumulated in a plurality of different accumulation periods within a full frame period from each of the first counter and the second counter, the count value of a first accumulation period is read out by the end of a second accumulation period that is longer than the first accumulation period; performing control so that the readout timing for each accumulation period differs between the first counter and the second counter; generating a difference image based on a difference between signals corresponding to the count values ​​output from the first memory and the second memory, A processing method comprising:

14. A computer program for causing a photoelectric conversion device having a plurality of pixels, each of which includes a sensor unit that emits a pulse at a frequency corresponding to a frequency of receiving photons, a counter unit that counts the number of the pulses, and a memory unit that stores a count value of the counter unit, to execute a process, the counter unit includes a first counter and a second counter, the memory unit includes a first memory and a second memory for storing the count values ​​of the first counter and the second counter, The process comprises: a step of controlling the plurality of pixels so that, in an operation of reading out the count values ​​accumulated in a plurality of different accumulation periods within a full frame period from each of the first counter and the second counter, the count value of a first accumulation period is read out by the end of a second accumulation period that is longer than the first accumulation period; performing control so that the readout timing for each accumulation period differs between the first counter and the second counter; generating a difference image based on a difference between signals corresponding to the count values ​​output from the first memory and the second memory, A computer program comprising:

Images