Photoelectric conversion apparatus, moving body, control method, and non-transitory computer-readable storage medium

Abstract

To provide a photoelectric conversion device capable of outputting a recognition processing result to the outside with low delay.SOLUTION: A photoelectric conversion apparatus includes a plurality of pixels each including a sensor unit configured to generate a pulse at a frequency corresponding to a light reception frequency of a photon, a counter configured to count the number of pulses, and a memory configured to store a count value of the counter, a recognition unit configured to perform recognition processing on an image signal generated based on a difference between count values of the counter at a start and an end of an accumulation period, and a control unit configured to cause a result of the recognition processing to be output to an outside as the recognition processing is completed by the recognition unit.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to a photoelectric conversion device, a moving object, a control method, a computer program, and the like. [Background technology]

[0002] In recent years, photoelectric conversion devices have been developed that digitally count the number of photons incident on an avalanche photodiode (APD) and output the counted value from a pixel as a photoelectrically converted digital signal.

[0003] Furthermore, for example, Patent Document 1 describes a configuration in which a photoelectric conversion device having an APD can output multiple images whose accumulation periods overlap each other, thereby enabling continuous shooting even under low illumination. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7223070 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when considering the image sensor of an in-vehicle camera installed on a moving object, for example, normal sensor driving performs recognition processing on a frame-by-frame basis, so in the case of 30 fps, for example, recognition processing can only be performed every 33.3 ms. Therefore, even if an object appears immediately after a frame change in an in-vehicle camera, recognition processing cannot be performed until the end of the frame.

[0006] Furthermore, in order to suppress flicker caused by traffic lights, in-vehicle cameras often use an accumulation period of at least 11 ms, and in particular in low light conditions, the accumulation period is extended to capture brighter images. However, because the accumulation period is long, subject blur occurs when capturing fast-moving objects, reducing the recognition rate and making it difficult to obtain recognition processing results quickly.

[0007] Therefore, one object of the present invention is to provide a photoelectric conversion device that can output the recognition processing results to the outside with low delay. [Means for solving the problem]

[0008] A photoelectric conversion device according to one aspect of the present invention comprises: a sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons; a counter that counts the number of pulses; a plurality of pixels each including a memory for storing a count value of the counter; a recognition unit that performs a recognition process on an image signal generated based on the difference between the count values ​​of the counter at the start and end of an accumulation period; and control means for outputting the result of the recognition process to the outside as the recognition means completes the recognition process. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a photoelectric conversion device that can output the recognition processing result to the outside with low delay. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating a configuration example of a photoelectric conversion element according to an embodiment of the present invention. [Figure 2] 2 is a diagram showing an example of the configuration of a sensor substrate 11. FIG. [Figure 3] 2 is a diagram showing an example of the configuration of a circuit board 21. FIG. [Figure 4] 2 and 3, and shows an example of an equivalent circuit of the pixel 101 and the signal processing circuit 103 corresponding to the pixel 101. FIG. [Figure 5] 2 is a diagram schematically illustrating the relationship between the operation of an APD 201 and an output signal. FIG. [Figure 6] 1 is a functional block diagram of a photoelectric conversion device 600 and a moving object 700 according to an embodiment. [Figure 7]10A and 10B are diagrams for explaining an example of photoelectric conversion processing by a camera control unit 605 according to an embodiment. [Figure 8] 10 is a diagram for explaining an example of recognition processing by a camera control unit 605 according to an embodiment. FIG. [Figure 9] 10A to 10C are diagrams illustrating the recognition process performed by the recognition unit 604 according to the embodiment. [Figure 10] FIG. 10 is a diagram showing an example of an image of a plurality of divided frames. [Figure 11] FIG. 2 is a diagram illustrating a relationship between a memory circuit and a buffer according to the embodiment. [Figure 12] 10 is a flowchart showing details of an example of driving a photoelectric conversion element according to an embodiment. [Figure 13] 13 is a continuation of the flowchart in FIG. 12. [Figure 14] 7 is a diagram for explaining an example of a specific control method for a moving object 700 by an ECU 701 according to the embodiment. [Figure 15] 10 is a flowchart illustrating an example of an object recognition operation according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions will be omitted or simplified.

[0012] 1 is a diagram showing an example of the configuration of a photoelectric conversion element according to an embodiment of the present invention. In the following, the photoelectric conversion element 100 will be described as an example of a photoelectric conversion device having a so-called stacked structure, which is configured by stacking and electrically connecting two substrates, a sensor substrate 11 and a circuit substrate 21.

[0013] However, a so-called non-laminated structure may be used in which the components included in the sensor substrate and the components included in the circuit substrate are arranged on a common semiconductor layer. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes signals detected in the pixel region 12.

[0014] 2 is a diagram showing an example of the configuration of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally in rows and columns. Each pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter referred to as an APD).

[0015] Here, the photoelectric conversion unit 102 functions as a sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons. The number of rows and columns of the pixel array that forms the pixel region 12 is not particularly limited.

[0016] 3 is a diagram showing an example of the configuration of the circuit board 21. The circuit board 21 has a signal processing circuit 103 that processes charges photoelectrically converted by each photoelectric conversion unit 102 in FIG. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114.

[0017] The vertical scanning circuit 110 receives control pulses supplied from the control pulse generating unit 115 and supplies the control pulses to a plurality of pixels arranged in the row direction, row by row, in sequence. The vertical scanning circuit 110 uses logic circuits such as a shift register and an address decoder.

[0018] The signal output from the photoelectric conversion unit 102 of each pixel is processed by each signal processing circuit 103. The signal processing circuit 103 is provided with a counter, memory, etc., and digital values ​​are stored in the memory. In order to read the signal from the memory of each pixel where the digital signal is stored, the horizontal scanning circuit 111 inputs a control pulse that sequentially selects each column to the signal processing circuit 103.

[0019] A signal is output to the vertical signal line 113 from the signal processing circuit 103 of the pixel of the row selected by the vertical scanning circuit 110. The signal output to the vertical signal line 113 is output to the outside of the photoelectric conversion element 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has a plurality of buffers built in, which are connected to each vertical signal line 113.

[0020] 2 and 3, a plurality of signal processing circuits 103 are arranged in an area overlapping the pixel area 12 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged so as to overlap between an end of the sensor substrate 11 and an end of the pixel area 12 in a plan view.

[0021] In other words, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in a region overlapping the non-pixel region in a plan view.

[0022] The arrangement of the vertical signal lines 113, the readout circuits 112, and the output circuits 114 is not limited to the example shown in Fig. 3. For example, the vertical signal lines 113 may be arranged extending in the row direction, and the readout circuits 112 may be arranged at the ends of the vertical signal lines 113. Furthermore, it is not necessary to provide one signal processing circuit 103 for each photoelectric conversion unit, and one signal processing circuit may be shared by multiple photoelectric conversion units to perform signal processing sequentially.

[0023] 2 and 3, and an example of an equivalent circuit of the photoelectric conversion unit 102 and signal processing circuit 103 included in the pixel 101. As shown in Fig. 4, each pixel 101 includes the photoelectric conversion unit 102 as a sensor unit, a counter circuit 211, a memory circuit 212, etc.

[0024] The APD 201 included in the photoelectric conversion unit 102 generates charge pairs in response to incident light through photoelectric conversion. One of the two nodes of the APD 201 (anode) is connected to a power supply line that supplies a voltage VL (first voltage). The other of the two nodes of the APD 201 (cathode) is connected to a power supply line that supplies a voltage VH (second voltage) that is higher than the voltage VL.

[0025] A reverse bias voltage is applied to the anode and cathode of the APD 201 so that the APD 201 performs avalanche multiplication. With this voltage applied, the charge generated by the incident light undergoes avalanche multiplication, generating an avalanche current.

[0026] When a reverse bias voltage is supplied, there are two modes: Geiger mode, in which the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, in which the voltage difference between the anode and cathode is close to or less than the breakdown voltage. An APD operating in Geiger mode is called a SPAD. In the case of a SPAD, the voltage VL (first voltage) is, for example, -30 V, and the voltage VH (second voltage) is, for example, 1 V.

[0027] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212. The quench element 202 is connected to a power supply line to which a voltage VH is supplied and one of the anode and cathode nodes of the APD 201.

[0028] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, suppressing the voltage supplied to the APD 201 and suppressing avalanche multiplication (quench operation).The quench element 202 also functions to return the voltage supplied to the APD 201 to voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0029] In the equivalent circuit shown in FIG. 4, the signal processing circuit 103 has a waveform shaping section 210, a counter circuit 211, and a memory circuit 212 in addition to the quench element 202.

[0030] The waveform shaping unit 210 shapes the voltage change at the cathode of the APD 201 obtained when photons are detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. While Fig. 4 shows an example in which one inverter is used as the waveform shaping unit 210, a circuit in which multiple inverters are connected in series, or another circuit that has a waveform shaping effect, may also be used.

[0031] The counter circuit 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. When a control pulse RES is supplied via a drive line 213, the signal held in the counter circuit 211 is reset. Here, the counter circuit 211 generates an image signal based on the difference between the count values ​​at the start and end of the accumulation period.

[0032] 3 is supplied to the memory circuit 212 via a drive line 214 (not shown in FIG. 3) in FIG. 4, and switches between electrical connection and disconnection between the counter circuit 211 and the vertical signal line 113. The memory circuit 212 functions as a memory that temporarily stores the count value of the counter, and outputs an output signal from the counter circuit 211 of the pixel to the vertical signal line 113.

[0033] Note that electrical connections may be switched by disposing a switch such as a transistor between the quench element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103. Similarly, the supply of the voltage VH or the voltage VL to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0034] 5 is a diagram schematically illustrating the relationship between the operation of the APD 201 and the output signal. The input side of the waveform shaping unit 210 is designated node A, and the output side is designated node B. Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage at node A drops.

[0035] When the voltage drop amount further increases and the potential difference applied to the APD 201 decreases, the avalanche multiplication of the APD 201 stops as at time t2, and the voltage level of node A does not drop below a certain value.

[0036] After that, between time t2 and time t3, a current flows through node A to compensate for the voltage drop from voltage VL, and at time t3, node A settles to its original potential level. At this time, the part of the output waveform at node A that exceeds a certain threshold is shaped by waveform shaping unit 210 and output as a pulse signal at node B.

[0037] Next, a photoelectric conversion device 600 and a mobile object 700 according to this embodiment will be described. Fig. 6 is a functional block diagram of the photoelectric conversion device 600 and the mobile object 700 according to this embodiment. Note that some of the functional blocks shown in Fig. 6 are realized by causing a computer (not shown) included in the photoelectric conversion device 600 and the mobile object 700 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0038] However, some or all of these functions may be implemented by hardware. Examples of hardware that can be used include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs). Furthermore, the functional blocks shown in Fig. 6 do not have to be built into the same housing, and may be configured as separate devices connected to each other via signal paths.

[0039] 1 to 5, an imaging optical system 601, an image processing unit 603, a recognition unit 604, a camera control unit 605, a storage unit 606, a communication unit 607, etc. The photoelectric conversion element 100 is configured by an avalanche photodiode for photoelectrically converting an optical image, as described in FIGS.

[0040] The photoelectric conversion device of this embodiment is mounted on a moving body 700, and a camera unit consisting of a set of an imaging optical system 601 and a photoelectric conversion element 100 is disposed so as to capture an image in at least one direction, for example, in front, behind, or to the side of the moving body. Note that a plurality of camera units may be provided on the moving body 700.

[0041] The image processing unit 603 performs image processing such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaic processing, and data compression on the image signal acquired by the photoelectric conversion element 100 to generate a final image signal. Note that if each pixel of the photoelectric conversion element 100 has an on-chip color filter such as RGB, the image processing unit 603 also performs processing such as white balance correction and color conversion.

[0042] Furthermore, when the read image is to be used for multiple purposes, such as for visual confirmation and recognition, it is desirable to store different image processing parameters in advance for each purpose for which the image is to be used, and to switch the image processing parameters as the purpose for which the image is to be used is switched.

[0043] The output of the image processing unit 603 is supplied to a recognition unit 604, an ECU (Electric Control Unit) 701 of the moving object 700, and a camera control unit 605. The recognition unit 604 functions as a recognition means that recognizes subjects by performing image recognition based on the image signal, and recognizes subjects such as people, animals, vehicles, traffic lights, signs, etc. around the moving object 700 and identifies their types.

[0044] At that time, the reliability indicating the accuracy of the recognition result and coordinate information indicating the position and size of the subject in the image are also acquired. Specifically, the coordinate information acquired is the X coordinate offset and size, and the Y coordinate offset and size. In this embodiment, three types of recognition results are acquired: subject type, reliability, and coordinate information, but other information may also be acquired.

[0045] The recognition result acquired by the recognition unit 604 is output to the ECU 701. Note that, in this embodiment, the mobile body 700 will be described using an example of an automobile, but the mobile body may be any mobile body such as an airplane, a train, a ship, a drone, an AGV, or a robot.

[0046] The camera control unit 605 includes a CPU as a computer and a memory that stores a computer program, and controls each unit of the photoelectric conversion device 600 by the CPU executing the computer program stored in the memory.

[0047] The camera control unit 605 functions as a control means, and controls the length of the accumulation period (photoelectric conversion period) of each frame of the photoelectric conversion element 100 and the timing of the control signal CLK, for example, via a control pulse generation unit of the photoelectric conversion element 100. The camera control unit 605 also has a function to set various image processing parameters in the image processing unit 603, a function to control the recognition unit 604, and a function to obtain the recognition results.

[0048] The storage unit 606 includes a recording medium such as a memory card or a hard disk, and can store and read image signals. The communication unit 607 has a wireless or wired interface, and outputs the generated image signals and recognition results to the outside of the photoelectric conversion device 600, and receives various signals from the outside.

[0049] The communication unit 607 may perform wired communication in accordance with standards such as SPI or I2C as an interface, or may perform communication using a wireless LAN system such as Wi-Fi or Bluetooth (registered trademark).

[0050] The ECU 701 incorporates a CPU as a computer and a memory that stores a computer program, and controls each part of the moving object 700 by the CPU executing the computer program stored in the memory.

[0051] The output of the ECU 701 is supplied to a vehicle control unit 702 and a display unit 703. The vehicle control unit 702 functions as a movement control means that controls the movement (driving, stopping, direction control, etc.) of the vehicle as a moving body based on the output of the ECU 701. Furthermore, the vehicle control unit 702 as a movement control means controls the moving body based on the recognition result output from the recognition unit.

[0052] The display unit 703 functions as a display means, includes a display element such as a liquid crystal device or an organic EL, and is mounted on the mobile object 700. Based on the output of the ECU 701, the display unit 703 displays to the driver of the mobile object 700, for example, using a GUI, images acquired by the photoelectric conversion element 100, recognition results acquired by the recognition unit 604, various information related to the vehicle's running state, etc.

[0053] 7 is a diagram illustrating an example of photoelectric conversion processing by the camera control unit 605 according to the embodiment. In this embodiment, photoelectric conversion is periodically driven at, for example, 30 full frames per second. Furthermore, a frame corresponding to one vertical period having a length of 33.3 ms is called a full frame, and each of the four divisions of a full frame is called a frame.

[0054] 7, full frame 1 is divided into frames 1_1, 1_2, 1_3, and 1_4, each having an equal period (8.33 ms). Then, at times T1, T2, T3, and T4, which are the end points of frames 1_1, 1_2, 1_3, and 1_4, respectively, count values ​​are read out from the counter circuit 211. Meanwhile, the counter circuit 211 is reset at T0 (T4), which is the end point of a full frame period (one vertical period).

[0055] Therefore, the count value C1_1 read out from the counter circuit 211 at time T1, which is the end of frame 1_1, is the count value for the accumulation period from time T0 to time T1. Also, the count value C1_2 read out from the counter circuit 211 at time T2, which is the end of frame 1_2, is the count value for the accumulation period from time T0 to time T2.

[0056] Furthermore, the count value C1_3 read out from the counter circuit 211 at time T3, which is the end of frame 1_3, is the count value for the accumulation period from time T0 to time T3. Furthermore, the count value C1_4 read out from the counter circuit 211 at time T4, which is the end of frame 1_4, is the count value for the accumulation period from time T0 to time T4.

[0057] First, a processing flow of an image read out from the counter circuit 211 at time T1, which is the end of frame 1_1, will be described. The counter circuit 211 is reset at time T0, and a count value C1_1 of the counter circuit 211 is acquired at time T1. The acquired count value C1_1 is temporarily stored in the memory circuit 212.

[0058] 8 is a diagram for explaining an example of recognition processing by the camera control unit 605 according to the embodiment. The count value C1_1 temporarily stored in the memory circuit 212 is read from the photoelectric conversion element and output row by row to the image processing unit 603 via a buffer in the read circuit 112 during the period from time T1 to time T2.

[0059] The image processing unit 603 performs the various image processing operations described above and outputs the processing results as they are to the recognition unit 604. The image of frame 1_1 is output to the recognition unit 604 during the period from time T1 to time T2, and output of image signals for all lines is completed at the timing of time T2. Then, at the timing of time T2, the recognition unit 604 starts object recognition processing.

[0060] In this embodiment, the recognition process starts at time T2 when all image signals for one frame are received, but the recognition process may be performed line by line or pixel by pixel. By reducing the processing unit for recognition, the delay time until the recognition result is obtained can be reduced.

[0061] Then, as soon as the recognition process of the image of frame 1_1 is completed, the recognition result is promptly output to ECU 701. In this embodiment, the shortest time required for the recognition process and output of the result is set to Td. At this time, the output of the recognition result (Rcg1_1 in FIG. 8) based on the image of frame 1_1 is completed at the timing of time T2+Td.

[0062] 9(A) to 9(C) are diagrams for explaining the recognition processing by the recognition unit 604 according to the embodiment, and the details of the recognition processing will be explained using Fig. 9. Fig. 9(A) is a diagram showing an example of an image output from the image processing unit 603 to the recognition unit 604 at timing T2. A road extends in the vertical direction within the image, with a vehicle appearing in the upper part of the image and a person running out onto the road appearing in the lower left part of the image.

[0063] Fig. 9(B) is a diagram showing an example of the results of subject recognition processing performed on frame 1_1 by the recognition unit 604. As shown in Fig. 9(B), two subjects are recognized in frame 1_1, and the recognition unit 604 outputs the subject type, the reliability of image recognition, and coordinate information for each subject as data information.

[0064] That is, the subject type of the first recognized subject is "person," and the reliability of image recognition is 90%. Furthermore, the coordinate information for the pixel area in which the "person" was recognized is that the X and Y offsets are 200 pix and 400 pix, respectively, and the size is 600 pix and 500 pix. That is, it can be seen that the X coordinate is in the range of 200 to 800 pix, and the Y coordinate is in the range of 400 to 900 pix.

[0065] The subject type of the second recognized subject is "vehicle," and the reliability of image recognition is 80%. Furthermore, the coordinate information for the pixel area in which the "vehicle" is recognized is that the X and Y offsets are 600 pix and 50 pix, respectively, and the size is 600 pix and 500 pix. In other words, it can be seen that the X coordinate is in the range of 600 to 1200 pix, and the Y coordinate is in the range of 50 to 550 pix.

[0066] 9B can be output to the ECU 701 at the timing of time T2+Td when the recognition unit 604 acquires the information shown in FIG. 9B. Therefore, the recognition result can be supplied to the ECU 701 of the moving object 700 in the shortest time (Td) from the start of recognition to the output of the recognition result.

[0067] In this embodiment, the data information of Fig. 9(B) is supplied to the ECU 701 with the highest priority given to reducing the delay time, but other information may also be supplied to the ECU 701 as needed. For example, as shown in Fig. 9(C), the image signal of Fig. 9(A) may be superimposed with the recognition result shown in Fig. 9(B) as information and supplied to the ECU 701.

[0068] Alternatively, only the image of the area where the subject is recognized (within the white frame in FIG. 9C) may be cut out and supplied together with the recognition result to the ECU 701. However, if image information is also supplied to the ECU 701, the amount of data increases, and transmission to the ECU 701 may take time.

[0069] That is, the delay time until the output of the recognition result is completed increases. Therefore, it is desirable to determine the content of the information supplied from the photoelectric conversion device 600 to the ECU 701 taking into consideration the information required by the moving object 700, the allowable delay time, etc. This completes the explanation of the processing flow for the image of frame 1_1.

[0070] Subsequently, readout of image signals starts at times T2, T3, and T4 at the end of frames 1_2, 1_3, and 1_4, respectively, and is read out sequentially during periods from time T2 to time T3, from time T3 to time T4, and from time T4 to time T1, respectively. Then, respective recognition processes start sequentially at timings T3, T4, and T1, and supply of respective recognition results to ECU 701 is completed at timings T3+Td, T4+Td, and T1+Td.

[0071] Fig. 10 is a diagram showing an example of images of a plurality of divided frames. As shown in Fig. 9, the image read out at time T1 at the end of frame 1_1 is dark because the accumulation period is short, but there is little subject blur of the person who suddenly appears.

[0072] On the other hand, the images read out at the end of frames 1_2, 1_3, and 1_4 are more likely to have subject blur due to the longer accumulation periods. Note that stopped vehicles and white lines are less likely to have blur, and the longer the accumulation period, the more likely the contrast is to improve.

[0073] As described above, in this embodiment, one full frame period has a first accumulation period (e.g., from time T0 to time T1) and a second accumulation period (e.g., from time T0 to time T4), and the first accumulation period is shorter than the second accumulation period. Also, the signal generated during the first accumulation period is controlled to be output between the end of the first accumulation period (e.g., time T1) and the end of the second accumulation period (e.g., time T4).

[0074] In this embodiment, the first and second accumulation periods overlap and start at the same time. Furthermore, the end of the second accumulation period is at the end of a full frame period (one vertical period), and the second accumulation period is an integer multiple of the first accumulation period (four times in the example of FIG. 7).

[0075] However, the second accumulation period does not need to be an integer multiple of the first accumulation period, but rather the second accumulation period may be longer than the first accumulation period and may end after the first accumulation period.

[0076] That is, an image with a short accumulation period and an image with a long accumulation period are created, and the timing at which the short accumulation period ends is set earlier than the timing at which the long accumulation period ends. As soon as the short accumulation period ends, that image is output and sent to a downstream recognition unit. The subject is recognized based on signals generated during at least the first accumulation period. The recognition unit 604 as a recognition means recognizes the subject based on signals generated during at least the first accumulation period.

[0077] Therefore, while image recognition was only possible in a full-frame period in the prior art, in this embodiment, image recognition can be performed in every 1 / 4 full-frame period, and for example, when a moving object is moving at high speed, obstacles and the like can be quickly recognized. Furthermore, if the accumulation period is further shortened, image recognition can be performed more frequently.

[0078] Furthermore, by outputting the recognition results to the ECU 701 with low latency, it is possible to apply automatic brakes quickly, or to avoid obstacles early. Also, since image recognition can be performed every 1 / 4 full frame period, obstacles that appear during that time can be recognized in a timely manner.

[0079] Furthermore, the display device of this embodiment displays, as an image, a signal generated during at least the second accumulation period. Note that an image generated during the second accumulation period, which has a longer accumulation period, can improve contrast and is therefore suitable as a display image. That is, an image generated during the first accumulation period, which has a shorter accumulation period, is suitable for rapid subject recognition, and an image generated during the second accumulation period, which has a longer accumulation period, is suitable as a display image.

[0080] In this embodiment, since an APD is used, unlike a CMOS sensor, the accumulated charge does not deteriorate due to the readout operation, so the accumulation periods can be overlapped. Also, since there is no readout noise, the original signal does not deteriorate no matter how many times it is read out from a single accumulation.

[0081] Fig. 11 is a diagram showing the relationship between memory circuits and buffers in an embodiment. Fig. 11 shows a state in which memory circuits 212 in signal processing circuit 103 in Fig. 3 are arranged in N rows and M columns, and each memory circuit is represented as memory 1-1 to memory NM. Also, buffer 1 to buffer M in Fig. 11 represent buffers included in read circuit 112 in Fig. 3. Output circuit 114 in Fig. 11 corresponds to output circuit 114 in Fig. 3.

[0082] Fig. 12 is a flowchart showing details of an example of driving a photoelectric conversion element in an embodiment, and Fig. 13 is a flowchart that follows Fig. 12. Note that the operations of the steps in the flowcharts of Fig. 12 and Fig. 13 are performed sequentially by a CPU or the like serving as a computer in the camera control unit 605 executing a computer program stored in memory.

[0083] In step S101 of Fig. 12, i is set to 1. Next, in step S102, the count value Count of the counter circuit 211 at time Ti is output to the memory circuit 212. At this time, output is simultaneously performed to the memory circuits of all pixels. This operation corresponds to the operation at time T1 in Fig. 7.

[0084] Next, in step S103, j=1 is set, and in step S104, the count value Count(jki) in memory circuit jk in Fig. 11 is output to buffer k. At this time, columns 1 to M are output to the buffer simultaneously. This operation means that the count value of the first row in Fig. 11 is taken into the buffer.

[0085] Next, in step S105, k is set to 1, and in step S106, the count value Count(jki) of buffer k is output to output circuit 114. This operation corresponds to the operation of reading out the signal of the buffer in the leftmost column in FIG. 11 from the output circuit.

[0086] Next, proceed to step S107 in FIG. 13 via A, and in step S107, determine whether k < M. If Yes, then in step S108, increment k by 1 such that k = k + 1, return to step S106 via B, and perform the operation of step S106. This operation corresponds to the operation of reading the signal of the buffer in the second column from the left in FIG. 11 from the output circuit.

[0087] If it becomes No in step S107, that is, if k = M, it means that the signal of the buffer in the M-th column of FIG. 11 has been read from the output circuit. Next, proceed to step S109 and determine whether j < N. If Yes in step S109, then in step S110, increment j by 1 such that j = j + 1, and return to step S104 via C. This corresponds to the operation for starting the reading of the next row.

[0088] If it is determined to be No in step S109, it means that the reading of all rows has been completed. So, proceed to step S111 and determine whether j < 4. If it is determined to be Yes in step S111, then proceed to step S112, increment i by 1 such that i = i + 1, and return to step S102 via D. This operation corresponds to the operation for starting the reading at the next time T2.

[0089] If it is determined to be No in step S111, it means that the reading at time T4 has been completed. So, proceed to step S113 and reset the counter circuit 211 with a reset signal. This operation corresponds to the reset operation of the counter circuit 211 at time T4 in FIG. 7. In the above manner, the signals accumulated in the photoelectric conversion element 100 can be sequentially read out.

[0090] FIG. 14 is a diagram for explaining an example of a specific control method of the moving body 700 by the ECU 701 according to the embodiment. In FIG. 14, the control method of the photoelectric conversion device 600 is the same as that in FIG. 7, and the full frame is divided into frames of four equal periods, and the period of each frame corresponds to the frame period in FIG. 7.

[0091] In FIG. 14, the processing of each frame from when an image signal is read out from the readout circuit 112 in the photoelectric conversion element 100 until a final image is generated by the image processing unit 603 is defined as "image generation."

[0092] Furthermore, in this embodiment, as shown in "Image Output" in Figure 14, only the image signals of frames 0_4 and 1_4, which have the longest accumulation period (33.3 ms), are output from the image processing unit 603 to the ECU 701 as images to be displayed on the display unit 703 of the moving body 700.

[0093] Furthermore, the recognition unit 604 performs recognition processing on all frames, and supplies the results of the recognition processing to the ECU 701 ("recognition result output" in FIG. 14). The recognition results are displayed on the display unit 703 of the moving object 700, or are used for driving control by the vehicle control unit 702 ("driving control" in FIG. 14). Note that if the subject is not recognized in the recognition processing, it is not necessarily necessary to output the recognition results to the ECU 701, and as will be described later, they may be output as necessary.

[0094] Next, a description will be given of the process from outputting a display image from photoelectric conversion device 600 to moving object 700 to causing ECU 701 to display the image on display unit 703. In Fig. 14, at time T1 of full frame 1, various types of image processing are performed in image processing unit 603 within photoelectric conversion device 600, and the final "image generation" of frame 0_4 is performed.

[0095] Next, during the period from time T1 to time T2, the image signal of frame 0_4 is "image output" from the image processing unit 603 to the ECU 701 one horizontal line at a time, and is temporarily stored in a memory (not shown). At the timing of time T2 when all the image signals of frame 0_4 have been acquired, the ECU 701 causes the display unit 703 to start "image display" of the image of frame 0_4.

[0096] Since the image displayed on the display unit 703 is updated once per full frame period (four frame period), the image of frame 0_4 updated at time T2 of full frame 1 continues to be "displayed" until time T2 of full frame 2.

[0097] Next, the processing of the recognition result from outputting the object recognition result by the recognition unit 604 to the moving body 700 until the ECU 701 controls the vehicle control unit 702 and the display unit 703 will be described with reference to FIGS.

[0098] 15 is a flowchart showing an example of a control method for object recognition, etc. according to the embodiment. The operation of each step in the flowchart in FIG. 15 is performed sequentially by a CPU or the like serving as a computer in the camera control unit 605 executing a computer program stored in memory.

[0099] 15, the recognition unit 604 acquires an image of frame 0_4 at time T1 of full frame 1. Then, in step S152, the "recognition process" of the subject is started, and the recognition result (Rcg0_4) is acquired as shown in FIG. 14. Here, step S152 functions as a recognition step that performs recognition process on an image signal generated based on the difference between the count values ​​of the counter at the start and end of the accumulation period.

[0100] Then, in step S153, it is determined whether the subject has been recognized. If the determination in step S153 is No, the process proceeds to step S155, and if the determination is Yes, the process proceeds to step S154.

[0101] In step S154, the recognition unit 604 starts transmitting the recognition processing result Rcg0_4 by interrupt processing to the ECU 701. The transmission of the recognition processing result Rcg0_4 is completed at the timing of time T1+Td ("recognition result output" in FIG. 14).

[0102] Here, step S154 functions as a control step in which, as the recognition step (recognition means) completes the recognition process, the camera control unit 605 as control means outputs the result of the recognition process to the external ECU 701 or the like.

[0103] At the timing of time T1+Td when Rcg0_4 is acquired, the ECU 701 transmits a signal for performing vehicle control to the vehicle control unit 702. The vehicle control unit 702 drives, stops, controls the direction, etc. of the moving object 700 based on the received vehicle control signal ("driving control" in FIG. 14).

[0104] In this way, by controlling the moving body 700 immediately after receiving Rcg0_4, even if an obstacle enters the moving body while it is moving at high speed, it is possible to perform driving control such as quickly applying the brakes or changing course to avoid the obstacle early.

[0105] Furthermore, almost simultaneously with transmitting the vehicle control signal from the ECU 701 to the vehicle control unit 702, the ECU 701 transmits the acquired information on Rcg0_4 as recognition result information to the display unit 703. Based on the received recognition result information, the display unit 703 superimposes information such as the subject type, reliability, and coordinate information as shown in Fig. 9(C) on the image being displayed using a GUI or the like ("recognition result display" in Fig. 14).

[0106] It is desirable to display the coordinate information using a frame or the like so that it is visually easy for the driver to see, but a display method other than a frame may also be used. It is also desirable to use a color that is easy for the driver to see. In this way, by transmitting and displaying the recognition result information to the display unit 703 immediately after the ECU 701 receives Rcg0_4, the driver can be promptly notified of the approaching subject or obstacle information.

[0107] In this embodiment, during the period from time T2 of full frame 1 to time T2 of full frame 2, the image of frame 0_4 accumulated in full frame 0 is displayed on the display unit 703. However, before the displayed image switches from full frame 0 to full frame 1, the recognition results of Rcg1_1, Rcg1_2, Rcg1_3, and Rcg1_4 can be received in advance and displayed on the display unit 703.

[0108] The image of frame 0_4, which starts to be transmitted in step S154, is temporarily stored in a memory (not shown) and then displayed after time T2. However, as described above, the image displayed on the display unit 703 is an image of each full frame such as frame 0_4 or frame 1_4, and in this embodiment, the images of frames obtained in between are not used for display.

[0109] In step S155 following step S154, the process waits for the synchronization signal for the next frame to arrive, and when it arrives, the process proceeds to step S156. In step S156, it is determined whether an end instruction has been issued by the user or by turning off the power of the imaging device, and if the answer is Yes, the flow in FIG.

[0110] If the determination in step S156 is No, the process returns to step S151 and the above series of operations is repeated, thereby sequentially performing new recognition processing based on the image acquired in each frame, and displaying the recognition results on the currently displayed image or reflecting them in driving control.

[0111] Therefore, subject information can be notified to the driver or vehicle control unit 702 at a shorter cycle and earlier timing than the image update cycle. In other words, if a subject or obstacle is approaching, the driver can quickly apply the brakes or change course, making it possible to avoid danger.

[0112] In this embodiment, the content to be output is determined according to the result of subject recognition, and whether or not the result of the recognition process is output to the outside is switched depending on whether or not the subject is recognized by the recognition means. In other words, when the recognition means recognizes a subject from image signals of a predetermined period, the image signals of the predetermined period are also output to the outside together with the result of the recognition process.

[0113] For example, when the recognition unit 604 recognizes an object, the recognition result is displayed superimposed on the image signal, and when the object is not recognized, only the image signal is displayed. This prevents the display from becoming too complicated for the driver of the moving object.

[0114] The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications are possible based on the spirit of the present invention, and these modifications are not excluded from the scope of the present invention.

[0115] For example, in the above embodiment, the image signal and the recognition result are output using separate signal lines, but they may be output using the same signal line. That is, the image signal and the recognition result may be output simultaneously using an output signal line from the image processing unit 603 to the ECU 701.

[0116] Alternatively, the image signal and the recognition result may be output simultaneously using an output signal line from the recognition unit 604 to the ECU 701. Alternatively, one or both pieces of information may be output using the communication unit 607 via the camera control unit 605. The present invention also includes the following combinations.

[0117] (Configuration 1) A photoelectric conversion device characterized by having a plurality of pixels each having a sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons, a counter that counts the number of said pulses, and a memory that stores the count value of said counter, a recognition means that performs a recognition process on an image signal generated based on the difference between the count values ​​of said counter at the start and end of an accumulation period, and a control means that outputs the results of said recognition process to the outside as said recognition means completes said recognition process.

[0118] (Configuration 2) The photoelectric conversion device according to Configuration 1, wherein the control means has a first accumulation period and a second accumulation period during a full frame period, the first accumulation period is shorter than the second accumulation period, and controls the signal generated during the first accumulation period to be output between the end of the first accumulation period and the end of the second accumulation period.

[0119] (Configuration 3) The photoelectric conversion device described in configuration 1 or 2, characterized in that the control means switches whether or not to output the results of the recognition processing to the outside depending on whether or not the subject is recognized by the recognition means.

[0120] (Configuration 4) A photoelectric conversion device according to any one of configurations 1 to 3, characterized in that when the recognition means recognizes a subject from an image signal of a predetermined period, the image signal of the predetermined period is also output to the outside together with the result of the recognition process.

[0121] (Configuration 5) The photoelectric conversion device according to any one of configurations 1 to 4, wherein the sensor section includes an avalanche photodiode.

[0122] (Configuration 6) A moving body comprising the photoelectric conversion device according to any one of configurations 1 to 5, and movement control means for controlling the movement of the moving body.

[0123] (Configuration 7) The moving body according to configuration 6, wherein the movement control means controls the moving body based on the recognition result output from the recognition means.

[0124] (Configuration 8) The moving body according to configuration 6 or 7, further comprising a display unit for displaying the result of the recognition processing in the recognition means.

[0125] (Method) A control method for a photoelectric conversion device having a plurality of pixels, each pixel having a sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons, a counter that counts the number of said pulses, and a memory that stores the count value of said counter, characterized in that the control method includes a recognition step that performs recognition processing on an image signal generated based on the difference between the count values ​​of said counter at the start and end of an accumulation period, and a control step that outputs the results of said recognition processing to the outside as the recognition processing is completed by said recognition step.

[0126] (Program) A computer program for controlling the photoelectric conversion device according to any one of configurations 1 to 5 or each means of the moving body according to any one of configurations 6 to 8 by a computer.

[0127] Note that a computer program that realizes part or all of the control in this embodiment and the functions of the above-described embodiment may be supplied to the photoelectric conversion device via a network or various storage media. Then, a computer (or a CPU, MPU, etc.) in the photoelectric conversion device may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0128] 11: Sensor board 12: Pixel area 21: Circuit board 22: Circuit area 100: Photoelectric conversion element 101: Pixel 102: Photoelectric conversion unit 103: Signal processing circuit 110: Vertical scanning circuit 111: Horizontal scanning circuit 112: Readout circuit 113: Vertical signal line 114: Output circuit 115: Control pulse generation unit 201: Avalanche photodiode 202: Quench element 210: Waveform shaping section 211: Counter circuit 212: Memory circuit 213: Drive line 214: Drive line 600: Photoelectric conversion device 601: Imaging optical system 603: Image processing unit 604: Recognition part 605: Camera control unit 606: Storage section 607: Communications Department 700: Mobile 701:ECU 702: Vehicle control unit 703: Display section

Claims

1. a sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons; a counter that counts the number of pulses; a plurality of pixels each including a memory for storing a count value of the counter; a recognition unit that performs a recognition process on an image signal generated based on the difference between the count values ​​of the counter at the start and end of an accumulation period; and control means for outputting a result of the recognition process to the outside as the recognition means completes the recognition process.

2. 2. The photoelectric conversion device according to claim 1, wherein the control means controls the device so that a full frame period has a first accumulation period and a second accumulation period, the first accumulation period is shorter than the second accumulation period, and the signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period.

3. 2. The photoelectric conversion device according to claim 1, wherein the control means switches whether or not to output the result of the recognition process to the outside, depending on whether or not the subject has been recognized by the recognition means.

4. 2. The photoelectric conversion device according to claim 1, wherein when the recognition means recognizes an object from the image signal of a predetermined period, the image signal of the predetermined period is also output to the outside together with the result of the recognition process.

5. 2. The photoelectric conversion device according to claim 1, wherein the sensor unit includes an avalanche photodiode.

6. The photoelectric conversion device according to any one of claims 1 to 5, and a movement control means for controlling the movement of the moving body.

7. 7. The mobile body according to claim 6, wherein said movement control means controls the mobile body based on the recognition result output from said recognition means.

8. 7. A moving body according to claim 6, further comprising a display unit for displaying the results of the recognition processing in said recognition means.

9. a sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons; a counter that counts the number of pulses; a memory that stores the count value of the counter; and a plurality of pixels each including the counter, a recognition step of performing a recognition process on an image signal generated based on a difference between the count values ​​of the counter at the start and end of an accumulation period; a control step of outputting a result of the recognition process to an outside as the recognition process is completed by the recognition step.

10. A computer program for controlling the photoelectric conversion device according to any one of claims 1 to 5 or each means of the moving body according to any one of claims 6 to 8 by a computer.

Images