Photoelectric conversion device, movable body, control method, and computer program

Abstract

To provide a photoelectric conversion device that can change the recognition frequency of image recognition in a short time according to subject information.SOLUTION: A photoelectric conversion device 600 has: a photoelectric conversion element that has a plurality of pixels each including a photoelectric conversion part emitting pulses at a frequency according to the frequency of photon reception, a counter counting the number of pulses, and a memory storing the count value of the counter; a camera control unit that controls to generate a signal on the basis of the difference in the count value of the counter between the start and the end of a storage period, and output the signal which has a first storage period and a second storage period in one frame, the first storage period being shorter than the second storage period, and is generated in the first storage period, from the end of the first storage period until the end of the second storage period; and a recognition unit that performs recognition processing on the basis of an image signal. The recognition unit changes recognition frequency in the one frame on the basis of the subject information.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a photoelectric conversion device, a moving object, a control method, a computer program, and the like. [Background technology]

[0002] In recent years, photoelectric conversion devices have been developed that digitally count the number of photons incident on an avalanche photodiode (APD) and output the counted value from a pixel as a photoelectrically converted digital signal. For example, Patent Document 1 describes a configuration in which a photoelectric conversion device having an APD can output multiple images whose accumulation periods overlap with each other, thereby enabling continuous shooting even in low illuminance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7223070 Summary of the Invention [Problem to be solved by the invention]

[0004] However, for example, when considering the image sensor of an in-vehicle camera installed on a moving object, because recognition processing is performed on a frame-by-frame basis in normal sensor driving, recognition processing can only be performed every 33.3 ms in the case of 30 fps. Therefore, even if an object appears immediately after a frame change in an in-vehicle camera, recognition processing cannot be performed until the end of the frame.

[0005] In addition, in order to suppress flicker caused by traffic lights, in-vehicle cameras often have a certain accumulation period (11 ms or more), and in particular, in low-illumination conditions, the accumulation period is extended to capture brighter images. However, because the accumulation period is long, subject blurring may occur depending on the vehicle's conditions and the subject's conditions, which may result in a lower recognition rate.

[0006] In addition, if there is a subject approaching in the direction of travel of the vehicle, it is necessary to pay attention to the subject because of the danger. In this case, the frequency of checking the subject's situation may be insufficient with frame-by-frame recognition processing, and there is a possibility that emergency response will be delayed.

[0007] SUMMARY OF THE PRESENT EMBODIMENTS An object of the present invention is to provide a photoelectric conversion device capable of changing the recognition frequency of image recognition in a short period of time in accordance with subject information. [Means for solving the problem]

[0008] A photoelectric conversion device according to one aspect of the present invention comprises: A photoelectric conversion unit that emits pulses at a frequency corresponding to the frequency of receiving photons; A counter that counts the number of pulses; A plurality of pixels each including a memory for storing a count value of the counter; generating a signal based on a difference between the count values ​​of the counter at the start and end of an accumulation period; A control means for controlling the signal generation circuit so that a first accumulation period and a second accumulation period are provided within one frame, the first accumulation period being shorter than the second accumulation period, and a signal generated during the first accumulation period is output during a period from the end of the first accumulation period to the end of the second accumulation period; A recognition processing unit that performs recognition processing based on an image signal, The recognition processing unit changes the recognition frequency within one frame based on subject information. Effect of the Invention

[0009] According to the present invention, it is possible to realize a photoelectric conversion device capable of changing the recognition frequency of image recognition in a short period of time in accordance with subject information. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing a configuration example of a photoelectric conversion element according to an embodiment of the present invention. [Diagram 2]2 is a diagram showing a configuration example of a sensor substrate 11. FIG. [Diagram 3] 2 is a diagram showing an example of the configuration of a circuit board 21. FIG. [Figure 4] 2 and 3, and shows an equivalent circuit of a pixel 101 and a signal processing circuit 103 corresponding to the pixel 101. [Diagram 5] 2 is a diagram illustrating a relationship between the operation of an APD 201 and an output signal. FIG. [Figure 6] 7 is a functional block diagram of a photoelectric conversion device 600 and a moving object 700 according to the embodiment. [Figure 7] 11 is a diagram for explaining a photoelectric conversion method by a camera control unit 605 according to an embodiment. FIG. [Figure 8] FIG. 13 is a diagram showing an example of an image divided into a plurality of frames. [Figure 9] FIG. 2 is a diagram illustrating a relationship between a memory circuit and a buffer in the embodiment. [Figure 10] 5 is a flowchart showing details of an example of driving a photoelectric conversion element in the embodiment. [Figure 11] This is a continuation of the flowchart in Figure 10. [Figure 12] FIG. 11 is a diagram for explaining the recognition frequency during normal driving in the embodiment. [Figure 13] 10A and 10B are diagrams for explaining the recognition frequency when a subject approaching in the traveling direction of the host vehicle is detected in an embodiment. [Figure 14] 10 is a flowchart illustrating a change in the recognition frequency when a subject approaching in the traveling direction of the host vehicle is detected in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified.

[0012] 1 is a diagram showing a configuration example of a photoelectric conversion element according to an embodiment of the present invention. In the following, a photoelectric conversion device having a so-called stacked structure in which a photoelectric conversion element 100 is configured by stacking and electrically connecting two substrates, a sensor substrate 11 and a circuit substrate 21, will be described as an example.

[0013] However, a so-called non-laminated structure in which the components included in the sensor substrate and the components included in the circuit substrate are disposed on a common semiconductor layer may be used. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes signals detected in the pixel region 12.

[0014] 2 is a diagram showing a configuration example of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally across a plurality of rows and columns. The pixels 101 include a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, APD), and the photoelectric conversion unit 102 emits a pulse at a frequency corresponding to the frequency of receiving photons. The number of rows and columns of the pixel array constituting the pixel region 12 is not particularly limited.

[0015] Fig. 3 is a diagram showing an example of the configuration of the circuit board 21. The circuit board 21 has a signal processing circuit 103 that processes charges photoelectrically converted by each photoelectric conversion unit 102 in Fig. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114.

[0016] The vertical scanning circuit 110 receives a control pulse supplied from a control pulse generating unit 115, and sequentially supplies the control pulse to a plurality of pixels arranged in the row direction. The vertical scanning circuit 110 uses logic circuits such as a shift register and an address decoder.

[0017] The signal output from the photoelectric conversion unit 102 of each pixel is processed by each signal processing circuit 103. The signal processing circuit 103 is provided with a counter, a memory, etc., and a digital value is stored in the memory. The horizontal scanning circuit 111 inputs a control pulse that sequentially selects each column to the signal processing circuit 103 in order to read the signal from the memory of each pixel in which the digital signal is stored.

[0018] A signal is output to the vertical signal line 113 from the signal processing circuit 103 of the pixel of the row selected by the vertical scanning circuit 110. The signal output to the vertical signal line 113 is output to the outside of the photoelectric conversion element 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has a plurality of buffers built therein and connected to the vertical signal line 113.

[0019] 2 and 3, a plurality of signal processing circuits 103 are arranged in a region overlapping the pixel region 12 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged so as to overlap between an end of the sensor substrate 11 and an end of the pixel region 12 in a plan view.

[0020] That is, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in a region overlapping the non-pixel region in a plan view.

[0021] The arrangement of the vertical signal lines 113, the readout circuits 112, and the output circuits 114 are not limited to the example shown in Fig. 3. For example, the vertical signal lines 113 may be arranged extending in the row direction, and the readout circuits 112 may be arranged at the ends of the vertical signal lines 113. Also, it is not necessary to provide one signal processing circuit 103 for each photoelectric conversion unit, and a configuration may be adopted in which one signal processing unit is shared by a plurality of photoelectric conversion units and performs signal processing sequentially.

[0022] FIG. 4 is a diagram showing an equivalent circuit of the pixel 101 in FIG. 2 and FIG.

[0023] The APD 201 included in the photoelectric conversion unit 102 generates a pair of electric charges according to incident light by photoelectric conversion. One of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VL (first voltage) is supplied. The other of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VH (second voltage) higher than the voltage VL is supplied.

[0024] In Fig. 4, one node of the APD 201 is the anode, and the other node of the APD is the cathode. A reverse bias voltage is supplied to the anode and cathode of the APD 201 so that the APD 201 performs avalanche multiplication. By supplying such a voltage, the charge generated by the incident light undergoes avalanche multiplication, generating an avalanche current.

[0025] When a reverse bias voltage is supplied, there are two modes: Geiger mode, in which the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, in which the voltage difference between the anode and cathode is close to or less than the breakdown voltage. An APD that operates in Geiger mode is called a SPAD. In the case of a SPAD, for example, the drive voltage VL (first voltage) is -30V, and the drive voltage VH (second voltage) is 1V.

[0026] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212. The quench element 202 is connected to a power supply line to which a driving voltage VH is supplied and one of the anode and cathode nodes of the APD 201.

[0027] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, and has a function of suppressing avalanche multiplication by suppressing the voltage supplied to the APD 201 (quench operation).The quench element 202 also has a function of returning the voltage supplied to the APD 201 to the drive voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0028] FIG. 4 shows an example in which the signal processing circuit 103 includes a waveform shaping section 210, a counter circuit 211, and a memory circuit 212 in addition to the quench element 202.

[0029] The waveform shaping unit 210 shapes the voltage change of the cathode of the APD 201 obtained when a photon is detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. Although an example in which one inverter is used as the waveform shaping unit 210 is shown in Fig. 4, a circuit in which a plurality of inverters are connected in series may be used, or another circuit having a waveform shaping effect may be used.

[0030] The counter circuit 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. When a control pulse RES is supplied via a drive line 213, the signal held in the counter circuit 211 is reset. Here, the counter circuit 211 generates a signal based on the difference between the count values ​​at the start and end of the accumulation period.

[0031] 3 to the memory circuit 212 via a drive line 214 (not shown in FIG. 3) in FIG. 4, and switches between electrical connection and disconnection between the counter circuit 211 and the vertical signal line 113. The memory circuit 212 functions as a memory that temporarily stores the count value of the counter, and outputs an output signal from the counter circuit 211 of the pixel to the vertical signal line 113.

[0032] Note that electrical connections may be switched by disposing a switch such as a transistor between the quench element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103. Similarly, the supply of the driving voltage VH or the driving voltage VL to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0033] 5 is a diagram showing a schematic diagram of the relationship between the operation of the APD 201 and the output signal. The input side of the waveform shaping unit 210 is nodeA, and the output side is nodeB. Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage at nodeA drops.

[0034] When the amount of voltage drop further increases and the potential difference applied to the APD 201 decreases, the avalanche multiplication of the APD 201 stops as at time t2, and the voltage level of nodeA does not drop below a certain value.

[0035] After that, between time t2 and time t3, a current flows through nodeA to compensate for the voltage drop from voltage VL, and at time t3, nodeA is stabilized to its original potential level. At this time, the portion of the output waveform at nodeA that exceeds a certain threshold is shaped by waveform shaping section 210 and is output as a pulse signal at nodeB.

[0036] Next, a photoelectric conversion device 600 and a moving object 700 according to an embodiment will be described. Fig. 6 is a functional block diagram of the photoelectric conversion device 600 and the moving object 700 according to the embodiment. Some of the functional blocks shown in Fig. 6 are realized by causing a computer (not shown) included in the photoelectric conversion device 600 and the moving object 700 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0037] However, some or all of these may be realized by hardware. For the hardware, a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP) may be used. Furthermore, each functional block shown in Fig. 6 does not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths.

[0038] 1 to 5, an imaging optical system 601, an image processing unit 603, a recognition unit 604, a camera control unit 605, a storage unit 606, a communication unit 607, etc. The photoelectric conversion unit of the photoelectric conversion element 100 is composed of an avalanche photodiode for photoelectrically converting an optical image, as described in FIGS.

[0039] The photoelectric conversion device of the embodiment is mounted on a moving body 700, and a camera unit consisting of a set of an imaging optical system 601 and a photoelectric conversion element 100 is configured to capture an image in at least one direction, for example, in front, behind, or to the side of the moving body. Note that a plurality of camera units may be provided on the moving body 700.

[0040] The image processing unit 603 performs image processing such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaic processing, data compression, etc. on the image signal acquired by the photoelectric conversion element 100 to generate a final image signal. Note that, if the photoelectric conversion element 100 has an on-chip color filter such as RGB, it is desirable for the image processing unit 603 to perform processing such as white balance correction and color conversion.

[0041] Moreover, the output of the image processing unit 603 is supplied to a recognition unit 604, an ECU (Electric Control Unit) 701 of the moving object 700, and a camera control unit 605. The recognition unit 604 functions as a recognition processing unit that performs recognition processing based on an image signal generated from a signal generated by the photoelectric conversion unit, and recognizes surrounding people, vehicles, etc. by performing image recognition based on the image signal, and issues a warning, etc. as necessary.

[0042] In this embodiment, the moving body 700 is described using an example of an automobile, but the moving body may be any moving body, such as an airplane, a train, a ship, a drone, an AGV, or a robot.

[0043] The camera control unit 605 has a built-in CPU as a computer and a memory that stores a computer program, and controls each unit of the photoelectric conversion device 600 by the CPU executing the computer program stored in the memory.

[0044] The camera control unit 605 functions as a control means, and for example, controls the length of the exposure period of each frame of the photoelectric conversion element 100 and the timing of the control signal CLK via a control pulse generation unit of the photoelectric conversion element 100. In addition, the camera control unit 605 also controls the recognition process of the recognition unit 604 based on an output from the ECU 701, as will be described in detail later.

[0045] The storage unit 606 includes a recording medium such as a memory card, a hard disk, etc., and can store and read image signals. The communication unit 607 includes a wireless or wired interface, and outputs the generated image signal to the outside of the photoelectric conversion device 600 and receives various signals from the outside.

[0046] The ECU 701 incorporates a CPU as a computer and a memory that stores a computer program, and controls each part of the moving body 700 by the CPU executing the computer program stored in the memory.

[0047] Also, a subject detection unit 704 is connected, and detection information of a subject is input to the ECU 701. Information detected by the subject detection unit 704 includes the subject's traveling direction, the type of subject, the amount of change of the subject per unit time, the moving speed of the subject, the position of the subject, and the like, based on the subject detection information.

[0048] In addition, in FIG. 6, detection of the subject is performed based on an image acquired by the photoelectric conversion device 600, but detection of the subject may also be performed based on an image signal output from a sensor other than the photoelectric conversion device 600.

[0049] The output of the ECU 701 is supplied to a vehicle control unit 702, a display unit 703, and a camera control unit 605. The vehicle control unit 702 functions as a movement control unit that drives, stops, controls the direction, etc. of the vehicle as a moving body based on the output of the ECU 701.

[0050] The vehicle control unit 702 as a movement control unit controls the operation of the moving body based on the recognition result by the recognition processing using the signal generated at least during the first accumulation period. The first accumulation period will be described later. The display unit 703 functions as a display means, and includes a display element such as a liquid crystal device or an organic EL, and is mounted on the moving body 700.

[0051] The display unit 703 displays images acquired by the photoelectric conversion element 100 and various information related to the vehicle's running state and the like to the driver of the vehicle 700 using, for example, a GUI based on the output of the ECU 701 .

[0052] Also, subject information detected by the subject detection unit 704 is input to the camera control unit 605 via the ECU 701, and a control signal for changing the recognition frequency of image recognition within one full frame is output. The change in recognition frequency will be described later.

[0053] In addition, the image processing unit 603, the recognition unit 604, etc. in Figure 6 do not have to be mounted on the moving body 700, and may be provided, for example, in an external terminal provided separately from the moving body 700 for remotely controlling the moving body 700 or for monitoring the traveling of the moving body.

[0054] 7 is a diagram for explaining a photoelectric conversion method by the camera control unit 605 according to the embodiment. In this embodiment, one frame having a length of 33.3 ms is called a full frame (or a main frame), and the full frame divided into four is called a frame (or a sub-frame). In this embodiment, one full frame having a length of 33.3 ms is divided into four.

[0055] That is, as shown in Fig. 7, full frame 1 is divided into frames 1_1, 1_2, 1_3, and 1_4 of equal duration (8.33 ms). In Fig. 7 and subsequent figures, frames 0_3, 0_4, 1_1, 1_2, 1_3, 1_4, 2_1, and 2_2 are represented as F0_3, 0_4, 1_1, 1_2, 1_3, 1_4, 2_1, and 2_2.

[0056] Frame 1_1 has an accumulation period from start time T0 of full frame 1 to time T1, and frame 1_2 has an accumulation period from time T0 to time T2. Frame 1_3 has an accumulation period from time T0 to time T3, and frame 1_4 has an accumulation period from time T0 to time T4. In this embodiment, for example, the accumulation period from time T0 to time T1 is called a first accumulation period, and the accumulation period from time T0 to time T4 is called a second accumulation period.

[0057] Then, at time T0, the counter circuit 211 is reset, and count values ​​C1_1, C1_2, C1_3, and C1_4 are obtained from the counter circuit 211 at times T1 to T4, respectively.

[0058] Moreover, the count values ​​C1_1, C1_2, C1_3, and C1_4 are temporarily stored in the memory circuit 212. Then, the signals for one row temporarily stored in the memory circuit 212 are sequentially output from the photoelectric conversion elements via the buffer of the readout circuit 112.

[0059] In this way, according to this embodiment, the signals accumulated during the period of frame 1_1 are read out from time T1 to T2 and are promptly processed by the recognition unit 604. Therefore, image recognition can be performed promptly. Similarly, the signals accumulated during the periods of frames 1_2, 1_3, and 1_4 are sequentially read out from time T2 to T3, T3 to T4, and T4 to T1, respectively, and image recognition can be performed repeatedly.

[0060] Fig. 8 is a diagram showing an example of images of a plurality of divided frames. As shown in Fig. 8, the image of frame 1_1 is dark because the accumulation time is short, but there is little subject blur of a person who jumps out in front of the camera. On the other hand, the accumulation time becomes longer in the order of frames 1_2, 1_3, and 1_4, so subject blur is more likely to occur. Note that stopped vehicles and white lines are less likely to be blurred, and the longer the accumulation time, the more likely the contrast is to improve.

[0061] As described above, in this embodiment, one frame has a first accumulation period and a second accumulation period, the first accumulation period is shorter than the second accumulation period, and the signal generated during the first accumulation period is controlled to be output between the end of the first accumulation period and the end of the second accumulation period.

[0062] In this embodiment, the first and second accumulation periods overlap, and start at the same time. Furthermore, the end of the second accumulation period is a frame boundary, and the second accumulation period is an integer multiple of the first accumulation period. That is, the end of the second accumulation period coincides with the end of the full frame.

[0063] However, the second accumulation period does not need to be an integer multiple of the first accumulation period; the second accumulation period only needs to be longer than the first accumulation period and the end of the second accumulation period needs to be after the end of the first accumulation period.

[0064] That is, an image with a short accumulation period and an image with a long accumulation period are created, the timing when the short accumulation period ends is set to be earlier than the timing when the long accumulation period ends, and as soon as the short accumulation period ends, the image is output and sent to the downstream recognition unit.Then, the subject is recognized based on at least a signal generated during the first accumulation period.The recognition unit 604 as a recognition means recognizes the subject based on at least a signal generated during the first accumulation period.

[0065] Therefore, while in the past it was not possible to recognize an image until one frame had passed, in this embodiment, image recognition can be performed after as little as a quarter frame period, and for example, when a moving body is moving at high speed, an obstacle or the like can be recognized quickly. Therefore, it is possible to apply the brakes quickly, or to avoid the obstacle early.

[0066] In addition, an image with a long accumulation time can improve the contrast, and can be used as a display image. That is, an image with a short accumulation time is suitable for quick image recognition, and an image with a long accumulation time is suitable for a display image. In this way, the display unit of this embodiment displays at least the signal generated during the second accumulation period as an image. In this embodiment, the recognition processing unit further recognizes the subject based on the signal generated during the second accumulation period.

[0067] In addition, since the present embodiment uses an APD, unlike a CMOS sensor, the accumulated charge does not deteriorate when read out, so the accumulation periods can be overlapped. Also, since there is no readout noise, the original signal does not deteriorate even if it is read out many times after one accumulation.

[0068] Fig. 9 is a diagram showing the relationship between memory circuits and buffers in an embodiment. Fig. 9 shows a state in which memory circuits 212 in signal processing circuit 103 in Fig. 3 are arranged in N rows and M columns, and each memory circuit is represented as memory 1-1 to memory NM. Also, buffer 1 to buffer M in Fig. 9 indicate buffers included in read circuit 112 in Fig. 3. Output circuit 114 in Fig. 9 corresponds to output circuit 114 in Fig. 3.

[0069] Fig. 10 is a flowchart showing details of an example of driving a photoelectric conversion element in an embodiment, and Fig. 11 is a flowchart continuing from Fig. 10. Note that the operations of the steps in the flowcharts of Fig. 10 and Fig. 11 are performed sequentially by a CPU or the like as a computer in the camera control unit 605 executing a computer program stored in a memory.

[0070] 10, the CPU in the camera control unit 605 sets i=1, and in step S102 outputs the count value Count of the counter circuit 211 at time Ti to the memory circuit 212. At this time, output is performed simultaneously to all memory circuits. This operation corresponds to the operation at time T1 in FIG.

[0071] Next, in step S103, the CPU in the camera control unit 605 sets j=1, and in step S104 outputs the count value Count(jki) in the memory circuit jk in Fig. 9 to the buffer. At this time, columns 1 to M are output to the buffer at the same time. This operation means that the count value of the first row in Fig. 9 is taken into the buffer.

[0072] Next, in step S105, the CPU in the camera control unit 605 sets k=1, and in step S106 outputs the count value Count(jki) of buffer k to the output circuit 114. This operation corresponds to the operation of reading out the signal of the buffer in the leftmost column in FIG. 9 from the output circuit.

[0073] Next, it proceeds to step S107 in FIG. 11 via A. In step S107, the CPU in the camera control unit 605 determines whether k < M. If Yes, then in step S108, k is incremented by 1 such that k = k + 1. After that, it returns to step S106 via B and performs the operation of step S106. This operation corresponds to the operation of reading the signal of the buffer in the second column from the left in FIG. 9 from the output circuit.

[0074] If it becomes No in step S107, that is, if k = M, it means that the signal of the buffer in the M-th column in FIG. 9 has been read from the output circuit. Next, it proceeds to step S109, and the CPU in the camera control unit 605 determines whether j < N.

[0075] If it is Yes in step S109, then in step S110, the CPU in the camera control unit 605 sets j = J + 1, increments j by 1, and returns to step S104 via C. This corresponds to the operation for starting the reading of the next row.

[0076] If it is determined to be No in step S109, it means that the reading of all rows has been completed. So, it proceeds to step S111, and the CPU in the camera control unit 605 determines whether j < 4. If it is determined to be Yes in step S111, it proceeds to step S112, and the CPU in the camera control unit 605 sets i = i + 1, increments i by 1, and returns to step S102 via D. This operation corresponds to the operation for starting the reading at the next time T2.

[0077] If it is determined to be No in step S111, it means that the reading at time T4 has been completed. So, it proceeds to step S113, and the CPU in the camera control unit 605 resets the counter circuit 211 with a reset signal. This operation corresponds to the reset operation of the counter circuit 211 at time T4 in FIG. 7. As described above, the signals accumulated in the photoelectric conversion element 100 can be sequentially read out.

[0078] Next, a method for changing the frequency of image recognition performed by the recognition unit 604 based on subject information detected by the subject detection unit 704 in this embodiment will be described with reference to Figs. 12 and 13. The subject information includes information such as the moving direction of the subject.

[0079] Fig. 12 is a diagram for explaining the recognition frequency in the case of normal driving in the embodiment, and explains the recognition frequency when the subject's traveling direction is moving away from the traveling direction of the vehicle. As shown in Fig. 7, full frame 1 is divided into frames 1_1, 1_2, 1_3, and 1_4 of equal periods (8.33 ms).

[0080] At this time, the ECU 701 judges whether the subject is moving away from the traveling direction of the vehicle based on the position information and traveling direction information of the subject detected by the subject detection unit 704. If it is judged that the subject is moving away, the ECU 701 notifies the camera control unit 605 to read out the signals of the accumulation periods of frames 1_2 and 1_4 and perform image recognition twice.

[0081] Regarding the traveling direction of the vehicle, the road may be detected from the video signal acquired by the photoelectric conversion device 600 and the traveling direction of the vehicle may be determined based on the detection information, or the traveling direction of the vehicle may be determined based on the road detection result from the video signal and the steering angle of the vehicle. Also, the traveling direction of the vehicle may be determined based on map information from a car navigation system or the like and GPS information.

[0082] In this embodiment, even when the frequency of image recognition is reduced, image recognition can be performed every 1 / 2 full frame period, so that obstacles and the like can be quickly recognized when a moving object is moving. This makes it possible to quickly apply the brakes or quickly avoid the obstacles.

[0083] Next, Fig. 13 is a diagram for explaining the recognition frequency when a subject is detected approaching the traveling direction of the vehicle in the embodiment, and explains the recognition frequency when the traveling direction of the subject is a direction approaching the traveling direction of the vehicle. In this case, as in Fig. 7, full frame 1 is divided into frames 1_1, 1_2, 1_3, and 1_4 of equal periods (8.33 ms).

[0084] In this embodiment, when the traveling direction of the subject detected by the subject detection unit 704 is a direction approaching the traveling direction of the vehicle, the signals of the accumulation periods of frames 1_1, 1_2, 1_3, and 1_4 are read out, respectively. Then, the ECU 701 notifies the camera control unit 605 to perform image recognition four times.

[0085] In this manner, in this embodiment, the intervals at which the recognition processing unit starts recognition are set to be equal. The recognition processing unit performs the recognition process based on the signals read out for each first accumulation period within the second accumulation period.

[0086] Conventionally, image recognition was not possible until one full frame had passed, but in this embodiment, image recognition can be performed by reading out signals multiple times every 1 / 4 frame period, making it possible to quickly make a more accurate judgment on the movement of an obstacle, etc., or to avoid the obstacle early.

[0087] 14 is a flowchart for explaining a change in the recognition frequency when a subject approaching in the traveling direction of the vehicle is detected in the embodiment. The CPU or the like as a computer in the ECU 701 executes a computer program stored in a memory, and the operations of the steps in the flowchart in FIG. 14 are performed in sequence.

[0088] In step S141 in Fig. 14, the CPU of the ECU performs an initial setting of the frequency of the recognition process. Since the subject has not yet been detected, the frequency of the recognition process is set to the initial setting of every 1 / 2 full frame period. When the initialization of the recognition process is completed, the process proceeds to step S142.

[0089] Next, in step S142, the CPU of the ECU detects an object by the object detection unit 704, obtains the output thereof in the ECU 701, and proceeds to step S143.

[0090] Next, in step S143, the CPU of the ECU calculates the traveling direction of the subject based on the output of the subject detection unit 704, and proceeds to step S144.

[0091] Next, in step S144, the CPU of the ECU determines whether the subject is moving toward the subject relative to the vehicle's traveling direction, and displays the result. If the subject is moving toward the subject (Yes), the process proceeds to step S145, and if the subject is moving away (No), the process proceeds to step S146.

[0092] If the process proceeds to step S145, the ECU CPU sets the recognition process to be performed every 1 / 4 full frame period so that image recognition is performed four times within one full frame. After setting the recognition process within one frame for every 1 / 4 full frame period, the process proceeds to step S147.

[0093] In step S147, the CPU of the ECU 701 notifies the camera control unit 605 of the recognition frequency of the recognition process within one full frame set in step S145, and the process proceeds to step S142 to continue. The camera control unit 605, which has received the notification from the CPU of the ECU 701, outputs a control signal to the recognition unit 604 to change the recognition frequency.

[0094] Next, a description will be given of the process in the case where the process proceeds from step S144 to step S146. When the process proceeds to step S146, the CPU of the ECU sets the recognition process to be performed every 1 / 2 full frame period so that image recognition is performed twice within one frame.

[0095] Thereafter, in step S147, the CPU of the ECU notifies the camera control unit 605 from the ECU 701 of the recognition frequency of the recognition process within one full frame that was set in step S146, and the process proceeds to step S142 to continue the process.

[0096] That is, when the subject is moving away, the support by image recognition is kept to a minimum, warnings by the recognition process are not issued frequently, and the processing load is reduced.In addition, when the subject is approaching, the frequency of image recognition is increased to avoid accidents.

[0097] In this way, in the recognition unit 604 according to this embodiment, the recognition frequency of image recognition within one full frame can be changed based on the subject's situation (subject information), thereby optimizing the subject recognition accuracy.

[0098] In this embodiment, the recognition frequency of image recognition is changed when a specified switching condition is met based on the subject movement direction, i.e., whether or not the subject is approaching, as subject information. However, the subject information is not limited to this.

[0099] For example, as an example of subject information for changing the recognition frequency of image recognition while a vehicle is traveling, the moving speed of the subject may be detected and the switching condition may be changed according to the moving speed of the subject. For example, the recognition frequency may be controlled to be higher as the moving speed of the subject increases. Alternatively, the recognition frequency may be changed based on both the moving speed and direction of the subject. For example, the recognition frequency may be controlled to be higher as the relative approach speed of the subject to the vehicle increases.

[0100] In addition, in this embodiment, no limitations are placed on the type of subject, but the detected subjects may be limited to animals, cars, bicycles, people, etc., and the switching conditions may be changed depending on the type of subject.

[0101] That is, for example, the recognition frequency of a detected subject may be controlled to be higher when the detected subject is a person or a vehicle with a person in it than when the detected subject is an animal. In this way, the subject information includes at least one of the moving speed, moving direction, and type of the subject.

[0102] Also, for example, in the above embodiment, accumulation is performed for a 1 / 4 full frame period at the shortest, but accumulation for a 1 / N full frame period (N is an integer) may be used. Even if the readout cycle is set to every 1 / N full frame period, the effective accumulation period may be made shorter than the 1 / N full frame period by resetting the counter circuit midway through the 1 / N full frame period depending on the brightness of the subject, the image recognition accuracy, etc.

[0103] Alternatively, the counter circuit may be reset once at time T1 in Fig. 7. This may allow the count values ​​read out at the following times T2, T3, T4, etc. to be adjusted.

[0104] Although the present invention has been described in detail based on the preferred embodiment, the present invention is not limited to the above embodiment, and various modifications are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. Note that this embodiment includes the following combinations.

[0105] (Configuration 1) A photoelectric conversion device comprising: a plurality of pixels, each having a photoelectric conversion unit that emits a pulse at a frequency corresponding to the frequency of receiving photons, a counter that counts the number of the pulses, and a memory that stores the count value of the counter; a control means that generates a signal based on the difference between the count values ​​of the counter at the start and end of an accumulation period, has a first accumulation period and a second accumulation period within one frame, the first accumulation period is shorter than the second accumulation period, and controls so that the signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period; and a recognition processing unit that performs recognition processing based on an image signal, wherein the recognition processing unit changes the recognition frequency within one frame based on subject information.

[0106] (Configuration 2) The photoelectric conversion device according to configuration 1, wherein the recognition processing unit starts recognition at equal intervals.

[0107] (Configuration 3) The photoelectric conversion device according to configuration 1 or 2, wherein the first accumulation period and the second accumulation period overlap each other.

[0108] (Configuration 4) The photoelectric conversion device according to any one of configurations 1 to 3, wherein the first accumulation period and the second accumulation period start simultaneously.

[0109] (Configuration 5) The photoelectric conversion device according to any one of configurations 1 to 4, wherein the end of the second accumulation period coincides with the end of a frame.

[0110] (Configuration 6) The photoelectric conversion device according to any one of configurations 1 to 5, wherein the recognition processing section recognizes a subject based on a signal generated during at least the first accumulation period.

[0111] (Configuration 7) The photoelectric conversion device according to configuration 6, wherein the recognition processing section further recognizes the subject based on a signal generated during the second accumulation period.

[0112] (Configuration 8) The photoelectric conversion device according to any one of configurations 1 to 7, wherein the recognition processing unit performs the recognition processing based on a signal read out for each of the first accumulation periods within the second accumulation period.

[0113] (Configuration 9) The photoelectric conversion device according to any one of configurations 1 to 8, further comprising a display section that displays, as an image, at least a signal generated during the second accumulation period.

[0114] (Configuration 10) The photoelectric conversion device according to any one of configurations 1 to 9, wherein the photoelectric conversion section includes an avalanche photodiode.

[0115] (Configuration 11) The photoelectric conversion device according to any one of configurations 1 to 10, wherein the subject information includes a moving direction of the subject.

[0116] (Configuration 12) The photoelectric conversion device according to any one of configurations 1 to 11, wherein the subject information includes a moving speed of the subject.

[0117] (Configuration 13) The photoelectric conversion device according to any one of configurations 1 to 12, wherein the subject information includes a type of subject.

[0118] (Configuration 14) A moving body comprising: the photoelectric conversion device according to any one of configurations 1 to 10; and a movement control unit that controls the operation of the moving body based on the recognition result of the recognition processing unit.

[0119] (Method) A control method for a photoelectric conversion device having a plurality of pixels, each of which has a photoelectric conversion unit that emits a pulse at a frequency corresponding to the frequency of receiving photons, a counter that counts the number of the pulses, and a memory that stores the count value of the counter, the control method generating a signal based on the difference between the count values ​​of the counter at the start and end of an accumulation period, having a first accumulation period and a second accumulation period within one frame, the first accumulation period being shorter than the second accumulation period, controlling so that the signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period, and performing recognition processing based on an image signal, and changing the recognition frequency within one frame based on subject information.

[0120] (Program) A computer program for controlling each part of the photoelectric conversion device according to any one of configurations 1 to 13 or the moving body according to configuration 14 by a computer.

[0121] A computer program for implementing the functions of the above-described embodiment as part or all of the control in this embodiment may be supplied to the photoelectric conversion device via a network or various storage media. Then, a computer (or a CPU, MPU, etc.) in the photoelectric conversion device may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0122] 11: Sensor board 12: Pixel area 21: Circuit board 22: Circuit area 100: Photoelectric conversion element 101: Pixel 102: Photoelectric conversion unit 103: Signal processing circuit 110: Vertical scanning circuit 111: Horizontal scanning circuit 112: Readout circuit 113: Vertical signal line 114: Output circuit 115: Control pulse generation unit 201: Avalanche photodiode 202: Quench element 210: Waveform shaping section 211: Counter circuit 212: Memory circuit 213: Drive wire 214: Drive line 600: Photoelectric conversion device 601: Imaging optical system 603: Signal processing unit 604: Recognition part 605: Camera control unit 606: Storage section 607: Communications Department 700: Mobile 701: ECU 702: Vehicle control unit 703: Display section 704: Object detection section

Claims

1. A photoelectric conversion unit that emits pulses at a frequency corresponding to the frequency of receiving photons; A counter that counts the number of pulses; A plurality of pixels each including a memory for storing a count value of the counter; generating a signal based on a difference between the count values ​​of the counter at the start and end of an accumulation period; A control means for controlling a first accumulation period and a second accumulation period within one frame, the first accumulation period being shorter than the second accumulation period, so that a signal generated during the first accumulation period is output during a period from the end of the first accumulation period to the end of the second accumulation period; A recognition processing unit that performs recognition processing based on an image signal, The photoelectric conversion device according to claim 1, wherein the recognition processing unit changes a recognition frequency within one frame based on subject information.

2. 2. The photoelectric conversion device according to claim 1, wherein the recognition processing unit starts recognition at equal intervals.

3. 2. The photoelectric conversion device according to claim 1, wherein the first accumulation period and the second accumulation period overlap each other.

4. 2. The photoelectric conversion device according to claim 1, wherein the first accumulation period and the second accumulation period start simultaneously.

5. 2. The photoelectric conversion device according to claim 1, wherein the end of the second accumulation period coincides with the end of a frame.

6. 2. The photoelectric conversion device according to claim 1, wherein the recognition processing unit recognizes a subject based on a signal generated during at least the first accumulation period.

7. 7. The photoelectric conversion device according to claim 6, wherein the recognition processing section further recognizes the subject based on a signal generated during the second accumulation period.

8. 2. The photoelectric conversion device according to claim 1, wherein the recognition processing section performs the recognition processing based on a signal that is read out for each of the first accumulation periods within the second accumulation period.

9. 2. The photoelectric conversion device according to claim 1, further comprising a display unit that displays, as an image, at least a signal generated during the second accumulation period.

10. The photoelectric conversion device according to claim 1 , wherein the photoelectric conversion section includes an avalanche photodiode.

11. 2. The photoelectric conversion device according to claim 1, wherein the object information includes a moving direction of the object.

12. 2. The photoelectric conversion device according to claim 1, wherein the object information includes a moving speed of the object.

13. The photoelectric conversion device according to claim 1 , wherein the subject information includes a type of the subject.

14. The photoelectric conversion device according to any one of claims 1 to 10, A moving body comprising: a movement control unit that controls an operation of the moving body based on a recognition result of the recognition processing unit.

15. A photoelectric conversion unit that emits pulses at a frequency corresponding to the frequency of receiving photons; A counter that counts the number of pulses; A method for controlling a photoelectric conversion device having a plurality of pixels each including a memory that stores a count value of the counter, the method comprising: generating a signal based on a difference between the count values ​​of the counter at the start and end of an accumulation period; A first accumulation period and a second accumulation period are provided within one frame, the first accumulation period is shorter than the second accumulation period, and a signal generated during the first accumulation period is controlled to be output during a period from the end of the first accumulation period to the end of the second accumulation period, Recognition processing is performed based on the image signal, A control method comprising changing the recognition frequency within one frame based on subject information.

16. A computer program for controlling each part of the photoelectric conversion device according to any one of claims 1 to 13 or the moving body according to claim 14 by a computer.

Images