Imaging system, movable body, imaging method, and computer program

Abstract

To provide an imaging system that can determine the reliability of each of a plurality of image recognition results according to a driving situation.SOLUTION: An imaging system has a plurality of pixels each including a sensor part that emits pulses at a frequency according to the frequency of receiving photons, and a counter that counts the number of pulses. The imaging system generates a signal on the basis of the difference between a counter value of the counter at the start of a storage period and the counter value of the counter at the end of the storage period, performs storage operations in a first storage period and a second storage period in one full frame, wherein the first storage period is shorter than the second storage period, outputs a signal generated in the first storage period within a period from the end of the first storage period until the end of the second storage period, performs recognition processing on the basis of the generated signal, and determines the reliability of a recognition result based on the signal in the first storage period and the reliability of a recognition result based on the signal in the second storage period on the basis of driving situation information related to the driving situation of a movable body.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present invention relates to an imaging system, a moving object, an imaging method, a computer program, and the like. [Background technology]

[0002] In recent years, imaging devices have been developed that digitally count the number of photons incident on an avalanche photodiode (APD) and output the counted value from a pixel as a photoelectrically converted digital signal. For example, Patent Document 1 describes a configuration in which an imaging device having an APD can output multiple images whose accumulation periods overlap each other, thereby enabling continuous shooting even in low illuminance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7223070 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when considering the image sensor of an in-vehicle camera installed on a moving object, for example, because recognition processing is performed on a frame-by-frame basis in normal sensor driving, recognition processing can only be performed every 33.3 ms in the case of 30 fps, for example. Therefore, even if an object appears immediately after a frame change in an in-vehicle camera, recognition processing cannot be performed until the end of the frame.

[0005] In addition, in order to suppress flicker caused by traffic lights, in-vehicle cameras often have a certain accumulation period (11 ms or more), and in particular in low-light conditions, the accumulation period is extended to capture brighter images. However, because the accumulation period is long, subject blurring occurs for fast-moving objects, lowering the recognition rate.

[0006] In addition, when driving on a bumpy road that causes vertical shaking, when driving with a steering angle greater than a certain level, or when it is raining or snowing heavily and lines appear in the image, accurate recognition results cannot be obtained if recognition processing is performed using images accumulated for a long period of time.

[0007] Therefore, an object of the present invention is to provide an imaging system capable of determining the reliability of each of the recognition results of a plurality of images according to the driving situation. [Means for solving the problem]

[0008] One aspect of the present invention provides an imaging system, comprising: A sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons; A plurality of pixels each including a counter for counting the number of the pulses; generating a signal based on a difference between the count values ​​of the counter at the start and end of an accumulation period; a control unit that performs an accumulation operation of a first accumulation period and a second accumulation period within one full frame, the first accumulation period being shorter than the second accumulation period, and controls so that the signal generated during the first accumulation period is output during a period from the end of the first accumulation period to the end of the second accumulation period; The present invention is characterized by having a recognition unit that performs recognition processing based on the generated signal and determines the reliability of the recognition result based on the signal in the first accumulation period and the reliability of the recognition result based on the signal in the second accumulation period based on driving situation information regarding the driving situation of the moving body. Effect of the Invention

[0009] According to the present invention, it is possible to provide an imaging system capable of determining the reliability of each of the recognition results of a plurality of images according to the driving situation. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a photoelectric conversion element according to a first embodiment of the present invention. [Diagram 2]2 is a diagram showing a configuration example of a sensor substrate 11. FIG. [Diagram 3] 2 is a diagram showing an example of the configuration of a circuit board 21. FIG. [Figure 4] 2 and 3, and shows an equivalent circuit of a pixel 101 and a signal processing circuit 103 corresponding to the pixel 101. [Diagram 5] 2 is a diagram illustrating a relationship between the operation of an APD 201 and an output signal. FIG. [Figure 6] FIG. 6 is a functional block diagram of an imaging device 600 and a moving object 700 according to the first embodiment. [Figure 7] FIG. 11 is a diagram for explaining an imaging method by a camera control unit 605 according to the first embodiment. [Figure 8] FIG. 13 is a diagram showing an example of an image divided into a plurality of frames. [Figure 9] FIG. 2 is a diagram illustrating a relationship between a memory circuit and a buffer in the embodiment. [Figure 10] 5 is a flowchart showing details of an example of driving a photoelectric conversion element in the embodiment. [Figure 11] This is a continuation of the flowchart in Figure 10. [Figure 12] 10 is a flowchart showing an example of an imaging method for detecting an operating state and changing a reliability in the first embodiment. [Figure 13] FIG. 7 is a functional block diagram of an imaging device 800 and a moving object 700 according to a second embodiment. [Figure 14] 13 is a flowchart showing an example of an imaging method for detecting the amount of incident light and an operating state to change the reliability in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified.

[0012] <Embodiment 1> 1 is a diagram showing a configuration example of a photoelectric conversion element according to an embodiment of the present invention. In the following, the photoelectric conversion element 100 will be described taking as an example an imaging device having a so-called stacked structure, in which two substrates, a sensor substrate 11 and a circuit substrate 21, are stacked and electrically connected.

[0013] However, a so-called non-laminated structure in which the components included in the sensor substrate and the components included in the circuit substrate are disposed on a common semiconductor layer may be used. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes signals detected in the pixel region 12.

[0014] 2 is a diagram showing a configuration example of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally across a plurality of rows and columns. Each pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, APD).

[0015] Here, the photoelectric conversion unit 102 functions as a sensor unit that emits a pulse at a frequency corresponding to the frequency of receiving photons. The number of rows and columns of the pixel array that constitutes the pixel region 12 is not particularly limited.

[0016] Fig. 3 is a diagram showing an example of the configuration of the circuit board 21. The circuit board 21 has a signal processing circuit 103 that processes charges photoelectrically converted by each photoelectric conversion unit 102 in Fig. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114.

[0017] The vertical scanning circuit 110 receives a control pulse supplied from a control pulse generating unit 115, and sequentially supplies the control pulse to a plurality of pixels arranged in the row direction. The vertical scanning circuit 110 uses logic circuits such as a shift register and an address decoder.

[0018] The signal output from the photoelectric conversion unit 102 of each pixel is processed by each signal processing circuit 103. The signal processing circuit 103 is provided with a counter, a memory, etc., and a digital value is stored in the memory. The horizontal scanning circuit 111 inputs a control pulse that sequentially selects each column to the signal processing circuit 103 in order to read the signal from the memory of each pixel in which the digital signal is stored.

[0019] A signal is output to the vertical signal line 113 from the signal processing circuit 103 of the pixel of the row selected by the vertical scanning circuit 110. The signal output to the vertical signal line 113 is output to the outside of the photoelectric conversion element 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has a plurality of buffers built therein and connected to the vertical signal line 113.

[0020] 2 and 3, a plurality of signal processing circuits 103 are arranged in a region overlapping the pixel region 12 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged so as to overlap between an end of the sensor substrate 11 and an end of the pixel region 12 in a plan view.

[0021] In other words, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in a region overlapping the non-pixel region in a plan view.

[0022] The arrangement of the vertical signal lines 113, the readout circuits 112, and the output circuits 114 are not limited to the example shown in Fig. 3. For example, the vertical signal lines 113 may be arranged extending in the row direction, and the readout circuits 112 may be arranged at the ends of the vertical signal lines 113. Also, it is not necessary to provide one signal processing circuit 103 for each photoelectric conversion unit, and one signal processing circuit may be shared by a plurality of photoelectric conversion units to perform signal processing sequentially.

[0023] Fig. 4 is a diagram showing an equivalent circuit of the pixel 101 in Fig. 2 and Fig. 3 and the signal processing circuit 103 corresponding to the pixel 101. As shown in Fig. 4, each pixel 101 includes a photoelectric conversion unit 102 as a sensor unit, a counter circuit 211, a memory circuit 212, etc.

[0024] The APD 201 included in the photoelectric conversion unit 102 generates a pair of electric charges according to incident light by photoelectric conversion. One of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VL (first voltage) is supplied. The other of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VH (second voltage) higher than the driving voltage VL is supplied.

[0025] In Fig. 4, one node of the APD 201 is the anode, and the other node of the APD is the cathode. A reverse bias voltage is supplied to the anode and cathode of the APD 201 so that the APD 201 performs avalanche multiplication. By supplying such a voltage, the charge generated by the incident light undergoes avalanche multiplication, generating an avalanche current.

[0026] When a reverse bias voltage is supplied, there are two modes: Geiger mode, in which the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, in which the voltage difference between the anode and cathode is close to or less than the breakdown voltage. An APD that operates in Geiger mode is called a SPAD. In the case of a SPAD, for example, the drive voltage VL (first voltage) is -30V, and the drive voltage VH (second voltage) is 1V.

[0027] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212. The quench element 202 is connected to a power supply line to which a driving voltage VH is supplied and one of the anode and cathode nodes of the APD 201.

[0028] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, and has a function of suppressing avalanche multiplication by suppressing the voltage supplied to the APD 201 (quench operation).The quench element 202 also has a function of returning the voltage supplied to the APD 201 to the drive voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0029] FIG. 4 shows an example in which the signal processing circuit 103 includes a waveform shaping section 210, a counter circuit 211, and a memory circuit 212 in addition to the quench element 202.

[0030] The waveform shaping unit 210 shapes the voltage change of the cathode of the APD 201 obtained when a photon is detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. Although an example in which one inverter is used as the waveform shaping unit 210 is shown in Fig. 4, a circuit in which a plurality of inverters are connected in series may be used, or another circuit having a waveform shaping effect may be used.

[0031] The counter circuit 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. When a control pulse RES is supplied via a drive line 213, the signal held in the counter circuit 211 is reset. Here, the counter circuit 211 generates a signal based on the difference between the count values ​​at the start and end of the accumulation period.

[0032] 3 to the memory circuit 212 via a drive line 214 (not shown in FIG. 3) in FIG. 4, and switches between electrical connection and disconnection between the counter circuit 211 and the vertical signal line 113. The memory circuit 212 functions as a memory that temporarily stores the count value of the counter, and outputs an output signal from the counter circuit 211 of the pixel to the vertical signal line 113.

[0033] Note that electrical connections may be switched by disposing a switch such as a transistor between the quench element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103. Similarly, the supply of the driving voltage VH or the driving voltage VL to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0034] 5 is a diagram showing a schematic diagram of the relationship between the operation of the APD 201 and the output signal. The input side of the waveform shaping unit 210 is nodeA, and the output side is nodeB. Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage at nodeA drops.

[0035] When the amount of voltage drop further increases and the potential difference applied to the APD 201 decreases, the avalanche multiplication of the APD 201 stops as at time t2, and the voltage level of nodeA does not drop below a certain value.

[0036] After that, between time t2 and time t3, a current flows through nodeA to compensate for the voltage drop from the drive voltage VL, and at time t3, nodeA is stabilized to its original potential level. At this time, the portion of the output waveform at nodeA that exceeds a certain threshold is shaped by the waveform shaping section 210 and is output as a pulse signal at nodeB.

[0037] 6 is a functional block diagram of an image capturing apparatus 600 and a moving object 700 according to embodiment 1. In this embodiment, some of the functional blocks of the image capturing apparatus 600 and the moving object 700 constitute an image capturing system.

[0038] Some of the functional blocks shown in FIG. 6 are realized by causing a computer (not shown) included in the imaging device 600 and the moving object 700 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0039] However, some or all of them may be realized by hardware. A dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP) may be used as the hardware. Furthermore, the functional blocks shown in Fig. 6 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths. The above explanation regarding Fig. 6 also applies to Fig. 13 described later.

[0040] The imaging device 600 includes a photoelectric conversion element 100, an imaging optical system 601, an image processing unit 603, a recognition unit 604, a camera control unit 605, a storage unit 606, and a communication unit 607. The photoelectric conversion element 100 includes the avalanche photodiode described with reference to FIGS. 1 to 5 for photoelectrically converting an optical image.

[0041] The imaging device of the embodiment is mounted on a moving body 700, and a camera unit consisting of a set of an imaging optical system 601 and a photoelectric conversion element 100 is configured to capture images in at least one direction, for example, in front, behind, or to the side of the moving body. Note that a plurality of camera units may be provided on the moving body 700.

[0042] The image processing unit 603 performs image processing on the signals output from the pixels. Moreover, the image processing unit 603 performs predetermined image processing on the image signals based on the area information output from the camera control unit 605 to generate a final image signal. The predetermined image processing includes, for example, contour enhancement, black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaic processing, data compression, and other predetermined image processing.

[0043] When the photoelectric conversion element 100 has an on-chip color filter such as RGB, it is desirable to perform processing such as white balance correction and color conversion in the image processing unit 603. The output of the image processing unit 603 is supplied to a recognition unit 604, an ECU (Electric Control Unit) 701 of the moving object 700, and a camera control unit 605.

[0044] The recognition unit 604 performs image recognition based on the result of the image processing. That is, the recognition unit 604 recognizes people, vehicles, etc. in the vicinity by performing image recognition based on the image signal that has been image-processed by the image processing unit 603, and issues a warning, etc., as necessary.

[0045] In this embodiment, the moving body 700 is described using an example of an automobile, but the moving body may be any moving body, such as an airplane, a train, a ship, a drone, an AGV, or a robot.

[0046] The camera control unit 605 has a built-in CPU as a computer and a memory that stores a computer program, and controls each unit of the imaging device 600 by the CPU executing the computer program stored in the memory.

[0047] The camera control unit 605 functions as a control means, and controls the length of the exposure period of each frame of the photoelectric conversion element 100 and the timing of the control signal CLK, for example, via a control pulse generation unit of the photoelectric conversion element 100. In addition, the recognition process of the recognition unit 604 may also be controlled based on the output from the ECU 701.

[0048] The storage unit 606 includes a recording medium such as a memory card, a hard disk, etc., and can store and read image signals. The communication unit 607 includes a wireless or wired interface, and outputs the generated image signal to the outside of the imaging device 600 and receives various signals from the outside.

[0049] The ECU 701 incorporates a CPU as a computer and a memory that stores a computer program, and controls each part of the moving body 700 by the CPU executing the computer program stored in the memory.

[0050] The output of the ECU 701 is supplied to a vehicle control unit 702 and a display unit 703. The vehicle control unit 702 functions as a movement control means that performs drive, stop, direction control, etc. of the vehicle as a moving body based on the output of the ECU 701.

[0051] The vehicle control unit 702 as a movement control means controls the movement of the moving body based on at least one of the recognition processing based on the signal generated in the first accumulation period and the recognition processing based on the signal generated in the second accumulation period. The first accumulation period and the second accumulation period will be described later.

[0052] The display unit 703 includes a display element such as a liquid crystal device or an organic EL, and is mounted on the moving body 700. The display unit 703 displays images acquired by the photoelectric conversion element 100 and various information related to the running state of the vehicle to the driver of the moving body 700 using, for example, a GUI based on the output of the ECU 701.

[0053] The operating status detection unit 704 detects the operating status of the mobile object 700 and outputs a signal to the ECU 701. The operating status includes at least one of the following, for example: the running speed of the mobile object 700, the steering angle of the mobile object 700, the weather (rain, snow) when the mobile object 700 is operating, the amount of shaking of the mobile object 700 itself, and the like.

[0054] These operating conditions are output from the operating condition detection unit 704 to the ECU 701. Taking the traveling speed as an example, when the ECU 701 acquires the traveling speed of the moving body 700 and determines that the traveling speed is equal to or greater than a certain threshold v (e.g., 80 km / h), the ECU 701 increases the reliability x of the recognition result for the output image of the first accumulation period, or decreases the reliability y of the recognition result for the output image of the second accumulation period, or performs both.

[0055] This is because when the traveling speed of the moving object exceeds the threshold v, the reliability of the recognition result decreases in the main image accumulated for a long time due to the large subject blur, whereas the recognition reliability of the recognition result of the image accumulated for a short time is high. Note that when the moving object is traveling at a speed below the threshold v, the main image accumulated for a long time has relatively little subject blur, so a more accurate recognition result can be obtained than an image accumulated for a short time that has low contrast and insufficient brightness.

[0056] Incidentally, the output of the operating condition detection unit 704 may be configured to be supplied to the camera control unit 605 via, for example, the ECU 710, and the camera control unit 605 may determine whether the traveling speed is equal to or greater than the threshold value v.

[0057] 6 may not be mounted on the moving body 700, and may be provided, for example, in an external terminal or the like provided separately from the moving body 700 for remotely controlling the moving body 700 or for monitoring the traveling of the moving body. The imaging system of this embodiment includes one composed of a plurality of devices connected by such communication paths.

[0058] 7 is a diagram for explaining an imaging method by the camera control unit 605 according to the first embodiment. In this embodiment, photoelectric conversion is performed periodically at, for example, 30 full frames / second. Note that in this embodiment, one frame having a length of, for example, 33.3 ms is called a full frame (or main frame), and a full frame divided is called a frame (or sub-frame).

[0059] That is, as shown in FIG. 7, a full frame 1 having a length of 33.3 ms from time T0 to time T12 is divided into two, and as shown in FIG. 7, the full frame 1 is divided into a frame 1_1 and a frame 1_2.

[0060] In this embodiment, frame 1_1 is set to the position at 8.33 ms, but may be set to another position. For convenience, frames 0_1, 0_2, 1_1, 1_2, 2_1, and 2_2 are represented as F0_1, 0_2, 1_1, 1_2, 2_1, and 2_2 in Fig. 7 and subsequent figures.

[0061] Frame 1_1 has an accumulation period (e.g., 8.33 ms) from start time T0 to time T1 of full frame 1, and frame 1_2 has an accumulation period (e.g., 33.3 ms) from time T0 to time T2. In this embodiment, for example, a short accumulation period from time T0 to time T1 is called a first accumulation period, and a long accumulation period from time T0 to time T4 is called a second accumulation period.

[0062] At time T0, the counter circuit 211 is reset, and count values ​​C1_1 and C1_2 are obtained from the counter circuit 211 at times T1 and T2, respectively.

[0063] Moreover, the count values ​​C1_1 and C1_2 are temporarily stored in the memory circuit 212. Then, the signals for one row temporarily stored in the memory circuit 212 are sequentially output from the photoelectric conversion elements via the buffer of the readout circuit 112.

[0064] In this way, according to this embodiment, the signal accumulated during the period of frame 1_1 is read out from time T1 to T2 and is promptly processed by the recognition unit 604. Therefore, image recognition can be performed promptly. Similarly, the signal accumulated during the period of frame 1_2 is read out from time T2 to T1 and image recognition can be performed repeatedly.

[0065] Fig. 8 is a diagram showing an example of images of a plurality of divided frames. As shown in Fig. 8, the image of frame 1_1 is dark because the accumulation time is short, but there is little subject blur of a person who jumps out in front of the camera. On the other hand, frame 1_2 has a longer accumulation time, so subject blur is more likely to occur. Note that stopped vehicles and white lines are less likely to be blurred, and the longer the accumulation time, the more likely the contrast is to improve.

[0066] In this manner, in the control step in the camera control unit 605 of this embodiment, accumulation operations are performed for the first accumulation period and the second accumulation period within one full frame. Also, the first accumulation period is shorter than the second accumulation period, and the signal generated during the first accumulation period is controlled to be output between the end of the first accumulation period and the end of the second accumulation period.

[0067] In this embodiment, the first and second accumulation periods overlap, and start at the same time. Furthermore, the end of the second accumulation period is a frame boundary, and the second accumulation period is an integer multiple of the first accumulation period. That is, the end of the second accumulation period coincides with the end of the full frame.

[0068] However, the second accumulation period does not need to be an integer multiple of the first accumulation period. The second accumulation period only needs to be longer than the first accumulation period (the first accumulation period is shorter than the second accumulation period) and the second accumulation period only needs to end after the first accumulation period.

[0069] That is, an image with a short accumulation period and an image with a long accumulation period are created, the timing at which the short accumulation period ends is set to be earlier than the timing at which the long accumulation period ends, and as soon as the short accumulation period ends, the image is output and sent to a downstream recognition section.

[0070] The subject is recognized based on the signals generated during the first accumulation period and the second accumulation period. A recognition unit 604 as a recognition means recognizes the subject based on the signals generated during the first accumulation period and the second accumulation period.

[0071] Therefore, while in the past it was not possible to recognize an image until one full frame had elapsed, in this embodiment, image recognition can be performed in as little as 8.33 ms. That is, for example, when a moving object is moving at high speed, obstacles and the like can be recognized quickly. Therefore, it is possible to apply the brakes quickly, or to avoid obstacles at an early stage.

[0072] In addition, since an image with a long accumulation time can improve the contrast, it can be used not only as an image for recognition but also as an image for display. That is, an image with a short accumulation time is suitable for quick subject recognition such as a sudden jump-out, and an image with a long accumulation time is also suitable for an image for display. In this way, the display unit 703 of this embodiment displays the signal generated during at least the second accumulation period as an image.

[0073] In addition, since the present embodiment uses an APD, unlike a CMOS sensor, the accumulated charge does not deteriorate due to readout, so the accumulation periods can be overlapped. Also, since there is no readout noise, the original signal does not deteriorate no matter how many times it is read out from a single accumulation.

[0074] Fig. 9 is a diagram showing the relationship between memory circuits and buffers in an embodiment. Fig. 9 shows a state in which memory circuits 212 in signal processing circuit 103 in Fig. 3 are arranged in N rows and M columns, and each memory circuit is represented as memory 1-1 to memory NM. Also, buffer 1 to buffer M in Fig. 9 indicate buffers included in read circuit 112 in Fig. 3. Output circuit 114 in Fig. 9 corresponds to output circuit 114 in Fig. 3.

[0075] Fig. 10 is a flowchart showing details of an example of driving a photoelectric conversion element in an embodiment, and Fig. 11 is a flowchart continuing from Fig. 10. Note that the operations of the steps in the flowcharts of Fig. 10 and Fig. 11 are performed sequentially by a CPU or the like as a computer in the camera control unit 605 executing a computer program stored in a memory.

[0076] 10, set i=1. Next, in step S102, the count value Count of the counter circuit 211 at time Ti is output to the memory circuit 212. At this time, the output is performed simultaneously to all the memory circuits. This operation corresponds to the operation at time T1 in FIG.

[0077] Next, in step S103, j=1 is set, and in step S104, the count value Count(jki) in the memory circuit jk in Fig. 9 is output to the buffer. At this time, columns 1 to M are output to the buffer at the same time. This operation means that the count value of the first row in Fig. 9 is taken into the buffer.

[0078] Next, in step S105, k is set to 1, and in step S106, the count value Count(jki) of buffer k is output to the output circuit 114. This operation corresponds to the operation of reading out the signals of the buffers in the leftmost column in FIG.

[0079] Next, proceed to step S107 in FIG. 11 via A, and in step S107, determine whether k < M. If Yes, then in step S108, increment k by 1 as k = k + 1, return to step S106 via B, and perform the operation of step S106. This operation corresponds to the operation of reading the signal of the buffer in the second column from the left in FIG. 9 from the output circuit.

[0080] If it becomes No in step S107, that is, if k = M, it means that the signal of the buffer in the M-th column of FIG. 9 has been read from the output circuit. Next, proceed to step S109 and determine whether j < N. If Yes in step S109, then in step S110, increment j by 1 as j = j + 1, and return to step S104 via C. This corresponds to the operation for starting to read the next row.

[0081] If it is determined to be No in step S109, it means that the reading of all rows has been completed. So, proceed to step S111 and determine whether j < 2. If it is determined to be Yes in step S111, then proceed to step S112, increment i by 1 as i = i + 1, and return to step S102 via D. This operation corresponds to the operation for starting to read at the next time T2.

[0082] If it is determined to be No in step S111, it means that the reading at time T2 has been completed. So, proceed to step S113 and reset the counter circuit 211 with a reset signal. This operation corresponds to the reset operation of the counter circuit 211 at time T2 in FIG. 7. In the above manner, the signals accumulated in the photoelectric conversion element 100 can be sequentially read out.

[0083] FIG. 12 is a flowchart showing an example of an imaging method for detecting the operating situation in Embodiment 1 and changing the reliability, and shows an example of changing the reliability for the recognition results of the images with short-time accumulation and long-time accumulation respectively.

[0084] The CPU as a computer in the ECU 701 executes a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowchart in Fig. 12. Note that some of the steps in Fig. 12 may be performed by the CPU in the camera control unit 605.

[0085] 12, the power source of the moving body 700 is in a started state. In step S200, the CPU of the ECU 701 sets the reliability of the recognition result of the short-term accumulated image output from the recognition unit 604 and the reliability of the recognition result of the long-term accumulated image to initial values.

[0086] In step S201, the CPU of the ECU 701 detects the traveling speed of the moving object 700 by the operating status detection unit 704, and proceeds to the process of step S202.

[0087] In step S202, the CPU of the ECU 701 acquires the traveling speed of the moving object 700 acquired by the operating status detection unit 704 in step S201, and proceeds to step S203.

[0088] In step S203, the CPU of the ECU 701 determines whether the traveling speed of the moving object 700 acquired from the operating status detection unit 704 in step S202 is equal to or higher than a threshold value v (e.g., 80 km / h). If the traveling speed of the moving object 700 is equal to or higher than the threshold value v, the process proceeds to step S204. If the traveling speed of the moving object 700 is lower than the threshold value v, the process proceeds to step S205.

[0089] In step S204, the CPU of the ECU 701 changes the reliability of the recognition result of the short-time accumulation image acquired from the recognition unit 604 and the reliability of the recognition result of the long-time accumulation image, respectively.

[0090] That is, in step S204, the reliability x of the recognition result for the output image in the first accumulation period is increased from its initial value. Alternatively, the reliability y of the recognition result for the output image in the second accumulation period is decreased from its initial value. Alternatively, both may be changed as described above. After the reliability of each recognition result is changed in step S204, proceed to step S206.

[0091] In step S205, similar to step S200, the reliability of the recognition result of the short-time accumulation image and the reliability of the recognition result of the long-time accumulation image are set to initial values. After the reliability of each recognition result is set in step S205, the process proceeds to step S206.

[0092] In step S206, the CPU of the ECU 701 integrates the recognition result for the output image of the first accumulation period (short-time accumulation) weighted by reliability x and the recognition result for the output image of the second accumulation period (short-time accumulation) weighted by reliability y. This calculates the final recognition result. Note that integration includes weighted addition and processing to select the one with higher reliability.

[0093] Then, based on the final recognition result calculated in step S206, vehicle control unit 702 controls the operation of the moving object. Also, based on the final recognition result, a subject frame, the reliability of the integrated recognition result, a warning, etc. may be displayed on display unit 703. After the process of step S206, the process returns to step S201, and the flow of Fig. 12 is periodically repeated while the power of the moving object is ON.

[0094] Here, steps S203 to S205 function as a recognition step for performing recognition processing based on the generated signal. In the recognition step, the reliability of the recognition result based on the signal in the first accumulation period and the reliability of the recognition result based on the signal in the second accumulation period are determined based on driving situation information related to the driving situation of the moving object.

[0095] <Embodiment 2> 13 is a functional block diagram of an imaging device 800 and a moving object 700 according to embodiment 2. In embodiment 2, an imaging system is configured by some functional blocks of the imaging device 800 and the moving object 700. The imaging device 800 includes a photoelectric conversion element 100, an imaging optical system 601, an image processing unit 603, a recognition unit 604, a camera control unit 605, a storage unit 606, a communication unit 607, an incident light amount detection unit 801, and the like.

[0096] A moving object 700 includes an ECU 701, a vehicle control unit 702, a display unit 703, and an operating status detection unit 704. In Fig. 13, the same reference numerals as in Fig. 6 denote the same functional blocks, and therefore a description thereof will be omitted.

[0097] The incident light amount detection unit 801 detects the amount of light incident on the imaging device 800, and outputs light amount information to the ECU 701. Having acquired the light amount information, the ECU 701 determines whether the light amount is equal to or greater than a certain threshold z. If the light amount exceeds the threshold z, the ECU 701 changes the reliability of the recognition results of the short-time accumulated image and the long-time accumulated image based on operating status information from the operating status detection unit 704, as in the first embodiment.

[0098] If it is determined that the amount of light is not greater than or equal to the threshold value z, the brightness of the short-time accumulation image is insufficient, so the reliability of the recognition results of the short-time accumulation image and the long-time accumulation image is not changed regardless of the operating conditions.

[0099] FIG. 14 is a flowchart showing an example of an imaging method in which the reliability is changed by detecting the amount of incident light and the operating conditions in the second embodiment, and shows an example of changing the reliability of the recognition results for images accumulated for a short time and images accumulated for a long time.

[0100] The CPU serving as a computer within the ECU 701 executes a computer program stored in the memory, whereby the operations of the steps in the flowchart of FIG. 14 are performed in sequence.

[0101] 14 is performed, the power source of the moving body 700 is in a started state. In step S300, the CPU of the ECU 701 sets the reliability of the recognition result of the short-term accumulated image output from the recognition unit 604 and the reliability of the recognition result of the long-term accumulated image to initial values.

[0102] In step S301, the CPU of the ECU 701 detects the amount of light incident on the imaging device 800 using the incident light amount detection unit 801, and the process proceeds to step S302.

[0103] In step S302, the CPU of the ECU 701 acquires the light amount information on the imaging device 800 acquired by the incident light amount detection unit 801 in step S301, and the process proceeds to step S303.

[0104] In step S303, the CPU of the ECU 701 determines whether the incident light amount information on the image pickup device 800 acquired from the incident light amount detection unit 801 in step S302 is equal to or greater than the threshold z. If the incident light amount on the image pickup device 800 is equal to or greater than the threshold z, the process proceeds to step S304. If the incident light amount on the image pickup device 800 is less than the threshold z, the process proceeds to step S308.

[0105] In step S304, the CPU of the ECU 701 detects the traveling speed of the moving object 700 by the operating status detection unit 704, and proceeds to step S305.

[0106] In step S305, the CPU of the ECU 701 acquires the traveling speed of the moving object 700 acquired by the operating status detection unit 704 in step S304, and proceeds to step S306.

[0107] In step S306, the CPU of the ECU 701 determines whether the traveling speed of the moving object 700 acquired from the operating status detection unit 704 in step S305 is equal to or higher than a threshold value v (e.g., 80 km / h). If the traveling speed of the moving object 700 is equal to or higher than the threshold value v, the process proceeds to step S307. If the traveling speed of the moving object 700 is lower than the threshold value v, the process proceeds to step S308.

[0108] In step S307, the CPU of the ECU 701 changes the reliability of the recognition result of the short-time accumulation image acquired from the recognition unit 604 and the reliability of the recognition result of the long-time accumulation image, respectively.

[0109] That is, in step S307, the reliability x of the recognition result for the output image in the first accumulation period is increased from the initial value. Alternatively, the reliability y of the recognition result for the output image in the second accumulation period is decreased from the initial value. Alternatively, both may be changed as described above. After the processing of step S307, the process proceeds to step S309.

[0110] In step S308, similarly to step S300, the CPU of the ECU 701 sets the reliability of the recognition result of the short-time accumulation image acquired from the recognition unit 604 and the reliability of the recognition result of the long-time accumulation image to initial values. After the processing of step S308, the process proceeds to step S309.

[0111] In step S309, the CPU of the ECU 701 integrates the recognition result for the output image of the first accumulation period (short-time accumulation) weighted by reliability x and the recognition result for the output image of the second accumulation period (long-time accumulation) weighted by reliability y. This calculates the final recognition result. Note that integration includes weighted addition and processing to select the one with the higher reliability.

[0112] Then, based on the final recognition result calculated in step S307, vehicle control unit 702 controls the operation of the moving object. Also, based on the final recognition result, a subject frame, the reliability of the integrated recognition result, a warning, etc. may be displayed on display unit 703. After the process of step S309, the process returns to step S301, and the flow of Fig. 14 is periodically repeated while the power of the moving object is ON.

[0113] Here, steps S306 to S308 function as a recognition step for performing recognition processing based on the generated signal. In the recognition step, the reliability of the recognition result based on the signal in the first accumulation period and the reliability of the recognition result based on the signal in the second accumulation period are determined based on driving situation information related to the driving situation of the moving object.

[0114] For example, in the above embodiment, the operating condition detection unit 704 detects the traveling speed of the moving body 700, and changes the reliability of the recognition result when the traveling speed is determined to be equal to or greater than a predetermined threshold. However, the operating condition detection unit 704 may detect the steering angle of the moving body 700, and change the reliability of the recognition result when the steering angle is equal to or greater than a predetermined threshold.

[0115] Alternatively, the operating condition detection unit 704 may detect vertical or horizontal shaking (vibration) of the moving body 700, and change the reliability of the recognition result when the shaking reaches or exceeds a certain threshold. Alternatively, the operating condition detection unit 704 may detect the amount of rain or snowfall, and change the reliability of the recognition result when the amount of rain or snowfall reaches or exceeds a predetermined threshold.

[0116] Alternatively, the operating condition detection unit 704 may detect the acceleration of the moving object, and if the acceleration exceeds a predetermined threshold, it may be possible that the object is accelerating suddenly, and therefore the reliability of the recognition result may be changed.

[0117] If the steering angle, vibration, rainfall, snowfall, or acceleration exceeds the respective thresholds, the reliability x of the recognition result for the output image in the first accumulation period is increased from the initial value, or the reliability y of the recognition result for the output image in the second accumulation period is decreased from the initial value, or both may be changed as described above.

[0118] This is because when the steering angle, vibration, amount of rainfall, snowfall, or acceleration is large, the rate of change and degree of degradation of the image (image quality degradation due to rain or snow) is large, so it is estimated that images accumulated for a short period of time will have a relatively higher reliability in the recognition process.

[0119] In this manner, in this embodiment, at least one of the reliability x of the recognition result for the output image in the first accumulation period and the reliability y of the recognition result for the output image in the second accumulation period is changed according to the driving condition information of the moving body. Note that the driving condition information in this embodiment includes at least one of the traveling speed of the moving body, the acceleration of the moving body, the steering angle of the moving body, the vibration of the moving body, the amount of rainfall or snowfall around the moving body, etc.

[0120] In the above embodiment, only one threshold value is set for the driving condition information, but multiple threshold values ​​may be set to gradually change the reliability in stages, or the reliability may be continuously changed according to the change in the driving condition.

[0121] In addition, in the second embodiment, a flow is shown in which the reliability during normal driving is used when the amount of light is insufficient, but when the amount of light is extremely low, processing may be performed to lower the reliability of the short-time accumulated image below the reliability during normal driving or below the initial value.

[0122] In addition, although accumulation is performed for a ¼ full frame period at the shortest, the length of the shortest accumulation period may be changed to, for example, a ⅕ full frame period or a ⅓ full frame period depending on the recognition accuracy in the recognition unit 604. Alternatively, the length of the shortest accumulation period may be changed depending on the brightness of the subject.

[0123] Furthermore, even if the readout timing is set to 1 / 4 full frame period, the counter circuit may be reset midway through the accumulation period of frame 1_1 in Fig. 7 depending on the brightness of the subject, the image recognition accuracy, etc. As a result, the substantial accumulation period may be made shorter than 1 / 4 full frame period.

[0124] Alternatively, the counter circuit may be reset once at time T1 in Fig. 7. This may adjust the count value read out at time T2.

[0125] Although the present invention has been described in detail based on the preferred embodiment, the present invention is not limited to the above embodiment, and various modifications are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. The present invention includes the following combinations.

[0126] (Configuration 1) An imaging system comprising: a plurality of pixels, each pixel having a sensor unit that emits a pulse at a frequency corresponding to a frequency of receiving photons, and a counter that counts the number of the pulses; a control unit that generates a signal based on the difference between the count values ​​of the counter at the start and end of an accumulation period, and performs accumulation operations for a first accumulation period and a second accumulation period within one full frame, the first accumulation period being shorter than the second accumulation period, and controls so that the signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period; and a recognition unit that performs recognition processing based on the generated signal, and determines the reliability of a recognition result based on the signal during the first accumulation period and the reliability of a recognition result based on the signal during the second accumulation period based on driving situation information regarding the driving situation of a moving body.

[0127] (Configuration 2) The imaging system according to configuration 1, wherein the first accumulation period and the second accumulation period overlap each other.

[0128] (Configuration 3) The imaging system according to configuration 2, wherein the first accumulation period and the second accumulation period start simultaneously.

[0129] (Configuration 4) The imaging system according to any one of configurations 1 to 3, wherein the end of the second accumulation period coincides with the end of a frame.

[0130] (Configuration 5) The imaging system according to any one of configurations 1 to 4, further comprising a display section that displays, as an image, the signal generated during at least the second accumulation period.

[0131] (Configuration 6) The imaging system according to any one of configurations 1 to 5, wherein the sensor section includes an avalanche photodiode.

[0132] (Configuration 7) An imaging system described in any one of configurations 1 to 6, characterized in that the driving condition information includes at least one of the traveling speed of the moving body, the acceleration of the moving body, the steering angle of the moving body, the vibration of the moving body, and the amount of rainfall or snowfall around the moving body.

[0133] (Configuration 8) A moving body comprising the imaging system according to any one of configurations 1 to 7, and a movement control means for controlling the operation of the moving body based on the result of the recognition processing using the signal generated during at least the first accumulation period.

[0134] (Method) An imaging method in an imaging system having multiple pixels, each pixel including a sensor unit that emits a pulse at a frequency corresponding to the frequency of receiving photons, and a counter that counts the number of the pulses, the imaging method comprising: a control step of generating a signal based on the difference between the count values ​​of the counter at the start and end of an accumulation period, performing accumulation operations for a first accumulation period and a second accumulation period within one full frame, the first accumulation period being shorter than the second accumulation period, and controlling so that the signal generated in the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period, and a recognition step of performing recognition processing based on the generated signal, and determining the reliability of the recognition result based on the signal in the first accumulation period and the reliability of the recognition result based on the signal in the second accumulation period based on driving situation information regarding the driving situation of a moving body.

[0135] (Program) A computer program for controlling each part of the imaging system according to any one of configurations 1 to 7 or the moving body according to configuration 8 by a computer.

[0136] A computer program for implementing the functions of the above-described embodiment, part or all of the control in this embodiment, may be supplied to the imaging system via a network or various storage media. Then, a computer (or a CPU, MPU, etc.) in the imaging system may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0137] 11: Sensor board 12: Pixel area 21: Circuit board 22: Circuit area 100: Photoelectric conversion element 101: Pixel 102: Photoelectric conversion unit 103: Signal processing circuit 110: Vertical scanning circuit 111: Horizontal scanning circuit 112: Readout circuit 113: Vertical signal line 114: Output circuit 115: Control pulse generation unit 201: Avalanche photodiode 202: Quench element 210: Waveform shaping section 211: Counter circuit 212: Memory circuit 213: Drive line 214: Drive line 600: Imaging device 601: Imaging optical system 603: Signal processing unit 604: Recognition part 605: Camera control unit 606: Storage section 607: Communications Department 700: Mobile 701: ECU 702: Vehicle control unit 703: Display section 704: Operation status detection unit 800: Imaging device 801: Incident light amount detection unit

Claims

1. A sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons; A plurality of pixels each including a counter for counting the number of the pulses; generating a signal based on a difference between the count values ​​of the counter at the start and end of an accumulation period; a control unit that performs an accumulation operation of a first accumulation period and a second accumulation period within one full frame, the first accumulation period being shorter than the second accumulation period, and that controls so that the signal generated during the first accumulation period is output during a period from the end of the first accumulation period to the end of the second accumulation period; an image capturing system comprising: a recognition unit that performs recognition processing based on the generated signal, and determines the reliability of a recognition result based on the signal in the first accumulation period and the reliability of a recognition result based on the signal in the second accumulation period based on driving situation information relating to the driving situation of a moving body.

2. 2. The imaging system according to claim 1, wherein the first accumulation period and the second accumulation period overlap each other.

3. 3. The imaging system according to claim 2, wherein the first accumulation period and the second accumulation period start simultaneously.

4. 2. The imaging system according to claim 1, wherein the end of the second accumulation period coincides with the end of a frame.

5. 2. The imaging system according to claim 1, further comprising a display unit that displays, as an image, the signal generated during at least the second accumulation period.

6. 2. The imaging system according to claim 1, wherein the sensor unit includes an avalanche photodiode.

7. The imaging system according to claim 1 , wherein the driving condition information includes at least one of a traveling speed of the moving body, an acceleration of the moving body, a steering angle of the moving body, a vibration of the moving body, and an amount of rainfall or an amount of snowfall around the moving body.

8. An imaging system according to any one of claims 1 to 7; A moving body comprising a movement control means for controlling the movement of the moving body based on a result of the recognition process using the signal generated during at least the first accumulation period.

9. An imaging method for an imaging system having a plurality of pixels, each of which includes a sensor unit that emits a pulse at a frequency corresponding to a frequency of receiving photons, and a counter that counts the number of the pulses, comprising: a control step of generating a signal based on a difference between count values ​​of the counter at the start and end of an accumulation period, performing accumulation operations for a first accumulation period and a second accumulation period within one full frame, the first accumulation period being shorter than the second accumulation period, and controlling so that the signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period; a recognition step of performing a recognition process based on the generated signal and determining a reliability of a recognition result based on the signal in the first accumulation period and a reliability of a recognition result based on the signal in the second accumulation period based on driving situation information regarding a driving situation of a moving body.

10. A computer program for controlling each part of the imaging system according to any one of claims 1 to 7 or the moving body according to claim 8 by a computer.

Images