Photoelectric conversion device, mobile body, photoelectric conversion method, and computer program

Abstract

To acquire both an image for recognition and an image for display from a single sensor.SOLUTION: A photoelectric conversion device has a plurality of pixels each including a sensor unit for emitting a pulse at a frequency corresponding to a frequency of receiving photons, a counter for counting the number of pulses, and a memory for storing a count value of the counter, and control means for generating an image signal based on the difference between the count values of the counter at the start time and end time of an accumulation period. The control means performs a pre-reading operation for reading out a pre-read image signal during a full frame period and performs readout processing of the image signal such that the time for the pre-reading operation is shorter than the time for reading out a main image signal during the full frame period.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a photoelectric conversion device, a moving object, a photoelectric conversion method, a computer program, and the like. [Background technology]

[0002] In recent years, photoelectric conversion devices have been developed that digitally count the number of photons incident on an avalanche photodiode (APD) and output the counted value from a pixel as a photoelectrically converted digital signal. For example, Patent Document 1 describes a configuration in which a photoelectric conversion device having an APD can output multiple images whose accumulation periods overlap with each other, thereby enabling continuous shooting even in low illuminance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7223070 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when considering the image sensor of an in-vehicle camera installed on a moving object, for example, because recognition processing is performed on a frame-by-frame basis in normal sensor operation, recognition processing can only be performed every 33.3 ms in the case of 30 fps, for example. Therefore, if an object appears immediately after a frame is read in the in-vehicle camera, it will take about one frame's worth of time (e.g., 33.3 ms) before recognition processing can begin.

[0005] Generally, when the period of one frame is shortened, the accumulation period (exposure period) is also shortened, and the image captured by the camera may be unsuitable for display purposes, as it is likely to have a poor S / N ratio. In particular, in low-illumination environments such as at night or inside a tunnel, it is desirable to capture an image with good visibility by lengthening the period of one frame (accumulation period). In this way, the properties required for applications in which objects are recognized from images and those required for applications in which images are displayed may differ.

[0006] Therefore, one of the objects of the present invention is to obtain both an image for recognition and an image for display from a single sensor. [Means for solving the problem]

[0007] A photoelectric conversion device according to one aspect of the present invention comprises: A sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons; A counter that counts the number of pulses; A plurality of pixels each including a memory for storing a count value of the counter; a control means for generating an image signal based on a difference between the count values ​​of the counter at the start and end of an accumulation period; The control means performs a pre-read operation to read out a pre-read image signal in the middle of a full frame period, and performs a read process of the image signal so that the time for the pre-read operation is shorter than the time for reading out the main image signal during the full frame period. Effect of the Invention

[0008] According to the present invention, both images for recognition and images for display can be acquired from a single sensor. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram showing a configuration example of a photoelectric conversion element according to an embodiment of the present invention. [Diagram 2] 2 is a diagram showing a configuration example of a sensor substrate 11. FIG. [Diagram 3] 2 is a diagram showing an example of the configuration of a circuit board 21. FIG. [Figure 4] 2 and 3, and shows an equivalent circuit of a pixel 101 and a signal processing circuit 103 corresponding to the pixel 101. [Diagram 5] 2 is a diagram illustrating a relationship between the operation of an APD 201 and an output signal. FIG. [Figure 6] 7 is a functional block diagram of a photoelectric conversion device 600 and a moving object 700 according to the embodiment. [Figure 7] 11 is a diagram for explaining the processing from photoelectric conversion to storage in a memory by the camera control unit 605 according to the embodiment. FIG. [Figure 8] 11 is a diagram for explaining the processing of image output and recognition result output by the camera control unit 605 according to the embodiment. FIG. [Figure 9] FIG. 13 is a diagram showing an example of an image divided into a plurality of frames. [Figure 10] 4 is a diagram showing a relationship between a memory circuit and a buffer according to the embodiment; [Figure 11] 5 is a flowchart showing a part of an example of driving a photoelectric conversion element according to an embodiment. [Figure 12] 12 is a continuation of the flowchart in FIG. 11. [Figure 13] 12 is another flowchart showing a continuation of FIG. 11. [Figure 14] 14 is a flowchart continuing from FIG. 13. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified.

[0011] 1 is a diagram showing a configuration example of a photoelectric conversion element according to an embodiment of the present invention. In the following, a photoelectric conversion device having a so-called stacked structure in which a photoelectric conversion element 100 is configured by stacking and electrically connecting two substrates, a sensor substrate 11 and a circuit substrate 21, will be described as an example.

[0012] However, a so-called non-laminated structure in which the components included in the sensor substrate and the components included in the circuit substrate are disposed on a common semiconductor layer may be used. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes signals detected in the pixel region 12.

[0013] 2 is a diagram showing an example of the configuration of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally across a plurality of rows and columns. Each pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, APD).

[0014] Here, the photoelectric conversion unit 102 functions as a sensor unit that emits a pulse at a frequency corresponding to the frequency of receiving photons. Although an example of a pixel arrangement of 3 rows and 4 columns is shown in Fig. 2, the number of rows and columns of the pixel array constituting the pixel region 12 is not particularly limited.

[0015] Fig. 3 is a diagram showing an example of the configuration of the circuit board 21. The circuit board 21 has a signal processing circuit 103 that processes charges photoelectrically converted by each photoelectric conversion unit 102 in Fig. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114.

[0016] The vertical scanning circuit 110 receives a control pulse supplied from a control pulse generating unit 115, and sequentially supplies the control pulse to a plurality of pixels arranged in the row direction. The vertical scanning circuit 110 uses logic circuits such as a shift register and an address decoder.

[0017] The signal output from the photoelectric conversion unit 102 of each pixel is processed by each signal processing circuit 103. The signal processing circuit 103 is provided with a counter, a memory, etc., and a digital value is stored in the memory. The horizontal scanning circuit 111 inputs a control pulse that sequentially selects each column to the signal processing circuit 103 in order to read the signal from the memory of each pixel in which the digital signal is stored.

[0018] A signal is output to the vertical signal line 113 from the signal processing circuit 103 of the pixel of the row selected by the vertical scanning circuit 110. The signal output to the vertical signal line 113 is output to the outside of the photoelectric conversion element 100 via the read circuit 112 and the output circuit 114.

[0019] The readout circuit 112 includes a plurality of buffers connected to each vertical signal line 113. In addition, the control pulse generating unit 115 can control the vertical scanning circuit 110 to perform non-additive readout that selects all rows.

[0020] In addition, when reducing the amount of data of the signal to be read, control can be performed to perform thinning readout, in which every few rows are selected for reading. That is, a preceding readout operation for reading a preceding image signal may be performed in the middle of a full frame period, and the amount of data may be reduced by thinning out pixels in rows or columns when generating the preceding image signal.

[0021] Alternatively, it is possible to control to perform averaging (summing) readout in which a plurality of adjacent pixel signals are subjected to averaging or other processing and read out as a single pixel signal. That is, when generating a preceding image signal, the amount of data may be reduced by adding up the count values ​​of a plurality of pixels. In this case, the number of pixels to be averaged may be freely changed in both the row and column directions, such as 2×2 pixels, 4×4 pixels, etc.

[0022] Alternatively, in order to reduce the amount of data of the read signal, control may be performed to perform crop readout by selecting only a portion of the image area (angle of view). That is, when generating a preceding image signal, the amount of data may be reduced by cutting out and reading out pixels of a portion of the area.

[0023] 2 and 3, a plurality of signal processing circuits 103 are arranged in a region that overlaps vertically with the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in a region that overlaps vertically with a peripheral region (non-pixel region) of the pixel region 12.

[0024] The arrangement of the vertical signal lines 113, the readout circuits 112, and the output circuits 114 are not limited to the example shown in Fig. 3. For example, the vertical signal lines 113 may be arranged extending in the row direction, and the readout circuits 112 may be arranged at the ends of the vertical signal lines 113. Also, it is not necessary to provide one signal processing circuit 103 for each photoelectric conversion unit, and one signal processing circuit may be shared by a plurality of photoelectric conversion units to perform signal processing sequentially.

[0025] FIG. 4 is a diagram showing an equivalent circuit of the pixel 101 in FIG. 2 and FIG.

[0026] The APD 201 included in the photoelectric conversion unit 102 generates a pair of electric charges according to incident light by photoelectric conversion. One of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VL (first voltage) is supplied. The other of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VH (second voltage) higher than the driving voltage VL is supplied.

[0027] In Fig. 4, one node of the APD 201 is the anode, and the other node of the APD is the cathode. A reverse bias voltage is supplied to the anode and cathode of the APD 201 so that the APD 201 performs avalanche multiplication. By supplying such a voltage, the charge generated by the incident light undergoes avalanche multiplication, generating an avalanche current.

[0028] When a reverse bias voltage is supplied, there are two modes: Geiger mode, in which the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, in which the voltage difference between the anode and cathode is close to or less than the breakdown voltage. An APD that operates in Geiger mode is called a SPAD. In the case of a SPAD, for example, the drive voltage VL (first voltage) is -30V, and the drive voltage VH (second voltage) is 1V.

[0029] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212. The quench element 202 is connected to a power supply line to which a driving voltage VH is supplied and one of the anode and cathode nodes of the APD 201.

[0030] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, and has a function of suppressing avalanche multiplication by suppressing the voltage supplied to the APD 201 (quench operation).The quench element 202 also has a function of returning the voltage supplied to the APD 201 to the drive voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0031] In the configuration shown in FIG. 4, the signal processing circuit 103 includes a waveform shaping section 210, a counter circuit 211, and a memory circuit 212 in addition to the quench element 202.

[0032] The waveform shaping unit 210 shapes the voltage change of the cathode of the APD 201 obtained when a photon is detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. Although an example in which one inverter is used as the waveform shaping unit 210 is shown in Fig. 4, a circuit in which a plurality of inverters are connected in series may be used, or another circuit having a waveform shaping effect may be used.

[0033] The counter circuit 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. When a control pulse RES is supplied via a drive line 213, the signal held in the counter circuit 211 is reset. Here, the counter circuit 211 generates a signal based on the difference between the count values ​​at the start and end of the accumulation period.

[0034] A control pulse SEL is supplied to the memory circuit 212 from the vertical scanning circuit 110 in FIG. 3 via a drive line 214 in FIG. 4 (not shown in FIG. 3), and switches between electrical connection and disconnection between the counter circuit 211 and the vertical signal line 113.

[0035] The memory circuit 212 functions as a memory that temporarily stores the count value of the counter, and outputs an output signal from the counter circuit 211 of the pixel to the vertical signal line 113. Note that the memory circuit 212 may be configured to be electrically connectable to a counter circuit in a signal processing circuit 103 (not shown in FIG. 4) adjacent to the signal processing circuit 103 shown in FIG. 4 to perform processing such as addition (average).

[0036] In this way, a plurality of pixel signals generated by a plurality of photoelectric conversion units can be added (averaged) or the like and output as a single pixel signal, thereby reducing the amount of data in the pixel output signal.

[0037] Alternatively, the memory circuit 212 may be selectively switched between a non-addition processing mode in which the memory circuit 212 is connected to only one counter circuit 211 and an averaging processing mode in which the memory circuit 212 is connected to a plurality of counter circuits.

[0038] Note that electrical connections may be switched by disposing a switch such as a transistor between the quench element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103. Similarly, the supply of the driving voltage VH or the driving voltage VL to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0039] 5 is a diagram showing a schematic diagram of the relationship between the operation of the APD 201 and the output signal. The input side of the waveform shaping unit 210 is nodeA, and the output side is nodeB. Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage at nodeA drops.

[0040] When the amount of voltage drop further increases and the potential difference applied to the APD 201 decreases, the avalanche multiplication of the APD 201 stops as at time t2, and the voltage level of nodeA does not drop below a certain value.

[0041] After that, between time t2 and time t3, a current flows through nodeA to compensate for the voltage drop from voltage VL, and at time t3, nodeA is stabilized to its original potential level. At this time, the portion of the output waveform at nodeA that exceeds a certain threshold is shaped by waveform shaping section 210 and is output as a pulse signal at nodeB.

[0042] Next, a photoelectric conversion device 600 and a moving object 700 according to an embodiment will be described. Fig. 6 is a functional block diagram of the photoelectric conversion device 600 and the moving object 700 according to the embodiment. Some of the functional blocks shown in Fig. 6 are realized by causing a computer (not shown) included in the photoelectric conversion device 600 and the moving object 700 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0043] However, some or all of these may be realized by hardware. For the hardware, a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP) may be used. Furthermore, each functional block shown in Fig. 6 does not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths.

[0044] 1 to 5, an imaging optical system 601, an image processing unit 603, a recognition unit 604, a camera control unit 605, a storage unit 606, a communication unit 607, etc. The photoelectric conversion element 100 is composed of an avalanche photodiode for photoelectrically converting an optical image, as described in FIGS.

[0045] Based on a control signal from the camera control unit 605, the photoelectric conversion element 100 can selectively switch between a plurality of readout methods, such as a non-additive readout mode, an additive (average) readout mode, and a thinning readout mode.

[0046] The non-additive read mode is a mode in which all pixels are read as is, the additive (average) read mode is a mode in which adjacent pixels are averaged and read, and the thinning read mode is a mode in which pixels are thinned out and read.

[0047] In this embodiment, the preliminary image used for recognition is read out in a 2x2 pixel additive (average) readout mode, and the main image used for viewing is read out in a non-additive readout mode. That is, the amount of data read out for the preliminary image is 1 / 4 of that for the main image, so the time required for readout is also 1 / 4. In other words, it is possible to read out pixel signals from the preliminary image four times faster than the main image.

[0048] The photoelectric conversion device of the embodiment is mounted on a moving body 700, and a camera unit consisting of a set of an imaging optical system 601 and a photoelectric conversion element 100 is configured to capture an image in at least one direction, for example, in front, behind, or to the side of the moving body. Note that a plurality of camera units may be provided on the moving body 700.

[0049] The image processing unit 603 performs image processing such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaic processing, and data compression on the image signal acquired by the photoelectric conversion element 100, and generates a final image signal.

[0050] In addition, when the photoelectric conversion element 100 has an on-chip color filter such as RGB, it is desirable to perform processing such as white balance correction and color conversion in the image processing unit 603. Furthermore, when the read image is used for multiple purposes such as visual confirmation and recognition, it is desirable to store different image processing parameters for each use of the image in advance and switch the image processing parameters when the use of the image is switched.

[0051] The output of the image processing unit 603 is supplied to a recognition unit 604, an ECU (Electric Control Unit) 701 of the moving object 700, and a camera control unit 605. The recognition unit 604 recognizes the subject by performing image recognition based on the image signal. Specifically, it identifies the type of subject, such as people, animals, vehicles, traffic lights, signs, etc., around the moving object 700.

[0052] In addition, the reliability indicating the accuracy of the recognition result and coordinate information indicating the position and size of the subject in the image are also obtained. Specifically, the offset and size of the X coordinate, and the offset and size of the Y coordinate are obtained as the coordinate information.

[0053] In this embodiment, three types of information, for example, object type, reliability, and coordinate information, are acquired as the recognition result, but other information may be acquired. The recognition result acquired by the recognition unit 604 is output to the ECU 701. Note that, in this embodiment, the moving body 700 is described using an example of an automobile, but the moving body may be any moving object, such as an airplane, a train, a ship, a drone, an AGV, or a robot.

[0054] The camera control unit 605 has a built-in CPU as a computer and a memory that stores a computer program, and controls each unit of the photoelectric conversion device 600 by the CPU executing the computer program stored in the memory.

[0055] The camera control unit 605 functions as a control means, and controls the length of the accumulation period of each frame of the photoelectric conversion element 100 and the timing of the control signal CLK, for example, via a control pulse generation unit of the photoelectric conversion element 100. For the sake of explanation, the period of one frame at the frame rate of the video displayed on the display unit 703 described below will be referred to as one full frame or a full frame period.

[0056] In addition, the camera control unit 605 transmits a control signal for switching the readout mode (readout method) when reading out pixel signals to the photoelectric conversion element 100. In this embodiment, for example, two readout methods, an additive (average) readout mode and a non-additive readout mode, are selectively switched within one full frame.

[0057] Furthermore, the camera control unit 605 has a function of setting various image processing parameters in the image processing unit 603, a function of controlling the recognition unit 604, and a function of acquiring the recognition results. The storage unit 606 includes a recording medium such as a memory card or a hard disk, and can store and read out image signals.

[0058] The communication unit 607 includes a wireless or wired interface, and outputs the generated image signal and the recognition result to the outside of the photoelectric conversion device 600, and receives various signals from the outside. Note that the interface may be a wired communication conforming to a standard such as SPI or I2C, or may use a wireless LAN system such as Wi-Fi, Bluetooth (registered trademark), or the like.

[0059] The ECU 701 incorporates a CPU as a computer and a memory that stores a computer program, and controls each part of the moving body 700 by the CPU executing the computer program stored in the memory.

[0060] The output of the ECU 701 is supplied to a vehicle control unit 702 and a display unit 703. The vehicle control unit 702 functions as a movement control means for driving, stopping, controlling the direction, etc. of the vehicle as a moving object based on the output of the ECU 701. Here, the vehicle control unit 702 as a movement control means controls the operation of the moving object based on the result of recognition processing using a preceding image signal.

[0061] The display unit 703 functions as a display means, includes a display element such as a liquid crystal device or an organic EL, and is mounted on the moving body 700. The display unit 703 displays, for example, an image acquired by the photoelectric conversion element 100, a recognition result acquired by the recognition unit 604, various information related to the running state of the vehicle, and the like, to the driver of the moving body 700 using, for example, a GUI based on the output of the ECU 701.

[0062] In addition, the image processing unit 603, the recognition unit 604, etc. in Figure 6 do not have to be mounted on the moving body 700, and may be provided, for example, in an external terminal provided separately from the moving body 700 for remotely controlling the moving body 700 or for monitoring the movement of the moving body.

[0063] Next, the photoelectric conversion method according to this embodiment will be described with reference to Fig. 7 and Fig. 8. Fig. 7 is a diagram for explaining the processing from photoelectric conversion to storage in memory by the camera control unit 605 according to the embodiment, and Fig. 8 is a diagram for explaining the processing of image output and recognition result output by the camera control unit 605 according to the embodiment.

[0064] In this embodiment, one full frame having a length of 33.3 ms is divided into 13. That is, the photoelectric conversion element 100 sequentially generates 12 frames (frame 1_1, frame 1_2, to frame 1_12) as a preliminary image for recognition, and one full frame (frame 1_13) as a main image for viewing. The frame simply described as "frame" here is different from a full frame and is an expression including both a read period of the preliminary image and a read period of the main image.

[0065] The 12 frames of the preview image are read out in additive (average) readout mode, and the 1 full frame of the main image is read out in non-additive readout mode. As mentioned above, the preview image for recognition has 1 / 4 the amount of data of the main image for viewing, so the image readout time is also 1 / 4 the length. In other words, the readout time for the main image is 8.33 ms, and the readout time for the preview image is 2.08 ms, allowing the readout process to be completed quickly.

[0066] Moreover, the main image is output from the image processing unit 603 to the ECU 701, and the preceding image is subjected to recognition processing by the recognition unit 604, and the recognition result is output from the recognition unit 604 to the ECU 701. In this embodiment, the main image of frame 1_13, which has been accumulated for the longest period of time, is displayed on a display means (display unit) for visual confirmation, but the image of frame 1_13 can be used not only for visual confirmation but also for recognition.

[0067] However, since the image of frame 1_13 is an output signal in a non-additive readout mode, the amount of data is large and the recognition process may take time, so the image of frame 1_13 may be subjected to recognition process as necessary. If the recognition process is performed but the amount of data is not enough, the image processing unit 603 may be used to reduce the amount of data by compression or the like, and the reduced data may be used for the recognition process.

[0068] Also, frame 1_1 has an accumulation period from start time T0 of frame 1 to time T1, and frame 1_2 has an accumulation period from time T0 to time T2. Similarly for subsequent frames, frame 1_N (N is a natural number equal to or less than 13) has an accumulation period from time T0 to time TN. That is, frames 1_1 to 1_13 have the first to thirteenth accumulation periods, respectively.

[0069] First, a processing flow of frame 1_1 (pilot image) used for recognition will be described. At time T0, the counter circuit 211 is reset, and at time T1, a count value C1_1 is obtained in an addition (average) read mode.

[0070] When the pixel 101 is a monochrome pixel, the count value C1_1 calculated by averaging the count values ​​of the counter circuit 211 and the four adjacent pixels is temporarily stored in the memory circuit 212.

[0071] For example, if the pixel 101 has a color filter in a Bayer array of R, Gr, Gb, and B, the count value C1_1 is calculated by averaging the count values ​​of the closest pixels of the same color. Then, the count value C1_1 temporarily stored in the memory circuit 212 is output from the photoelectric conversion element to the image processing unit 603 via the buffer of the readout circuit 112, as shown in FIG.

[0072] The image processing unit 603 performs the various image processes described above, and outputs the processed results as is to the recognition unit 604. As shown in Fig. 8, the image of frame 1_1 is output to the recognition unit 604 during the period from time T1 to time T2, and the output of image signals for all lines is completed at the timing of time T2.

[0073] Then, at the timing of time T2, the object recognition process is started by the recognition unit 604. Note that, in the present embodiment, an example has been described in which the recognition process is started at the timing of time T2 when all image signals of frame 1_1 are available, but the recognition process may be performed sequentially for each line or for each pixel.

[0074] As described above, by making the processing unit for recognition smaller, the time required to obtain the recognition result can be shortened. As soon as the recognition process is completed, the recognition result is promptly output to the ECU 701. In this embodiment, the shortest time required for the recognition process and output of the result is set to Td, as shown in FIG. 8.

[0075] At this time, the output of the recognition result based on the image of the frame 1_1 can be transmitted to the ECU 701 of the moving object 700 at the earliest timing of the time T2+Td. The description of the processing flow of the frame 1_1 is as above.

[0076] Thereafter, the preceding images of frame 1_2, frame 1_3, . . . frame 1_12 are sequentially read out in the periods from time T2 to time T3, from time T3 to time T4, . . . from time T12 to time T13, similar to frame 1_1. Then, the recognition processes are sequentially performed at the timings of time T3, time T4, . . . time T13, and the recognition results are sequentially output from recognition unit 604 to ECU 701 at the timings of time T3+Td, time T4+Td, . . . time T13+Td.

[0077] Next, a processing flow of frame 1_13 (main image) used for visual recognition will be described. At time T0 of full frame 2, a count value C1_13 is read out and acquired from the counter circuit 211 in the non-addition mode.

[0078] The acquired count value C1_13 is temporarily stored in the memory circuit 212. Then, the count value C1_13 temporarily stored in the memory circuit 212 is output from the photoelectric conversion element to the image processing unit 603 via the buffer of the read circuit 112.

[0079] The image processing unit 603 performs the various image processing operations described above, and outputs the processing results as they are to the recognition unit 604. As shown in Fig. 8, the image of frame 1_13 is promptly output to the ECU 701 in the moving object 700 during the period from time T1 to time T5 of full frame 2, and the output of image signals for all lines is completed at the timing of time T5.

[0080] In this embodiment, in consideration of the large amount of data of the image signal of frame 1_13, the recognition process using the recognition unit 604 is not performed, but the recognition process may be performed as necessary. The image signal of frame 1_13 has high contrast, low noise, and suppressed LED flicker, so it is given top priority to be used for visual confirmation.

[0081] However, by performing the recognition process in parallel, the time until an object is recognized when switching full frames can be shortened. That is, in the "recognition result output" at the bottom of Fig. 8, C1_12 to C2_1 are blank periods during which no recognition results are obtained, but by performing the recognition process on the image of C1_13, the blank period can be shortened. This concludes the explanation of the processing flow for frame 1_13.

[0082] Fig. 9 is a diagram showing an example of images of a plurality of divided frames. Fig. 9 shows four frames, including three preceding images of frame 1_1, frame 1_5, and frame 1_9, and a main image of frame 1_13, out of a total of 13 images acquired within one full frame (33.3 ms).

[0083] As shown in Fig. 9, the image of frame 1_1 is dark because the accumulation period is short, but there is little blurring of the person who jumps out into the distance. On the other hand, the accumulation periods of frames 1_5, 1_9, and 1_13 are longer in that order, so blurring of the subject is more likely to occur. Note that blurring of stopped vehicles and white lines is less likely to occur, and the longer the accumulation period, the more likely the contrast is to improve.

[0084] In this manner, in this embodiment, one full frame period has the first to thirteenth readout timings (T1 to T13). Also, the first to twelfth readout timings (T1 to T12) are controlled by the camera control unit 605 so as to be earlier than the thirteenth readout timing (T13).

[0085] The readout time (time required for readout) of each of the first to twelfth image signals as the preliminary image signal is controlled to be shorter than the readout time of the thirteenth image signal as the main image signal.

[0086] That is, a preceding read operation for reading out a preceding image signal is performed in a full frame period, and the period of the preceding read operation is set shorter than the period for reading out the main image signal in the full frame period. Also, between the preceding read operation (for example, the read operation of T1) and the read operation of the main image signal (the read operation of T13), another image signal is further read out.

[0087] In this embodiment, the first to twelfth accumulation periods overlap with the thirteenth accumulation period, and the first to twelfth accumulation periods start at the same time as the thirteenth accumulation period. Furthermore, the end of the thirteenth accumulation period is the end timing of a full frame.

[0088] That is, an image with a short accumulation period and an image with a long accumulation period are created, the timing at which the short accumulation period ends is set earlier than the timing at which the long accumulation period ends, and as soon as the short accumulation period ends, the image is output and sent to the downstream recognition unit.The subject is recognized based on a signal generated during at least the first accumulation period.The recognition unit 604 as a recognition means recognizes the subject based on a preceding image signal generated during at least the first accumulation period.

[0089] Therefore, while in the past, image recognition was not possible until one full frame had elapsed, in this embodiment, image recognition can be performed immediately after the readout time of the 13th image has elapsed at the earliest. That is, when a moving object is moving at high speed, obstacles and the like can be recognized quickly. Furthermore, by outputting the recognition result to the ECU 701 with low delay, it is possible to apply the brakes quickly, or to avoid obstacles early.

[0090] In addition, an image with a long accumulation period can improve the contrast, and can be used as a display image. That is, an image with a short accumulation period is suitable for quick subject recognition, and an image with a long accumulation period is suitable for a display image. In this way, the display device of this embodiment displays the signal generated during at least the 13th accumulation period as an image.

[0091] In addition, since the present embodiment uses an APD, unlike a CMOS sensor, the accumulated charge does not deteriorate when read out, so the accumulation periods can be overlapped. Also, since there is no readout noise, the original signal does not deteriorate even if it is read out many times after one accumulation.

[0092] Fig. 10 is a diagram showing the relationship between memory circuits and buffers in an embodiment. Fig. 10 shows a state in which memory circuits 212 in signal processing circuit 103 in Fig. 3 are arranged in N rows and M columns, and each memory circuit is represented as memory 1-1 to memory NM.

[0093] Buffers 1 to M in Fig. 10 indicate buffers included in read circuit 112 in Fig. 3. Output circuit 114 in Fig. 10 corresponds to output circuit 114 in Fig. 3.

[0094] Fig. 11 is a flowchart showing a part of an example of driving a photoelectric conversion element according to the embodiment, Fig. 12 is a flowchart continuing from Fig. 11, Fig. 13 is another flowchart showing a continuation from Fig. 11, and Fig. 14 is a flowchart continuing from Fig. 13. That is, the entire example of driving a photoelectric conversion element according to the embodiment is shown by the flowcharts of Figs. 11 to 14.

[0095] The CPU serving as a computer in the camera control unit 605 executes a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowcharts of FIGS.

[0096] 11, the CPU in the camera control unit 605 sets i=1, where i corresponds to frame 1_i. Next, in step S102, the CPU in the camera control unit 605 determines whether the current frame is a frame for performing addition processing when outputting the count value to the memory, that is, a frame for recognition processing.

[0097] In this embodiment, i = 1 to 12 is Yes, and i = 13 is No. The determination in step S102 may be made by setting an ID indicating additive (average) readout mode, non-additive readout mode, etc. for each frame in advance and looking at the ID, or by counting the number of frames and looking at the count value.

[0098] By using IDs, if you want to selectively change the frames to be read out using additive (average) readout or non-additive readout within one full frame, you can use the same algorithm to process the frames by simply reassigning the IDs of the frames. If the result of the determination in step S102 is Yes, proceed to step S103, and if it is No, proceed to step S112 in FIG. 13 via A.

[0099] Next, in step S103, the CPU in the camera control unit 605 performs averaging on the count value Count of the counter circuit 211 at time Ti, for a total of four pixels, two pixels in each of the horizontal and vertical directions, and outputs the result to the memory circuit 212. In this embodiment, since a monochrome pixel 101 is assumed, the averaging result of the four pixels is output only to the memory circuit 212 where both N rows and M columns are odd numbers.

[0100] That is, there is no count value in the memory circuit 212 where any one of the N rows or M columns is an even number. For example, in the case of a color filter having a Bayer array of R, Gr, Gb, and B, rather than monochrome, there is one count value for every four pixels in both the horizontal and vertical directions for each color. Also, when the count value is output to the memory circuit 212, it is output to all memory circuits at the same time. This operation corresponds to the operation at time T1 in FIG. 7.

[0101] Next, in step S104, the CPU in the camera control unit 605 sets j=1, and in step S105, the CPU in the camera control unit 605 outputs the count value Count(jki) in the memory circuit jk in Fig. 10 to the buffer. At this time, columns 1 to M are output to the buffer at the same time. However, since count values ​​exist only in odd-numbered columns, the number of data of the count value Count(jki) actually output to the buffer is M / 2.

[0102] That is, the count values ​​of the even-numbered columns may or may not be output to the buffer. This operation means that the count value of the first row in FIG.

[0103] Next, in step S106, the CPU in the camera control unit 605 sets k=1, and in step S107, the CPU in the camera control unit 605 outputs the count value Count(jki) of buffer k to the output circuit 114. This operation corresponds to the operation of reading out the signals of the buffers in the leftmost column in FIG. 10 from the output circuit.

[0104] Next, in step S108 of FIG. 12, the CPU in the camera control unit 605 determines whether k < M - 1. If Yes, in step S109, the CPU in the camera control unit 605 increments k by 2 as k = k + 2, returns to step S107, and performs the operation of step S107.

[0105] The reason for not incrementing k by 1 is that, as described above, there is no count value in the even columns. This operation corresponds to the operation of reading the signal of the buffer in the third column from the left in FIG. 10 from the output circuit.

[0106] If it is No in step S108, that is, if k = M - 1 or k = M, it means that the signal of the buffer in the (M - 1)-th column or the M-th column of FIG. 10 has been read from the output circuit. Then, next, it proceeds to step S110, and the CPU in the camera control unit 605 determines whether j < N - 1.

[0107] If it is Yes in step S110, then in step S111, the CPU in the camera control unit 605 increments j by 2 as j = j + 2 and returns to step S105. This corresponds to the operation for starting the reading of the next row.

[0108] If it is determined to be No in step S110, it means that the reading of all rows has been completed. So, it proceeds to step S121, and the CPU in the camera control unit 605 determines whether i < 13. If it is determined to be Yes in step S121, it proceeds to step S122, and the CPU in the camera control unit 605 increments i by 1 as i = i + 1 and returns to step S102. This operation corresponds to the operation for starting the reading at the next time T2.

[0109] In this embodiment, the process determined as Yes in step S102 is repeated 12 times. That is, the flow from step S103 to step S121 is repeated 12 times. This operation corresponds to the operation of reading at times T1 to T12. Thereafter, in the state where i = 13 in step S122, the process returns to step S102. In step S102, it is determined as No by the CPU in the camera control unit 605, and the process proceeds to step S112 in FIG. 13.

[0110] Next, in step S112, the CPU in the camera control unit 605 outputs the count value Count of the counter circuit 211 at time Ti (without performing addition average processing) as it is to the memory circuit 212. At this time, the output is performed simultaneously for all the memory circuits. This operation corresponds to the operation at time T13 in FIG. 7.

[0111] Next, in step S113, the CPU in the camera control unit 605 sets j = 1, and in step S114, the CPU in the camera control unit 605 outputs the count value Count(j - k - i) in the memory circuit j - k in FIG. 10 to the buffer. At this time, the output is performed to the buffer for 1 to M columns simultaneously. Different from step S105, in step S114, count values exist in both even columns and odd columns, and the number of data is M. This operation means the operation of taking in the count values in the first row in FIG. 10 into the buffer.

[0112] Next, in step S115, the CPU in the camera control unit 605 sets k = 1, and in step S116, the CPU in the camera control unit 605 outputs the count value Count(j - k - i) of buffer k to the output circuit 114. This operation corresponds to the operation of reading the signal of the buffer in the leftmost column in FIG. 10 from the output circuit.

[0113] Next, in step S117 in FIG. 14, the CPU in the camera control unit 605 determines whether k < M. If No, in step S118, the CPU in the camera control unit 605 increments k by 1 as k = k + 1, returns to step S116, and performs the operation of step S116.

[0114] In step S109, k is incremented by 2, while in step S116, k is incremented by 1 because, as described above, there are count values for both even and odd columns. This operation corresponds to the operation of reading the signal of the buffer in the second column from the left in FIG. 10 from the output circuit.

[0115] When it becomes No in step S117, that is, when k = M, it means that the signal of the buffer in the Mth column in FIG. 10 has been read from the output circuit. Next, it proceeds to step S119, and the CPU in the camera control unit 605 determines whether j < N.

[0116] In step S119, when it is determined to be Yes, in step S120, the CPU in the camera control unit 605 sets j = j + 1, increments j by 1, and returns to step S114. This corresponds to the operation for starting to read the next row.

[0117] When it is determined to be No in step S119, it means that the reading of all rows has been completed. So, it proceeds to step S121, and the CPU in the camera control unit 605 determines whether i < 13. Since the 13th frame is in the state of i = 13, it is determined to be No in step S121.

[0118] When it is determined to be No in step S121, it means that the reading at time T13 has been completed. So, the reading of all the image signals accumulated in the first full frame has been completed. In the above manner, the signals accumulated in the photoelectric conversion element 100 can be sequentially read out.

[0119] In addition, for example, in the above embodiment, an example of performing a 4-pixel addition average of 2 pixels in the horizontal direction and 2 pixels in the vertical direction for the preceding image for recognition has been described. However, the number of pixels for addition average can be arbitrarily set both in the horizontal direction and the vertical direction.

[0120] Similarly, when thinning out and reading out, the number of pixels and the number of rows to be thinned out (thinning rate) may be set arbitrarily. In addition, in the present embodiment, an example of the monochrome pixel 101 has been described, but when performing averaging or thinning out, color filters in a Bayer array of R, Gr, Gb, B, etc. are provided, and averaging or thinning out may be performed for each pixel of the same color.

[0121] Alternatively, the recognition unit 604 may perform subject recognition based on the preceding image signal, calculate the recognition accuracy based on the result, and determine the number of averaging pixels in pixel averaging readout mode and the thinning rate in thinning readout mode according to the recognition accuracy.

[0122] That is, if the recognition unit 604 can output a recognition result with high accuracy even if the resolution of the acquired image is low, the number of pixels to be added and the thinning rate are increased to reduce the amount of image data, thereby realizing high speed processing.

[0123] Also, whether to use the additive (average) read mode or the thinning read mode may be determined according to the brightness of the ambient light. That is, the data amount reduction rate may be controlled based on the brightness of the ambient light.

[0124] For example, when the ambient light is equal to or greater than a predetermined value, the amount of data can be reduced by switching to a thinning readout mode, which has a simple readout process, that is, a mode in which pixels in rows or columns are thinned out, since the noise level does not significantly increase even if the signal level is reduced. On the other hand, when the ambient light is less than a predetermined value, the amount of data can be reduced by switching to an addition (average) readout mode, which adds (averages) the count values ​​of multiple pixels to improve the noise level.

[0125] As described above, in this embodiment, in addition to an image acquired in full frame (33.3 ms) units (hereinafter, the main image), an image read out in the middle of accumulation (hereinafter, the preceding image) is acquired and subjected to recognition processing. This makes it possible to reduce the time from full frame switching to recognition. In addition, since the accumulation period of the preceding image is shorter than that of the main image and there is relatively little subject blur, the recognition rate can be improved even for objects that could not be recognized using only the main image. Furthermore, by increasing the frequency of acquiring the preceding image, the time from full frame switching to recognition is further shortened.

[0126] In this embodiment, the photoelectric conversion device is mounted on a moving body and used for visual confirmation to display the surrounding environment as an image to the driver using a display means such as an electronic mirror, and for object recognition to avoid obstacles and apply the brakes in automatic driving, driving assistance, etc. In this case, since the image is used as the image for visual confirmation, high image quality such as high contrast and low noise can be obtained.

[0127] On the other hand, the resolution of the preliminary image for recognition is lowered to reduce the amount of image data in order to achieve high-speed processing, so that transmission with low latency is possible. Furthermore, the image resolution is optimized according to the subject recognition accuracy, so that it is possible to achieve a resolution sufficient for subject recognition with a minimum amount of data.

[0128] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to the above-mentioned embodiments, and various modifications are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. In addition, the above-mentioned embodiments may be appropriately combined. In addition, the present invention includes the following combinations.

[0129] (Configuration 1) A photoelectric conversion device comprising a plurality of pixels, each pixel having a sensor unit that emits a pulse at a frequency corresponding to the frequency of receiving photons, a counter that counts the number of the pulses, and a memory that stores the count value of the counter, and a control means that generates an image signal based on the difference between the count values ​​of the counter at the start and end of an accumulation period, wherein the control means performs a pre-reading operation to read out a pre-reading image signal in the middle of a full frame period, and performs a readout process of the image signal so that the time for the pre-reading operation is shorter than the time for reading out the main image signal during the full frame period.

[0130] (Configuration 2) The photoelectric conversion device according to configuration 1, further comprising: a recognition means for performing a recognition process for recognizing a subject based on the preceding image signal; and a display means for displaying the main image signal for visual confirmation.

[0131] (Configuration 3) The photoelectric conversion device according to configuration 1 or 2, wherein the control means further reads out another image signal between the preceding read operation and the read operation of the main image signal.

[0132] (Configuration 4) The photoelectric conversion device according to any one of configurations 1 to 3, wherein the control means reduces an amount of data when generating the preceding image signal.

[0133] (Configuration 5) The photoelectric conversion device according to configuration 4, wherein the control means reduces the amount of data by adding up count values ​​of a plurality of pixels when generating the preceding image signal.

[0134] (Configuration 6) The photoelectric conversion device according to configuration 4 or 5, wherein the control means reduces the amount of data by thinning out pixels in rows or columns when generating the preceding image signal.

[0135] (Configuration 7) The photoelectric conversion device according to any one of configurations 4 to 6, wherein the control means reduces the amount of data by cutting out and reading out pixels in a partial area when generating the pilot image signal.

[0136] (Configuration 8) The photoelectric conversion device according to any one of configurations 4 to 7, wherein the control means determines the data amount reduction rate based on the accuracy of subject recognition based on the preceding image signal.

[0137] (Configuration 9) The photoelectric conversion device according to any one of configurations 4 to 8, wherein the control means controls the data amount reduction rate based on the brightness of ambient light.

[0138] (Configuration 10) The photoelectric conversion device described in Configuration 9, wherein the control means reduces the amount of data by thinning out pixels in a row or column when the ambient light is equal to or greater than a predetermined value, and reduces the amount of data by adding up count values ​​of multiple pixels when the ambient light is less than the predetermined value.

[0139] (Configuration 11) The photoelectric conversion device according to any one of configurations 1 to 10, wherein the sensor section includes an avalanche photodiode.

[0140] (Configuration 12) A photoelectric conversion device comprising a plurality of pixels, each pixel having a sensor unit that emits a pulse at a frequency corresponding to the frequency of receiving photons, a counter that counts the number of the pulses, and a memory that stores the count value of the counter, and a control means that generates an image signal based on the difference between the count values ​​of the counter at the start and end of an accumulation period, wherein the control means performs a precedence readout operation to read out a precedence image signal in the middle of a full frame period, and performs a readout process of the image signal so that the resolution of the precedence image signal read out by the precedence readout operation is lower than the resolution of the main image signal during the full frame period.

[0141] (Configuration 13) A moving body comprising: a photoelectric conversion device according to any one of configurations 1 to 12; and a movement control means for controlling the movement of the moving body based on the result of recognition processing using the preceding image signal.

[0142] (Method) A photoelectric conversion method for performing photoelectric conversion using a plurality of pixels, each of which has a sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons, a counter that counts the number of the pulses, and a memory that stores the count value of the counter, characterized in that the photoelectric conversion method generates an image signal based on the difference between the count values ​​of the counter at the start and end of an accumulation period, performs a pre-reading operation to read out a pre-reading image signal in the middle of a full frame period, and performs a readout process of the image signal so that the time for the pre-reading operation is shorter than the time for reading out the main image signal during the full frame period.

[0143] (Program) A computer program for controlling each means of the photoelectric conversion device according to any one of configurations 1 to 12 or the moving body according to configuration 13 by a computer.

[0144] In order to realize a part or all of the control in the above-described embodiment, a computer program for realizing the functions of the above-described embodiment may be supplied to a photoelectric conversion device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the photoelectric conversion device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0145] 11: Sensor board 12: Pixel area 21: Circuit board 22: Circuit area 100: Photoelectric conversion element 101: Pixel 102: Photoelectric conversion unit 103: Signal processing circuit 110: Vertical scanning circuit 111: Horizontal scanning circuit 112: Readout circuit 113: Vertical signal line 114: Output circuit 115: Control pulse generation unit 201: Avalanche photodiode 202: Quench element 210: Waveform shaping section 211: Counter circuit 212: Memory circuit 213: Drive wire 214: Drive line 600: Photoelectric conversion device 601: Imaging optical system 603: Signal processing unit 604: Recognition part 605: Camera control unit 606: Storage section 607: Communications Department 700: Mobile 701: ECU 702: Vehicle control unit 703: Display section

Claims

1. A sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons; A counter that counts the number of pulses; A plurality of pixels each including a memory for storing a count value of the counter; a control means for generating an image signal based on a difference between the count values ​​of the counter at the start and end of an accumulation period; The control means performs a pre-reading operation to read out a pre-read image signal in the middle of a full frame period, and performs a read process of the image signal so that the time for the pre-reading operation is shorter than the time for reading out the main image signal during the full frame period.

2. 2. The photoelectric conversion device according to claim 1, further comprising: a recognition unit that performs a recognition process for recognizing a subject based on the preceding image signal; and a display unit that displays the main image signal for visual confirmation.

3. 2. The photoelectric conversion device according to claim 1, wherein the control means further reads out another image signal between the preceding read operation and the read operation of the main image signal.

4. 2. The photoelectric conversion device according to claim 1, wherein the control means reduces an amount of data when generating the preceding image signal.

5. 5. The photoelectric conversion device according to claim 4, wherein the control means reduces the amount of data by adding up count values ​​of a plurality of pixels when generating the preceding image signal.

6. 5. The photoelectric conversion device according to claim 4, wherein the control means reduces the amount of data by thinning out pixels in rows or columns when generating the preceding image signal.

7. 5. The photoelectric conversion device according to claim 4, wherein said control means reduces the amount of data by cutting out and reading out pixels in a partial area when generating said preceding image signal.

8. 5. The photoelectric conversion device according to claim 4, wherein the control means determines the data amount reduction rate based on the accuracy of subject recognition based on the preceding image signal.

9. 5. The photoelectric conversion device according to claim 4, wherein the control means controls the data amount reduction rate based on the brightness of ambient light.

10. The photoelectric conversion device according to claim 9, characterized in that the control means reduces the amount of data by thinning out pixels in a row or column when the ambient light is greater than or equal to a predetermined value, and reduces the amount of data by adding up count values ​​of multiple pixels when the ambient light is less than the predetermined value.

11. 2. The photoelectric conversion device according to claim 1, wherein the sensor portion includes an avalanche photodiode.

12. A sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons; A counter that counts the number of pulses; A plurality of pixels each including a memory for storing a count value of the counter; a control means for generating an image signal based on a difference between the count values ​​of the counter at the start and end of an accumulation period; The control means performs a preceding read operation to read out a preceding image signal in the middle of a full frame period, and performs an image signal read process so that the resolution of the preceding image signal read out by the preceding read operation is lower than the resolution of the main image signal during the full frame period.

13. The photoelectric conversion device according to any one of claims 1 to 12, and a movement control means for controlling the movement of the moving body based on a result of the recognition processing using the preceding image signal.

14. A photoelectric conversion method for performing photoelectric conversion using a plurality of pixels, each of which includes a sensor unit that emits a pulse at a frequency corresponding to a frequency of receiving photons, a counter that counts the number of the pulses, and a memory that stores a count value of the counter, comprising: generating an image signal based on a difference between count values ​​of the counter at the start and end of an accumulation period; A photoelectric conversion method characterized in that a preceding readout operation is performed to read out a preceding image signal during a full frame period, and an image signal readout process is performed so that the time for the preceding readout operation is shorter than the time for reading out a main image signal during the full frame period.

15. A computer program for controlling the photoelectric conversion device according to any one of claims 1 to 12 or each means of the moving body according to claim 13 by a computer.

Images