Photoelectric conversion device, movable body, control method, and computer program

Abstract

To provide a photoelectric conversion device that can obtain high night vision performance and a wide dynamic range.SOLUTION: A photoelectric conversion device 600 comprises: a photoelectric conversion element that has a plurality of pixels each including a photoelectric conversion part emitting pulses at a frequency according to the frequency of photon reception, a counter counting the number of pulses, and a memory storing the count value of the counter; a camera control unit that controls to generate an image signal on the basis of the difference in the count values of the counter between the start and the end of a storage period, and output a first to (N-1)-th image signals having, in one full frame, a first to N-th (N≥2) storage periods, the first to (N-1)-th storage periods being shorter than the N-th storage period, and generated in the first to (N-1)-th storage periods, from the end of the first storage period until the end of the N-th storage period; and an image processing unit that combines two or more image signals in different storage periods of the first to N-th image signals to generate a composite image signal.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a photoelectric conversion device, a moving object, a control method, a computer program, and the like. [Background technology]

[0002] Conventionally, a technique for expanding the dynamic range by synthesizing multiple images captured with different accumulation periods is well known. For example, Patent Document 1 describes a technique for expanding the dynamic range by capturing an image with a proper exposure, capturing a second image if it is determined that the dynamic range needs to be expanded, and synthesizing the images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 8-214211 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when using the technology described in Patent Document 1, the timing of the first and second shots is different, so if the subject moves, a composite image is generated in which the subject's position is shifted. The faster the subject moves, the greater the shift in subject position becomes, resulting in an unnatural composite image.

[0005] Furthermore, in order to expand the dynamic range, it is necessary to change the length of the accumulation period between the first and second shots, which means that the amount of blur of the subject will differ between the two images. This means that when combining images, an image that looks unnatural near the contours of the subject may be generated. Specifically, the resulting image will have clear contours in bright subject areas and blurred contours in dark areas.

[0006] Furthermore, expanding the dynamic range requires images to be accumulated in a short time to preserve the gradation of bright subjects, so if LED light enters the subject, a flicker phenomenon will occur. For example, in automotive applications, LED traffic lights and signs may become unrecognizable, and subjects illuminated by headlights may become dark.

[0007] Therefore, an object of the present invention is to provide a photoelectric conversion device that can provide high night vision performance and a wide dynamic range. [Means for solving the problem]

[0008] A photoelectric conversion device according to one aspect of the present invention comprises: A photoelectric conversion unit that emits pulses at a frequency corresponding to the frequency of receiving photons; A counter that counts the number of pulses; A plurality of pixels each including a memory for storing a count value of the counter; generating an image signal based on a difference between count values ​​of the counter at the start and end of an accumulation period; a control unit which has first to N-th (N≧2) accumulation periods within one full frame, the first to (N-1) accumulation periods being shorter than the N accumulation period, and which controls the first to (N-1)-th image signals generated during the first to (N-1) accumulation periods to be output between the end of the first accumulation period and the end of the N accumulation period; The image signal processing device further comprises an image synthesis unit that synthesizes two or more image signals having different accumulation periods from among the first to Nth image signals to generate a synthesized image signal. Effect of the Invention

[0009] According to the present invention, it is possible to provide a photoelectric conversion device that can obtain high night vision performance and a wide dynamic range. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing a configuration example of a photoelectric conversion element according to an embodiment of the present invention. [Diagram 2] 2 is a diagram showing a configuration example of a sensor substrate 11. FIG. [Diagram 3] 2 is a diagram showing an example of the configuration of a circuit board 21. FIG. [Figure 4] 2 and 3, and shows an equivalent circuit of a pixel 101 and a signal processing circuit 103 corresponding to the pixel 101. [Diagram 5] 2 is a diagram illustrating a relationship between the operation of an APD 201 and an output signal. FIG. [Figure 6] FIG. 2 is a functional block diagram for explaining functions of a photoelectric conversion device 600 and a moving object 700 according to the embodiment. [Figure 7] 1A is a functional block diagram showing an example of the configuration of an image processing unit 602, and FIG. 1B is a functional block diagram showing an example of the configuration of a short-second image generating unit 608. FIG. [Figure 8] 6A is a functional block diagram showing an example of the configuration of an HDR synthesis unit 609, FIG. 6B is a diagram explaining the processing contents of a brightness correction unit 6091, and FIG. 6C is a diagram explaining a method of generating a synthetic image using an image synthesis unit 6092. [Figure 9] 10 is a diagram for explaining the processing from photoelectric conversion to image output by the camera control unit 604 according to the embodiment. [Figure 10] 10(A) is a diagram showing an example of images of multiple divided frames. FIG. 10(B) is a diagram showing images C1(T0-T1), C1(T1-T2), C1(T2-T3), and C1(T3-T4) generated by the short second image generating unit 608 using the image signal (count value) of FIG. 10(A). FIG. 10(C) is a diagram showing images of the short second image signal generated by the short second image generating unit 608 and the long second image signal output from the emission crosstalk correction unit 607. [Figure 11] 4 is a diagram showing a relationship between a memory circuit and a buffer according to the embodiment; [Figure 12] 5 is a flowchart showing details of an example of driving a photoelectric conversion element according to the embodiment. [Figure 13] This is a continuation of the flowchart in FIG. 12. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified.

[0012] 1 is a diagram showing a configuration example of a photoelectric conversion element according to an embodiment of the present invention. In the following, a photoelectric conversion device having a so-called stacked structure in which a photoelectric conversion element 100 is configured by stacking and electrically connecting two substrates, a sensor substrate 11 and a circuit substrate 21, will be described as an example.

[0013] However, a so-called non-laminated structure in which the components included in the sensor substrate and the components included in the circuit substrate are disposed on a common semiconductor layer may be used. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes signals detected in the pixel region 12.

[0014] 2 is a diagram showing a configuration example of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally across a plurality of rows and columns. The pixels 101 include a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, APD), and the photoelectric conversion unit 102 emits a pulse at a frequency corresponding to the frequency of receiving photons. The number of rows and columns of the pixel array constituting the pixel region 12 is not particularly limited.

[0015] Fig. 3 is a diagram showing an example of the configuration of the circuit board 21. The circuit board 21 has a signal processing circuit 103 that processes charges photoelectrically converted by each photoelectric conversion unit 102 in Fig. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114.

[0016] The vertical scanning circuit 110 receives a control pulse supplied from a control pulse generating unit 115, and sequentially supplies the control pulse to a plurality of pixels arranged in the row direction. The vertical scanning circuit 110 uses logic circuits such as a shift register and an address decoder.

[0017] The signal output from the photoelectric conversion unit 102 of each pixel is processed by each signal processing circuit 103. The signal processing circuit 103 is provided with a counter, a memory, etc., and a digital value is stored in the memory. The horizontal scanning circuit 111 inputs a control pulse that sequentially selects each column to the signal processing circuit 103 in order to read the signal from the memory of each pixel in which the digital signal is stored.

[0018] A signal is output to the vertical signal line 113 from the signal processing circuit 103 of the pixel of the row selected by the vertical scanning circuit 110. The signal output to the vertical signal line 113 is output to the outside of the photoelectric conversion element 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has a plurality of buffers built therein and connected to the vertical signal line 113.

[0019] 2 and 3, a plurality of signal processing circuits 103 are arranged in a region overlapping the pixel region 12 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged so as to overlap between an end of the sensor substrate 11 and an end of the pixel region 12 in a plan view.

[0020] That is, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in a region overlapping the non-pixel region in a plan view.

[0021] The arrangement of the vertical signal lines 113, the readout circuits 112, and the output circuits 114 are not limited to the example shown in Fig. 3. For example, the vertical signal lines 113 may be arranged extending in the row direction, and the readout circuits 112 may be arranged at the ends of the vertical signal lines 113. Also, it is not necessary to provide one signal processing circuit 103 for each photoelectric conversion unit, and a configuration may be adopted in which one signal processing unit is shared by a plurality of photoelectric conversion units and performs signal processing sequentially.

[0022] FIG. 4 is a diagram showing an equivalent circuit of the pixel 101 in FIG. 2 and FIG.

[0023] The APD 201 included in the photoelectric conversion unit 102 generates a pair of electric charges according to incident light by photoelectric conversion. One of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VL (first voltage) is supplied. The other of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VH (second voltage) higher than the voltage VL is supplied.

[0024] In Fig. 4, one node of the APD 201 is the anode, and the other node of the APD is the cathode. A reverse bias voltage is supplied to the anode and cathode of the APD 201 so that the APD 201 performs avalanche multiplication. By supplying such a voltage, the charge generated by the incident light undergoes avalanche multiplication, generating an avalanche current.

[0025] When a reverse bias voltage is supplied, there are two modes: Geiger mode, in which the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, in which the voltage difference between the anode and cathode is close to or less than the breakdown voltage. An APD that operates in Geiger mode is called a SPAD. In the case of a SPAD, for example, the drive voltage VL (first voltage) is -30V, and the drive voltage VH (second voltage) is 1V.

[0026] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212. The quench element 202 is connected to a power supply line to which a driving voltage VH is supplied and one of the anode and cathode nodes of the APD 201.

[0027] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, and has a function of suppressing avalanche multiplication by suppressing the voltage supplied to the APD 201 (quench operation).The quench element 202 also has a function of returning the voltage supplied to the APD 201 to the drive voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0028] FIG. 4 shows an example in which the signal processing circuit 103 includes a waveform shaping section 210, a counter circuit 211, and a memory circuit 212 in addition to the quench element 202.

[0029] The waveform shaping unit 210 shapes the voltage change of the cathode of the APD 201 obtained when a photon is detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. Although an example in which one inverter is used as the waveform shaping unit 210 is shown in Fig. 4, a circuit in which a plurality of inverters are connected in series may be used, or another circuit having a waveform shaping effect may be used.

[0030] The counter circuit 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. When a control pulse RES is supplied via a drive line 213, the signal held in the counter circuit 211 is reset. Here, the counter circuit 211 generates a signal based on the difference between the count values ​​at the start and end of the accumulation period.

[0031] 3 to the memory circuit 212 via a drive line 214 (not shown in FIG. 3) in FIG. 4, and switches between electrical connection and disconnection between the counter circuit 211 and the vertical signal line 113. The memory circuit 212 functions as a memory that temporarily stores the count value of the counter, and outputs an output signal from the counter circuit 211 of the pixel to the vertical signal line 113.

[0032] Note that electrical connections may be switched by disposing a switch such as a transistor between the quench element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103. Similarly, the supply of the driving voltage VH or the driving voltage VL to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0033] 5 is a diagram showing a schematic diagram of the relationship between the operation of the APD 201 and the output signal. The input side of the waveform shaping unit 210 is nodeA, and the output side is nodeB. Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage at nodeA drops.

[0034] When the amount of voltage drop further increases and the potential difference applied to the APD 201 decreases, the avalanche multiplication of the APD 201 stops as at time t2, and the voltage level of nodeA does not drop below a certain value.

[0035] After that, between time t2 and time t3, a current flows through nodeA to compensate for the voltage drop from voltage VL, and at time t3, nodeA is stabilized to its original potential level. At this time, the portion of the output waveform at nodeA that exceeds a certain threshold is shaped by waveform shaping section 210 and is output as a pulse signal at nodeB.

[0036] Next, a photoelectric conversion device 600 and a moving object 700 according to an embodiment will be described. Fig. 6 is a functional block diagram of the photoelectric conversion device 600 and the moving object 700 according to the embodiment. Some of the functional blocks shown in Fig. 6 are realized by causing a computer (not shown) included in the photoelectric conversion device 600 and the moving object 700 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0037] However, a part or all of these may be realized by hardware, which may be a dedicated circuit (ASIC) or a processor (a reconfigurable processor, DSP), etc.

[0038] In addition, the functional blocks shown in Fig. 6 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths. The above explanation of Fig. 6 also applies to the functional blocks of Figs. 7(A), (B) and Fig. 8(A) described later.

[0039] The photoelectric conversion device 600 has a photoelectric conversion element 100 including an avalanche photodiode for photoelectrically converting an optical image as described in Figures 1 to 5. The photoelectric conversion device 600 also has an imaging optical system 601, an image processing unit 602, a frame memory 603, a camera control unit 604, a storage unit 605, a communication unit 606, and the like.

[0040] The photoelectric conversion device of the embodiment is mounted on a moving body 700, and a camera unit consisting of a set of an imaging optical system 601 and a photoelectric conversion element 100 is configured to capture an image in at least one direction, for example, in front, behind, or to the side of the moving body. Note that a plurality of camera units may be provided on the moving body 700.

[0041] In this embodiment, the moving body 700 is described using an example of an automobile, but the moving body may be any moving body, such as an airplane, a train, a ship, a drone, an AGV, or a robot.

[0042] The function of the image processing unit 602 will be described in detail. Fig. 7(A) is a functional block diagram showing an example of the configuration of the image processing unit 602. As shown in Fig. 7(A), the image processing unit 602 includes at least a light emission crosstalk correction unit 607, a short-second image generation unit 608, and an HDR (high dynamic range) synthesis unit 609.

[0043] The light emission crosstalk correction unit 607 is for correcting light emission crosstalk, which is a signal detected by an adjacent pixel due to light emission in a high electric field region. The light emission crosstalk correction method may be a known correction method based on theoretical values, or another method based on actual measurement results, etc.

[0044] After the light emission crosstalk correction unit 607 corrects the light emission crosstalk, the image signal is output to the frame memory 603, the short-second image generation unit 608, and the HDR synthesis unit 609. The functional blocks to which the image signal is output will be described later.

[0045] The frame memory 603 is for temporarily storing image signals when various image processes are performed using the image processing unit 602. For example, when generating a new image using multiple images, the frame memory 603 is used to temporarily store multiple image signals that are to become original images. In this embodiment, the frame memory 603 is used when generating a short-second image signal, which will be described later, and when performing HDR composition.

[0046] The short-second image generation unit 608 generates a short-second image signal required for generating an HDR image, and generates the short-second image signal based on the image signal output from the light emission crosstalk correction unit 607 and the image signal read from the frame memory 603. Details of the method of generating a short-second image signal will be described later with reference to Fig. 7(B). The short-second image signal generated by the short-second image generation unit 608 is output to an HDR synthesis unit 609.

[0047] The HDR synthesis unit 609 generates a final HDR image using the short-time image signal generated by the short-time image generation unit 608 and the long-time image signal output from the light emission crosstalk correction unit 607. Details of the method for generating an HDR image will be described later with reference to FIG.

[0048] In this embodiment, the image processing unit 602 is composed of three blocks: a light emission crosstalk correction unit 607, a short-second image generation unit 608, and an HDR synthesis unit 609. However, for example, image processing such as black level correction, gamma curve correction, noise reduction, digital gain adjustment, demosaic processing, and data compression may be performed on the image signal output from the photoelectric conversion element 100.

[0049] Furthermore, when the photoelectric conversion element 100 has an on-chip color filter such as RGB, it is desirable to perform processing such as white balance correction and color conversion in the image processing unit 602. The HDR image generated by the HDR synthesis unit 609 in the image processing unit 602 is output to the camera control unit 604 and an ECU (Electric Control Unit) 701 in the moving object 700.

[0050] The camera control unit 604 incorporates a CPU as a computer and a memory that stores a computer program, and controls each unit of the photoelectric conversion device 600 by the CPU executing the computer program stored in the memory.

[0051] The camera control unit 604 controls the length of the exposure period of each frame of the photoelectric conversion element 100 and the timing of the control signal CLK via a control pulse generation unit of the photoelectric conversion element 100, for example.

[0052] The camera control unit 604 also performs various processes for generating an HDR image using the image processing unit 602 and the frame memory 603, and controls writing of an image signal output from the HDR synthesis unit 609 to the storage unit 605. Furthermore, the camera control unit 604 also communicates with the ECU 701 in the moving object 700.

[0053] The storage unit 605 includes a recording medium such as a memory card or a hard disk, and can store and read out the final image signal. The communication unit 606 includes a wireless or wired interface, and outputs the generated image signal to the outside of the photoelectric conversion device 600 and receives various signals from the outside. As the interface, wired communication according to a standard such as SPI or I2C may be performed, or a wireless LAN system such as Wi-Fi or Bluetooth (registered trademark) may be used.

[0054] The ECU 701 incorporates a CPU as a computer and a memory that stores a computer program, and controls each part of the moving body 700 by the CPU executing the computer program stored in the memory.

[0055] The output of the ECU 701 is supplied to a vehicle control unit 702 and a display unit 703. The vehicle control unit 702 functions as a movement control unit that performs movement control such as driving, stopping, and directional control of a vehicle as a moving body based on the output of the ECU 701.

[0056] The display unit 703 includes a display element such as a liquid crystal device or an organic EL, and is mounted on the moving body 700. The display unit 703 displays images acquired by the photoelectric conversion element 100, various information related to the running state of the vehicle, and the like, to the driver of the moving body 700 using, for example, a GUI based on the output of the ECU 701.

[0057] 6 may not be mounted on the moving body 700. For example, they may be provided in an external terminal or the like provided separately from the moving body 700 for remotely controlling the moving body 700 or for monitoring the traveling of the moving body. Also, a recognition unit may be provided in either the photoelectric conversion device 600 or the moving body 700, and the recognition result may be used for controlling the vehicle or notifying the driver.

[0058] FIG. 9 is a diagram for explaining the processing from photoelectric conversion to image output by the camera control unit 604 according to the embodiment.

[0059] In this embodiment, one frame having a length of 33.3 ms is called a full frame (or a main frame), and a full frame divided into four is called a frame (or a subframe). In this embodiment, one full frame having a length of 33.3 ms is divided into four.

[0060] That is, as shown in Fig. 9, a full frame 1 is divided into frames 1_1, 1_2, 1_3, and 1_4 of equal duration (8.33 ms). In Fig. 9 and subsequent figures, frames 1_1, 1_2, 1_3, 1_4, etc. are represented as F1_1, 1_2, 1_3, 1_4, etc.

[0061] Frame 1_1 has an accumulation period from start time T0 of full frame 1 to time T1, frame 1_2 has an accumulation period from time T0 to time T2, frame 1_3 has an accumulation period from time T0 to time T3, and frame 1_4 has an accumulation period from time T0 to time T4.

[0062] The number of divisions may be N (an integer). Therefore, in this embodiment, it is sufficient to have first to Nth (N≧2) accumulation periods within one full frame. The first to (N-1)th accumulation periods are shorter than the Nth accumulation period, and the first to (N-1)th image signals generated during the first to (N-1)th accumulation periods are controlled to be output between the end of the first accumulation period and the end of the Nth accumulation period.

[0063] 9, the first to (N-1)th accumulation periods overlap with the Nth accumulation period, and the first to (N-1)th accumulation periods start at the same time as the Nth accumulation period. Also, the end of the Nth accumulation period coincides with the end of a full frame.

[0064] First, the processing flow of frame 1_1 will be described. At time T0, the camera control unit 604 resets the counter circuit 211 in the photoelectric conversion element 100, accumulates data for the period from time T0 to time T1, and then reads out the count value C1_1 for the period from time T1 to time T2. Specifically, C1_1 is temporarily stored in the memory circuit 212, and then output from the photoelectric conversion element to the image processing unit 602 via the buffer of the read circuit 112.

[0065] C1_1 read out from the photoelectric conversion element 100 undergoes the above-mentioned light emission crosstalk correction by the light emission crosstalk correction unit 607 in the image processing unit 602, and the corrected image signal is promptly written in the frame memory 1 area in the frame memory 603.

[0066] That is, the count value C1_1 of each pixel of frame 1_1 starts to be written to the frame memory 603 at the timing of time T1, and writing of C1_1 for all lines is completed at the timing of time T2. Then, the image information of C1_1 is held in the frame memory 603 as it is.

[0067] In this embodiment, for the sake of convenience, it is assumed that the delay time associated with processing in the image processing unit 602 is short and can be ignored. However, if it cannot be ignored, it is not necessarily necessary to complete writing to the frame memory 603 by time T2.

[0068] Subsequently, for frame 1_2, accumulation is similarly performed during the period from time T0 to time T2, and then the count value C1_2 is read out during the period from time T2 to time T3, and after various processes are performed, the count value C1_2 is written into the area of ​​the frame memory 2 in the frame memory 603. Then, the image information of frame 1_2 is held in the frame memory 603.

[0069] Similarly, for frame 1_3, accumulation is performed during the period from time T0 to time T3, and then the count value C1_3 is read out during the period from time T3 to time T4, and after various processes are performed, the count value C1_3 is written into the frame memory 3 area in the frame memory 603. Then, image information of frame 1_3 is held in the frame memory 603.

[0070] For frame 1_4, accumulation is performed during the period from time T0 to time T4 of full frame 1, and then a count value C1_4 is obtained during the period from time T0 to time T1 of full frame 2. After various processes are performed, the count value C1_4 is output to the short-second image generation unit 608.

[0071] The reason why C1_4 is not written to the frame memory 603 is to quickly generate a short-second image signal by the short-second image generation unit 608 and perform HDR synthesis processing, thereby shortening the time until image output as much as possible and reducing the capacity of the frame memory 603.

[0072] However, if the processing speed for generating the short-second image signal described below is slower than the output speed of the image signal output from the emission crosstalk correction unit 607, a new area called frame memory 4 may be provided within the frame memory 603 to hold image information.

[0073] A method by which the short-second image generating unit 608 generates a short-second image signal will be described with reference to Fig. 7(B). Fig. 7(B) is a functional block diagram showing an example of the configuration of the short-second image generating unit 608. Generation of a short-second image signal starts when all image information from frames 1_1 to 1_4 is available.

[0074] In this embodiment, a short-second image signal is generated using four count values. Specifically, the count values ​​Cn(T0-T1), Cn(T1-T2), Cn(T2-T3), and Cn(T3-T4) in the four accumulation periods of time T0-T1, time T1-T2, time T2-T3, and time T3-T4 are used. Note that n indicates the full frame number, and in this embodiment, n=1.

[0075] These four count values ​​can be calculated using Cn_ 1 , Cn_ 2 , and Cn_ 3 stored in the frame memory 603 , and Cn_ 4 output from the emission crosstalk correction unit 607 .

[0076] The calculation formulas are as shown in the following formulas 1 to 4. In this embodiment, the calculations are performed using subtractors 6081, 6082, and 6083 shown in Fig. 7B. In full frame 1, C1(T0 to T1), C1(T1 to T2), C1(T2 to T3), and C1(T3 to T4) are generated using C1_1, C1_2, C1_3, and C1_4.

[0077] Cn(T0~T1)=Cn_1 (Eq. 1) Cn(T1~T2)=Cn_2-Cn_1...(Formula 2) Cn(T2~T3)=Cn_3-Cn_2...(Formula 3) Cn(T3~T4)=Cn_4-Cn_3...(Formula 4)

[0078] Next, an averaging process is performed using the four calculated count values ​​C1(T0-T1), C1(T1-T2), C1(T2-T3), and C1(T3-T4). Specifically, the four count values ​​are added using an adder 6084 shown in Fig. 7(B), and then multiplied by 4 using a multiplier 6085.

[0079] As a result, it is possible to generate a short-second image signal with an accumulation period (8.33 ms) that is 1 / 4 that of a long-second image signal (33.3 ms).In other words, even for subjects whose count value saturates during the accumulation period of a long-second image signal, the short-second image signal makes it possible to express the brightness of the subject as gradations (however, the upper limit is up to four times the brightness at which the long-second image signal saturates).

[0080] In addition, since the count values ​​acquired during four consecutive accumulation periods are averaged, the period during which the short-time image signal captures the subject is virtually the same as the capture period of the long-time image signal, so the position of the subject present in the image signal does not shift.

[0081] In addition, even if a fast-moving subject is present, the amount of blur is the same, so there is no visual discomfort even when HDR compositing, which will be described later, is performed. Furthermore, even if a subject that periodically blinks, such as an LED or fluorescent light, is present, the accumulation period of the short-second image signal is artificially extended, so it is possible to avoid the LED or fluorescent light being completely turned off.

[0082] For in-vehicle applications, it is desirable that the accumulation period of temporally consecutive short-period image signals is 11 ms or more. In other words, when generating an image signal with a short accumulation period by averaging the first to Nth image signals, it is desirable to select images among the first to Nth image signals with consecutive accumulation periods of 11 ms or more as target images for averaging. This makes it possible to avoid LEDs or fluorescent lights from being turned off completely.

[0083] The short-second image signal generated by the short-second image generation unit 608 is output to the HDR synthesis unit 609. That is, in this embodiment, an image signal having a shorter accumulation period than the Nth image signal is generated using the first to Nth image signals, and a synthetic image signal is generated using the image signal having the shorter accumulation period and the Nth image signal.

[0084] The HDR synthesis unit 609 synthesizes the long-time image signal output from the light emission crosstalk correction unit 607 and the short-time image signal generated by the short-time image generation unit 608 to generate a final HDR image.

[0085] Next, the HDR synthesis method in this embodiment will be described with reference to Fig. 8(A). Fig. 8(A) is a functional block diagram showing a configuration example of an HDR synthesis unit 609. The HDR synthesis unit 609 has a luminance correction unit 6091 and an image synthesis unit 6092, and receives two image signals, a long-time image signal (Cn_4) output from the light emission crosstalk correction unit 607 and a short-time image signal output from the short-time image generation unit 608.

[0086] The luminance correction unit 6091 adjusts the count value of the image signal accumulated in a short time to a level equivalent to the count value of the image signal accumulated in a long time based on the ratio of the accumulation periods. By adjusting the count values ​​of the long time image signal and the short time image signal using the luminance correction unit 6091, the two images can be synthesized without any awkwardness.

[0087] Fig. 8(B) is a diagram for explaining the processing content of the brightness correction unit 6091, where the horizontal axis indicates the amount of light of the subject, and the vertical axis indicates the count value of the image input to the HDR synthesis unit 609. In Fig. 8(B), 609A is a graph showing the count value of the long-time image signal relative to the amount of light of the subject, and 609B is a graph showing the count value of the short-time image signal relative to the amount of light of the subject.

[0088] Also, 609B′ indicates a graph after the count value of 609B has been subjected to luminance correction processing by a luminance correction unit 6091. Moreover, Sat written on the vertical axis indicates the saturation count value of the photoelectric conversion element 100.

[0089] Since 609A has a long accumulation period, it quickly reaches the Sat level when the amount of light on the subject increases. Since 609B has a short accumulation period, it reaches the Sat level when the amount of light on the subject is greater than that of 609A.

[0090] The luminance correction unit 6091 multiplies the count value of 609B by a predetermined gain (amplification factor) to match the count values ​​of 609A and 609B. Specifically, the luminance correction unit 6091 calculates the gain by dividing the accumulation period during long-second accumulation by the accumulation period during short-second accumulation, and directly multiplies the count value of the short-second image signal by this gain.

[0091] That is, in this embodiment, 33.3 ms is divided by 8.33 ms to calculate a Gain value of 4.0, and the count value of the short time duration image signal is multiplied by 4.0. By doing so, the count value of the long time duration image signal and the count value of the short time duration image signal will be the same count value (unless the count value is saturated).

[0092] The image synthesis unit 6092 synthesizes the count value of 609A and the count value of 609B' to generate a single image with expanded dynamic range by synthesizing the long time image signal and the short time image signal. In this manner, the image synthesis unit 6092 synthesizes two or more image signals having different accumulation periods among the first to Nth image signals to generate a synthesized image signal.

[0093] Fig. 8(C) is a diagram explaining a method for generating a composite image using the image synthesis unit 6092, where the horizontal axis represents the count value of the input image, and the vertical axis represents the synthesis ratio of the long exposure image signal and the short exposure image signal. The count values ​​α to β on the horizontal axis are count values ​​just near the switchover point between the long exposure image signal and the short exposure image signal, and ideally the count value β should be set to a count value near Sat in Fig. 8(B).

[0094] If the count value of input image 609A is equal to or less than α, the count value of input image 609A of the long exposure image signal is used 100%. If the count value of input image 609A is between α and β, the count value of input image 609A of the long exposure image signal and the count value 609B' of the short exposure image signal are each multiplied by the combination ratio shown in the graph of Figure 8(C), and the results are added together.

[0095] If the count value of the input image 609A is equal to or greater than β (saturated), the count value of the short-second image signal 609B' is used as 100%. By performing this type of processing, it is possible to synthesize the long-second image signal acquired by long-second accumulation and the short-second image signal acquired by short-second accumulation to generate a single HDR image with an expanded dynamic range.

[0096] In this embodiment, in order to explain the functions of the short-second image generation unit 608 and the HDR synthesis unit 609 separately, the short-second image generation unit 608 multiplied by 1 / 4 and the HDR synthesis unit 609 multiplied by a gain of 4.0, but these two processes may be omitted. However, if the short-second image generation unit 608 does not perform averaging, these two processes cannot be omitted.

[0097] For example, when generating an HDR image using one of a plurality of acquired short-second image signals, division processing is not required in the short-second image generating unit 608, and therefore a gain multiplication process is required in the HDR synthesis unit 609. The final HDR image generated by the image synthesis unit 6092 is output to the camera control unit 604 and the ECU 701 in the moving object 700.

[0098] Fig. 10(A) is a diagram showing an example of images of a plurality of divided frames. As shown in Fig. 10(A), the image of frame 1_1 is dark because the accumulation period is short, and there is little movement (blur) of the subject of a person who has jumped out.

[0099] On the other hand, the accumulation period becomes longer in the order of frame 1_2, frame 1_3, and frame 1_4, so the movement of the subject of a person who jumps out into the frame appears blurred. Note that blurring is unlikely to occur for stopped vehicles or white lines, and the longer the accumulation period, the easier it is to improve contrast.

[0100] Figure 10(B) is a diagram showing images C1(T0 to T1), C1(T1 to T2), C1(T2 to T3), and C1(T3 to T4) generated by the short-second image generation unit 608 using the image signal (count value) of Figure 10(A).

[0101] Since the accumulation period is 8.33 ms for all images, the brightness of the subject (count value) and the amount of blur are the same. However, the position of the person who jumps out has changed. Specifically, the position of the person moves to the right in the order of C1 (T0-T1), C1 (T1-T2), C1 (T2-T3), and C1 (T3-T4).

[0102] Fig. 10(C) is a diagram showing, as images, the short time image signal generated by the short time image generation unit 608 and the long time image signal output from the light emission crosstalk correction unit 607. Since the short time image signal is obtained by adding all four count values ​​in Fig. 10(B) once, the movement (amount of blur) of the person who has jumped out is large and matches the amount of blur in the long time image signal.

[0103] In addition, after the count value is added, it is divided by the number of frames, so the count value after addition and averaging will be 1 / 4 the brightness of the long-exposure image signal.

[0104] Fig. 11 is a diagram showing the relationship between memory circuits and buffers in an embodiment. Fig. 11 shows a state in which memory circuits 212 in signal processing circuit 103 in Fig. 3 are arranged in N rows and M columns, and each memory circuit is represented as memory 1-1 to memory NM.

[0105] Buffers 1 to M in Fig. 11 indicate buffers included in read circuit 112 in Fig. 3. Output circuit 114 in Fig. 11 corresponds to output circuit 114 in Fig. 3.

[0106] Fig. 12 is a flowchart showing details of an example of driving a photoelectric conversion element in this embodiment, and Fig. 13 is a flowchart continuing from Fig. 12. Note that the operations of the steps in the flowcharts of Fig. 12 and Fig. 13 are performed sequentially by a CPU or the like as a computer in the camera control unit 604 executing a computer program stored in a memory.

[0107] 12, set i=1. Next, in step S1202, the count value Count of the counter circuit 211 at time Ti is output to the memory circuit 212. At this time, the output is performed simultaneously to all the memory circuits. This operation corresponds to the operation at time T1 in FIG.

[0108] Next, in step S1203, j=1 is set, and in step S1204, the count value Count(jki) in the memory circuit jk in Fig. 11 is output to the buffer. At this time, columns 1 to M are output to the buffer at the same time. This operation means that the count value of the first row in Fig. 11 is taken into the buffer.

[0109] Next, in step S1205, set k = 1, and in step S1206, output the count value Count(j - k - i) of buffer k to the output circuit 114. This operation corresponds to the operation of reading the signal of the buffer in the leftmost column of FIG. 11 from the output circuit.

[0110] Next, proceed to step S1207 in FIG. 13 via A, and in step S1207, determine whether k < M. If No, then in step S1208, increment k by 1 as k = k + 1, return to step S1206 via B, and perform the operation of step S1206. This operation corresponds to the operation of reading the signal of the buffer in the second column from the left in FIG. 11 from the output circuit.

[0111] When it becomes No in step S1207, that is, when k = M, it means that the signal of the buffer in the M-th column of FIG. 11 has been read out from the output circuit. Next, proceed to step S1209 and determine whether j < N.

[0112] If Yes in step S1209, then in step S1210, increment j by 1 as j = j + 1, and return to step S1204 via C. This corresponds to the operation for starting the reading of the next row.

[0113] If it is determined to be No in step S1209, it means that the reading of all rows has been completed. So, proceed to step S1211 and determine whether j < 4. If it is determined to be Yes in step S1211, then proceed to step S1212, increment i by 1 as i = i + 1, and return to step S1202 via D. This operation corresponds to the operation of starting the reading at the next time T2.

[0114] If it is determined to be No in step S1211, it means that the reading at time T4 has been completed. So, proceed to step S1213 and reset the counter circuit 211 with a reset signal. This operation corresponds to the reset operation of the counter circuit 211 at time T4 in FIG. 9. In the above manner, the signals accumulated in the photoelectric conversion element 100 can be sequentially read out.

[0115] In the above embodiment, the accumulation period of a full frame is divided into four, but it is not necessary to divide it into four. If it is desired to enhance the effect of HDR, the dynamic range can be further expanded by increasing the number of divisions.

[0116] For example, if the full frame accumulation period for acquiring a short-time image signal is divided into 10 parts, it becomes possible to express gradations up to an amount of light 10 times the brightness at which the long-time image signal saturates (4.0 times in this embodiment).

[0117] In addition, if a CMOS sensor is used and the number of divisions of the accumulation period is increased, read noise occurs every time an image is read out, which reduces night vision performance, whereas in this embodiment, a SPAD sensor is used and no read noise occurs. Therefore, when the number of divisions is large, it is desirable to use a SPAD sensor.

[0118] Also, if you want to reduce the capacity of the frame memory, you can reduce the number of divisions of the accumulation period for acquiring the short-second image signal. By doing so, the number of frames of the image signal that are temporarily stored can be reduced, making it possible to reduce the capacity of the frame memory. Also, if only one short-second image signal is used, the short-second image generation unit 608 is not necessary.

[0119] In this embodiment, the full frame period is divided equally into four to obtain a short-time image signal. That is, the accumulation period of one full frame is divided equally into N accumulation periods to output N image signals, but it is not necessary to divide it equally. The accumulation period of the short-time image signal may be determined according to the required dynamic range expansion ratio.

[0120] Also, in this embodiment, the short-second image signal is generated by averaging four image signals (the number of divisions of the accumulation period). That is, when generating an image signal with a shorter accumulation period than the Nth image signal, the first to Nth image signals are averaged to generate an image signal with a shorter accumulation period. However, it is not necessarily necessary to use all images corresponding to the number of divisions of the accumulation period. Only one image signal may be used, or any other number of images may be used.

[0121] For example, one image signal accumulated during the period closest to the center of the accumulation period of the long-period image signal may be used. That is, when generating an image signal having an accumulation period shorter than the Nth image signal, an image signal having an accumulation period shorter than the Nth accumulation period and closest to the center of the Nth accumulation period may be used to generate a composite image signal. This makes it possible to reduce the capacity of the frame memory.

[0122] Alternatively, a composite image may be generated using three image signals accumulated during the start, center, and last three periods of the accumulation period of a long-exposure image signal, thereby achieving a good balance between reducing the frame memory capacity and suppressing the positional deviation of the subject.

[0123] Although the present invention has been described in detail based on the preferred embodiment, the present invention is not limited to the above embodiment, and various modifications are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. Note that this embodiment includes the following combinations.

[0124] (Configuration 1) A photoelectric conversion device comprising: a plurality of pixels, each pixel having a photoelectric conversion unit that emits a pulse at a frequency corresponding to a frequency of receiving photons, a counter that counts the number of the pulses, and a memory that stores the count value of the counter; a control unit that generates an image signal based on the difference between the count values ​​of the counter at the start and end of an accumulation period, having first to Nth accumulation periods (N≧2) within one full frame, the first to (N-1)th accumulation periods being shorter than the Nth accumulation period, and controlling the first to (N-1)th image signals generated during the first to (N-1)th accumulation periods to be output between the end of the first accumulation period and the end of the Nth accumulation period; and an image synthesis unit that synthesizes two or more image signals having different accumulation periods among the first to Nth image signals to generate a synthesized image signal.

[0125] (Configuration 2) The photoelectric conversion device according to configuration 1, wherein the first to (N-1)th accumulation periods and the Nth accumulation period overlap each other.

[0126] (Configuration 3) The photoelectric conversion device according to configuration 1 or 2, wherein the first to (N-1)th accumulation periods and the Nth accumulation period start simultaneously.

[0127] (Configuration 4) The photoelectric conversion device according to any one of configurations 1 to 3, wherein the end of the Nth accumulation period coincides with the end of a full frame.

[0128] (Configuration 5) The photoelectric conversion device described in any one of configurations 1 to 4, characterized in that the control unit generates an image signal having a shorter accumulation period than the Nth image signal using the first to Nth image signals, and generates a composite image signal using the image signal having the shorter accumulation period and the Nth image signal.

[0129] (Configuration 6) The photoelectric conversion device according to any one of configurations 1 to 5, wherein the control unit equally divides an accumulation period of one full frame into N accumulation periods and outputs N image signals.

[0130] (Configuration 7) The photoelectric conversion device according to configuration 6, characterized in that when generating an image signal having a shorter accumulation period than the Nth image signal, the image signal having the shorter accumulation period is generated by averaging the first to Nth image signals.

[0131] (Configuration 8) The photoelectric conversion device according to configuration 7, characterized in that when generating an image signal having a short accumulation period by averaging the first to Nth image signals, images among the first to Nth image signals having a continuous accumulation period of 11 ms or more are used as target images for averaging.

[0132] (Configuration 9) A photoelectric conversion device according to any one of configurations 1 to 8, characterized in that when generating an image signal having an accumulation period shorter than that of the Nth image signal, a composite image signal is generated using an image signal having an accumulation period shorter than the Nth accumulation period and closest to the center of the Nth accumulation period.

[0133] (Configuration 10) The photoelectric conversion device according to any one of configurations 1 to 9, wherein the photoelectric conversion section includes an avalanche photodiode.

[0134] (Configuration 11) A moving body comprising: the photoelectric conversion device according to any one of configurations 1 to 10; and a movement control unit for controlling the movement of the moving body.

[0135] (Method) A control method for controlling a photoelectric conversion device having a plurality of pixels, each pixel including a photoelectric conversion unit that emits pulses at a frequency corresponding to a frequency of receiving photons, a counter that counts the number of the pulses, and a memory that stores the count value of the counter, the control method comprising the steps of: generating an image signal based on the difference between the count values ​​of the counter at the start and end of an accumulation period; having first to Nth accumulation periods (N≧2) within one full frame, the first to (N-1)th accumulation periods being shorter than the Nth accumulation period; controlling the first to (N-1)th image signals generated during the first to (N-1)th accumulation periods to be output between the end of the first accumulation period and the end of the Nth accumulation period; and combining two or more image signals having different accumulation periods among the first to Nth image signals to generate a composite image signal.

[0136] (Program) A computer program for controlling each part of the photoelectric conversion device according to any one of configurations 1 to 10 or the moving object according to configuration 11 by a computer.

[0137] A computer program for implementing the functions of the above-described embodiment as part or all of the control in this embodiment may be supplied to the photoelectric conversion device via a network or various storage media. Then, a computer (or a CPU, MPU, etc.) in the photoelectric conversion device may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0138] 11: Sensor board 12: Pixel area 21: Circuit board 22: Circuit area 100: Photoelectric conversion element 101: Pixel 102: Photoelectric conversion unit 103: Signal processing circuit 110: Vertical scanning circuit 111: Horizontal scanning circuit 112: Readout circuit 113: Vertical signal line 114: Output circuit 115: Control pulse generation unit 201: Avalanche photodiode 202: Quench element 210: Waveform shaping section 211: Counter circuit 212: Memory circuit 213: Drive wire 214: Drive line 600: Photoelectric conversion device 601: Imaging optical system 602: Image processing unit 603: Frame memory 604: Camera control unit 605: Storage section 606: Communications Department 607: Luminescence crosstalk correction unit 608: Short-time image generation unit 609: HDR (High Dynamic Range) synthesis section 700: Mobile 701: ECU 702: Vehicle control unit 703: Display section 6081: Subtractor 6082: Subtractor 6083: Subtractor 6084: Adder 6085: Divider 6091: Brightness correction section 6092: Image synthesis section

Claims

1. A photoelectric conversion unit that emits pulses at a frequency corresponding to the frequency of receiving photons; A counter that counts the number of pulses; A plurality of pixels each including a memory for storing a count value of the counter; generating an image signal based on a difference between count values ​​of the counter at the start and end of an accumulation period; a control unit which has first to N-th (N≧2) accumulation periods within one full frame, the first to (N−1)-th accumulation periods being shorter than the N-th accumulation period, and which controls the first to (N−1)-th image signals generated during the first to (N−1)-th accumulation periods to be output during a period from the end of the first accumulation period to the end of the N-th accumulation period; an image synthesis unit that synthesizes two or more image signals having different accumulation periods among the first to Nth image signals to generate a synthesized image signal.

2. 2. The photoelectric conversion device according to claim 1, wherein the first to (N-1)th accumulation periods and the Nth accumulation period overlap each other.

3. 2. The photoelectric conversion device according to claim 1, wherein the first to (N-1)th accumulation periods and the Nth accumulation period start simultaneously.

4. 2. The photoelectric conversion device according to claim 1, wherein the end of the Nth accumulation period coincides with the end of a full frame.

5. The photoelectric conversion device according to claim 1, characterized in that the control unit generates an image signal having a shorter accumulation period than the Nth image signal using the first to Nth image signals, and generates a composite image signal using the image signal having the shorter accumulation period and the Nth image signal.

6. 2. The photoelectric conversion device according to claim 1, wherein the control unit equally divides an accumulation period of one full frame into N accumulation periods and outputs N image signals.

7. The photoelectric conversion device according to claim 6, characterized in that, when generating an image signal having a shorter accumulation period than the Nth image signal, the image signal having the shorter accumulation period is generated by averaging the first to Nth image signals.

8. The photoelectric conversion device according to claim 7, characterized in that, when generating an image signal having a short accumulation period by averaging the first to Nth image signals, images among the first to Nth image signals having a continuous accumulation period of 11 ms or more are used as target images for averaging.

9. 2. The photoelectric conversion device according to claim 1, wherein when generating an image signal having an accumulation period shorter than that of the Nth image signal, a composite image signal is generated using an image signal having an accumulation period shorter than the Nth accumulation period and closest to the center of the Nth accumulation period.

10. The photoelectric conversion device according to claim 1 , wherein the photoelectric conversion section includes an avalanche photodiode.

11. The photoelectric conversion device according to any one of claims 1 to 10, A moving body comprising: a movement control unit for controlling the movement of the moving body.

12. A photoelectric conversion unit that emits pulses at a frequency corresponding to the frequency of receiving photons; A counter that counts the number of pulses; A control method for controlling a photoelectric conversion device having a plurality of pixels each including a memory that stores a count value of the counter, the method comprising: generating an image signal based on a difference between count values ​​of the counter at the start and end of an accumulation period; The image capturing apparatus has first to N-th (N≧2) accumulation periods within one full frame, the first to (N−1)-th accumulation periods being shorter than the N-th accumulation period, and controls the first to (N−1)-th image signals generated during the first to (N−1)-th accumulation periods to be output during a period from the end of the first accumulation period to the end of the N-th accumulation period, A control method comprising: generating a composite image signal by combining two or more image signals having different accumulation periods among the first to Nth image signals.

13. A computer program for controlling each part of the photoelectric conversion device according to any one of claims 1 to 10 or the moving body according to claim 11 by a computer.

Images