Photoelectric conversion device, mobile device, control method, storage medium and computer product

Abstract

The invention provides a photoelectric conversion device, a mobile device, a control method, a storage medium and a computer product. The photoelectric conversion device includes: a photoelectric conversion unit that emits a pulse corresponding to a photon; a counter that counts the number of pulses; and a plurality of pixels each including a memory storing a count value of the counter; wherein an image signal is generated based on a difference between count values of the counter at the start and end of the accumulation period, and during a period from the end of the first accumulation period to the end of the Nth accumulation period, first to (N-1)-th image signals generated during the first to (N-1)-th (N > = 2) accumulation periods are output, the first to Nth accumulation periods are within one full frame period, and each of the first to (N-1) th accumulation periods is shorter than the Nth accumulation period, and a synthesized image signal is generated by synthesizing two or more image signals having different accumulation periods among the first to Nth image signals.

Description

Technical Field

[0001] The present invention relates to a photoelectric conversion device, a movable device, a control method, a storage medium, etc. Background Art

[0002] Since the past, the technology of expanding the dynamic range by synthesizing a plurality of images captured at different accumulation periods is well known. For example, Japanese Patent Laid-Open No. 8-214211 discloses a technology of expanding the dynamic range by capturing an image with appropriate exposure, performing a second image capture if it is determined that the dynamic range needs to be expanded, and synthesizing the image.

[0003] However, if the technology disclosed in Japanese Patent Laid-Open No. 8-214211 is used, the timings of the first and second image capture are different from each other, so if the subject moves, a composite image in which the position of the subject is shifted is generated. When the movement of the subject becomes faster, the displacement amount of the subject's position also increases, resulting in a composite image that gives a sense of incongruity.

[0004] Furthermore, in order to expand the dynamic range, it is necessary to change the length of the accumulation period between the first and second image captures, thereby making the blur amount of the subject different between the two images. That is, when performing image synthesis, it is possible to generate an image that gives a sense of incongruity near the outline of the subject. Specifically, the resulting synthesized image has a clear outline in the bright area of ​​the subject and a blurred outline in the dark area of ​​the subject.

[0005] In addition, in order to expand the dynamic range, images accumulated in a short time period need to be kept in the grayscale of bright subjects, so if LED light enters the subject, flickering will occur. For example, in automotive applications, traffic signs and signs using LEDs may become unrecognizable, and subjects illuminated by headlights may become dark. Summary of the invention

[0006] According to one aspect of the present invention, there is provided a photoelectric conversion device, comprising:

[0007] a photoelectric conversion unit that emits a pulse corresponding to the photon;

[0008] a counter for counting the number of said pulses;

[0009] a plurality of pixels, each pixel comprising a memory storing a count value of the counter;

[0010] one or more memories storing instructions; and

[0011] One or more processors that execute the instructions to:

[0012] generating an image signal based on a difference between count values ​​of the counter at the beginning and the end of an accumulation period;

[0013] outputting, during a period from the end of the 1st accumulation period to the end of the Nth (N≥2)th accumulation period, the 1st image signal to the (N-1)th image signal generated during the 1st accumulation period to the N-1th accumulation period, wherein the 1st accumulation period to the Nth accumulation period are within one full frame period and each of the 1st accumulation period to the (N-1)th accumulation period is shorter than the Nth accumulation period; and

[0014] A synthesized image signal is generated by synthesizing two or more image signals having different accumulation periods among the first to Nth image signals.

[0015] Other features of the present invention will become apparent from the following description of embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a diagram illustrating a configuration example of a photoelectric conversion element according to an embodiment of the present invention.

[0017] Figure 2 is a diagram illustrating a configuration example of the sensor substrate 11 .

[0018] Figure 3 2 is a diagram illustrating a configuration example of the circuit substrate 21 .

[0019] Figure 4 It's a picture. Figure 2 and Figure 3 FIG. 1 is a diagram of an equivalent circuit of a pixel 101 in FIG. 1 and a signal processing circuit 103 corresponding to the pixel 101.

[0020] Figure 5 is a diagram schematically illustrating the relationship between the operation of the APD 201 and the output signal.

[0021] Figure 6 is a functional block diagram illustrating functions of a photoelectric conversion apparatus 600 and a movable device 700 according to an embodiment.

[0022] Fig. 7A is a functional block diagram illustrating a configuration example of the image processing unit 602, and Figure 7B 2 is a functional block diagram illustrating a configuration example of the short second image generating unit 608 .

[0023] Fig. 8A is a functional block diagram illustrating a configuration example of the HDR synthesis unit 609, Figure 8B is a diagram illustrating the processing content of the illumination correction unit 6091, and Figure 8C is a diagram illustrating a method of generating a composite image using the image synthesis unit 6092.

[0024] Fig. 9 is a diagram illustrating processing from photoelectric conversion to image output performed by the camera control unit 604 according to the embodiment.

[0025] Fig. 10A is a diagram illustrating an example of images of a plurality of divided frames. Fig. 10B It is a diagram showing the use of Fig. 10A The image signals (count values) of the images C1 (T0 to T1), C1 (T1 to T2), C1 (T2 to T3), and C1 (T3 to T4) generated by the short second image generating unit 608 are shown. Fig. 10C is a diagram illustrating an image of a short seconds image signal generated by the short seconds image generation unit 608 and an image of a long seconds image signal output from the light emission crosstalk correction unit 607 .

[0026] Fig.11 is a diagram illustrating a relationship between a memory loop and a buffer according to an embodiment.

[0027] Fig.12 is a flowchart illustrating details of a driving example of a photoelectric conversion element according to an embodiment.

[0028] Fig.13 yes Fig.12 Continuation of the flowchart in . DETAILED DESCRIPTION

[0029] Hereinafter, with reference to the accompanying drawings, an advantageous mode of the present invention will be described using an embodiment. In each figure, the same reference numerals are applied to the same components or elements, and repeated descriptions will be omitted or simplified.

[0030] Figure 1 1 is a diagram illustrating a configuration example of a photoelectric conversion element according to an embodiment of the present invention. Hereinafter, a photoelectric conversion device having a so-called stacked structure in which a photoelectric conversion element 100 is configured by stacking and electrically connecting two substrates (a sensor substrate 11 and a circuit substrate 21) will be described as an example.

[0031] However, a so-called non-stacked structure in which components included in the sensor substrate and components included in the circuit substrate are arranged on a common semiconductor layer may be used. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes a signal detected in the pixel region 12.

[0032] Figure 21 is a diagram illustrating a configuration example of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally across a plurality of rows and columns. The pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, APD) that emits pulses at a frequency corresponding to a photon reception frequency. Meanwhile, the number of rows and columns of the pixel array constituting the pixel region 12 is not particularly limited.

[0033] Figure 3 1 is a diagram illustrating a configuration example of the circuit substrate 21. The circuit substrate 21 includes a signal processing circuit 103, a readout circuit 112, a control pulse generating unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114. Figure 2 The charge photoelectrically converted by each photoelectric conversion unit 102 in FIG.

[0034] The vertical scanning circuit 110 receives the control pulse supplied from the control pulse generating unit 115 and sequentially supplies the control pulse to a plurality of pixels arranged in a row direction. In the vertical scanning circuit 110, a logic circuit such as a shift register and an address decoder is used.

[0035] The signal output from the photoelectric conversion unit 102 of each pixel is processed by each signal processing circuit 103. The signal processing circuit 103 is provided with a counter, a memory, etc., and holds a digital value in the memory. The horizontal scanning circuit 111 inputs a control pulse for sequentially selecting each column to the signal processing circuit 103 so as to read the signal from the memory of each pixel holding the digital signal.

[0036] A signal is output from the signal processing circuit 103 of the pixel in the row selected by the vertical scanning circuit 110 to the vertical signal line 113. The signal output to the vertical signal line 113 is output to the outside of the photoelectric conversion element 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has a plurality of buffers built therein and connected to the vertical signal line 113.

[0037] like Figure 2 and Figure 3 As shown, a plurality of signal processing circuits 103 are arranged in a region overlapping with the pixel region 12 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in a manner overlapping between an end portion of the sensor substrate 11 and an end portion of the pixel region 12 in a plan view.

[0038] That is, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. The vertical scanning circuit 110, the horizontal scanning circuit 111, the readout circuit 112, the output circuit 114 and the control pulse generating unit 115 are arranged in a region overlapping the non-pixel region in a plan view.

[0039] Meanwhile, the configuration of the vertical signal line 113 and the configuration of the readout circuit 112 and the output circuit 114 are not limited to Figure 3 For example, the vertical signal line 113 may be arranged to extend in the row direction, and the readout circuit 112 may be arranged at the end of the vertical signal line 113. In addition, it is not necessary to set a signal processing circuit 103 for each photoelectric conversion unit, and a structure in which one signal processing unit is shared by multiple photoelectric conversion units and performs signal processing sequentially may be used.

[0040] Figure 4 It's a picture. Figure 2 and Figure 3 FIG. 1 is a diagram of an equivalent circuit of a pixel 101 in FIG. 1 and a signal processing circuit 103 corresponding to the pixel 101.

[0041] The APD 201 included in the photoelectric conversion unit 102 generates a pair of charges corresponding to the incident light through photoelectric conversion. One of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VL (first voltage) is supplied. In addition, the other of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VH (second voltage) higher than the voltage VL is supplied.

[0042] exist Figure 4 In the embodiment, one node of the APD 201 is an anode and the other node of the APD is a cathode. A reverse bias voltage is provided to the anode and cathode of the APD 201 so that the APD 201 performs an avalanche multiplication operation. By providing such a voltage, the charge generated by the incident light causes avalanche multiplication and generates an avalanche current.

[0043] Meanwhile, if a reverse bias voltage is provided, there are two modes: a Geiger mode in which operation is performed when the voltage difference between the anode and the cathode is greater than the breakdown voltage; and a linear mode in which operation is performed when the voltage difference between the anode and the cathode is close to or less than the breakdown voltage. An APD operating in the Geiger mode is called a SPAD. For example, in the case of a SPAD, the driving voltage VL (first voltage) is -30V, and the driving voltage VH (second voltage) is 1V.

[0044] The signal processing circuit 103 includes a quenching element 202, a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212. The quenching element 202 is connected to a power supply line to which a driving voltage VH is supplied and one of an anode node and a cathode node of the APD 201.

[0045] During signal multiplication using avalanche multiplication, the quenching element 202 functions as a load loop (quenching loop) and has a function of suppressing the voltage supplied to the APD 201 and suppressing avalanche multiplication (quenching operation). In addition, the quenching element 202 has a function of returning the voltage supplied to the APD 201 to the driving voltage VH by passing a current corresponding to the amount of voltage drop caused by the quenching operation (charging operation).

[0046] Figure 4 An example is shown in which the signal processing circuit 103 includes a waveform shaping unit 210 , a counter circuit 211 , and a memory circuit 212 in addition to the quenching element 202 .

[0047] The waveform shaping unit 210 shapes the change in cathode voltage of the APD 201 obtained during photon detection and outputs a pulse signal. As the waveform shaping unit 210, for example, an inverter circuit is used. Figure 4 An example of using one inverter as the waveform shaping unit 210 is shown, but a circuit having a plurality of inverters connected in series may be used, or any other circuit having a waveform shaping effect may be used.

[0048] The counter circuit 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. In addition, when the control pulse RES is supplied via the driving line 213, the signal held in the counter circuit 211 is reset. Here, the counter circuit 211 generates a signal based on the difference between the count values ​​at the start and end of the accumulation period.

[0049] The control pulse SEL is Figure 4 The drive line 214 in Figure 3 ) from Figure 3 The vertical scanning circuit 110 in the pixel is provided to the memory circuit 212 to switch the electrical connection and non-connection between the counter circuit 211 and the vertical signal line 113. The memory circuit 212 plays the role of a memory for temporarily storing the count value of the counter, and outputs an output signal from the counter circuit 211 of the pixel to the vertical signal line 113.

[0050] Meanwhile, a switch such as a transistor may be arranged between the quenching element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103 to switch the electrical connection. Similarly, the drive voltage VH or the drive voltage VL provided to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0051] Figure 5 2 is a diagram schematically illustrating the relationship between the operation and output signal of the APD 201. The input side of the waveform shaping unit 210 is defined as a node A, and the output side thereof is defined as a node B. Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201. If a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quenching element 202, and the voltage of the node A drops.

[0052] If the amount of voltage drop becomes larger and the potential difference applied to the APD 201 becomes smaller, as shown at time t2, avalanche multiplication of the APD 201 stops, and the voltage level of the node A does not drop below a certain value.

[0053] After that, between time t2 and time t3, current flows through node A to compensate for the voltage drop from voltage VL, and node A settles at the original potential level at time t3. In this case, the portion of the output waveform at node A that exceeds a specific threshold is shaped by the waveform shaping unit 210 and output as a pulse signal at node B.

[0054] Next, the photoelectric conversion device 600 and the movable device 700 according to the embodiment will be described. Figure 6 is a functional block diagram of a photoelectric conversion device 600 and a movable device 700 according to an embodiment. Figure 6 Some of the functional blocks shown are realized by causing a computer (not shown) included in the photoelectric conversion apparatus 600 and the movable device 700 to execute a computer program stored in a memory (not shown) as a storage medium.

[0055] However, some or all of the functional blocks may be implemented by hardware. Examples of hardware that can be used include application specific circuits (ASIC), processors (reconfigurable processors, DSP), and the like.

[0056] also, Figure 6 The functional blocks shown do not have to be built into the same housing and may be constructed as independent devices connected to each other via signal paths. Figure 6 The above description also applies to the following description Fig. 7A , Figure 7B and Fig. 8A Function block.

[0057] The photoelectric conversion device 600 includes a photoelectric conversion element 100, which includes a photoelectric conversion element for performing photoelectric conversion on an optical image. Figures 1 to 5 In addition, the photoelectric conversion device 600 includes an imaging optical system 601, an image processing unit 602, a frame memory 603, a camera control unit 604, a storage unit 605, a communication unit 606, and the like.

[0058] The photoelectric conversion device of the embodiment is mounted on the mobile device 700, and the camera unit composed of a set of imaging optical system 601 and photoelectric conversion element 100 is configured to capture images in at least one direction (for example, in front, behind and to the side of the mobile device). At the same time, a plurality of camera units can be provided in the mobile device 700.

[0059] Meanwhile, in the present embodiment, for example, the movable device 700 will be described using an example of a car, but the movable device may be any other type, such as an airplane, a train, a ship, a drone, an AGV, or a robot, as long as it can move.

[0060] The function of the image processing unit 602 will be described in detail. Fig. 7A 602 is a functional block diagram illustrating a configuration example of the image processing unit 602. Fig. 7A As shown, the image processing unit 602 includes at least a light emission crosstalk correction unit 607 , a short second (short period) image generation unit 608 , and an HDR (High Dynamic Range) synthesis unit 609 .

[0061] The light emission crosstalk correction unit 607 is used to correct the light emission crosstalk in which signals are detected in adjacent pixels due to light emission in the high electric field region. As a method of correcting the light emission crosstalk, a known correction method based on theoretical values ​​can be used, or other methods based on actual measurement results can be used.

[0062] After light emission crosstalk is corrected by the light emission crosstalk correction unit 607, the image signal is output to the frame memory 603, the short second image generation unit 608, and the HDR synthesis unit 609. The functional blocks to which the image signal is output will be described later.

[0063] The frame memory 603 is used to temporarily store image signals when performing various types of image processing using the image processing unit 602. For example, when a new image is generated using a plurality of images, it is used to temporarily store a plurality of image signals as original images. In the present embodiment, the frame memory 603 is used when generating a short second image signal to be described later or performing HDR synthesis.

[0064] The short second image generation unit 608 is used to generate a short second image signal required when generating an HDR image, and generates a short second image signal based on the image signal output from the light emission crosstalk correction unit 607 and the image signal read out from the frame memory 603. Figure 7B The method of generating the short second image signal will be described in detail. The short second image signal generated by the short second image generating unit 608 is output to the HDR combining unit 609 .

[0065] The HDR synthesis unit 609 generates a final HDR image using the short second image signal generated by the short second image generation unit 608 and the long second (long period) image signal output from the light emission crosstalk correction unit 607. Fig. 8A The details of the method for generating an HDR image are described.

[0066] Meanwhile, in the present embodiment, the image processing unit 602 is composed of three blocks, namely, a light emission crosstalk correction unit 607, a short second image generation unit 608, and an HDR synthesis unit 609. However, for example, image processing such as black level correction, gamma curve correction, noise reduction, digital gain adjustment, demosaicing, and data compression may be performed on the image signal output from the photoelectric conversion element 100.

[0067] In addition, if the photoelectric conversion element 100 has an on-chip color filter such as RGB, it is desirable to perform processing such as white balance correction and color conversion in the image processing unit 602. The HDR image generated by the HDR synthesis unit 609 in the image processing unit 602 is output to the camera control unit 604 and the electrical control unit (ECU) 701 in the movable device 700.

[0068] The camera control unit 604 incorporates a CPU as a computer and a memory in which a computer program is stored, and controls each unit of the photoelectric conversion apparatus 600 by the CPU executing the computer program stored in the memory.

[0069] Meanwhile, the camera control unit 604 controls the length of the exposure period of each frame of the photoelectric conversion element 100 , the timing of the control signal CLK, and the like, for example, through a control pulse generating unit of the photoelectric conversion element 100 .

[0070] Furthermore, the camera control unit 604 also performs various processes for generating an HDR image using the image processing unit 602 and the frame memory 603, controls writing of the image signal output from the HDR synthesis unit 609 to the storage unit 605, etc. In addition, it also communicates with the ECU 701 in the movable device 700.

[0071] For example, the storage unit 605 includes a recording medium such as a memory card or a hard disk, and can store and read out the final image signal. The communication unit 606 includes a wireless or wired interface, and outputs the generated image signal to the outside of the photoelectric conversion device 600 and receives various signals from the outside. For example, as an interface, wired communication according to a standard such as SPI or I2C can be used, or a wireless LAN system such as Wi-Fi, Bluetooth (registered trademark), etc. can be used.

[0072] The ECU 701 incorporates a CPU as a computer and a memory in which a computer program is stored, and controls each unit of the movable device 700 by the CPU executing the computer program stored in the memory.

[0073] The output of ECU 701 is supplied to vehicle control unit 702 and display unit 703. Vehicle control unit 702 functions as a motion control unit that performs motion control such as driving, stopping, and direction control on a vehicle as a movable device based on the output of ECU 701.

[0074] In addition, the display unit 703 includes a display element such as a liquid crystal device or an organic EL, for example, and is mounted on the movable device 700. The display unit 703 displays an image acquired by the photoelectric conversion element 100 and various types of information related to the driving condition of the vehicle, etc. to the driver of the movable device 700 using, for example, a GUI based on the output of the ECU 701.

[0075] at the same time, Figure 6 The image processing unit 602, the frame memory 603, etc. in the mobile device 700 do not have to be mounted on the mobile device 700. For example, they can be set in an external terminal that is set separately from the mobile device 700 and is used to remotely control the mobile device 700 or to monitor the travel of the mobile device. In addition, a recognition unit can be set in either the photoelectric conversion device 600 or the mobile device 700, and the recognition result can be used to control the vehicle, notify the driver, etc.

[0076] Fig. 9 is a diagram illustrating processing from photoelectric conversion to image output performed by the camera control unit 604 according to the embodiment.

[0077] Meanwhile, in the present embodiment, a frame having a length of 33.3 ms is referred to as a full frame (or main frame), and a full frame divided into four parts is referred to as a frame (or subframe). In the present embodiment, a full frame having a length of 33.3 ms is divided into four parts.

[0078] That is to say, Fig. 9As shown, the full frame 1 is divided into frames 1_1, 1_2, 1_3 and 1_4 of equal duration (8.33ms). Fig. 9 And in subsequent figures, frame 1_1, frame 1_2, frame 1_3, frame 1_4, etc. are represented as F1_1, F1_2, F1_3, F1_4, etc.

[0079] Meanwhile, frame 1_1 has an accumulation period from time T0 to time T1 of the start of full frame 1, frame 1_2 has an accumulation period from time T0 to time T2. In addition, frame 1_3 has an accumulation period from time T0 to time T3, and frame 1_4 has an accumulation period from time T0 to time T4.

[0080] At the same time, the number of divisions can be N (integer). Therefore, the present embodiment only needs to have the first to Nth (N≥2) accumulation periods in one full frame. The 1st to (N-1)th accumulation periods are shorter than the Nth accumulation period, and the 1st to (N-1)th image signals generated during the 1st to (N-1)th accumulation periods only need to be controlled to be output during the period from the end of the 1st accumulation period to the end of the Nth accumulation period.

[0081] At the same time, if Fig. 9 As shown, the 1st to (N-1)th accumulation periods overlap with the Nth accumulation period, and the 1st to (N-1)th accumulation periods start simultaneously with the Nth accumulation period. In addition, the end of the Nth accumulation period coincides with the end of the full frame.

[0082] First, the processing flow of frame 1_1 will be described. At time T0, the camera control unit 604 resets the counter circuit 211 in the photoelectric conversion element 100, performs period accumulation for the period from time T0 to time T1, and then reads out the count value C1_1 for the period from time T1 to time T2. Specifically, after C1_1 is temporarily stored in the memory circuit 212, it is output from the photoelectric conversion element to the image processing unit 602 via the buffer of the readout circuit 112.

[0083] C1_1 read out from the photoelectric conversion element 100 is subjected to the above-described light emission crosstalk correction by the light emission crosstalk correction unit 607 within the image processing unit 602 , and the corrected image signal is quickly written in the area of ​​the frame memory 1 within the frame memory 603 .

[0084] That is, the count value C1_1 of each pixel of frame 1_1 starts to be written into frame memory 603 at time T1, and writing of C1_1 of the entire row is completed at time T2. The image information of C1_1 is retained in frame memory 603 as it is.

[0085] In this embodiment, for ease of explanation, it is assumed that the delay time associated with the processing in the image processing unit 602 is short and negligible. However, if the delay time is not negligible, writing in the frame memory 603 does not necessarily need to be completed before time T2.

[0086] Subsequently, for frame 1_2, the period accumulation of the period from time T0 to time T2 is similarly performed, and then the count value C1_2 for the period from time T2 to time T3 is read out. After performing various processes, the count value is written to the area of ​​frame memory 2 within frame memory 603. Then, the image information of frame 1_2 is then held in frame memory 603.

[0087] For frame 1_3, the period accumulation of the period from time T0 to time T3 is similarly performed, and then the count value C1_3 for the period from time T3 to time T4 is read out. After performing various processes, the count value is written in the area of ​​frame memory 3 within frame memory 603. Then, the image information of frame 1_3 is held in frame memory 603.

[0088] For frame 1_4, period accumulation of the period from time T0 to time T4 of full frame 1 is performed, and then count value C1_4 of the period from time T0 to time T1 for full frame 2 is acquired. After performing various processes, the count value is output to the short second image generation unit 608.

[0089] The reason for not temporarily writing C1_4 into the frame memory 603 is to quickly generate a short second image signal by the short second image generation unit 608 and perform HDR synthesis processing, shorten the time until the image is output as much as possible, and reduce the capacity of the frame memory 603.

[0090] However, if the processing speed of generating a short second image signal to be described later is lower than the output speed of the image signal output from the light emission crosstalk correction unit 607, an area called frame memory 4 is newly set in the frame memory 603 to hold image information.

[0091] Will refer to Figure 7B The method by which the short second image generating unit 608 generates the short second image signal will be described. Figure 7B : is a functional block diagram illustrating a configuration example of the short second image generating unit 608. At a timing when all the image information of the frame 1_1 to the frame 1_4 is provided, generation of the short second image signal is started.

[0092] In this embodiment, the four count values ​​are used to generate a short second image signal. Specifically, the count values ​​Cn (T0 to T1), Cn (T1 to T2), Cn (T2 to T3) and Cn (T3 to T4) of the four accumulation periods of time T0 to time T1, time T1 to time T2, time T2 to time T3 and time T3 to time T4 are used. At the same time, n represents the full frame number, and in this embodiment, a relationship of n=1 is established.

[0093] The four count values ​​can be calculated using Cn_1 , Cn_2 , and Cn_3 held in the frame memory 603 and Cn_4 output from the light emission crosstalk correction unit 607 .

[0094] The calculation expressions are shown in the following expressions 1 to 4, and in the present embodiment, using Figure 7B The calculation is performed by subtractors 6081, 6082, and 6083. In full frame 1, C1(T0 to T1), C1(T1 to T2), C1(T2 to T3), and C1(T3 to T4) are generated using C1_1, C1_2, C1_3, and C1_4.

[0095] Cn(T0 to T1)=Cn_1...(Expression 1)

[0096] Cn(T1 to T2)=Cn_2-Cn_1... (Expression 2)

[0097] Cn(T2 to T3)=Cn_3-Cn_2 (Expression 3)

[0098] Cn(T3 to T4)=Cn_4-Cn_3 (Expression 4)

[0099] Next, the addition average process is performed using the four calculated count values ​​C1 (T0 to T1), C1 (T1 to T2), C1 (T2 to T3), and C1 (T3 to T4). Specifically, using Figure 7B The adder 6084 shown adds the four count values ​​and further multiplies them by 1 / 4 using the multiplier 6085.

[0100] Therefore, a short second image signal having an accumulation period (8.33 ms) equivalent to 1 / 4 of the long second image signal (33.3 ms) can be generated. That is, even for a subject whose count value is saturated during the accumulation period of the long second image signal, the brightness of the subject can be expressed as grayscale by using the short second image signal (however, the upper limit is up to four times the brightness when the long second image signal is saturated).

[0101] Furthermore, since the count values ​​obtained during four consecutive accumulation periods are added and averaged, the period during which the image of the subject is captured by the short-second image signal becomes the same as the image capturing period of the long-second image signal in a pseudo manner, and thus the position of the subject existing in the image signal does not shift.

[0102] In addition, even if there is a subject moving at high speed, the amount of blur will be the same, so there will be no visual incongruity even if HDR synthesis to be described later is performed. In addition, even if there is a subject that flickers periodically, such as an LED or fluorescent lamp, the accumulation period of the short-second image signal can be extended in a pseudo manner, so that the LED or fluorescent lamp can be prevented from being completely turned off.

[0103] In vehicle-mounted applications, it is desirable that the accumulation period of temporarily continuous short-second image signals should be 11 ms or longer. That is, when generating an image signal with a short accumulation period by adding and averaging the 1st to Nth image signals, it is desirable to use an image with a continuous accumulation period of 11 ms or longer among the 1st to Nth image signals as a target image for addition and averaging. This makes it possible to prevent the LED or fluorescent lamp from being completely turned off.

[0104] The short second image signal generated by the short second image generation unit 608 is output to the HDR synthesis unit 609. That is, in the present embodiment, an image signal having an accumulation period shorter than that of the Nth image signal is generated using the first to Nth image signals, and a synthesized image signal is generated using the image signal having a short accumulation period and the Nth image signal.

[0105] The HDR synthesis unit 609 generates a final HDR image by synthesizing the long second image signal output from the light emission crosstalk correction unit 607 and the short second image signal generated by the short second image generation unit 608 .

[0106] Next, we will refer to Fig. 8A The HDR synthesis method of this embodiment is described. Fig. 8A is a functional block diagram illustrating a configuration example of the HDR synthesis unit 609. The HDR synthesis unit 609 includes an illumination correction unit 6091 and an image synthesis unit 6092, and has two image signals input thereto: a long second image signal (Cn_4) output from the light emission crosstalk correction unit 607 and a short second image signal output from the short second image generation unit 608.

[0107] The illumination correction unit 6091 is used to match the count value of the image signal accumulated in short seconds to a level equivalent to the count value of the image signal accumulated in long seconds (long period) based on the ratio of the accumulation period. By matching the count value of the long second image signal and the count value of the short second image signal using the illumination correction unit 6091, the two images can be synthesized without any sense of incongruity.

[0108] Figure 8B 6091 is a diagram illustrating the processing content of the illumination correction unit 6091, wherein the horizontal axis represents the light amount of the subject and the vertical axis represents the count value of the image input to the HDR synthesis unit 609. Figure 8B 609A is a graph illustrating count values ​​of long-second image signals relative to the light amount of an object, and 609B is a graph illustrating count values ​​of short-second image signals relative to the light amount of an object.

[0109] Furthermore, 609B′ represents a graph after illuminance correction processing is performed on the count value of 609B by the illuminance correction unit 6091. Furthermore, Sat written on the vertical axis represents the saturation count value of the photoelectric conversion element 100.

[0110] Since 609A has a long accumulation period, if the light amount of the subject increases, 609A quickly reaches the Sat level. Since 609B has a short accumulation period, if the light amount of the subject is greater than that of 609A, 609B reaches the Sat level.

[0111] The illumination correction unit 6091 multiplies the count value of 609B by a predetermined gain (amplification factor) to match the count values ​​of 609A and 609B. Specifically, the result obtained by dividing the accumulation period of the long second accumulation period by the accumulation period of the short second accumulation period is calculated as the gain, and the gain is directly multiplied by the count value of the short second image signal.

[0112] That is, in this embodiment, 4.0 obtained by dividing 33.3ms by 8.33ms is calculated as the gain value, and the count value of the short second image signal is multiplied by 4.0. By doing so, the count value of the long second image signal and the count value of the short second image signal become the same count value (unless the count value is saturated).

[0113] The image synthesis unit 6092 synthesizes the count value of 609A and the count value of 609B' to generate a single dynamic range expanded image obtained by synthesizing the long second image signal and the short second image signal. In this way, the image synthesis unit 6092 generates a synthesized image signal by synthesizing two or more image signals having different accumulation periods in the first to Nth image signals.

[0114] Figure 8C 6 is a diagram illustrating a method of generating a composite image using the image synthesis unit 6092, wherein the horizontal axis represents the count value of the input image and the vertical axis represents the synthesis ratio of the long second image signal and the short second image signal. In addition, the count values ​​α to β on the horizontal axis are count values ​​just near the switching between the long second image signal and the short second image signal, and ideally, the count value of β is preferably set to Figure 8BCount values ​​near Sat in .

[0115] If the count value of the input image 609A is equal to or less than α, the count value of the input image 609A of the long second image signal is used 100%. If the count value of the input image 609A is between α and β, the count value of the input image 609A of the long second image signal and the count value 609B' of the short second image signal are multiplied by Figure 8C The composite ratios are shown in the graph and the results are added together.

[0116] If the count value of the input image 609A is equal to or greater than β (saturated), the count value of 609B' of the short second image signal is used 100%. By performing this processing, the long second image signal obtained by long second accumulation and the short second image signal obtained by short second accumulation can be synthesized, and a single HDR image with an expanded dynamic range can be generated.

[0117] In this embodiment, in order to separately explain the functions of the short second image generation unit 608 and the HDR synthesis unit 609, the short second image generation unit 608 is multiplied by 1 / 4 and the HDR synthesis unit 609 is multiplied by a gain of 4.0, but these two processes can be omitted. However, if the addition averaging process is not performed in the short second image generation unit 608, these two processes cannot be omitted.

[0118] For example, if any one of a plurality of acquired short second image signals is used to generate an HDR image, a division process is not required in the short second image generation unit 608, and thus a process of multiplying a gain is required in the HDR synthesis unit 609. The final HDR image generated by the image synthesis unit 6092 is output to the camera control unit 604 and the ECU 701 in the movable device 700.

[0119] Fig. 10A is a diagram illustrating an example of an image of a plurality of segmented frames. Fig. 10A As shown, because the accumulation period is short, the image of frame 1_1 is dark, and there is almost no motion (blur) of the subject of the person who jumps out.

[0120] On the other hand, the accumulation period becomes longer in the order of frame 1_2, frame 1_3, frame 1_4, so the motion of the subject of the person jumping out appears blurred. At the same time, the stopped vehicle and the white line are less likely to be blurred, and as the accumulation period becomes longer, the contrast is more likely to be improved.

[0121] Fig. 10B It is a diagram showing the use of Fig. 10A The image signals (count values) of the short second image generation unit 608 generate images C1 (T0 to T1), C1 (T1 to T2), C1 (T2 to T3), and C1 (T3 to T4).

[0122] Since the accumulation period for all images is 8.33ms, the brightness (count value) and blur amount of the subject are the same. However, the position of the person who jumped out has changed. Specifically, the position of the person moves to the right in the order of C1 (T0 to T1), C1 (T1 to T2), C1 (T2 to T3), and C1 (T3 to T4).

[0123] Fig. 10C is a diagram illustrating a short second image signal generated by the short second image generation unit 608 and a long second image signal output from the light emission crosstalk correction unit 607 as an image. Fig. 10B It is obtained by adding all 4 count values ​​in once, so the movement (blur amount) of the person who jumps out is large and consistent with the blur amount of the long second image signal.

[0124] Furthermore, after the count values ​​are added, they are divided by the number of frames so that the brightness of the count values ​​after addition and averaging becomes 1 / 4 of the brightness of the long second image signal.

[0125] Fig.11 is a diagram illustrating a relationship between a memory loop and a buffer according to an embodiment. Fig.11 Shows the Figure 3 The memory loops 212 in the signal processing loop 103 are arranged in a state of N rows and M columns, wherein each memory loop is represented as memory 1-1 to memory NM.

[0126] also, Fig.11 The buffers 1 to M in the table represent the buffers included in Figure 3 The buffer in the readout circuit 112 in . Fig.11 The output circuit 114 in corresponds to Figure 3 The output loop 114 in.

[0127] Fig.12 is a flowchart illustrating details of a driving example of a photoelectric conversion element in this embodiment, and Fig.13 yes Fig.12 At the same time, the CPU or the like as a computer in the camera control unit 604 executes the computer program stored in the memory, thereby sequentially executing Fig.12 and Fig.13 The operations of the steps in the flowchart.

[0128] exist Fig.12 In step S1201, the relationship i=1 is set. Next, in step S1202, the count value Count of the counter loop 211 at the time Ti is output to the memory loop 212. In this case, all memory loops perform output at the same time. This operation corresponds to Fig. 9 The operation at time T1.

[0129] Next, in step S1203, the relationship j = 1 is set, and in step S1204, the Fig.11 count value Count(j - k - i) in the memory circuit j - k in is output to the buffer. In this case, columns 1 to M are output to the buffer simultaneously. This operation means that Fig.11 the count value of the first row in is fetched into the buffer.

[0130] Next, in step S1205, the relationship k = 1 is set, and in step S1206, the count value Count(j - k - i) of buffer k is output to the output circuit 114. This operation corresponds to the operation of reading the signal of the buffer in the leftmost column in Fig.11 .

[0131] Next, the process proceeds via A to Fig.13 step S1207 in, where it is determined whether k < M. If so, then in step S1208, k is incremented by 1 as k = k + 1, the process returns via B to step S1206, and the operation of step S1206 is executed. This operation corresponds to the operation of reading the signal of the buffer in the second column from the left in Fig.11 .

[0132] If the answer in step S1207 is no, that is, if k = M, it means that the signal of the buffer in the Mth column in Fig.11 has been read out. Next, the process proceeds to step S1209 to determine whether j < N.

[0133] If the answer in step S1209 is yes, then in step S1210, the relationship j = J + 1 is set, j is incremented by 1, and the process returns via C to step S1204. This corresponds to the operation of starting to read the next row.

[0134] If it is determined in step S1209 that the answer is no, it means that the reading of all rows has been completed. Therefore, the process proceeds to step S1211 to determine whether i < 4. If it is determined in step S1211 that the answer is yes, the process proceeds to step S1212, the relationship i = i + 1 is set, i is incremented by 1, and the process returns via D to step S1202. This operation corresponds to the operation of starting the next read at time T2.

[0135] If it is determined in step S1211 that the answer is no, it means that the reading at time T4 is completed. Therefore, the process proceeds to step S1213, where the counter circuit 211 is reset using a reset signal. This operation corresponds to the counter circuit 211 at Fig. 9In this way, the signals accumulated in the photoelectric conversion element 100 can be sequentially read out.

[0136] Meanwhile, in the above embodiment, the accumulation period of the full frame is divided into four parts, but it is not necessary to divide the accumulation period into four parts. If it is desired to enhance the effect based on HDR, the dynamic range can be further expanded by increasing the number of divisions.

[0137] For example, if the accumulation period for acquiring a full frame of a short second image signal is divided into ten parts, grayscale can be expressed to a light amount 10 times (4.0 times in the present embodiment) the saturated brightness of a long second image signal.

[0138] Meanwhile, if the number of divisions of the accumulation period is increased by using a CMOS sensor, read noise will occur each time an image is read out, resulting in a decrease in night vision performance, whereas in this embodiment, since a SPAD sensor is used, read noise will not occur. Therefore, if the number of divisions is large, it is desirable to use a SPAD sensor.

[0139] Furthermore, if it is desired to reduce the capacity of the frame memory, the number of divisions of the accumulation period for acquiring the short second image signal can be reduced. By doing so, the number of frames of the image signal for temporarily storing the image signal can be reduced, which makes it possible to reduce the capacity of the frame memory. Furthermore, if only one short second image signal is used, the short second image generation unit 608 is not required.

[0140] In addition, in this embodiment, in order to obtain a short second image signal, the full frame period is divided into four parts. That is, the accumulation period of a full frame is divided into N accumulation periods and N image signals are output, but it is not necessary to divide the accumulation period into N periods. The accumulation period of the short second image signal can be determined according to the expansion ratio required for the dynamic range.

[0141] In addition, in the present embodiment, a short second image signal is generated by adding and averaging four (the number of divisions of the accumulation period) image signals. That is, when an image signal having an accumulation period shorter than the Nth image signal is generated, the 1st to Nth image signals are added and averaged to generate an image signal having a shorter accumulation period. However, it is not necessary to use all images corresponding to the number of divisions of the accumulation period. Only one image signal may be used, or any other number of images may be used.

[0142] For example, an image signal accumulated during a period closest to the center of the accumulation period of the long second image signal may be used. That is, when an image signal having an accumulation period shorter than that of the Nth image signal is generated, an image signal having an accumulation period shorter than that of the Nth accumulation period and closest to the center of the Nth accumulation period may be used to generate a synthetic image signal. This makes it possible to reduce the capacity of the frame memory.

[0143] Alternatively, a composite image can be generated using three image signals accumulated during three periods in the accumulation period of the long second image signal, that is, at the beginning, center and end of the accumulation period. This makes it possible to achieve a good balance between reducing the frame memory capacity and suppressing the position deviation of the subject.

[0144] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

[0145] In addition, as a part or the whole of the control according to the embodiment, a computer program that realizes the functions of the above-mentioned embodiment can be provided to the photoelectric conversion device, etc. through a network or various storage media. Then, the computer (or CPU, MPU, etc.) of the photoelectric conversion device, etc. can be configured to read and execute the program. In this case, the program and the storage medium storing the program configure the present invention.

[0146] In addition, the present invention includes those implemented using at least one processor or circuit configured to perform the functions of the above-mentioned embodiments. For example, multiple processors can be used for distributed processing to perform the functions of the above-mentioned embodiments.

[0147] This application claims priority from Japanese Patent Application No. 2023-147319 filed on September 12, 2023, the entire contents of which are incorporated herein by reference.

Claims

1. A photoelectric conversion device, comprising: a photoelectric conversion unit that emits a pulse corresponding to the photon; a counter for counting the number of said pulses; a plurality of pixels, each pixel comprising a memory storing a count value of the counter; a control device for generating an image signal based on a difference between count values ​​of the counter at the start and end of an accumulation period, wherein, during a period from the end of the 1st accumulation period to the end of the Nth accumulation period, the control device outputs 1st to (N-1)th image signals generated during the 1st to (N-1)th accumulation periods, wherein the 1st to Nth accumulation periods are within one full frame period and each of the 1st to (N-1)th accumulation periods is shorter than the Nth accumulation period, wherein N≥2; and An image synthesis device generates a synthesized image signal by synthesizing two or more image signals having different accumulation periods among the 1st to Nth image signals.

2. The photoelectric conversion device according to claim 1, wherein The 1st to (N-1)th accumulation periods and the Nth accumulation period overlap with each other.

3. The photoelectric conversion device according to claim 1, wherein The 1st to (N-1)th accumulation periods start simultaneously with the Nth accumulation period.

4. The photoelectric conversion device according to claim 1, wherein The end of the Nth accumulation period coincides with the end of the full frame.

5. The photoelectric conversion device according to claim 1, wherein The control device generates an image signal having a shorter accumulation period than the Nth image signal using the 1st to Nth image signals, and generates a composite image signal using the image signal having the short accumulation period and the Nth image signal.

6. The photoelectric conversion device according to claim 1, wherein The control device equally divides an accumulation period of one full frame into N accumulation periods and outputs N image signals.

7. The photoelectric conversion device according to claim 6, wherein: When generating an image signal having a shorter accumulation period than the Nth image signal, the control means generates an image signal having a short accumulation period by adding and averaging the first to Nth image signals.

8. The photoelectric conversion device according to claim 7, wherein: When generating an image signal with a short accumulation period by adding and averaging the 1st to Nth image signals, the control device uses an image with a continuous accumulation period of 11 ms or longer among the 1st to Nth image signals as a target image for adding and averaging.

9. The photoelectric conversion device according to claim 1, wherein When generating an image signal having an accumulation period shorter than the Nth image signal, the control means generates a synthetic image signal using an image signal having an accumulation period shorter than the Nth accumulation period and closest to a center of the Nth accumulation period.

10. The photoelectric conversion device according to claim 1, wherein The photoelectric conversion unit includes an avalanche photodiode.

11. A movable device comprising: a photoelectric conversion unit that emits a pulse corresponding to the photon; a counter for counting the number of said pulses; a plurality of pixels, each pixel comprising a memory storing a count value of the counter; a control device for generating an image signal based on a difference between count values ​​of the counter at the start and end of an accumulation period, wherein, during a period from the end of the 1st accumulation period to the end of the Nth accumulation period, the control device outputs 1st to (N-1)th image signals generated during the 1st to (N-1)th accumulation periods, wherein the 1st to Nth accumulation periods are within one full frame period and each of the 1st to (N-1)th accumulation periods is shorter than the Nth accumulation period, wherein N≥2; and an image synthesizing device that generates a synthesized image signal by synthesizing two or more image signals having different accumulation periods among the 1st to Nth image signals; and A motion control device is used to control the motion of the movable device.

12. A method for controlling a photoelectric conversion device, the photoelectric conversion device comprising: a photoelectric conversion unit that emits a pulse corresponding to the photon; a counter for counting the number of said pulses; and a plurality of pixels, each pixel comprising a memory storing a count value of the counter, Wherein, the method comprises: generating an image signal based on a difference between count values ​​of the counter at the beginning and the end of an accumulation period; outputting, during a period from the end of the 1st accumulation period to the end of the Nth accumulation period, 1st to (N-1)th image signals generated during the 1st to (N-1)th accumulation periods, wherein the 1st to Nth accumulation periods are within one full frame period and each of the 1st to (N-1)th accumulation periods is shorter than the Nth accumulation period, wherein N≥2; and A synthesized image signal is generated by synthesizing two or more image signals having different accumulation periods among the 1st to Nth image signals.

13. A non-transitory computer-readable storage medium storing a computer program including instructions for performing the following processing: To control a photoelectric conversion device including a photoelectric conversion unit that emits pulses corresponding to photons, a counter that counts the number of the pulses, and a plurality of pixels each including a memory that stores count values ​​of the counter, generating an image signal based on a difference between count values ​​of the counter at the start and end of an accumulation period; During a period from the end of the 1st accumulation period to the end of the Nth accumulation period, the 1st to (N-1)th image signals generated during the 1st to (N-1)th accumulation periods are output, wherein, The first to Nth accumulation periods are within a full frame period, and each of the first to (N-1)th accumulation periods is shorter than the Nth accumulation period, wherein N≥2; as well as A synthesized image signal is generated by synthesizing two or more image signals having different accumulation periods among the 1st to Nth image signals.

14. A computer program product comprising a computer program, the computer program comprising instructions for performing the following processing: To control a photoelectric conversion device including a photoelectric conversion unit that emits pulses corresponding to photons, a counter that counts the number of the pulses, and a plurality of pixels each including a memory that stores count values ​​of the counter, generating an image signal based on a difference between count values ​​of the counter at the start and end of an accumulation period; During a period from the end of the 1st accumulation period to the end of the Nth accumulation period, the 1st to (N-1)th image signals generated during the 1st to (N-1)th accumulation periods are output, wherein, The 1st to Nth accumulation periods are within one full frame period, and each of the 1st to (N-1)th accumulation periods is shorter than the Nth accumulation period; as well as A synthesized image signal is generated by synthesizing two or more image signals having different accumulation periods among the 1st to Nth image signals.

Images