Photoelectric conversion device, photoelectric conversion system, mobile object, and apparatus

Abstract

To provide a technique for suppressing impact due to alias (IR alias) resulting from IR light incident on RGB pixels.SOLUTION: A photoelectric conversion device according to this invention includes a first pixel having an avalanche photodiode for acquiring IR light information and a second pixel having an avalanche photodiode for acquiring visible light information. The first pixel and the second pixel respectively include a photon detection circuit for detecting a photon signal and outputting a photon detection signal, and a pixel counter for counting the photon detection signal. The first pixel transfers the photon signal or the photon detection signal in the first pixel to the second pixel, and causes the pixel counter in the second pixel not to change the count value or to decrement the count value.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a photoelectric conversion device, a photoelectric conversion system, a mobile object, and an apparatus. [Background technology]

[0002] RGB-IR sensors are known that can simultaneously capture infrared (IR) images and visible light images (red / green / blue, or RGB). In RGB-IR sensors, an IR filter for capturing IR images and an RGB filter for capturing visible light images are mounted on each pixel. In recent years, imaging sensors using avalanche photodiodes (APDs) have been widely proposed. Patent Document 1 describes a single photon absorber (SPAD) as an imaging element. The configuration of an RGB-IR sensor using an avalanche diode element is disclosed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2022-549577 Summary of the Invention [Problem to be solved by the invention]

[0004] In order for an RGB-IR sensor to simultaneously capture an IR image and a visible light image, IR light and visible light are incident on the sensor surface at the same time. Because RGB pixels have IR sensitivity, when IR light is incident, the IR light is mixed in as a false signal, causing the problem of not being able to obtain a correct RGB image signal. Patent Document 1 describes the incorporation of light incident on adjacent pixels, but does not consider in detail the mixed false signal.

[0005] The present invention has been made in view of the above problems, and its purpose is to provide a technology for suppressing the influence of false signals (IR false signals) caused by IR light incident on RGB pixels in a photoelectric conversion device including an avalanche photodiode. [Means for solving the problem]

[0006] A first aspect of the present invention is a photoelectric conversion device comprising a first pixel having an avalanche photodiode that acquires IR light information, and a second pixel having an avalanche photodiode that acquires visible light information, wherein each of the first pixel and the second pixel has a photon detection circuit that detects a photon signal and outputs a photon detection signal, and a pixel counter that counts the photon detection signal, and wherein the first pixel transmits the photon signal or photon detection signal in the first pixel to the second pixel, and either does not change the count value of the pixel counter in the second pixel, or subtracts the count value.

[0007] A second aspect of the present invention is a photoelectric conversion device comprising: a first pixel having an avalanche photodiode that acquires IR light information; and a second pixel having an avalanche photodiode that acquires visible light information; each of the first pixel and the second pixel having a photon detection circuit that detects a photon signal and outputs a photon detection signal; and a pixel counter that counts the photon detection signals; the first pixel transmits flag information corresponding to the photon signal or photon detection signal of the first pixel to the second pixel; and the second pixel outputs the flag information together with a count value of the photon detection signal of the second pixel.

[0008] A third aspect of the present invention is a photoelectric conversion system comprising the above-described photoelectric conversion device and a signal processing unit that generates an image using a signal output by the photoelectric conversion device.

[0009] A fourth aspect of the present invention is a mobile body equipped with the above-mentioned photoelectric conversion device, characterized in that it has a control unit that controls the movement of the mobile body using a signal output by the photoelectric conversion device.

[0010] A fifth aspect of the present invention is an apparatus characterized by having the above-mentioned photoelectric conversion device and at least one of an optical device corresponding to the photoelectric conversion device, a control device that controls the photoelectric conversion device, a processing device that processes signals output from the photoelectric conversion device, a display device that displays information obtained by the photoelectric conversion device, a memory device that stores information obtained by the photoelectric conversion device, and a mechanical device that operates based on information obtained by the photoelectric conversion device. [Effects of the Invention]

[0011] According to the present invention, the influence of IR aliasing signals can be suppressed in a photoelectric conversion device including an avalanche photodiode. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 2 is a diagram showing a color filter layout according to an embodiment. [Figure 2] FIG. 1 is a SPAD pixel circuit diagram of a single pixel according to an embodiment. [Figure 3] FIG. 1 is a pixel circuit diagram of 2×2 pixels according to a first embodiment (active driving). [Figure 4] FIG. 1 is a pixel circuit diagram of 2×2 pixels according to a first embodiment (passive driving). [Figure 5] FIG. 10 is a pixel circuit diagram of 2×2 pixels according to the second embodiment. [Figure 6] FIG. 10 is a pixel circuit diagram of 2×2 pixels according to the third embodiment. [Figure 7] FIG. 10 is a pixel circuit diagram of 2×2 pixels according to a fourth embodiment. [Figure 8] FIG. 11 is a timing chart of pixel driving according to the fifth embodiment. [Figure 9] FIG. 10 is a diagram illustrating a photoelectric conversion system according to a sixth embodiment. [Figure 10] FIG. 13 is a diagram illustrating a photoelectric conversion system and a moving object according to a seventh embodiment. [Figure 11] 13A and 13B are diagrams illustrating a range image sensor according to an eighth embodiment. [Figure 12] 13A and 13B are diagrams illustrating an endoscopic surgery system according to a ninth embodiment. [Figure 13] FIG. 20 is a diagram illustrating smart glasses according to a tenth embodiment. [Figure 14] FIG. 20 is a diagram illustrating an electronic device according to an eleventh embodiment. [Figure 15] FIG. 22 is a diagram illustrating a device according to a twelfth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] The embodiments shown below are intended to embody the technical concept of the present invention and are not intended to limit the present invention. The size and positional relationship of components shown in each drawing may be exaggerated for clarity. In the following description, the same components may be designated by the same reference numerals and their description may be omitted.

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, terms indicating specific directions or positions (for example, "up," "down," "right," "left," and other terms containing these terms) will be used as necessary. The use of these terms is intended to facilitate understanding of the embodiments with reference to the drawings, and the meaning of these terms does not limit the technical scope of the present invention.

[0015] A configuration common to all embodiments of the photoelectric conversion device according to the present invention will be described with reference to Figures 1 and 2. Hereinafter, for convenience of explanation, a circuit including an avalanche photodiode may be referred to as a SPAD pixel or a SPAD pixel circuit, but in this disclosure, the avalanche photodiode is not limited to a SPAD.

[0016] Figure 1 shows the color filter arrangement of an RGB-IR sensor using a SPAD element, viewed from above. The RGB-IR sensor has an R pixel filter 1, a G pixel filter 2, a B pixel filter 3, and an IR pixel filter 4. The R, G, and B pixels are collectively referred to as RGB pixels. A pixel with an IR pixel filter and three pixels with RGB pixel filters are arranged in a two-dimensional array, with 2 x 2 pixels as one repeating unit.

[0017] Figure 2 is a diagram of a SPAD pixel circuit for a single pixel. The SPAD pixel circuit includes an APD (avalanche photodiode) 5, a quench / recharge circuit 6, and an inverter (buffer) circuit 7. When light is incident on the APD 5, photoelectric conversion occurs in the photosensitive region, generating charge. A voltage VL is supplied to the anode of the APD 5. A voltage VH, higher than the voltage VL supplied to the anode, is supplied to the cathode of the APD 5. A reverse bias voltage is supplied to the anode and cathode so that the APD 5 performs avalanche multiplication. With this voltage applied, avalanche multiplication occurs when the charge generated in the APD 5 passes through a high-electric field region within the element, generating a large amount of charge. The voltage change due to this charge multiplication is then detected as a photon detection signal via the inverter circuit 7.

[0018] When a reverse bias voltage is supplied to an APD, there is a Geiger mode in which the potential difference between the anode and cathode is greater than the breakdown voltage. There is also a linear mode in which the potential difference between the anode and cathode is close to or less than the breakdown voltage. An APD operating in Geiger mode is called a SPAD. For example, the voltage VL is -30V and the voltage VH is 1V. Or, for example, the voltage VL is 0V and the voltage VH is 33V.

[0019] To restore the APD 5 to a state where it can detect photons again after photon detection, the quench / recharge circuit 6 controls the voltage applied to the APD 5. The quench / recharge circuit can have the following configuration, for example: A switch is connected to a power supply line that supplies the drive voltage VH and to the APD 5. The switch is connected to one of the anode and cathode nodes of the APD 5. The switch then switches the potential difference between the anode and cathode of the APD 5 between a first potential difference at which avalanche multiplication occurs and a second potential difference at which avalanche multiplication does not occur. Hereinafter, switching from the second potential difference to the first potential difference is referred to as "switch on," and switching from the first potential difference to the second potential difference is referred to as "switch off." The switch functions as a quenching element.

[0020] The switch can be configured, for example, by a MOS transistor. A control signal for the switch is applied to the gate electrode of the MOS transistor that configures the switch. By controlling the voltage applied to the gate electrode of the switch, it is possible to control the on and off of the switch.

[0021] The inverter circuit 7 shapes the potential change at the cathode of the APD 5 obtained when a photon is detected and outputs a pulse signal. The node on the input side of the inverter circuit 7 is designated node A, and the node on the output side is designated node B. The inverter circuit 7 changes the output potential from node B depending on whether the input potential to node A is above or below a predetermined value. While Figure 2 shows an example using a single inverter, a circuit in which multiple inverters are connected in series or another circuit with a waveform shaping effect may also be used.

[0022] The photoelectric conversion devices of the respective embodiments will be described below.

[0023] (First to third embodiments) In the first to third embodiments, the photoelectric conversion device includes a first pixel having an IR pixel filter disposed on the light incident surface side and an avalanche photodiode for acquiring IR light information. Also, an RGB pixel filter disposed on the light incident surface side and an avalanche photodiode for acquiring visible light information. The first pixel corresponds to an IR pixel, and the second pixel corresponds to an RGB pixel.

[0024] When the IR pixel (first pixel) detects a photon signal due to IR light, it transmits the signal itself (photon signal) detected by the avalanche photodiode of the IR pixel, or a signal (photon detection signal) that has passed through the IR pixel's photon detection circuit, to the RGB pixel (second pixel).Then, the count value of the pixel counter in the RGB pixel (second pixel) is either left unchanged or is decremented.

[0025] IR light incident on an RGB pixel (second pixel) has a luminous flux with a certain area. In particular, in pixels with small pixel size, IR light is thought to be incident on multiple adjacent pixels and be counted as a false signal. For RGB pixels (second pixel) adjacent to an IR pixel (first pixel) that detects IR light, the photon detection signal output from that RGB pixel is treated as a false signal and not counted, thereby suppressing the effects of IR false signals.

[0026] Among RGB pixels adjacent to an IR pixel, an RGB pixel that detects IR light is deemed to have mistakenly counted a photon signal based on IR light as a photon signal based on visible light. The RGB pixel operates to subtract a count value of the photon detection signal at the RGB pixel based on the photon signal or photon detection signal transmitted from the adjacent IR pixel. On the other hand, an RGB pixel whose adjacent IR pixel did not detect IR light counts the photon detection signal at the RGB pixel regardless of the photon signal or photon detection signal transmitted from the IR pixel. Note that "adjacent" means that pixels are adjacent to each other, and some element may exist between the IR pixel and the RGB pixel.

[0027] The first to third embodiments will be described in detail below.

[0028] (First embodiment) In the first embodiment, the RGB pixel (second pixel) has a signal control circuit. The signal control circuit can control the input of the pixel counter of the RGB pixel (second pixel) based on a photon signal or photon detection signal transmitted from the IR pixel (first pixel). When the IR pixel (first pixel) detects a photon signal due to IR light, the signal control circuit operates to either not change the count value of the pixel counter of the RGB pixel (second pixel) or to subtract from the count value. This configuration is expected to be effective in suppressing the effects of IR aliasing.

[0029] As an example, Figure 3 shows a pixel circuit diagram of a 2x2 pixel RGB-IR sensor. The RGB-IR sensor includes an R pixel 8, a G pixel 9, a B pixel 10, and an IR pixel 11, each with a pixel counter 12 and a photon detection circuit 14. Furthermore, the R pixel 8, the G pixel 9, and the B pixel 10 each have a signal control circuit 13. The arrangement of the pixels (8, 9, 10, 11) in Figure 3 corresponds to the arrangement of the pixels (1, 2, 3, 4) in Figure 1. A single-pixel SPAD pixel circuit, as shown in Figure 2, is provided in each pixel, and the photon detection signal of each pixel is counted by the corresponding pixel counter 12 and then output to the outside of the sensor. The output node of the photon detection circuit 14 of the IR pixel 11 is connected to the signal control circuit 13 of each of the R pixel 8, the G pixel 9, and the B pixel 10.

[0030] As shown in FIG. 3, when the photon detection circuit 14 of the IR pixel (first pixel) detects a photon signal due to IR light, it transmits the photon detection signal of the IR pixel to the signal control circuit 13 of the RGB pixel (second pixel). Then, the signal control circuit 13 of the RGB pixel operates so as not to change the count value of the pixel counter in the RGB pixel (second pixel) or to subtract the count value. With this configuration, it is expected that the effect of suppressing the influence of IR spurious signals can be suppressed. do.

[0031] An example of the signal control circuit 13 is an AND circuit to which a photon detection signal from an RGB pixel (second pixel) and a photon detection signal from an IR pixel (first pixel) via a NOT circuit are input. In the description of this embodiment, the photon detection signal output from the photon detection circuit 14 of the IR pixel (first pixel) is input to the signal control circuit 13 of the RGB pixel (second pixel), but the photon signal from the IR pixel (first pixel) may also be transmitted to the RGB pixel (second pixel).

[0032] 3 is a circuit diagram assuming active driving in which a clock signal is input to the quench / recharge circuit 6 and the photon detection circuit 14 to recharge the SPAD element and detect photons at regular intervals. This configuration makes it possible to capture images favorably even when the subject is highly bright. This configuration can be applied not only to the first embodiment, but also to the second to fifth embodiments described below, and similar effects can be expected.

[0033] On the other hand, Figure 4 is a circuit diagram assuming passive drive, in which the SPAD element operates passively when a photon is incident, without inputting a clock signal to the quench / recharge circuit 6 and the photon detection circuit 15. The effect of the circuit diagram shown in Figure 4 is the same as that of the circuit diagram in Figure 3 described above, and it can suppress the effects of IR aliasing.

[0034] The IR aliasing suppression effect is expected in both the active driving and passive driving methods described above. While the drawings used in the following embodiments will explain the active driving method, the same configuration can be applied to the passive driving method, and similar effects can be expected. Furthermore, the pixel arrangement and pixel circuit can be applied to cases other than a 2x2 pixel arrangement for the RGB pixels adjacent to the IR pixel.

[0035] (Second embodiment) In the second embodiment, the RGB pixels (second pixels) do not have a signal control circuit. The photon detection circuits of the RGB pixels can control the pixel counters of the RGB pixels based on the photon signal or photon detection signal transmitted from the IR pixel (first pixel). When the IR pixel (first pixel) detects a photon signal due to IR light, the count value of the pixel counter in the RGB pixel (second pixel) remains unchanged or is decremented. This configuration is also expected to be effective in suppressing the effects of IR aliasing.

[0036] An example of the second embodiment will be described with reference to FIG. 5. The second embodiment is a modification of the first embodiment. The configuration shown in FIG. 5 differs from the first embodiment in that each of the RGB pixels does not have a signal control circuit 13, and the photon detection circuit 16 has a three-input configuration. A signal based on the detection of a photon in the IR pixel 11 is input to the photon detection circuit 16 of each of the RGB pixels (8, 9, 10). The photon detection circuit 16 of each RGB pixel is configured to perform a logical operation on the signal input from the IR pixel 11, the output of the APD 5 of each RGB pixel, and a clock signal. This configuration is expected to suppress the influence of IR spurious signals without placing a signal control circuit 13 before the pixel counter 12, and can reduce the circuit area.

[0037] As shown in Figure 5, when an IR pixel (first pixel) detects a photon signal due to IR light, the IR pixel's photon signal or photon detection signal is transmitted to the photon detection circuit of the RGB pixel (second pixel). The photon detection circuit of the RGB pixel (second pixel) then operates to either not change the count value of the pixel counter in the second pixel or to subtract from the count value. This configuration is expected to suppress the effects of IR spurious signals.

[0038] (Third embodiment) In the third embodiment, the RGB pixels (second pixels) do not have a signal control circuit. An up-down counter is used as the pixel counter of the RGB pixels (second pixels). The up-down counter operates to count up when the RGB pixels detect a photon signal. When a photon signal or photon detection signal transmitted from the IR pixel (first pixel) is input to the pixel counter of the RGB pixels, the up-down counter operates to count down based on the photon signal or photon detection signal transmitted from the IR pixel. This configuration is also expected to be effective in suppressing the effects of IR aliasing.

[0039] An example of the third embodiment will be described with reference to FIG. 6. In the third embodiment, an up / down counter 17 is used as a pixel counter for RGB pixels. The output signal of the APD 5 of each RGB pixel and the output signal of the IR pixel are input to the up / down counter 17. In the first embodiment, the signal control circuit 13 controls the input to the counter, and in the second embodiment, the photon detection circuit 16 controls the input to the counter. On the other hand, in the third embodiment, the up / down counter 17 counts up the count value of photon detection in the RGB pixels (8, 9, 10), and counts down the count value of photon detection in the adjacent IR pixel 11. This configuration is expected to be effective in suppressing the influence of IR aliasing signals.

[0040] (Fourth embodiment) In the fourth embodiment, the photoelectric conversion device includes a first pixel having an avalanche photodiode for acquiring IR light information and a second pixel having an avalanche photodiode for acquiring visible light information. Each pixel also includes a photon detection circuit that detects photon signals and outputs photon detection signals, and a pixel counter that counts the photon detection signals. As in the first embodiment, the first pixel corresponds to an IR pixel, and the second pixel corresponds to an RGB pixel.

[0041] When an IR pixel (first pixel) detects a photon signal due to IR light, the IR pixel transmits flag information corresponding to the IR pixel's photon signal or photon detection signal to an adjacent RGB pixel (second pixel).The RGB pixel then outputs the flag information along with the count value of the RGB photon detection signal.This configuration is expected to be effective in suppressing the influence of IR aliasing signals.

[0042] An example of the fourth embodiment will be described with reference to FIG. 7. Each of the RGB pixels of the photoelectric conversion device according to the fifth embodiment has a flag information circuit 18. In this embodiment, all photon detection signals generated by the RGB pixels are counted, but when a photon signal is detected by an IR pixel, the result is transmitted as flag information to the flag information circuit 18 of the adjacent RGB pixel. Each of the RGB pixels then outputs flag information to the outside of the sensor along with the count value counted by the pixel counter 12. Using the flag information enables an ISP (image signal processor) that receives the signal output from the sensor to quickly remove IR aliases, which is expected to have the effect of suppressing the effects of IR aliases.

[0043] (Fifth embodiment) A modified example of the pixel driving method according to the first to fourth embodiments will be described with reference to FIG. 8. In the first to fourth embodiments described above, when the SPAD element is actively driven, the quench / recharge circuit 6 and the photon detection circuit 14 (15, 16) operate to enter a state in which photons can be detected at regular intervals. Then, the voltage applied to the APD 5 is recharged the same number of times as the number of pixel counter bits, using an exposure control clock within one frame.

[0044] As a modification of this, an example will be shown in which the exposure control clock is set to a number greater than the number of pixel counter bits (denoted as N in FIG. 8). That is, in the sixth embodiment, the number of photon detections in one frame is greater than the number of counter bits of each pixel. The IR pixel has a pixel counter bit After performing normal operation up to the number of times equivalent to the number of IR signals, the counter value is maintained and photon detection continues for the purpose of determining whether the RGB pixels are generating false signals. Meanwhile, the RGB pixels continue to detect photons and count the number of detections during one frame period, and perform sensor output after the end of one frame period. If an IR false signal is detected, the RGB pixels' counter does not cycle, so not all bits of the counter can be used for the number of detections equivalent to the pixel counter bits. However, by adopting the above driving method, it is possible to increase the number of photon detections and the number of detections counted for the RGB pixels in one frame.

[0045] The photoelectric conversion system, the mobile object, and the device according to the embodiment will be described below.

[0046] (Sixth embodiment) The photoelectric conversion system according to this embodiment will be described with reference to Fig. 9. Fig. 9 is a block diagram showing a schematic configuration of the photoelectric conversion system according to this embodiment.

[0047] The photoelectric conversion devices described in the first to fifth embodiments can be applied to various photoelectric conversion systems. A photoelectric conversion system includes at least the photoelectric conversion device according to the above embodiments and a signal processing unit that processes signals output from the photoelectric conversion device. Examples of devices to which such photoelectric conversion systems can be applied include digital still cameras, digital camcorders, surveillance cameras, copiers, fax machines, mobile phones, vehicle-mounted cameras, observation satellites, sensors, and measuring instruments. Camera modules equipped with an optical system such as a lens and an imaging device are also included in devices to which photoelectric conversion systems are applied. FIG. 9 illustrates a block diagram of a digital still camera as an example of such devices.

[0048] 9 includes an image pickup device 2504, which is an example of a photoelectric conversion device, and a lens 2502 that forms an optical image of a subject on the image pickup device 2504. The photoelectric conversion system also includes an aperture 2503 that adjusts the amount of light passing through the lens 2502, and a barrier 2501 that protects the lens 2502. The lens 2502 and the aperture 2503 form an optical system that focuses light on the image pickup device 2504. The image pickup device 2504 is a photoelectric conversion device (image pickup device) according to any of the above embodiments, and converts the optical image formed by the lens 2502 into an electrical signal.

[0049] The photoelectric conversion system also includes a signal processing unit 2507, which is an image generation unit that generates an image by processing an output signal output from the imaging device 2504. The signal processing unit 2507 performs various corrections and compressions as necessary to output image data. The signal processing unit 2507 may be formed on the same semiconductor substrate on which the imaging device 2504 is provided, or may be formed on a semiconductor substrate separate from the imaging device 2504. Furthermore, the imaging device 2504 and the signal processing unit 2507 may be formed on the same semiconductor substrate.

[0050] The photoelectric conversion system further includes a memory unit 2510 for temporarily storing image data, and an external interface unit (external I / F unit) 2513 for communicating with an external computer or the like. The photoelectric conversion system further includes a recording medium 2512 such as a semiconductor memory for recording or reading out imaging data, and a recording medium control interface unit (recording medium control I / F unit) 2511 for recording or reading out data from the recording medium 2512. The recording medium 2512 may be built into the photoelectric conversion system or may be detachable.

[0051] The photoelectric conversion system further includes an overall control and calculation unit 2509 that performs various calculations and controls the entire digital still camera, and a timing generation unit 2508 that outputs various timing signals to the image pickup device 2504 and the signal processing unit 2507. Here, timing signals and the like may be input from the outside, and the photoelectric conversion system includes at least the image pickup device 2504 and the signal processing unit 2507. It is sufficient to have a signal processing unit 2507 that processes the output signal.

[0052] The imaging device 2504 outputs an imaging signal to the signal processing unit 2507. The signal processing unit 2507 performs predetermined signal processing on the imaging signal output from the imaging device 2504 and outputs image data. The signal processing unit 2507 generates an image using the imaging signal.

[0053] As described above, according to this embodiment, it is possible to realize a photoelectric conversion system to which the photoelectric conversion device (imaging device) according to any one of the above embodiments is applied.

[0054] Seventh embodiment The photoelectric conversion system and the mobile object of this embodiment will be described with reference to Figures 10(A) and 10(B). Figure 10(A) is a diagram showing the configuration of the photoelectric conversion system of this embodiment, and Figure 10(B) is a diagram showing the configuration of the mobile object of this embodiment.

[0055] FIG. 10A shows an example of a photoelectric conversion system related to an in-vehicle camera. The photoelectric conversion system 2600 includes an image capturing device 2610. The image capturing device 2610 is the photoelectric conversion device (image capturing device) described in any of the above embodiments. The photoelectric conversion system 2600 includes an image processing unit 2612 that performs image processing on multiple pieces of image data acquired by the image capturing device 2610. The photoelectric conversion system 2600 also includes a distance acquisition unit 2616 that calculates the distance to an object, and a collision determination unit 2618 that determines whether or not there is a possibility of a collision based on the calculated distance. Here, the distance acquisition unit 2616 may acquire distance information to the object using ToF (Time Of Flight), or may acquire distance information using parallax information, etc. In other words, the distance information is information related to parallax, defocus amount, distance to the object, etc. The collision determination unit 2618 may determine the possibility of a collision using any of these pieces of distance information. The distance information acquisition unit may be realized by dedicated hardware or a software module. Furthermore, it may be realized by a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like, or may be realized by a combination of these.

[0056] The photoelectric conversion system 2600 is connected to a vehicle information acquisition device 2620 and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. The photoelectric conversion system 2600 is also connected to an ECU 2630, which is a control device (control unit) that outputs a control signal to generate a braking force for the vehicle based on the determination result of the collision determination unit 2618. The photoelectric conversion system 2600 is also connected to an alarm device 2640 that issues an alarm to the driver based on the determination result of the collision determination unit 2618. For example, if the determination result of the collision determination unit 2618 indicates a high possibility of a collision, the ECU 2630 performs vehicle control to avoid the collision and mitigate damage by applying the brakes, releasing the accelerator, suppressing engine output, etc. The alarm device 2640 warns the user by sounding an alarm, displaying alarm information on the screen of a car navigation system, etc., or vibrating the seat belt or steering wheel.

[0057] In this embodiment, the surroundings of the vehicle, for example, the front or rear, are imaged by the photoelectric conversion system 2600. Fig. 10(B) shows the photoelectric conversion system when imaging the area in front of the vehicle (imaging range 2650). A vehicle information acquisition device 2620 sends instructions to the photoelectric conversion system 2600 or the imaging device 2610. This configuration can further improve the accuracy of distance measurement.

[0058] Although the above describes an example of control to prevent collision with other vehicles, the present invention can also be applied to control of automatic driving to follow other vehicles, control of automatic driving to prevent the vehicle from straying from its lane, etc. Furthermore, the photoelectric conversion system is not limited to vehicles such as automobiles, but can also be applied to other vehicles such as ships, aircraft, etc. The present invention can be applied to a mobile object (mobile device) such as a robot or an industrial robot. The mobile object includes one or both of a driving force generating unit that generates a driving force mainly used to move the mobile object and a rotating object mainly used to move the mobile object. The driving force generating unit can be an engine, a motor, etc. The rotating object can be a tire, a wheel, a ship's screw, an aircraft's propeller, etc. In addition, the present invention can be applied not only to mobile objects but also to a wide range of devices that use object recognition, such as intelligent transport systems (ITS).

[0059] (Eighth embodiment) The photoelectric conversion system of this embodiment will be described with reference to Fig. 11. Fig. 11 is a block diagram showing an example of the configuration of a range image sensor, which is the photoelectric conversion system of this embodiment.

[0060] 11, the distance image sensor 2701 is configured to include an optical system 2707, a photoelectric conversion device 2708, an image processing circuit 2704, a monitor 2705, and a memory 2706. The distance image sensor 2701 can obtain a distance image according to the distance to the subject by receiving light (modulated light or pulsed light) that is projected toward the subject from a light source device 2709 and reflected from the surface of the subject.

[0061] The optical system 2707 is configured to have one or more lenses, and guides image light (incident light) from the subject to the photoelectric conversion device 2708, forming an image on the light receiving surface (sensor section) of the photoelectric conversion device 2708.

[0062] The photoelectric conversion device 2708 is the photoelectric conversion device of each of the above-described embodiments, and a distance signal indicating a distance determined from a light reception signal output from the photoelectric conversion device 2708 is supplied to the image processing circuit 2704.

[0063] The image processing circuit 2704 performs image processing to construct a distance image based on the distance signal supplied from the photoelectric conversion device 2708. The distance image (image data) obtained by this image processing is then supplied to a monitor 2705 for display, or supplied to a memory 2706 for storage (recording).

[0064] In the range image sensor 2701 configured in this way, by applying the above-described photoelectric conversion device, it is possible to obtain, for example, a more accurate range image as the pixel characteristics improve.

[0065] (Ninth embodiment) The photoelectric conversion system of this embodiment will be described with reference to Fig. 12. Fig. 12 is a diagram showing an example of the schematic configuration of an endoscopic surgery system, which is the photoelectric conversion system of this embodiment.

[0066] 12 shows a state in which an operator (doctor) 2831 is performing surgery on a patient 2832 on a patient bed 2833 using an endoscopic surgery system 2850. As shown in the figure, the endoscopic surgery system 2850 is composed of an endoscope 2800, a surgical tool 2810, and a cart 2834 on which various devices for endoscopic surgery are mounted.

[0067] The endoscope 2800 is composed of a lens barrel 2801, a region of a predetermined length from the tip of which is inserted into a body cavity of a patient 2832, and a camera head 2802 connected to the base end of the lens barrel 2801. In the example shown in the figure, the endoscope 2800 is configured as a so-called rigid lens barrel having a rigid lens barrel 2801, but the endoscope 2800 may also be configured as a so-called flexible lens barrel having a flexible lens barrel.

[0068] An opening into which an objective lens is fitted is provided at the tip of the lens barrel 2801. A light source device 2803 is connected to the endoscope 2800. Light generated by the light source device 2803 is guided to the tip of the lens barrel by a light guide extending inside the lens barrel 2801, and is irradiated via the objective lens towards an observation target inside the body cavity of the patient 2832. Note that the endoscope 2800 may be a direct-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0069] An optical system and a photoelectric conversion device are provided inside the camera head 2802, and light reflected from the observation object (observation light) is focused onto the photoelectric conversion device by the optical system. The observation light is photoelectrically converted by the photoelectric conversion device to generate an electrical signal corresponding to the observation light, i.e., an image signal corresponding to the observation image. The photoelectric conversion device may be any of the photoelectric conversion devices (imaging devices) described in the above-described embodiments. The image signal is transmitted as RAW data to a camera control unit (CCU) 2835.

[0070] The CCU 2835 is configured with a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and comprehensively controls the operations of the endoscope 2800 and the display device 2836. Furthermore, the CCU 2835 receives an image signal from the camera head 2802 and performs various image processing on the image signal, such as development processing (demosaic processing), to display an image based on the image signal.

[0071] The display device 2836, under the control of the CCU 2835, displays an image based on the image signal that has been subjected to image processing by the CCU 2835.

[0072] The light source device 2803 is configured from a light source such as an LED (Light Emitting Diode), and supplies the endoscope 2800 with irradiation light when photographing an operation site or the like.

[0073] The input device 2837 is an input interface for the endoscopic surgery system 2850. A user can input various information and instructions to the endoscopic surgery system 2850 via the input device 2837.

[0074] The treatment tool control device 2838 controls the driving of the energy treatment tool 2812 for cauterizing tissue, incising, sealing blood vessels, or the like.

[0075] The light source device 2803 that supplies illumination light to the endoscope 2800 when photographing the surgical site can be configured from a white light source configured from, for example, an LED, a laser light source, or a combination of these. When the white light source is configured from a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, making it possible to adjust the white balance of the captured image in the light source device 2803. In this case, it is also possible to capture images corresponding to each RGB in a time-division manner by irradiating the object of observation with laser light from each RGB laser light source in a time-division manner and controlling the drive of the image sensor of the camera head 2802 in synchronization with the irradiation timing. According to this method, a color image can be obtained without providing a color filter to the image sensor.

[0076] Furthermore, the driving of the light source device 2803 may be controlled so as to change the intensity of the light it outputs at predetermined time intervals. By controlling the driving of the image sensor of the camera head 2802 in synchronization with the timing of the change in the light intensity to acquire images in a time-division manner and combining these images, it is possible to generate an image with a high dynamic range that is free of so-called blocked-up shadows and blown-out highlights.

[0077] The light source device 2803 may also be configured to supply light in a predetermined wavelength band corresponding to special light observation. Special light observation utilizes, for example, the wavelength dependency of light absorption in body tissue. Specifically, by irradiating light with a narrower band than the light (i.e., white light) used in normal observation, a specific tissue, such as blood vessels on the surface of a mucous membrane, can be photographed with high contrast. Alternatively, special light observation may involve fluorescence observation, in which an image is obtained using fluorescence generated by irradiating excitation light. Fluorescence observation can involve irradiating excitation light onto a body tissue and observing the fluorescence from the body tissue, or locally injecting a reagent such as indocyanine green (ICG) into the body tissue and irradiating the body tissue with excitation light corresponding to the fluorescent wavelength of the reagent to obtain a fluorescent image. The light source device 2803 may be configured to supply narrow band light and / or excitation light corresponding to such special light observation.

[0078] (Tenth embodiment) The photoelectric conversion system of this embodiment will be described with reference to FIGS. 13(A) and 13(B). FIG. 13(A) illustrates glasses 2900 (smart glasses) which are the photoelectric conversion system of this embodiment. The glasses 2900 have a photoelectric conversion device 2902. The photoelectric conversion device 2902 is the photoelectric conversion device (imaging device) described in each of the above embodiments. A display device including a light-emitting device such as an OLED or LED may be provided on the back side of the lens 2901. The photoelectric conversion device 2902 may be one or more. A combination of multiple types of photoelectric conversion devices may also be used. The arrangement position of the photoelectric conversion device 2902 is not limited to that shown in FIG. 13(A).

[0079] The glasses 2900 further include a control device 2903. The control device 2903 functions as a power source that supplies power to the photoelectric conversion device 2902 and the display device. The control device 2903 also controls the operations of the photoelectric conversion device 2902 and the display device. The lens 2901 is formed with an optical system for focusing light onto the photoelectric conversion device 2902.

[0080] FIG. 13(B) illustrates glasses 2910 (smart glasses) according to one application example. The glasses 2910 include a control device 2912, which includes a photoelectric conversion device corresponding to the photoelectric conversion device 2902 and a display device. A lens 2911 includes an optical system for projecting light emitted from the photoelectric conversion device and the display device, and an image is projected onto the lens 2911. The control device 2912 functions as a power source for supplying power to the photoelectric conversion device and the display device, and controls the operation of the photoelectric conversion device and the display device. The control device may include a gaze detection unit for detecting the gaze of the wearer. Infrared light may be used for gaze detection. The infrared light emitting unit emits infrared light toward the eyeball of a user gazing at a displayed image. An imaging unit with a light receiving element detects the reflected light of the emitted infrared light from the eyeball, thereby obtaining an image of the eyeball. A reduction unit for reducing light from the infrared light emitting unit to the display unit in a planar view reduces degradation of image quality.

[0081] The gaze of the user relative to the displayed image is detected from an image of the eyeball obtained by capturing infrared light. Any known method can be used for gaze detection using an image of the eyeball. One example is a gaze detection method based on the Purkinje image formed by reflection of irradiated light on the cornea.

[0082] More specifically, gaze detection processing is performed based on the pupil-corneal reflex method, which calculates a gaze vector representing the direction (rotation angle) of the eyeball based on the pupil image and Purkinje image included in the captured image of the eyeball, thereby detecting the user's gaze.

[0083] The display device of this embodiment may have a photoelectric conversion device having a light receiving element, and may control the image displayed on the display device based on information about the user's line of sight from the photoelectric conversion device.

[0084] Specifically, the display device determines a first field of view area where the user gazes and a second field of view area other than the first field of view area based on the line-of-sight information. The first field of view area and the second field of view area may be determined by a control device of the display device, or may be determined by an external control device and received. In the display area of ​​the display device, the display resolution of the first field of view area may be controlled to be higher than the display resolution of the second field of view area. In other words, the resolution of the second field of view area may be lower than that of the first field of view area.

[0085] The display area may also include a first display area and a second display area different from the first display area, and a high-priority area may be determined from the first display area and the second display area based on line-of-sight information. The first display area and the second display area may be determined by a control device of the display device, or may be determined by an external control device and received. The resolution of the high-priority area may be controlled to be higher than the resolution of areas other than the high-priority area. In other words, the resolution of an area with a relatively low priority may be lowered.

[0086] Note that AI may be used to determine the first field of view area and areas with high priority. The AI ​​may be a model configured to estimate the angle of gaze and the distance to an object in the line of sight from an image of the eyeball, using as training data an image of the eyeball and the direction in which the eyeball in the image was actually looking. The AI ​​program may be included in the display device, the photoelectric conversion device, or an external device. If included in an external device, it is transmitted to the display device via communication.

[0087] When display control is performed based on visual recognition detection, the present invention is preferably applied to smart glasses that further include a photoelectric conversion device for capturing images of the outside world. The smart glasses can display captured external information in real time.

[0088] (Eleventh embodiment) The above-described photoelectric conversion device and photoelectric conversion system may be applied to electronic devices such as so-called smartphones and tablets.

[0089] 14(A) and 14(B) are diagrams showing an example of an electronic device 3000 equipped with a photoelectric conversion device. Fig. 14(A) shows the front side of the electronic device 3000, and Fig. 14(B) shows the back side of the electronic device 3000.

[0090] 14(A), a display 3010 for displaying an image is disposed in the center of the surface of electronic device 3000. Further, along the upper side of the surface of electronic device 3000, front cameras 3021 and 3022 using photoelectric conversion devices, an IR light source 3030 for emitting infrared light, and a visible light source 3040 for emitting visible light are disposed.

[0091] Also, as shown in Figure 14(B), rear cameras 3051 and 3052 using photoelectric conversion devices, an IR light source 3060 that emits infrared light, and a visible light source 3070 that emits visible light are arranged along the upper edge of the back of the electronic device 3000.

[0092] By applying the above-described photoelectric conversion device to the electronic device 3000 configured as described above, it is possible to capture higher quality images, for example. The photoelectric conversion device can also be applied to other electronic devices, such as infrared sensors, distance measurement sensors using active infrared light sources, security cameras, and personal or biometric authentication cameras. This can improve the accuracy and performance of these electronic devices.

[0093] (Twelfth embodiment) A photoelectric conversion system according to the twelfth embodiment will be described with reference to Fig. 15. Fig. 15 is a block diagram showing the schematic configuration of an imaging system SYS, which is a photoelectric conversion system according to the twelfth embodiment. 1 is a block diagram of an image capturing system SYS that includes at least the photoelectric conversion device according to the above embodiment and a signal processing unit that processes signals output from the photoelectric conversion device.

[0094] The imaging system SYS is an information terminal having a camera or imaging function. The imaging system SYS is constructed using an imaging device IS. The imaging device IS may further include a package PKG that houses an imaging device IC. The package PKG may include a base to which the imaging device IC is fixed and a lid that faces the imaging device IC. The package PKG may include a connecting member (a member that connects terminals provided on the base with terminals provided on the imaging device IC). The imaging device IS may also mount multiple imaging device ICs side by side in a common package PKG. The imaging device IS may also mount an imaging device IC and other semiconductor device ICs stacked on top of each other in a common package PKG.

[0095] The imaging system SYS may include an optical system OU (optical device) that forms an image on the imaging device IS. The imaging system SYS may also include at least one of a control device CU, a processing device PU, a display device DU, and a storage device MU. The control device CU controls the imaging device IS, and the processing device PU processes signals obtained from the imaging device IS. The display device DU displays images obtained from the imaging device IS, and the storage device MU stores images obtained from the imaging device IS.

[0096] (others) Although various devices have been described in the above embodiments, a mechanical device may also be provided. The mechanical device in the camera can drive optical components for zooming, focusing, and shutter operation. Alternatively, the mechanical device in the camera can move a photoelectric conversion device for vibration reduction.

[0097] The equipment may also be transportation equipment such as a vehicle, a ship, or an aircraft. A mechanical device in the transportation equipment category may be used as a moving device. Equipment serving as transportation equipment is suitable for transporting a photoelectric conversion device or for assisting and / or automating driving (piloting) using a photographing function. A processing device for assisting and / or automating driving (piloting) can perform processing for operating the mechanical device as a moving device based on information obtained by the photoelectric conversion device.

[0098] The embodiments described above can be modified as appropriate without departing from the spirit and scope of the present invention. The disclosure of this specification includes not only what is described in this specification, but also all matters that can be understood from this specification and the drawings attached hereto.

[0099] The disclosure of this embodiment includes the following configuration. (Configuration 1) a first pixel having an avalanche photodiode for acquiring IR light information; a second pixel having an avalanche photodiode for acquiring visible light information; and Each of the first pixel and the second pixel is a photon detection circuit that detects the photon signal and outputs a photon detection signal; a pixel counter that counts the photon detection signals; and The first pixel transmits a photon signal or a photon detection signal at the first pixel to the second pixel, and the count value of the pixel counter at the second pixel is not changed or is decreased. A photoelectric conversion device characterized by: (Configuration 2) the second pixel further includes a signal control circuit that controls a signal input to the pixel counter of the second pixel; The first pixel transmits a photon detection signal at the first pixel to the signal control circuit. The signal control circuit performs a logical operation on the transmitted photon detection signal and the photon detection signal of the second pixel. 2. The photoelectric conversion device according to configuration 1, (Configuration 3) The photon detection circuit of the second pixel operates to either not change or to decrement the counter value of the pixel counter in the second pixel. 2. The photoelectric conversion device according to configuration 1, (Configuration 4) The first pixel transmits a photon signal at the first pixel to the photon detection circuit. The photon detection circuit performs a logical operation on the transmitted photon detection signal and the photon detection signal of the second pixel. 4. The photoelectric conversion device according to configuration 3. (Configuration 5) the pixel counter of the second pixel is an up-down counter, The up-down counter counts up the count value of the pixel counter in the second pixel and counts down based on the photon signal or photon detection signal transmitted from the first pixel. 2. The photoelectric conversion device according to configuration 1, (Configuration 6) A clock signal is input to the photon detection circuit. 2. The photoelectric conversion device according to configuration 1, (Configuration 7) 2. The photoelectric conversion device according to configuration 1, wherein the number of photon detections in one frame is greater than the number of count bits of the counter. (Configuration 8) a first pixel having an avalanche photodiode for acquiring IR light information; a second pixel having an avalanche photodiode for acquiring visible light information; and Each of the first pixel and the second pixel is a photon detection circuit that detects the photon signal and outputs a photon detection signal; a pixel counter that counts photon detection signals; and the first pixel transmits flag information corresponding to the photon signal or the photon detection signal of the first pixel to the second pixel; The second pixel outputs the flag information together with a count value of the photon detection signal of the second pixel. A photoelectric conversion device characterized by: (Configuration 9) A clock signal is input to the photon detection circuit. 9. The photoelectric conversion device according to configuration 8, (Configuration 10) The number of photon detections in one frame is greater than the number of counter bits in each pixel 9. The photoelectric conversion device according to configuration 7 or 8. (Configuration 11) The photoelectric conversion device according to any one of configurations 1 to 10; a signal processing unit that generates an image using a signal output from the photoelectric conversion device; A photoelectric conversion system comprising: (Configuration 12) A moving object including the photoelectric conversion device according to any one of configurations 1 to 10, A moving body comprising a control unit that controls the movement of the moving body using a signal output from the photoelectric conversion device. (Configuration 13) The photoelectric conversion device according to any one of configurations 1 to 10; an optical device corresponding to the photoelectric conversion device; a control device that controls the photoelectric conversion device; a processing device that processes a signal output from the photoelectric conversion device; a display device that displays information obtained by the photoelectric conversion device; a storage device that stores information obtained by the photoelectric conversion device; and a mechanical device that operates based on information obtained by the photoelectric conversion device; At least one of An apparatus characterized by having: [Explanation of symbols]

[0100] 1: R pixel filter 2: G pixel filter 3: B pixel filter 4: IR pixel filter 5: APD (avalanche photodiode) 6: Quench and recharge circuit 7: Inverter (buffer) circuit 8: R pixel 9: G pixel 10: B pixel 11: IR pixel 12: Pixel counter 13: Signal control circuit 14-16: Photon detection circuit 17: Up / down counter 18: Flag information

Claims

1. a first pixel having an avalanche photodiode for acquiring IR light information; a second pixel having an avalanche photodiode for acquiring visible light information; and Each of the first pixel and the second pixel is a photon detection circuit that detects the photon signal and outputs a photon detection signal; a pixel counter that counts the photon detection signals; and The first pixel transmits a photon signal or a photon detection signal at the first pixel to the second pixel, and does not change the count value of the pixel counter at the second pixel, or decreases the count value. A photoelectric conversion device characterized by:

2. the second pixel further includes a signal control circuit that controls a signal input to the pixel counter of the second pixel; the first pixel transmits the photon detection signal at the first pixel to the signal control circuit; The signal control circuit performs a logical operation on the transmitted photon detection signal and the photon detection signal of the second pixel.

2. The photoelectric conversion device according to claim 1.

3. The photon detection circuit of the second pixel operates to either not change the count value of the pixel counter in the second pixel or to decrease the count value.

2. The photoelectric conversion device according to claim 1.

4. the first pixel transmits a photon signal at the first pixel to the photon detection circuit; The photon detection circuit performs a logical operation on the transmitted photon detection signal and the photon detection signal of the second pixel.

4. The photoelectric conversion device according to claim 3.

5. the pixel counter of the second pixel is an up-down counter, The up-down counter counts up the count value of the pixel counter in the second pixel and counts down based on the photon signal or photon detection signal transmitted from the first pixel.

2. The photoelectric conversion device according to claim 1.

6. A clock signal is input to the photon detection circuit.

2. The photoelectric conversion device according to claim 1.

7. The number of photon detections in one frame is greater than the number of counter bits of the pixel counter.

2. The photoelectric conversion device according to claim 1.

8. a first pixel having an avalanche photodiode for acquiring IR light information; a second pixel having an avalanche photodiode for acquiring visible light information; and Each of the first pixel and the second pixel is a photon detection circuit that detects the photon signal and outputs a photon detection signal; a pixel counter that counts photon detection signals; and the first pixel transmits flag information corresponding to the photon signal or the photon detection signal of the first pixel to the second pixel; The second pixel outputs the flag information together with a count value of the photon detection signal of the second pixel. A photoelectric conversion device characterized by:

9. A clock signal is input to the photon detection circuit.

9. The photoelectric conversion device according to claim 8.

10. The number of photon detections in one frame is greater than the number of counter bits of the pixel counter.

9. The photoelectric conversion device according to claim 8.

11. The photoelectric conversion device according to any one of claims 1 to 10; a signal processing unit that generates an image using a signal output from the photoelectric conversion device; A photoelectric conversion system comprising:

12. A moving object comprising the photoelectric conversion device according to any one of claims 1 to 10, A moving body comprising a control unit that controls the movement of the moving body using a signal output from the photoelectric conversion device.

13. The photoelectric conversion device according to any one of claims 1 to 10; an optical device corresponding to the photoelectric conversion device; a control device that controls the photoelectric conversion device; a processing device that processes a signal output from the photoelectric conversion device; a display device that displays information obtained by the photoelectric conversion device; a storage device that stores information obtained by the photoelectric conversion device; and a mechanical device that operates based on information obtained by the photoelectric conversion device; At least one of An apparatus characterized by having:

Images