Photoelectric conversion device

Abstract

To provide a highly functional photoelectric conversion device capable of outputting data of a plurality of frames or a result of data processing for a frame or a plurality of frames from a pixel, while keeping the circuit scale of the pixel small.SOLUTION: A photoelectric conversion device includes a pixel having a photoelectric conversion unit for outputting a pulse according to incidence of a photon and a pulse counting unit for counting pulses outputted from the photoelectric conversion unit. The pulse counting unit includes a first counter, a second counter, a selection circuit for selecting signals to be inputted respectively to the first counter and the second counter, and a control unit for controlling the selection circuit. The control unit is configured to perform a first connection mode for configuring a counter of a first bit number with the first counter and the second counter, and a second connection mode for configuring a counter of a second bit number less than the first bit number with at least either the first counter or the second counter by controlling the selection circuit.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a photoelectric conversion device.

Background Art

[0002] As photoelectric conversion elements, APD (Avalanche Photo Diode) and SPAD (Single Photon Avalanche Diode) that multiply charges generated by the incidence of photons by avalanche breakdown are known. These photoelectric conversion elements are used in photoelectric conversion devices having a function of counting the number of detected photons. Patent Document 1 describes a photoelectric conversion device that enables various signal processes including the process of data of a plurality of frames by providing a memory for data retention in a pixel. According to this method, various signals including signals obtained by processing data of a plurality of frames can be output from the pixel.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the photoelectric conversion device of Patent Document 1, it is necessary to provide a memory for accumulating data for a single or a plurality of frames for each pixel, and an increase in the area of the pixel circuit cannot be avoided.

[0005] An object of the present invention is to provide a highly functional photoelectric conversion device that can output data of a plurality of frames and results obtained by processing data of a single or a plurality of frames from a pixel while suppressing the circuit scale of the pixel to be small.

Means for Solving the Problems

[0006] According to one disclosure of this specification, a photoelectric converter is provided having a pixel having a photoelectric converter that outputs pulses in response to the incidence of photons, and a pulse counter that counts the pulses output from the photoelectric converter, wherein the pulse counter has a first counter, a second counter, a selection circuit that selects signals input to each of the first counter and the second counter, and a control unit that controls the selection circuit, and the control unit is configured to perform a first connection mode in which the first counter and the second counter constitute a counter with a first number of bits, and a second connection mode in which at least one of the first counter and the second counter constitutes a counter with a second number of bits less than the first number of bits. [Effects of the Invention]

[0007] According to the present invention, it is possible to output data from multiple frames, or the results of processing data from a single or multiple frames, from a pixel while keeping the circuit size of the pixel small. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the schematic configuration of a photoelectric conversion device according to a first embodiment of the present invention. [Figure 2] This is a block diagram showing an example of the pixel configuration in a photoelectric conversion device according to the first embodiment of the present invention. [Figure 3] This figure illustrates the basic operation of the photoelectric conversion element in a photoelectric conversion device according to a first embodiment of the present invention. [Figure 4] This is a block diagram showing an example configuration of the pulse counting unit in a photoelectric conversion device according to the first embodiment of the present invention. [Figure 5] This is a block diagram showing an example of the configuration of a sub-counter in a photoelectric conversion device according to the first embodiment of the present invention. [Figure 6] This is a perspective view showing an example configuration of a photoelectric conversion device according to the first embodiment of the present invention. [Figure 7]This is a timing diagram (part 1) showing an example of driving a photoelectric conversion device according to the first embodiment of the present invention. [Figure 8] This is a timing diagram (part 2) showing an example of driving a photoelectric conversion device according to the first embodiment of the present invention. [Figure 9] This is a diagram (part 1) illustrating the averaging process for two frames of image data. [Figure 10] This is a diagram (part 2) illustrating the averaging process for two frames of image data. [Figure 11] This is a timing diagram (part 3) showing an example of driving a photoelectric conversion device according to the first embodiment of the present invention. [Figure 12] This is a block diagram showing an example of the pixel configuration in a photoelectric conversion device according to a second embodiment of the present invention. [Figure 13] This is a block diagram showing an example of the configuration of a signal processing unit in a photoelectric conversion device according to a second embodiment of the present invention. [Figure 14] This is a block diagram showing an example of the pixel configuration in a photoelectric conversion device according to a third embodiment of the present invention. [Figure 15] This is a timing diagram showing an example of driving a photoelectric conversion device according to a third embodiment of the present invention. [Figure 16] This block diagram shows a schematic configuration of a photodetection system according to a fourth embodiment of the present invention. [Figure 17] This is a block diagram showing a schematic configuration of a distance image sensor according to a fifth embodiment of the present invention. [Figure 18] This is a schematic diagram showing an example configuration of an endoscopic surgical system according to the sixth embodiment of the present invention. [Figure 19] This is a schematic diagram showing an example of the configuration of a mobile body according to the seventh embodiment of the present invention. [Figure 20] This is a block diagram showing the schematic configuration of a photodetection system according to the seventh embodiment of the present invention. [Figure 21] This is a flowchart illustrating the operation of the photodetection system according to the seventh embodiment of the present invention. [Figure 22] This is a schematic diagram showing the general configuration of a photodetection system according to the eighth embodiment of the present invention.

Best Mode for Carrying Out the Invention

[0009] [First Embodiment] The schematic configuration of the photoelectric conversion device according to the first embodiment of the present invention will be described with reference to FIGS. 1 to 6. FIG. 1 is a block diagram showing the schematic configuration of the photoelectric conversion device according to the present embodiment. FIG. 2 is a block diagram showing a configuration example of pixels in the photoelectric conversion device according to the present embodiment. FIG. 3 is a diagram for explaining the basic operation of the photoelectric conversion element in the photoelectric conversion device according to the present embodiment. FIG. 4 is a block diagram showing a configuration example of the pulse counting unit in the photoelectric conversion device according to the present embodiment. FIG. 5 is a block diagram showing a configuration example of the sub-counter in the photoelectric conversion device according to the present embodiment. FIG. 6 is a perspective view showing a configuration example of the photoelectric conversion device according to the present embodiment.

[0010] As shown in FIG. 1, the photoelectric conversion device 100 according to the present embodiment includes a pixel region 10, a vertical selection circuit 40, a horizontal selection circuit 50, a register block 60, a signal processing circuit 70, an output circuit 80, and a drive control circuit 90.

[0011] In the pixel region 10, a plurality of pixels 12 are arranged in an array so as to form a plurality of rows and a plurality of columns. The number of pixels 12 constituting the pixel region 10 is not particularly limited. For example, the pixel region 10 can be constituted by a plurality of pixels 12 arranged in an array of several thousand rows × several thousand columns like a general digital camera. Alternatively, the pixel region 10 may be constituted by a plurality of pixels 12 arranged in one row or one column. Alternatively, the pixel region 10 may be constituted by one pixel 12.

[0012] Each row of the pixel array in the pixel region 10 has a control line 14 extending in a first direction (horizontal direction in Figure 1). The control line 14 is connected to each pixel 12 arranged in the first direction and forms a common signal line for these pixels 12. The first direction in which the control line 14 extends may be referred to as the row direction or horizontal direction. Each of the control lines 14 may include multiple signal lines for supplying multiple types of control signals to the pixels 12. The control lines 14 of each row are connected to the vertical selection circuit 40.

[0013] Furthermore, control lines 16 are provided in each column of the pixel array in the pixel region 10, extending in a second direction (vertical direction in Figure 1) that intersects the first direction. Each control line 16 is connected to a pixel 12 arranged in the second direction, forming a common signal line for these pixels 12. The second direction in which the control lines 16 extend may be referred to as the column direction or the vertical direction. Each control line 16 may include multiple signal lines for supplying multiple types of control signals to the pixels 12. The control lines 16 in each column are connected to the horizontal selection circuit 50.

[0014] Furthermore, each of the multiple pixels 12 arranged in the pixel area 10 is connected to a setting signal line 18. The setting signal line 18 may include multiple signal lines for supplying multiple types of signals to the pixels 12. The setting signal line 18 is connected to the register block 60. Also, each of the multiple pixels 12 arranged in the pixel area 10 is connected to a pixel signal output line 20. The pixel signal output line 20 may include multiple signal lines for transferring multi-bit digital signals output from the pixels 12 bit by bit. The pixel signal output line 20 may consist of multiple signal lines arranged in each column of the pixel array in the pixel area 10, or it may consist of multiple signal lines arranged in each row of the pixel array in the pixel area 10. The pixel signal output line 20 is connected to the signal processing circuit 70.

[0015] The vertical selection circuit 40 is a control circuit that receives a control signal output from the drive control circuit 90, generates a control signal for driving the pixels 12, and supplies it to the pixels 12 via the control line 14. Logic circuits such as a shift register or an address decoder may be used in the vertical selection circuit 40. The vertical selection circuit 40 sequentially supplies control signals to the pixels 12 of the pixel region 10 row by row, thereby sequentially driving the pixels 12 of the pixel region 10 row by row.

[0016] The horizontal selection circuit 50 is a control circuit that receives a control signal output from the drive control circuit 90, generates a control signal for driving the pixels 12, and supplies it to the pixels 12 via the control line 16. Logic circuits such as a shift register and an address decoder may be used in the horizontal selection circuit 50. The horizontal selection circuit 50 sequentially scans the pixels 12 within the pixel region 10 in column units and outputs the pixel signal held by each pixel 12 to the signal processing circuit 70 via the pixel signal output line 20.

[0017] The register block 60 receives data from systems located downstream of the photoelectric converter 100, such as an ISP (Image Signal Processor), ASIC (Application Specific Integrated Circuit), and FPGA (Field Programmable Gate Array). The register block 60 then stores these received data as setting values ​​for various operations of the photoelectric converter 100. The data communication means between the downstream system and the register block 60 is not particularly limited; for example, serial communication methods such as I2C (Inter-Integrated Circuit) or SPI (Serial Peripheral Interface) can be applied. The register block 60 stores all settings, such as the timing settings for the operation of the photoelectric converter 100, the conditions for pixel driving, and the drive capability settings for the analog driving section, and distributes them to each circuit via the setting signal line 18. For simplicity, Figure 1 only shows the setting signal line 18 connecting the register block 60 and the pixels 12.

[0018] The signal processing circuit 70 is a processing circuit that performs digital arithmetic processing, such as offset processing and digital gain processing, on the pixel signal input via the pixel signal output line 20. The digital arithmetic processing performed by the signal processing circuit 70 is not particularly limited, and various other digital arithmetic processing methods can be applied.

[0019] The output circuit 80 has a serializer that converts parallel data into serial data and an external interface circuit, and is a circuit for transmitting the pixel signal after digital processing in the signal processing circuit 70 to a downstream system such as an ISP, ASIC, FPGA, etc. As the external interface circuit, for example, a SerDes (SERializer / DESerializer) transmission circuit such as an LVDS (Low Voltage Differential Signaling) circuit or an SLVS (Scalable Low Voltage Signaling) circuit can be applied. However, the configuration of the output circuit 80 is not limited to this, and other configurations such as parallel output may also be used.

[0020] The drive control circuit 90 is a control circuit that generates and supplies control signals to each circuit block to control the operation and timing of the vertical selection circuit 40 and the horizontal selection circuit 50. Although omitted in Figure 1 for simplicity, it also simultaneously supplies control signals to control other circuit blocks besides the vertical selection circuit 40 and the horizontal selection circuit 50, and plays a role in controlling the overall operation of the photoelectric converter 100. The drive control circuit 90 may be driven by an externally input clock signal, or it may be driven using a clock signal such as a PLL circuit provided inside the photoelectric converter 100.

[0021] Each pixel 12, as shown in Figure 2, includes a photoelectric conversion element PD, a quench element 22, a waveform shaping unit 24, and a pulse counting unit 30.

[0022] The photoelectric conversion element PD may be an avalanche photodiode (hereinafter referred to as "APD"). The anode of the APD constituting the photoelectric conversion element PD is connected to a node to which voltage VSS is supplied. The cathode of the APD constituting the photoelectric conversion element PD is connected to one terminal of the quench element 22. The other terminal of the quench element 22 is connected to a node to which voltage VDD, which is higher than voltage VSS, is supplied. Voltages VSS and VDD are set so that a reverse bias voltage sufficient for the APD to perform avalanche multiplication operation is applied. In one example, a negative high voltage is applied as voltage VSS, and a positive voltage of about the power supply voltage is applied as voltage VDD. For example, voltage VSS is -30V and voltage VDD is 1V.

[0023] As mentioned above, the photoelectric conversion element (PD) can be composed of an APD (Augmented Photon Diode). By supplying the APD with a reverse bias voltage sufficient for avalanche multiplication, the charge generated by light incidence on the APD undergoes avalanche multiplication, generating an avalanche current. There are two operating modes when a reverse bias voltage is supplied to the APD: Geiger mode and linear mode. Geiger mode is an operating mode in which the voltage applied between the anode and cathode is a reverse bias voltage greater than the breakdown voltage of the APD. Linear mode is an operating mode in which the voltage applied between the anode and cathode is a reverse bias voltage near or below the breakdown voltage of the APD. An APD operating in Geiger mode is called a SPAD (Single Photon Avalanche Diode). The APD constituting the photoelectric conversion element (PD) may be configured to operate in linear mode or in Geiger mode.

[0024] The quench element 22 has the function of converting the change in avalanche current generated in the photoelectric converter PD into a voltage signal. Furthermore, the quench element 22 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, reducing the voltage applied to the photoelectric converter PD and suppressing avalanche multiplication. The operation by which the quench element 22 suppresses avalanche multiplication is called the quench operation. The quench element 22 also has the function of returning the voltage supplied to the photoelectric converter PD to voltage VDD by allowing current to flow to compensate for the voltage drop caused by the quench operation. The operation by which the quench element 22 returns the voltage supplied to the photoelectric converter PD to voltage VDD is called the recharge operation. The quench element 22 may be composed of one or more resistive elements such as MOS transistors or diffusion resistors.

[0025] The waveform shaping unit 24 has an input node connected to the connection node between the photoelectric conversion element PD and the quench element 22, and an output node. The waveform shaping unit 24 has the function of converting the analog signal supplied from the photoelectric conversion element PD into a pulse signal (signal P_PULSE). The waveform shaping unit 24 may be composed of logic circuits including NOT circuits (inverter circuits), NOR circuits, NAND circuits, etc. The output node of the waveform shaping unit 24 is connected to the pulse counting unit 30.

[0026] The pulse counting unit 30 is supplied with the signal P_PULSE from the waveform shaping unit 24, and the operating mode setting signal cnt_mode from the register block 60 via the setting signal line 18. The pulse counting unit 30 is also supplied with control signals such as the vertical synchronization signal s_vsync and the horizontal synchronization signal s_hsync from the vertical selection circuit 40 via the control line 14, and the control signal s_read from the horizontal selection circuit 50 via the control line 16. The pulse counting unit 30 outputs the signal P_DATA_O in response to these signals.

[0027] Figure 3 illustrates the basic operation of the photoelectric conversion unit, which consists of a photoelectric conversion element PD, a quench element 22, and a waveform shaping unit 24. Figure 3(a) is a circuit diagram of the photoelectric conversion unit, Figure 3(b) shows the signal waveform at the input node (node ​​A) of the waveform shaping unit 24, and Figure 3(c) shows the signal waveform at the output node (node ​​B) of the waveform shaping unit 24. For the sake of simplicity, it is assumed here that the waveform shaping unit 24 is configured as an inverter circuit.

[0028] At time t0, a reverse bias voltage with a potential difference equivalent to (VDD-VSS) is applied to the photoelectric converter PD. A reverse bias voltage sufficient to cause avalanche multiplication is applied between the anode and cathode of the APD constituting the photoelectric converter PD. However, when no photons are incident on the photoelectric converter PD, there are no carriers that serve as seeds for avalanche multiplication. Therefore, avalanche multiplication does not occur in the photoelectric converter PD, and no current flows through the photoelectric converter PD.

[0029] At the following time t1, assume that a photon is incident on the photoelectric element PD. When a photon is incident on the photoelectric element PD, electron-hole pairs are generated by photoelectric conversion, and avalanche multiplication occurs using these carriers as a seed, causing an avalanche multiplication current to flow through the photoelectric element PD. This avalanche multiplication current flows through the quench element 22, causing a voltage drop across the quench element 22, and the voltage at node A begins to drop. When the voltage drop at node A becomes large and avalanche multiplication stops at time t3, the voltage level at node A will no longer drop.

[0030] When avalanche multiplication in the photoelectric converter PD stops, a current flows from the node supplied with voltage VSS through the photoelectric converter PD to node A to compensate for the voltage drop, and the voltage at node A gradually increases. Subsequently, at time t5, node A settles back to its original voltage level.

[0031] The waveform shaping unit 24 binarizes the signal input from node A according to a predetermined threshold and outputs it from node B. Specifically, the waveform shaping unit 24 outputs a low-level signal from node B when the voltage level of node A exceeds the threshold, and outputs a high-level signal from node B when the voltage level of node A is below the threshold. For example, as shown in Figure 3(b), suppose the voltage of node A is below the threshold during the period from time t2 to time t4. In this case, as shown in Figure 3(c), the signal level at node B is low during the period from time t0 to time t2 and from time t4 to time t5, and high during the period from time t2 to time t4.

[0032] Thus, the analog signal input from node A is waveform-shaped into a digital signal by the waveform shaping unit 24. The pulse signal output from the waveform shaping unit 24 in response to the incidence of photons on the photoelectric conversion element PD is the photon detection pulse signal.

[0033] As shown in Figure 4, the pulse counting unit 30 includes selection circuits SEL1 and SEL2, sub-counters 32 and 34, a connection mode switching circuit 36, and an output switch 38.

[0034] The connection mode switching circuit 36 ​​is a control unit that controls the selection circuits SEL1, SEL2 and the sub-counters 32, 34. The connection mode switching circuit 36 ​​dynamically generates the selection signals sel1, sel2 and the counter reset signals res1, res2 in response to the operation mode setting signal cnt_mode, the vertical synchronization signal s_vsync, and the horizontal synchronization signal s_hsync.

[0035] The selection circuit SEL1 has two input nodes (node ​​a and node b) and one output node. The control node of the selection circuit SEL1 receives the selection signal sel1 from the connection mode switching circuit 36. The signal P_PULSE is input to node a. The reference signal (low level) is input to node b. The output node of the selection circuit SEL1 is connected to the input node CNT_IN of the sub-counter 32. The selection circuit SEL1 selects either the signal P_PULSE or the reference signal according to the selection signal sel1 and inputs the selected signal to the input node CNT_IN of the sub-counter 32.

[0036] The sub-counter 32 has two input nodes (input node CNT_IN, input node RST) and two output nodes (output node P_LASTBIT, output node P_DATA). Input node CNT_IN is connected to the output node of the selection circuit SEL1. Input node RST receives the counter reset signal res1 from the connection mode switching circuit 36. Output node P_LATBIT is connected to node d of the selection circuit SEL2. Output node P_DATA is connected to the output switch 38. The sub-counter 32 receives the output signal of the selection circuit SEL1 and the counter reset signal res1, and outputs the signals P_DATA_O[3:0] and P_LASTBIT.

[0037] As shown in Figure 5(a), the sub-counter 32 may be composed of four T-type flip-flops TFF0, TFF1, TFF2, and TFF3 connected in series. The input node CNT_IN of the sub-counter 32 is connected to the input node of the first-stage T-type flip-flop TFF0. The input node RST of the sub-counter 32 is connected to the reset nodes of each of the T-type flip-flops TFF0, TFF1, TFF2, and TFF3. The signal P_DATA[3:0] is a 4-bit digital signal where the output Q of the T-type flip-flop TFF0 is the least significant bit and the output Q of the T-type flip-flop TFF3 is the most significant bit. The signal P_LASTBIT is a 1-bit digital signal corresponding to the output Q of the T-type flip-flop TFF3.

[0038] The selection circuit SEL2 has three input nodes (nodes c, d, and e) and one output node. The control node of the selection circuit SEL2 receives the selection signal sel2 from the connection mode switching circuit 36. The signal P_PULSE is input to node c. The signal P_LASTBIT is input to node d. The reference signal (low level) is input to node e. The output node of the selection circuit SEL2 is connected to the input node CNT_IN of the subcounter 34. The selection circuit SEL2 selects one of the signals P_PULSE, P_LASTBIT, and the reference signal according to the selection signal sel2, and inputs the selected signal to the input node CNT_IN of the subcounter 34.

[0039] The sub-counter 34 has two input nodes (input node CNT_IN, input node RST) and one output node (output node P_DATA). Input node CNT_IN is connected to the output node of the selection circuit SEL2. The input node RST receives the counter reset signal res2 from the connection mode switching circuit 36. Output node P_DATA is connected to the output switch 38. The sub-counter 34 receives the output signal of the selection circuit SEL2 and the counter reset signal res2 and outputs the signal P_DATA_O[7:4].

[0040] As shown in Figure 5(b), the sub-counter 34 may be composed of four T-type flip-flops TFF4, TFF5, TFF6, and TFF7 connected in series. The input node CNT_IN of the sub-counter 34 is connected to the input node of the first-stage T-type flip-flop TFF4. The input node RST of the sub-counter 34 is connected to the reset nodes of each of the T-type flip-flops TFF4, TFF5, TFF6, and TFF7. The signal P_DATA[7:4] is a 4-bit digital signal with the output Q of the T-type flip-flop TFF4 as the least significant bit and the output Q of the T-type flip-flop TFF7 as the most significant bit.

[0041] The output switch 38 controls the output of the signal P_DATA_O[7:0], where the lower 4 bits are the signal P_DATA_O[3:0] output from the sub-counter 32 and the upper 4 bits are the signal P_DATA_O[7:4] output from the sub-counter 34. The output switch 38 is a switch whose connection state is switched according to the level of the control signal s_read supplied from the horizontal selection circuit 50. For example, the output switch 38 is on (connected) when the control signal s_read is at a high level, and off (disconnected) when the control signal s_read is at a low level.

[0042] In this embodiment, sub-counters 32 and 34 are configured using T-type flip-flops, but the configuration of sub-counters 32 and 34 is not limited to this as long as similar functionality can be achieved. Also, in this embodiment, the number of bits for sub-counters 32 and 34 is set to 4 bits, but this is not limited to this; for example, it could be 2 bits or 8 bits.

[0043] Furthermore, in this embodiment, the output data of subcounter 32 is assigned to the lower 4 bits and the output data of subcounter 34 is assigned to the upper 4 bits to generate the signal P_DATA_O[7:0], but the configuration of the signal P_DATA_O[7:0] is not limited to this. For example, the output data of subcounter 32 may be assigned to the upper 4 bits and the output data of subcounter 34 may be assigned to the lower 4 bits to generate the signal P_DATA_O[7:0].

[0044] The photoelectric converter 100 of this embodiment may be formed on a single substrate, or it may be configured as a stacked type photoelectric converter by stacking multiple substrates. In the latter case, for example, as shown in Figure 6, it can be configured as a stacked type photoelectric converter by stacking a sensor substrate 110 and a circuit board 120 and electrically connecting them. At least the photoelectric converter element PD, which is one of the components of the pixel 12, can be placed on the sensor substrate 110. The circuit board 120 can also contain the quench element 22, the waveform shaping unit 24, and the pulse counting unit 30, which are components of the pixel 12. The photoelectric converter element PD, the quench element 22, the waveform shaping unit 24, and the pulse counting unit 30 are electrically connected via connecting wiring provided for each pixel 12. Furthermore, the circuit board 120 can also contain a vertical selection circuit 40, a horizontal selection circuit 50, a register block 60, a signal processing circuit 70, an output circuit 80, a drive control circuit 90, and the like.

[0045] The photoelectric conversion element PD, quench element 22, waveform shaping unit 24, and pulse counting unit 30 of each pixel 12 may be arranged on the sensor substrate 110 and the circuit board 120 so as to overlap in a plan view. The vertical selection circuit 40, horizontal selection circuit 50, register block 60, signal processing circuit 70, output circuit 80, and drive control circuit 90 can be arranged around the pixel region 10 composed of multiple pixels 12. Here, "plan view" refers to viewing from a direction perpendicular to the surface of the sensor substrate 110.

[0046] By configuring a stacked photoelectric converter 100, the integration density of elements can be increased, leading to improved functionality. In particular, by arranging the photoelectric converter PD, quench element 22, waveform shaping unit 24, and pulse counting unit 30 on separate substrates, the photoelectric converter PD can be arranged at high density without sacrificing the light-receiving area of ​​the photoelectric converter PD, thereby improving photon detection efficiency.

[0047] Furthermore, the number of substrates constituting the photoelectric converter 100 is not limited to two; the photoelectric converter 100 may be constructed by stacking three or more substrates.

[0048] Furthermore, while Figure 6 assumes chips that have been diced as sensor substrate 110 and circuit board 120, the sensor substrate 110 and circuit board 120 are not limited to chips. For example, each of the sensor substrate 110 and circuit board 120 may be a wafer. Also, the sensor substrate 110 and circuit board 120 may be stacked in wafer form and then diced, or they may be made into chips and then stacked and bonded.

[0049] According to the above configuration of the photoelectric converter according to this embodiment, various counting results can be obtained by switching the counter connection mode. The counter connection modes that can be realized with the circuit configuration of this embodiment will be described below, with the first to fourth counter connection modes as examples.

[0050] First, let's explain the first counter connection mode. In this mode, two 4-bit subcounters 32 and 34 are connected in series by setting the selection signal sel1 to node a and the selection signal sel2 to node d, and they operate as a normal 8-bit counter. This connection mode is capable of 8-bit pulse counting and is effective, for example, in high-illumination scenes where a wide dynamic range is required.

[0051] Next, the second counter connection mode will be described. In the second counter connection mode, the input of sub-counter 32 is connected to the waveform shaping unit 24 and the input of sub-counter 34 is connected to a low level by setting the selection signal sel1 to node a and the selection signal sel2 to e. In this connection mode, sub-counter 32 counts the output pulses of the waveform shaping unit 24, and sub-counter 34 retains the count value up to that point. That is, the lower 4 bits signal P_DATA_O[3:0] output from sub-counter 32 indicates the count value after changing the selection signals sel1 and sel2. Also, the upper 4 bits signal P_DATA_O[7:4] output from sub-counter 34 indicates the count value before changing the selection signals sel1 and sel2.

[0052] Next, the third counter connection mode will be described. In the third counter connection mode, the input of sub-counter 34 is connected to the waveform shaping unit 24 and the input of sub-counter 32 is connected to a low level by setting the selection signal sel1 to node b and the selection signal sel2 to node c. In this connection mode, sub-counter 34 counts the output pulses of the waveform shaping unit 24, and sub-counter 32 holds the count value up to that point. That is, the upper 4-bit signal P_DATA_O[7:4] output from sub-counter 34 indicates the count value after changing the selection signals sel1 and sel2. Also, the lower 4-bit signal P_DATA_O[3:0] output from sub-counter 32 indicates the count value before changing the selection signals sel1 and sel2.

[0053] Next, the fourth counter connection mode will be described. The fourth counter connection mode is a mode in which the inputs of sub-counters 32 and 34 are both connected to the waveform shaping unit 24 by setting the selection signal sel1 to node a and the selection signal sel2 to node c. In this connection mode, sub-counters 32 and 34 count the output pulses of the waveform shaping unit 24 in parallel. The lower 4-bit signal P_DATA_O[3:0] output from sub-counter 32 and the upper 4-bit signal P_DATA_O[7:4] output from sub-counter 34 will output the same count value.

[0054] The first to fourth counter connection modes can achieve various effects by dynamically switching between modes. Below, the operating modes of the photoelectric converter 100 using dynamic switching of the counter connection modes will be explained, with the first to third operating modes being used as examples.

[0055] First, the first operating mode of the photoelectric converter according to this embodiment will be explained using Figure 7. Figure 7 is a timing diagram showing the first operating mode of the photoelectric converter according to this embodiment. The first operating mode is intended to operate similarly to general photon counting. Figure 7 shows the waveforms of the vertical synchronization signal s_vsync, the control signal s_read, and the counter reset signals res1 and res2, the values ​​of the selection signals sel1 and sel2, and the value of the signal P_DATA_O[7:0]. The vertical synchronization signal s_vsync is an internal synchronization signal synchronized with the operation of one frame. Here, the timing at which the vertical synchronization signal s_vsync transitions from a high level to a low level is considered to be the end timing of the previous frame and the start timing of the next frame.

[0056] At time t0, the vertical synchronization signal s_vsync transitions from a high level to a low level, and the operation of the nth frame begins. The connection mode switching circuit 36 ​​of the pulse counting unit 30 sets the selection signals sel1 and sel2 to nodes a and d respectively, according to the vertical synchronization signal s_vsync and the operation mode setting signal cnt_mode indicating the first operation mode (first counter connection mode). As a result, the sub-counters 32 and 34 are connected in series and operate as a single 8-bit counter.

[0057] The period from time t0 to time t1 is the exposure period (pulse count period) of the photoelectric conversion element PD in the nth frame. During the exposure period, the waveform shaping unit 24 outputs a photon detection pulse signal (signal P_PULSE) corresponding to the incidence of photons onto the photoelectric conversion element PD. Subcounters 32 and 34 count the number of pulses superimposed on the signal P_PULSE.

[0058] At time t1, the control signal s_read supplied from the horizontal selection circuit 50 to the output switch 38 of the pulse counting unit 30 via the control line 16 transitions from a low level to a high level. As a result, the output switch 38 turns on, and a signal P_DATA_O[7:0] indicating an 8-bit count value, with the lower 4 bits being the output of subcounter 32 and the upper 4 bits being the output of subcounter 34, is output to the pixel signal output line 20.

[0059] At time t2, after the signal P_DATA_O[7:0] is output to the signal processing circuit 70, the counter reset signals res1 and res2 supplied from the connection mode switching circuit 36 ​​to the sub-counters 32 and 34 transition from a low level to a high level. As a result, the count values ​​of the sub-counters 32 and 34 are reset, and the value of the signal P_DATA_O[7:0] becomes 0.

[0060] After this, the vertical synchronization signal s_vsync transitions from a high level to a low level, ending the operation of the nth frame and starting the operation of the (n+1)th frame. The same process is repeated thereafter until the operating mode of the photoelectric converter 100 switches.

[0061] Next, the second operating mode of the photoelectric converter according to this embodiment will be described using Figure 8. Figure 8 is a timing diagram showing the second operating mode of the photoelectric converter according to this embodiment. Figure 8 shows the waveforms of the vertical synchronization signal s_vsync, the control signal s_read, and the counter reset signals res1 and res2, the values ​​of the selection signals sel1 and sel2, and the values ​​of the signals P_DATA_O[3:0] and P_DATA_O[7:4]. The vertical synchronization signal s_vsync is an internal synchronization signal synchronized with the operation of one frame.

[0062] At time t0, the vertical synchronization signal s_vsync transitions from a high level to a low level, and the operation of the nth frame begins. The connection mode switching circuit 36 ​​of the pulse counting unit 30 sets the selection signals sel1 and sel2 to nodes a and e respectively, according to the vertical synchronization signal s_vsync and the operation mode setting signal cnt_mode indicating the second operation mode (second counter connection mode). As a result, the sub-counter 32 operates as a 4-bit counter that counts the number of pulses superimposed on the signal P_PULSE, and the sub-counter 34 holds the value of the (n-1)th frame until the counter reset signal res2 next goes high.

[0063] At the following time t1, the counter reset signal res1 supplied from the connection mode switching circuit 36 ​​to the sub-counter 32 transitions from a low level to a high level. As a result, the count value of the sub-counter 32 is reset, and the value of signal P_DATA_O[3:0] becomes 0. The period from time t1 to time t2, when the control signal s_read next becomes high, is the exposure period (pulse count period) of the photoelectric conversion element PD in the nth frame. At this time, the counter reset signal res2 remains at a low level, and the value of signal P_DATA_O[7:4] remains the same as the value of the (n-1)th frame.

[0064] At the following time t2, the control signal s_read, supplied from the horizontal selection circuit 50 to the output switch 38 of the pulse counting unit 30 via the control line 16, transitions from a low level to a high level. This turns on the output switch 38, and a signal P_DATA_O[7:0] is output to the pixel signal output line 20, with the lower 4 bits representing the output of subcounter 32 and the upper 4 bits representing the output of subcounter 34. In this signal P_DATA_O[7:0], the lower 4 bits consist of the count value of the nth frame, and the upper 4 bits consist of the count value of the (n-1)th frame. In other words, data for two frames can be obtained simultaneously.

[0065] Similarly, at time t2, the vertical synchronization signal s_vsync transitions from a high level to a low level, and the operation of the (n+1)th frame begins. The connection mode switching circuit 36 ​​of the pulse counting unit 30 sets the selection signals sel1 and sel2 to nodes b and c respectively, according to the vertical synchronization signal s_vsync and the operation mode setting signal cnt_mode indicating the second operation mode (third counter connection mode). As a result, the sub-counter 34 operates as a 4-bit counter that counts the number of pulses superimposed on the signal P_PULSE, and the sub-counter 32 holds the value of the nth frame until the counter reset signal res1 goes high.

[0066] At the following time t3, the counter reset signal res2 supplied from the connection mode switching circuit 36 ​​to the sub-counter 34 transitions from a low level to a high level. As a result, the count value of the sub-counter 34 is reset, and the value of signal P_DATA_O[7:4] becomes 0. The period from time t3 to time t4, when the control signal s_read next becomes high, is the exposure period (pulse count period) of the photoelectric conversion element PD in the (n+1)th frame. At this time, the counter reset signal res1 remains at a low level, and the value of signal P_DATA_O[3:0] remains the same as the value of the nth frame.

[0067] At the following time t4, the control signal s_read, supplied from the horizontal selection circuit 50 to the output switch 38 of the pulse counting unit 30 via the control line 16, transitions from a low level to a high level. This turns on the output switch 38, and a signal P_DATA_O[7:0] is output to the pixel signal output line 20, with the lower 4 bits being the output of subcounter 32 and the upper 4 bits being the output of subcounter 34. In this signal P_DATA_O[7:0], the lower 4 bits consist of the count value of the nth frame, and the upper 4 bits consist of the count value of the (n+1)th frame. In other words, data for two frames can be obtained simultaneously.

[0068] Similarly, at time t4, the vertical synchronization signal s_vsync transitions from a high level to a low level, and the operation of the (n+2)th frame begins. The connection mode switching circuit 36 ​​of the pulse counting unit 30 sets the selection signals sel1 and sel2 to nodes a and e respectively, according to the vertical synchronization signal s_vsync and the operation mode setting signal cnt_mode indicating the second operation mode (second counter connection mode). As a result, the sub-counter 32 operates as a 4-bit counter that counts the number of pulses superimposed on the signal P_PULSE, and the sub-counter 34 holds the value of the (n+1)th frame until the counter reset signal res1 next goes high.

[0069] At the following time t5, the counter reset signal res1 supplied from the connection mode switching circuit 36 ​​to the sub-counter 32 transitions from a low level to a high level. As a result, the count value of the sub-counter 32 is reset, and the value of signal P_DATA_O[3:0] becomes 0. The period from time t5 to time t6, when the control signal s_read next becomes high, is the exposure period (pulse count period) of the photoelectric conversion element PD in the (n+2)th frame. At this time, the counter reset signal res2 remains at a low level, and the value of signal P_DATA_O[7:4] remains the same as the value of the (n+1)th frame.

[0070] At the following time t6, the control signal s_read, supplied from the horizontal selection circuit 50 to the output switch 38 of the pulse counting unit 30 via the control line 16, transitions from a low level to a high level. This turns on the output switch 38, and a signal P_DATA_O[7:0] is output to the pixel signal output line 20, with the lower 4 bits representing the output of subcounter 32 and the upper 4 bits representing the output of subcounter 34. In this signal P_DATA_O[7:0], the lower 4 bits consist of the count value of the (n+2)th frame, and the upper 4 bits consist of the count value of the (n+1)th frame. In other words, data for two frames can be obtained simultaneously.

[0071] The second operating mode has a lower upper limit for the counter compared to the first operating mode, making it suitable for scenes with low photon counts, such as low-light photography. However, it has the advantage of being able to output two frames of image data simultaneously. Although the above explanation shows an example configuration using two sub-counters 32 and 34, the number of sub-counters can be three or more. By providing multiple sub-counters, it becomes possible to output multiple frames of image data corresponding to the number of sub-counters, one frame at a time, simultaneously.

[0072] Configuring the photoelectric converter to output image data for multiple frames simultaneously, one frame at a time, is useful for reducing the overall physical size of the system and lowering power consumption. The advantages of outputting image data for multiple frames simultaneously, one frame at a time, are explained below using Figures 9 and 10. Figures 9 and 10 illustrate the averaging process for two frames of image data.

[0073] Figure 9 shows the processing procedure in a system using a typical photoelectric converter that outputs one frame's worth of image data per frame. The photoelectric converter outputs one frame's worth of image data for each frame. When averaging the image data of the nth frame with the image data of the next (n+1)th frame, the image data of the nth frame is temporarily stored in an external frame memory, and the image data is retrieved from the frame memory when the (n+1)th frame is operated. Then, the image data of the nth frame read from the frame memory and the image data of the (n+1)th frame output from the photoelectric converter are averaged, and frame averaging of the two frames is performed. Frame averaging can be applied, for example, when performing noise reduction processing. In recent years, photoelectric converters have become higher pixel count, i.e., higher data capacity, making it difficult to hold frame image data using hardware resources within the ISP or ASIC. Therefore, systems that perform real-time image processing require additional elements such as DRAM to be used as external frame memory.

[0074] Figure 10 shows the processing procedure in a system using the photoelectric converter of this embodiment, which outputs image data for two frames per frame. In the second operating mode of the photoelectric converter according to this embodiment, image data for two frames can be output during the image output period of one frame. In the 8-bit pixel data of each pixel 12 output during the image output period of one frame, the upper 4 bits store the pixel data of the nth frame, and the lower 4 bits store the pixel data of the (n-1)th frame. Processing devices such as ISPs and ASICs located downstream of the photoelectric converter can simultaneously receive this image data for two frames, thus eliminating the need for external elements to store the data of the previous frame. This reduces the overall physical size and power consumption of the system, making it possible to construct a more compact system.

[0075] The first and second operating modes described above can be dynamically set according to the shooting scene in response to requests from downstream processing units such as ISPs, ASICs, and FPGAs. For example, in high-light scenes requiring a large dynamic range and a high count value, the system can operate in the first operating mode, and when the scene becomes low-light, the downstream system can detect this and switch to the second operating mode.

[0076] Next, the third operating mode of the photoelectric converter according to this embodiment will be described using Figure 11. Figure 11 is a timing diagram showing the third operating mode of the photoelectric converter according to this embodiment. Figure 11 shows the waveforms of the vertical synchronization signal s_vsync, the horizontal synchronization signal s_hsync, the control signal s_read, and the counter reset signals res1 and res2. Figure 11 also shows the values ​​of the selection signals sel1 and sel2 and the values ​​of the signals P_DATA_O[3:0] and P_DATA_O[7:4].

[0077] In the first and second operating modes, the operating mode is switched on a frame-by-frame basis, but the operating mode may also be switched within a single frame. In the third operating mode, the counter connection mode is controlled based on the horizontal synchronization signal s_hsync.

[0078] At time t0, the vertical synchronization signal s_vsync transitions from a high level to a low level, and the operation of the nth frame begins. The connection mode switching circuit 36 ​​sets the selection signals sel1 and sel2 to nodes a and c respectively, according to the vertical synchronization signal s_vsync, the horizontal synchronization signal s_hsync, and the operation mode setting signal cnt_mode indicating the third operation mode (fourth counter connection mode). As a result, sub-counters 32 and 34 each operate as 4-bit counters that count the number of pulses superimposed on the signal P_PULSE.

[0079] At the following time t1, the counter reset signals res1 and res2 supplied from the connection mode switching circuit 36 ​​to the sub-counter 32 transition from a low level to a high level. As a result, the count values ​​of the sub-counters 32 and 34 are reset, and the values ​​of signals P_DATA_O[3:0] and P_DATA_O[7:4] become 0. The period from time t1 to time t2, when the setting of the selection signal sel2 is switched to node e, is the first exposure period (pulse count period) of the photoelectric conversion element PD in the nth frame. The pulse counting during the first exposure period is performed by the sub-counter 34. The period from time t1 to time t3, when the control signal s_read becomes high, is the second exposure period (pulse count period) of the photoelectric conversion element PD in the nth frame. The pulse counting during the first exposure period is performed by the sub-counter 32.

[0080] At the following time t2, the connection mode switching circuit 36 ​​sets the selection signal sel2 to node e (second counter connection mode) according to the vertical synchronization signal s_vsync, the horizontal synchronization signal s_hsync, and the operation mode setting signal cnt_mode indicating the third operation mode. As a result, the sub-counter 34 holds the count value up to time t2 until the counter reset signal res2 goes to high level. At this time, the setting of the selection signal sel1 remains at node a, and the sub-counter 32 continues counting. The timing of switching the selection signal sel2 (time t2) can be set to any point in time within a frame. The switching of the selection signal sel2 is usually performed in synchronization with the horizontal synchronization signal s_hsync. For example, the timing of switching the selection signal sel2 can be set to the timing when the horizontal synchronization signal s_hsync transitions from high level to low level for the nth time after the start of the frame (n is an integer of 1 or more).

[0081] At the following time t3, the control signal s_read, supplied from the horizontal selection circuit 50 to the output switch 38 of the pulse counting unit 30 via the control line 16, transitions from a low level to a high level. This turns on the output switch 38, and a signal P_DATA_O[7:0] is output to the pixel signal output line 20, with the lower 4 bits representing the output of subcounter 32 and the upper 4 bits representing the output of subcounter 34. The lower 4 bits of this signal P_DATA_O[7:0] consist of the count value between time t1 and time t3, and the upper 4 bits consist of the count value between time t1 and time t2. In other words, data for two frames can be obtained simultaneously.

[0082] Similarly, at time t3, the vertical synchronization signal s_vsync transitions from a high level to a low level, and the operation of the (n+1)th frame begins. The connection mode switching circuit 36 ​​of the pulse counting unit 30 sets the selection signal sel2 to node c (fourth counter connection mode) according to the vertical synchronization signal s_vsync, the horizontal synchronization signal s_hsync, and the operation mode setting signal cnt_mode indicating the second operation mode. As a result, each of the sub-counters 32 and 34 operates as a 4-bit counter that counts the number of pulses superimposed on the signal P_PULSE.

[0083] At the following time t4, the counter reset signals res1 and res2 supplied from the connection mode switching circuit 36 ​​to the sub-counter 32 transition from a low level to a high level. As a result, the count values ​​of the sub-counters 32 and 34 are reset, and the values ​​of signals P_DATA_O[3:0] and P_DATA_O[7:4] become 0. The period from time t4 to time t5, when the setting of the selection signal sel2 is switched to node e, is the first exposure period (pulse count period) of the photoelectric conversion element PD in the (n+1)th frame. The pulse counting during the first exposure period is performed by the sub-counter 34. The period from time t4 to time t6, when the control signal s_read becomes high, is the second exposure period (pulse count period) of the photoelectric conversion element PD in the (n+1)th frame. The pulse counting during the first exposure period is performed by the sub-counter 32.

[0084] The behavior of each frame from the (n+2)th frame onward, starting from time t6, is the same as that of the nth and (n+1)th frames.

[0085] Thus, in the third operating mode, the upper four bits of the signal P_DATA_O[7:0] store data for a short pulse count period (first exposure period), and the lower four bits store data for a long pulse count period (second exposure period). In this way, by switching between the second counter connection mode and the fourth counter connection mode within a single frame, it is possible to output two types of image data with different pulse count periods.

[0086] According to the third operating mode, for example, two types of image data with different pulse count periods can be simultaneously output to a downstream system, and data suitable for image processing can be selected, for example, by selecting data that is not saturated. In this embodiment, sub-counters 32 and 34 are configured with the same number of bits, but sub-counters 32 and 34 may be configured with different numbers of bits, for example, one being 4 bits and the other 8 bits.

[0087] Thus, according to this embodiment, it is possible to output data from multiple frames, or the results of processing data from a single or multiple frames, from a pixel while keeping the circuit size of the pixel small.

[0088] [Second Embodiment] A photoelectric converter according to a second embodiment of the present invention will be described with reference to Figures 12 and 13. Figure 12 is a block diagram showing an example of the pixel configuration in the photoelectric converter according to this embodiment. Figure 13 is a block diagram showing an example of the signal processing unit configuration in the photoelectric converter according to this embodiment. In this embodiment, the explanation will focus on the differences between the photoelectric converter of this embodiment and the photoelectric converter of the first embodiment. Components similar to those of the photoelectric converter of the first embodiment will be denoted by the same reference numerals, and their descriptions will be omitted or simplified.

[0089] The photoelectric converter according to this embodiment is the same as the photoelectric converter according to the first embodiment, except that the configuration of the pixel 12 is different. That is, the pixel 12 of this embodiment further has a signal processing unit 26, as shown in Figure 12. The signal processing unit 26 is connected between the pulse counting unit 30 and the pixel signal output line 20. The signal processing unit 26 performs arithmetic processing such as addition and subtraction and signal processing such as bit division on the signal P_DATA_O output from the pulse counting unit 30, and outputs the processed signal P_DATA_PROC_O to the pixel signal output line 20. By performing signal processing such as addition within the pixel 12, it becomes possible to output a signal that has been processed such as frame averaging from the photoelectric converter, and it is possible to reduce the load on downstream systems such as ISPs, ASICs, FPGAs, etc.

[0090] Figure 13(a) shows an example configuration of the signal processing unit 26 when it performs a process of adding two frames of image data. The signal processing unit 26 can output a signal P_DATA_PROC_O[4:0] which is obtained by adding the lower 4 bits of the signal P_DATA_O[3:0] and the upper 4 bits of the signal P_DATA_O[7:4]. The addition process may also be performed after weighting the signals P_DATA_O[3:0] and P_DATA_O[7:4].

[0091] Figure 13(b) shows an example configuration of the signal processing unit 26 when performing subtraction processing on image data. The signal processing unit 26 can output a signal P_DATA_PROC_O[4:0] obtained by subtracting the upper 4 bits of the signal P_DATA_O[7:4] from the lower 4 bits of the signal P_DATA_O[3:0], which is the signal P_DATA_O[7:0]. The subtraction processing may also be performed after weighting the signals P_DATA_O[3:0] and P_DATA_O[7:4]. By equipping the pixel 12 with a subtraction circuit, it becomes possible to perform differential calculations between frames, which can be useful for motion detection, for example.

[0092] The above signal processing is just one example, and the signal processing unit 26 may also include other signal processing circuits. Furthermore, the pixel 12 may further include a circuit that switches between performing and not performing signal processing depending on the value of the image data.

[0093] Thus, according to this embodiment, it is possible to output data from multiple frames, or the results of processing data from a single or multiple frames, from a pixel while keeping the circuit size of the pixel small.

[0094] [Third Embodiment] A photoelectric converter according to a third embodiment of the present invention will be described with reference to Figures 14 and 15. Figure 14 is a block diagram showing an example of the pixel configuration in the photoelectric converter according to this embodiment. Figure 15 is a timing diagram showing an example of the operation of the photoelectric converter according to this embodiment. In this embodiment, the explanation will focus on the differences between the photoelectric converter of this embodiment and the photoelectric converter of the first embodiment. Components similar to those of the photoelectric converter of the first embodiment will be denoted by the same reference numerals, and their descriptions will be omitted or simplified.

[0095] The photoelectric converter according to this embodiment is the same as the photoelectric converter according to the first embodiment, except that the configuration of the pixel 12 is different. That is, the pixel 12 of this embodiment further has a pixel level determination unit 28, as shown in Figure 14. The pixel level determination unit 28 generates an operation mode setting signal cnt_mode based on the signal P_DATA_O received from the pulse counting unit 30 and outputs the generated operation mode setting signal cnt_mode to the pulse counting unit 30. In other words, the operation mode setting signal cnt_mode supplied to the pulse counting unit 30 (connection mode switching circuit 36) is output from the pixel level determination unit 28, not from the register block 60. That is, in the first embodiment, the switching of the operation mode was determined and instructed by a downstream system, whereas in this embodiment, the switching of the operation mode is determined and instructed within the pixel 12. By configuring the pixel 12 in this way, it is possible to switch the operation mode without receiving instructions from a downstream system.

[0096] The pixel level determination unit 28 monitors the output of the pulse counting unit 30 within the pixel 12 and checks the count value for each frame. Then, it switches the operating mode of the photoelectric converter according to the count value for each frame. For example, if the count value for each frame is low, it switches to the second operating mode described above. The second operating mode is effective in low-light conditions where the count is low. Also, if the count value for each frame is high, it switches to the first operating mode described above. The first operating mode is effective in high-light conditions.

[0097] Next, an example of the operation of the photoelectric converter according to this embodiment will be explained using Figure 15. Figure 15 shows the waveform of the vertical synchronization signal s_vsync, the operating mode, the signals P_DATA_O[7:0], P_DATA_O[3:0], and the count value comp_count. The operating mode represents the operating mode setting signal cnt_mode input to the connection mode switching circuit 36, where "1" corresponds to the setting signal for the first operating mode and "2" corresponds to the setting signal for the second operating mode. The count value comp_count represents the count value of the frame in which the output of the comparator in the pixel level determination unit 28 became 1.

[0098] At time t0, the photoelectric converter is assumed to be operating in the first operating mode. When operating in the first operating mode, the pixel level determination unit 28 monitors the signal P_DATA_O[7:0] and determines whether the value of the signal P_DATA_OUT[7:0] is less than a predetermined value, for example, less than 16. If the value of the signal P_DATA_O[7:0] is less than 16 in a given frame, the comparator in the pixel level determination unit 28 outputs 1, and the count value comp_count is incremented, counting this as 1. When it is detected that this state has continued for a predetermined period, for example, 4 frames, it is determined that a low-light scene is currently being captured, and the operating mode is switched to the second operating mode. For example, in the example in Figure 15, the period from time t0 to time t2 is low light, and the value of the signal P_DATA_O[7:0] is less than 16. At time t1, if there are four consecutive frames in which the value of signal P_DATA_O[7:0] is less than 16, the pixel level determination unit 28 switches the operating mode from the first mode to the second mode.

[0099] When operating in the second operating mode, the pixel level determination unit 28 monitors the value of the lower 4 bits of the signal P_DATA_O[3:0] output of the pulse counting unit 30 and determines whether the value of the signal P_DATA_O[3:0] is saturated, i.e., above a predetermined value. When the value of the signal P_DATA_O[3:0] is saturated in a given frame, the comparator in the pixel level determination unit 28 outputs 1, and the count value comp_count is incremented, counting this as 1. When it is detected that this state has continued for a predetermined period, for example, 4 frames, it is determined that a high-illumination scene is currently being captured, and the operating mode is switched to the first operating mode. For example, in the example in Figure 15, the period from time t2 onwards is high-illumination, and the value of the signal P_DATA_O[3:0] is saturated. At time t3, when 4 frames have been captured with a saturated signal P_DATA_O[3:0], the pixel level determination unit 28 switches the operating mode from the second mode to the first mode, which allows for shooting with a wide dynamic range.

[0100] Thus, according to this embodiment, the pixel level determination unit 28 within the pixel 12 determines the value of the pixel data and switches the operating mode, making it possible to set the optimal operating mode for each pixel while reducing the load on the subsequent processing system.

[0101] In this embodiment, a configuration combining the photoelectric converter of the first embodiment with the pixel level determination unit 28 has been described, but the pixel level determination unit 28 may also be combined with the photoelectric converter of the second embodiment. Furthermore, in this embodiment, the condition for switching the operating mode was set as four consecutive frames of low or high illumination, but the condition for switching the operating mode can be changed as appropriate. For example, the condition for switching the operating mode may be set as five or more predetermined frames of low or high illumination.

[0102] [Fourth Embodiment] A photodetection system according to a fourth embodiment of the present invention will be described with reference to Figure 16. Figure 16 is a block diagram showing the schematic configuration of the photodetection system according to this embodiment. In this embodiment, a photodetection sensor to which the photoelectric converter 100 described in any of the first to third embodiments is applied will be described.

[0103] The photoelectric converter 100 described in the first to third embodiments above is applicable to various light detection systems. Examples of applicable light detection systems include imaging systems such as digital still cameras, digital camcorders, surveillance cameras, photocopiers, fax machines, mobile phones, in-vehicle cameras, and observation satellites. Camera modules, which include optical systems such as lenses and imaging devices, are also included in light detection systems. Figure 16 shows a block diagram of a digital still camera as an example of these.

[0104] The photodetection system 200 illustrated in Figure 16 includes a photoelectric converter 201, a lens 202 for forming an optical image of a subject onto the photoelectric converter 201, an aperture 204 for varying the amount of light passing through the lens 202, and a barrier 206 for protecting the lens 202. The lens 202 and the aperture 204 form an optical system that focuses light onto the photoelectric converter 201. The photoelectric converter 201 is the photoelectric converter 100 described in any of the first to third embodiments, which converts the optical image formed by the lens 202 into image data.

[0105] The photodetection system 200 also includes a signal processing unit 208 that processes the output signal from the photoelectric converter 201. The signal processing unit 208 generates image data from the digital signal output by the photoelectric converter 201. The signal processing unit 208 also performs various corrections and compressions as needed before outputting the image data. The photoelectric converter 201 may include an AD conversion unit that generates the digital signal processed by the signal processing unit 208. The AD conversion unit may be formed on the semiconductor layer (semiconductor substrate) on which the photon detection element of the photoelectric converter 201 is formed, or on a semiconductor substrate separate from the semiconductor layer on which the photon detection element of the photoelectric converter 201 is formed. The signal processing unit 208 may also be formed on the same semiconductor substrate as the photoelectric converter 201.

[0106] The light detection system 200 further includes a buffer memory unit 210 for temporarily storing image data, and an external interface unit (external I / F unit) 212 for communicating with an external computer or the like. Furthermore, the light detection system 200 includes a recording medium 214 such as a semiconductor memory for recording or reading imaging data, and a recording medium control interface unit (recording medium control I / F unit) 216 for recording or reading data from the recording medium 214. The recording medium 214 may be built into the light detection system 200 or it may be detachable. In addition, communication between the recording medium control I / F unit 216 and the recording medium 214, and communication from the external I / F unit 212, may be performed wirelessly.

[0107] Furthermore, the photodetection system 200 includes an overall control / calculation unit 218 that controls various calculations and the entire digital still camera, and a timing generation unit 220 that outputs various timing signals to the photoelectric converter 201 and the signal processing unit 208. Here, the timing signals and the like may be input from an external source, and the photodetection system 200 only needs to have at least the photoelectric converter 201 and the signal processing unit 208 that processes the output signals output from the photoelectric converter 201. The timing generation unit 220 may be mounted on the photoelectric converter 201. In addition, the overall control / calculation unit 218 and the timing generation unit 220 may be configured to perform some or all of the control functions of the photoelectric converter 201.

[0108] The photoelectric converter 201 outputs the imaging signal to the signal processing unit 208. The signal processing unit 208 performs predetermined signal processing on the imaging signal output from the photoelectric converter 201 and outputs image data. The signal processing unit 208 generates an image using the imaging signal. The signal processing unit 208 may also be configured to perform distance measurement calculations on the signal output from the photoelectric converter 201.

[0109] Thus, according to this embodiment, by configuring a photodetection system using the photoelectric conversion device of the first to third embodiments, a photodetection system capable of acquiring higher quality images can be realized.

[0110] [Fifth Embodiment] A distance image sensor according to a fifth embodiment of the present invention will be described with reference to Figure 17. Figure 17 is a block diagram showing the schematic configuration of the distance image sensor according to this embodiment. In this embodiment, the distance image sensor will be described as an example of a photodetection system to which the photoelectric conversion device 100 described in any of the first to third embodiments is applied.

[0111] As shown in Figure 17, the distance image sensor 300 according to this embodiment may be configured to include an optical system 302, a photoelectric converter 304, an image processing circuit 306, a monitor 308, and a memory 310. This distance image sensor 300 receives light (modulated light or pulsed light) that is irradiated from a light source device 320 toward the subject 330 and reflected from the surface of the subject 330, and acquires a distance image corresponding to the distance to the subject 330.

[0112] The optical system 302 consists of one or more lenses and has the role of forming an image of the image light (incident light) from the subject 330 onto the light-receiving surface (sensor part) of the photoelectric converter 304.

[0113] The photoelectric converter 304 is the photoelectric converter 100 described in any of the first to third embodiments, and has the function of generating a distance signal indicating the distance to the subject 330 based on the image light from the subject 330, and supplying the generated distance signal to the image processing circuit 306.

[0114] The image processing circuit 306 has the function of performing image processing to construct a distance image based on the distance signal supplied from the photoelectric converter 304.

[0115] The monitor 308 has the function of displaying the distance image (image data) obtained by the image processing in the image processing circuit 306. The memory 310 also has the function of storing (recording) the distance image (image data) obtained by the image processing in the image processing circuit 306.

[0116] Thus, according to this embodiment, by configuring a distance image sensor using the photoelectric conversion device of the first to third embodiments, it is possible to realize a distance image sensor capable of acquiring distance images containing more accurate distance information, in conjunction with improving the characteristics of the pixel 12.

[0117] [Sixth Embodiment] An endoscopic surgical system according to the sixth embodiment of the present invention will be described with reference to Figure 18. Figure 18 is a schematic diagram showing an example of the configuration of the endoscopic surgical system according to this embodiment. In this embodiment, the endoscopic surgical system will be described as an example of a photodetection system to which the photoelectric converter 100 described in any of the first to third embodiments is applied.

[0118] Figure 18 illustrates a surgeon (physician) 460 performing surgery on a patient 472 on a patient bed 470 using an endoscopic surgical system 400.

[0119] As shown in Figure 18, the endoscopic surgical system 400 of this embodiment may consist of an endoscope 410, surgical instruments 420, and a cart 430 equipped with various devices for endoscopic surgery. The cart 430 may be equipped with a Camera Control Unit (CCU) 432, a light source device 434, an input device 436, a treatment instrument control device 438, a display device 440, and the like.

[0120] The endoscope 410 comprises a barrel 412, the tip of which is inserted into the body cavity of the patient 472 for a predetermined length, and a camera head 414 connected to the base end of the barrel 412. Figure 18 illustrates the endoscope 410 configured as a so-called rigid endoscope having a rigid barrel 412, but the endoscope 410 may also be configured as a so-called flexible endoscope having a flexible barrel. The endoscope 410 is held in a movable state by an arm 416.

[0121] An opening into which an objective lens is fitted is provided at the tip of the endoscope tube 412. A light source device 434 is connected to the endoscope 410, and the light generated by the light source device 434 is guided to the tip of the endoscope tube by a light guide extending inside the endoscope tube 412, and is irradiated through the objective lens towards the object to be observed inside the body cavity of the patient 472. The endoscope 410 may be a straight-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0122] The camera head 414 contains an optical system and a photoelectric converter (not shown), and reflected light from the object being observed (observation light) is focused by the optical system into the photoelectric converter. The photoelectric converter converts the observation light into electrical signals and generates an electrical signal corresponding to the observation light, i.e., an image signal corresponding to the observed image. The photoelectric converter 100 described in any of the first to third embodiments can be used as the photoelectric converter. The image signal is transmitted to the CCU 432 as RAW data.

[0123] The CCU432 consists of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and other components, and comprehensively controls the operation of the endoscope 410 and the display device 440. Furthermore, the CCU432 receives an image signal from the camera head 414 and performs various image processing operations on that image signal, such as development processing (demosaic processing), to display the image based on that image signal.

[0124] The display device 440 displays an image based on an image signal that has been processed by the CCU 432, under control from the CCU 432.

[0125] The light source device 434 consists of a light source such as an LED (Light Emitting Diode) and supplies illumination light to the endoscope 410 when photographing the surgical area, etc.

[0126] The input device 436 is an input interface for the endoscopic surgical system 400. The user can input various types of information and instructions to the endoscopic surgical system 400 via the input device 436.

[0127] The treatment instrument control device 438 controls the driving of the energy treatment instrument 450 for purposes such as tissue cauterization, incision, or blood vessel sealing.

[0128] The light source device 434 that supplies illumination light to the endoscope 410 when photographing the surgical area can be composed of, for example, an LED, a laser light source, or a combination thereof. When the white light source is composed of a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, so the white balance of the captured image can be adjusted in the light source device 434. In this case, it is also possible to capture images corresponding to each of the RGB colors in time-division by irradiating the observation target with laser light from each of the RGB laser light sources in time-division and controlling the drive of the image sensor of the camera head 414 in synchronization with the irradiation timing. According to this method, a color image can be obtained without providing a color filter on the image sensor.

[0129] Furthermore, the light source device 434 may be controlled to change the intensity of the light it outputs at predetermined time intervals. By controlling the drive of the image sensor of the camera head 414 in synchronization with the timing of the change in light intensity, images can be acquired in time-division order, and these images can be combined to generate high dynamic range images without so-called black crushing and white clipping.

[0130] Furthermore, the light source device 434 may be configured to supply light in a predetermined wavelength band corresponding to special light observation. In special light observation, for example, the wavelength dependence of light absorption in body tissue is utilized. Specifically, by irradiating with narrowband light compared to the irradiation light used during normal observation (i.e., white light), predetermined tissues such as blood vessels on the surface of mucosa can be imaged with high contrast. Alternatively, in special light observation, fluorescence observation may be performed to obtain an image from fluorescence generated by irradiation with excitation light. In fluorescence observation, excitation light can be irradiated onto body tissue and fluorescence from the body tissue can be observed, or a reagent such as indocyanine green (ICG) can be injected into body tissue and excitation light corresponding to the fluorescence wavelength of the reagent can be irradiated onto the body tissue to obtain a fluorescence image. The light source device 434 may be configured to supply narrowband light and / or excitation light corresponding to such special light observation.

[0131] Thus, according to this embodiment, by configuring an endoscopic surgical system using the photoelectric conversion device of the first to third embodiments, an endoscopic surgical system capable of acquiring higher quality images can be realized.

[0132] [Seventh Embodiment] A photodetection system and mobile body according to the seventh embodiment of the present invention will be described with reference to Figures 19 to 21. Figure 19 is a schematic diagram showing an example of the configuration of a mobile body according to this embodiment. Figure 20 is a block diagram showing a schematic configuration of the photodetection system according to this embodiment. Figure 21 is a flowchart showing the operation of the photodetection system according to this embodiment. In this embodiment, an example of application to an in-vehicle camera is shown as a photodetection system to which the photoelectric converter 100 described in any of the first to third embodiments is applied.

[0133] Figure 19 is a schematic diagram showing an example of the configuration of a mobile body (vehicle system) according to this embodiment. Figure 19 shows the configuration of a vehicle 500 (automobile) as an example of a vehicle system incorporating a photodetection system to which the photoelectric converter described in any of the first to third embodiments is applied. Figure 19(a) is a schematic front view of the vehicle 500, Figure 19(b) is a schematic top view of the vehicle 500, and Figure 19(c) is a schematic rear view of the vehicle 500. The vehicle 500 is equipped with a pair of photoelectric converters 502 on its front. Here, the photoelectric converters 502 are the photoelectric converters 100 described in any of the first to third embodiments. The vehicle 500 also includes an integrated circuit 503, an alarm device 512, and a main control unit 513.

[0134] Figure 20 is a block diagram showing an example configuration of a photodetection system 501 mounted on a vehicle 500. The photodetection system 501 includes a photoelectric converter 502, an image preprocessing unit 515, an integrated circuit 503, and an optical system 514. The photoelectric converter 502 is the photoelectric converter 100 described in any of the first to third embodiments. The optical system 514 forms an optical image of the subject on the photoelectric converter 502. The photoelectric converter 502 converts the optical image of the subject formed by the optical system 514 into an electrical signal. The image preprocessing unit 515 performs predetermined signal processing on the signal output from the photoelectric converter 502. The functions of the image preprocessing unit 515 may be incorporated into the photoelectric converter 502. The photodetection system 501 is provided with at least two sets of the optical system 514, photoelectric converter 502, and image preprocessing unit 515, and the output from the image preprocessing unit 515 of each set is input to the integrated circuit 503.

[0135] The integrated circuit 503 is an integrated circuit for imaging system applications and includes an image processing unit 504, an optical distance measuring unit 506, a disparity calculation unit 507, an object recognition unit 508, and an anomaly detection unit 509. The image processing unit 504 processes the image signal output from the image preprocessing unit 515. For example, the image processing unit 504 performs image processing such as development and defect correction on the output signal from the image preprocessing unit 515. The image processing unit 504 includes a memory 505 for temporarily holding the image signal. The memory 505 may store, for example, the location of known defective pixels in the photoelectric converter 502.

[0136] The optical distance measuring unit 506 focuses on and measures the distance of the subject. The parallax calculation unit 507 calculates distance information from multiple image data (parallax images) acquired by multiple photoelectric converters 502. Each of the photoelectric converters 502 may be configured to acquire various information such as distance information. The object recognition unit 508 recognizes subjects such as cars, roads, signs, and people. When the anomaly detection unit 509 detects an anomaly in the photoelectric converter 502, it notifies the main control unit 513 of the anomaly.

[0137] The integrated circuit 503 may be implemented by specially designed hardware, by a software module, or by a combination of these. It may also be implemented by an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a combination of these.

[0138] The main control unit 513 coordinates and controls the operation of the light detection system 501, vehicle sensor 510, control unit 520, etc. Note that the vehicle 500 does not necessarily have to have the main control unit 513. In this case, the photoelectric converter 502, vehicle sensor 510, and control unit 520 send and receive control signals via a communication network. For example, the CAN standard may be applied to the transmission and reception of these control signals.

[0139] The integrated circuit 503 has the function of receiving control signals from the main control unit 513 or transmitting control signals and set values ​​to the photoelectric converter 502 via its own control unit.

[0140] The light detection system 501 is connected to the vehicle sensor 510 and can detect the vehicle's driving conditions, such as vehicle speed, yaw rate, and steering angle, as well as the external environment and the state of other vehicles and obstacles. The vehicle sensor 510 also serves as a distance information acquisition means for acquiring distance information to objects. Furthermore, the light detection system 501 is connected to the driver assistance control unit 511, which performs various driving assistance functions such as automatic steering, automatic cruising, and collision avoidance. In particular, regarding the collision judgment function, it determines whether a collision with another vehicle or obstacle has occurred and estimates a collision based on the detection results of the light detection system 501 and the vehicle sensor 510. This enables avoidance control when a collision is estimated and activation of safety devices in the event of a collision.

[0141] Furthermore, the light detection system 501 is also connected to a warning device 512 that issues a warning to the driver based on the judgment result of the collision judgment unit. For example, if the collision judgment unit determines that there is a high probability of collision, the main control unit 513 performs vehicle control to avoid a collision or mitigate damage by applying the brakes, releasing the accelerator, or suppressing engine output. The warning device 512 warns the user by sounding an alarm, displaying warning information on a display screen such as the car navigation system or instrument panel, or vibrating the seat belt or steering wheel.

[0142] In this embodiment, the light detection system 501 captures images of the area around the vehicle, for example, the front or rear. Figure 19(b) shows an example of the arrangement of the light detection system 501 when it captures images of the area in front of the vehicle.

[0143] As described above, the photoelectric converter 502 is positioned in front of the vehicle 500. Specifically, it is preferable for acquiring distance information between the vehicle 500 and the object being photographed and for determining the possibility of collision if the two photoelectric converters 502 are positioned symmetrically with respect to the axis of symmetry, considering the center line with respect to the vehicle 500's direction of movement or external shape (e.g., vehicle width). Furthermore, it is preferable that the photoelectric converter 502 is positioned so as not to obstruct the driver's field of view when the driver is visually observing the situation outside the vehicle 500 from the driver's seat. It is preferable that the warning device 512 is positioned so as to be easily visible to the driver.

[0144] Next, the fault detection operation of the photoelectric converter 502 in the photodetection system 501 will be explained using Figure 21. The fault detection operation of the photoelectric converter 502 can be performed according to steps S110 to S180 shown in Figure 19.

[0145] Step S110 is a step in which the photoelectric converter 502 is configured for startup. Specifically, settings for the operation of the photoelectric converter 502 are transmitted from outside the photodetection system 501 (e.g., the main control unit 513) or from inside the photodetection system 501, and the imaging operation and fault detection operation of the photoelectric converter 502 are started.

[0146] Next, in step S120, a pixel signal is acquired from the active pixels. Also, in step S130, an output value is acquired from a fault detection pixel provided for fault detection. This fault detection pixel, like the active pixels, is equipped with a photoelectric conversion element. A predetermined voltage is written to this photoelectric conversion element. The fault detection pixel outputs a signal corresponding to the voltage written to this photoelectric conversion element. Note that steps S120 and S130 may be reversed.

[0147] Next, in step S140, a determination is made between the expected output value of the fault-detection pixel and the actual output value from the fault-detection pixel. If the determination in step S140 shows that the expected output value and the actual output value match, the process proceeds to step S150, where it is determined that the imaging operation is functioning normally, and the process moves to step S160. In step S160, the pixel signals of the scanned row are transmitted to the memory 505 for temporary storage. After that, the process returns to step S120 and continues the fault detection operation. On the other hand, if the determination in step S140 shows that the expected output value and the actual output value do not match, the process proceeds to step S170. In step S170, it is determined that there is an abnormality in the imaging operation, and an alarm is sent to the main control unit 513 or the alarm device 512. The alarm device 512 displays that an abnormality has been detected on its display unit. Subsequently, in step S180, the photoelectric converter 502 is stopped, and the operation of the photodetection system 501 is terminated.

[0148] In this embodiment, an example is shown where the flowchart is looped every row, but the flowchart may be looped every multiple rows, or the fault detection operation may be performed every frame. The alarm in step S170 may be notified to an external party via a wireless network.

[0149] Furthermore, although this embodiment describes control to avoid collisions with other vehicles, it can also be applied to control that automatically follows other vehicles or control that automatically drives without deviating from the lane. Moreover, the light detection system 501 can be applied not only to vehicles such as the vehicle itself, but also to moving objects (mobile devices) such as ships, aircraft, or industrial robots. In addition, it can be applied not only to moving objects, but also to a wide range of devices that utilize object recognition, such as intelligent transportation systems (ITS).

[0150] [Eighth Embodiment] An eighth embodiment of the present invention, a photodetection system, will be described with reference to Figure 22. Figure 22 is a schematic diagram showing an example of the configuration of the photodetection system according to this embodiment. In this embodiment, an example of application to eyeglasses (smart glasses) will be described as a photodetection system to which the photoelectric converter 100 described in any of the first to third embodiments is applied.

[0151] Figure 22(a) shows eyeglasses 600 (smart glasses) relating to one application example. The eyeglasses 600 includes a lens 601, a photoelectric converter 602, and a control device 603.

[0152] The photoelectric converter 602 is the photoelectric converter 100 described in any of the first to third embodiments and is provided on the lens 601. There may be one or more photoelectric converters 602. Furthermore, when using multiple photoelectric converters 602, multiple types of photoelectric converters 602 may be used in combination. The arrangement position of the photoelectric converter 602 is not limited to that shown in Figure 20(a). A display device (not shown) including a light-emitting device such as an OLED or LED may be provided on the back side of the lens 601.

[0153] The control device 603 functions as a power supply that provides power to the photoelectric converter 602 and the display device. The control device 603 also has a function to control the operation of the photoelectric converter 602 and the display device. The lens 601 is provided with an optical system for focusing light onto the photoelectric converter 602.

[0154] Figure 22(b) shows eyeglasses 610 (smart glasses) relating to another application example. The eyeglasses 610 have lenses 611 and a control device 612. The control device 612 may be equipped with a photoelectric converter (not shown) corresponding to a photoelectric converter 602 and a display device.

[0155] The lens 611 is equipped with a photoelectric converter within the control device 612 and an optical system for projecting light from the display device, thereby projecting an image. The control device 612 functions as a power supply to provide power to the photoelectric converter and the display device, and also has a function to control the operation of the photoelectric converter and the display device.

[0156] The control device 612 may further include a gaze detection unit that detects the wearer's gaze. In this case, the control device 612 can be provided with an infrared light emitter, and the infrared light emitted from the infrared light emitter can be used for gaze detection. Specifically, the infrared light emitter emits infrared light towards the eyeball of the user who is gazing at the displayed image. An image capture unit having a light-receiving element detects the reflected light from the eyeball of the emitted infrared light, thereby obtaining an image of the eyeball. By having a reduction means that reduces the light from the infrared light emitter to the display unit in a planar view, the deterioration of image quality can be reduced.

[0157] The user's gaze towards a displayed image can be detected from an image of the eyeball obtained by imaging with infrared light. Any known method can be applied to gaze detection using the image of the eyeball. As an example, a gaze detection method based on the Purkinje image obtained by the reflection of the irradiated light from the cornea can be used. More specifically, gaze detection processing based on the pupil-corneal reflection method is performed. Using the pupil-corneal reflection method, the user's gaze is detected by calculating a gaze vector representing the orientation (rotation angle) of the eyeball based on the pupil image and the Purkinje image contained in the image of the eyeball.

[0158] The display device of this embodiment may include a photoelectric converter having a light-receiving element and be configured to control the displayed image based on the user's gaze information from the photoelectric converter. Specifically, the display device determines a first field of view area that the user is fixated on and a second field of view area other than the first field of view area, based on the gaze information. The first and second field of view areas may be determined by the control device of the display device or by an external control device. If an external control device determines them, this information is transmitted to the display device via communication. In the display area of ​​the display device, the display resolution of the first field of view area may be controlled to be higher than the display resolution of the second field of view area. In other words, the resolution of the second field of view area may be lower than the resolution of the first field of view area.

[0159] Furthermore, the display area may have a first display area and a second display area different from the first display area, and may be configured to determine the area with higher priority from the first display area and the second display area based on gaze information. The first display area and the second display area may be determined by the control device of the display device or by an external control device. If the external control device makes the determination, it is communicated to the display device via communication. The resolution of the high-priority area may be controlled to be higher than the resolution of the areas other than the high-priority area. In other words, the resolution of areas with relatively lower priority may be lower.

[0160] AI may be used to determine the first field of view area and the areas with higher priority. The AI ​​may be a model configured to estimate the angle of gaze and the distance to the target object at the end of the line of sight from the image of the eye, using the image of the eye and the direction the eye was actually looking in that image as training data. The AI ​​program may be installed in the display device, the photoelectric converter, or an external device. If installed in an external device, it will be transmitted to the display device via communication.

[0161] When display control is based on visual detection, this method is preferably applicable to smart glasses that further include a photoelectric converter for capturing images of the surrounding environment. The smart glasses can display the captured external information in real time.

[0162] [Modified Embodiment] The present invention is not limited to the embodiments described above and can be modified in various ways. For example, an example in which a part of the configuration of one embodiment is added to another embodiment, or in which a part of the configuration of another embodiment is replaced, is also an embodiment of the present invention.

[0163] Furthermore, although the sub-counters 32 and 34 are configured as 4-bit counters in the above embodiment, the number of bits of the counters constituting sub-counters 32 and 34 is not limited to 4 bits. Also, the number of bits of sub-counters 32 and 34 do not necessarily have to be the same. For example, sub-counter 32 can be configured as an M-bit counter and sub-counter 34 can be configured as an N-bit counter. In this case, the pulse counting unit 30 operates as an (M+N)-bit counter in the first counter connection mode.

[0164] Furthermore, the circuit configuration of the pixel 12 is not limited to the above embodiment. For example, a switch such as a transistor may be provided between the photoelectric conversion element PD and the quench element 22, or between the photoelectric conversion element PD and the waveform shaping unit 24, to control the electrical connection state between them. Alternatively, a switch such as a transistor may be provided between the node to which voltage VDD is supplied and the quench element 22, and / or between the node to which voltage VSS is supplied and the photoelectric conversion element PD, to control the electrical connection state between them.

[0165] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0166] It should be noted that the above embodiments are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features. [Explanation of Symbols]

[0167] 12... pixels 14, 16… control lines 18…Setting signal line 20…Pixel signal output line 26... Signal Processing Unit 28...Pixel level determination unit 30...Pulse counting unit 32, 34… subcounters 36…Connection mode switching circuit 40…Vertical selection circuit 50... Horizontal selection circuit 60... Register Block 70... Signal processing circuits 80…Output circuit 90... Drive control circuit 100... Photoelectric converter

Claims

1. A pixel includes a photoelectric conversion unit that outputs a pulse in response to incidence of a photon, and a pulse counter that counts the pulses output from the photoelectric conversion unit, the pulse counting unit includes a first counter, a second counter, a selection circuit that selects a signal to be input to each of the first counter and the second counter, and a control unit that controls the selection circuit; the control unit is configured to control the selection circuit to execute a first connection mode in which the first counter and the second counter constitute a counter with a first number of bits, and a second connection mode in which at least one of the first counter and the second counter constitutes a counter with a second number of bits that is smaller than the first number of bits, The second connection mode includes a third connection mode in which the first counter and the second counter count the pulses in parallel. A photoelectric conversion device comprising:

2. The second connection mode further includes a fourth connection mode in which the first counter counts the pulses and the second counter does not count the pulses.

2. The photoelectric conversion device according to claim 1.

3. In the fourth connection mode, while the first counter is counting the pulses, the second counter holds its count value.

3. The photoelectric conversion device according to claim 2.

4. In a first frame, the second counter counts the pulses; In a second frame following the first frame, the first counter counts the pulses, and the second counter holds the count value in the first frame.

4. The photoelectric conversion device according to claim 3.

5. The pixel further includes a signal processing unit that processes a signal output from the pulse counter, The pulse counting unit simultaneously outputs the count value in the first frame held by the second counter and the count value in the second frame held by the first counter to the signal processing unit.

5. The photoelectric conversion device according to claim 4.

6. The third connection mode and the fourth connection mode are executed during one frame.

4. The photoelectric conversion device according to claim 2 or 3.

7. the first counter is an M-bit counter, the second counter is an N-bit counter, the first number of bits is (M+N) bits; The second number of bits is M bits or N bits.

7. The photoelectric conversion device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

8. The pixel further includes a signal processing unit that processes a signal output from the pulse counter.

8. The photoelectric conversion device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

9. The signal processing unit executes an arithmetic process between the count value of the first counter and the count value of the second counter.

9. The photoelectric conversion device according to claim 8.

10. The control unit controls a connection mode of the selection circuit based on a signal received from outside the photoelectric conversion device.

10. The photoelectric conversion device according to claim 1,

11. The pixel further includes a pixel level determination unit that determines a level of an output signal of the pulse counter, The control unit controls a connection mode of the selection circuit based on a signal received from the pixel level determination unit.

10. The photoelectric conversion device according to claim 1,

12. the pixel level determination unit determines whether the level of the output signal of the pulse counter unit is equal to or greater than a predetermined value; The control unit sets the selection circuit to the first connection mode when the output signal is equal to or greater than the predetermined value, and sets the selection circuit to the second connection mode when the output signal is less than the predetermined value.

12. The photoelectric conversion device according to claim 11.

13. The control unit switches the selection circuit to the first connection mode when a state in which a level of the output signal of the pulse counter unit is equal to or higher than the predetermined value continues for a predetermined number of frames while the selection circuit is operating in the second connection mode.

13. The photoelectric conversion device according to claim 12.

14. The control unit switches the selection circuit to the second connection mode when a state in which the level of the output signal of the pulse counter unit is less than the predetermined value continues for a predetermined number of frames while the selection circuit is operating in the first connection mode.

13. The photoelectric conversion device according to claim 12.

15. A plurality of the pixels are arranged in a plurality of rows and a plurality of columns.

15. The photoelectric conversion device according to claim 1,

16. The photoelectric conversion unit includes an avalanche photodiode that photoelectrically converts incident light and multiplies the generated charge by avalanche multiplication, a quenching element that suppresses avalanche multiplication of the avalanche photodiode, and a waveform shaping unit that converts an output signal of the avalanche photodiode into a pulse.

16. The photoelectric conversion device according to claim 1,

17. a first substrate on which the avalanche photodiode is provided; A second substrate on which the pulse counter is provided; 17. The photoelectric conversion device according to claim 16, further comprising:

18. The photoelectric conversion device according to any one of claims 1 to 17, a signal processing device that processes a signal output from the photoelectric conversion device; 1. An optical detection system comprising:

19. The signal processing device generates a distance image representing distance information to an object based on the signal.

20. The optical detection system of claim 18.

20. A mobile object, The photoelectric conversion device according to any one of claims 1 to 17, a distance information acquisition unit that acquires distance information to an object from a parallax image based on a signal output from the photoelectric conversion device; a control means for controlling the moving object based on the distance information; A moving object comprising:

Images