Photoelectric conversion device

Abstract

To provide a photoelectric conversion device that has improved accuracy.SOLUTION: A photoelectric conversion device has: a photon detection unit that generates pulses according to incidence of photons; a counting unit that counts the number of pulses output from the photon detection unit, and holds a count value obtained through the counting; and an output unit that reads the count value from the counting unit, and outputs frames based on the count value. The counting unit operates as a first counter and a second counter that each perform counting and hold the count value in a first mode. In the first mode, the count value is read out from the second counter in a first period during which the first counter performs the counting.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a photoelectric conversion device. [Background technology]

[0002] Patent Documents 1 and 2 disclose a photoelectric conversion device that detects photons by counting pulses generated by avalanche multiplication. Patent Documents 1 and 2 also disclose a method of achieving high performance by arranging multiple counters within a pixel. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-96472 A [Patent Document 2] JP 2023-102966 A Summary of the Invention [Problem to be solved by the invention]

[0004] In photoelectric conversion devices that count pulses using a counter as in Patent Documents 1 and 2, there is a demand for improved accuracy.

[0005] An object of the present invention is to provide a photoelectric conversion device with improved accuracy. [Means for solving the problem]

[0006] According to one disclosure of the present specification, there is provided a photoelectric conversion device comprising: a photon detection unit that generates a pulse in response to incident photons; a count unit that counts the pulses output from the photon detection unit and holds a count value obtained by the counting; and an output unit that reads out the count value from the count unit and outputs a frame based on the count value, wherein in a first mode, the count units operate as a first counter and a second counter that each perform the counting and hold the count value, and in the first mode, the count value is read out from the second counter during a first period during which the first counter performs the counting. Effect of the Invention

[0007] According to the present invention, a photoelectric conversion device with improved accuracy is provided. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of the configuration of a photoelectric conversion device according to a first embodiment. [Diagram 2] 1 is a schematic block diagram showing an example of the configuration of one pixel according to the first embodiment. FIG. [Diagram 3] 3A to 3C are diagrams illustrating the operation of the avalanche photodiode according to the first embodiment. [Figure 4] 2 is a functional block diagram of the photoelectric conversion device according to the first embodiment in a first mode and a second mode. FIG. [Diagram 5] 4 is a timing chart showing the operation of the photoelectric conversion device according to the first embodiment in a first mode and a second mode. FIG. [Figure 6] 10 is a timing chart showing the operation of a photoelectric conversion device according to a second embodiment in a first mode and a second mode. FIG. [Figure 7] 11 is a timing chart showing the operation of a photoelectric conversion device according to a third embodiment in a first mode and a second mode. FIG. [Figure 8] FIG. 13 is a block diagram of a light detection system according to a fourth embodiment. [Figure 9]FIG. 13 is a block diagram of a light detection system according to a fifth embodiment. [Figure 10] FIG. 13 is a schematic diagram of an endoscopic surgery system according to a sixth embodiment. [Figure 11] FIG. 13 is a schematic diagram of a light detection system according to a seventh embodiment. [Figure 12] FIG. 13 is a schematic diagram of a moving body according to a seventh embodiment. [Figure 13] 13 is a flowchart showing the operation of the light detection system according to the seventh embodiment. [Figure 14] FIG. 13 is a diagram showing a specific example of an electronic device according to the eighth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The same elements or corresponding elements in multiple drawings are denoted by the same reference numerals, and the description thereof may be omitted or simplified.

[0010] [First embodiment] Fig. 1 is a block diagram showing a configuration example of a photoelectric conversion device 1 according to this embodiment. The configuration example of the photoelectric conversion device 1 will be outlined with reference to Fig. 1. The photoelectric conversion device 1 may be, for example, an imaging device, a focus detection device, a distance measuring device, a ToF (Time-of-Flight) camera, etc.

[0011] The photoelectric conversion device 1 includes a pixel array 10, a control signal generating unit 21, a vertical scanning circuit 22, a readout circuit 23, a horizontal scanning circuit 24, and an output circuit 25. The pixel array 10 includes a plurality of pixels 100 arranged in a plurality of rows and a plurality of columns. Each of the plurality of pixels 100 includes a photoelectric conversion unit 110 including an avalanche photodiode (hereinafter referred to as APD) and a pixel signal processing unit 120. The photoelectric conversion unit 110 converts incident light into an electric signal. The pixels 100 in each column are provided for each column and connected to a pixel output signal line 27 extending in the column direction. The pixel signal processing unit 120 outputs the converted electric signal to the readout circuit 23 via the pixel output signal line 27 of the corresponding column. The row direction refers to the left-right direction in FIG. 1, and the column direction refers to the up-down direction in FIG. 1.

[0012] The vertical scanning circuit 22 supplies a control signal to each of the pixels 100 based on the control signal supplied from the control signal generation unit 21. The vertical scanning circuit 22 is provided for each row and supplies a control signal to the pixels 100 for each row via a group of drive lines 26 extending in the row direction. As will be described later, there may be multiple groups of drive lines 26 for each row. The vertical scanning circuit 22 may include logic circuits such as a shift register and an address decoder. In this way, the vertical scanning circuit 22 selects a row for outputting a signal from the pixel signal processing unit 120.

[0013] In each of the multiple pixels 100, a signal output from the photoelectric conversion unit 110 is processed by a pixel signal processing unit 120. The pixel signal processing unit 120 includes circuits such as a counter and a memory, and a digital value corresponding to incident light is held in the memory.

[0014] The horizontal scanning circuit 24 supplies a control signal to the readout circuit 23 based on the control signal supplied from the control signal generation unit 21. As a result, the horizontal scanning circuit 24 sequentially selects columns from which signals are to be output from the readout circuit 23. The horizontal scanning circuit 24 may include logic circuits such as a shift register and an address decoder. The readout circuit 23 outputs the signals of the selected columns to a storage unit or signal processing unit external to the photoelectric conversion device 1 via the output circuit 25.

[0015] The pixels 100 may be arranged one-dimensionally. Moreover, it is not necessary that one pixel signal processing unit 120 is provided for each pixel 100. For example, one pixel signal processing unit 120 may be shared by a plurality of pixels 100. In this case, the pixel signal processing unit 120 provides a signal processing function to the plurality of pixels 100 by sequentially processing signals output from the plurality of photoelectric conversion units 110.

[0016] 2 is a schematic block diagram showing an example of the configuration of one pixel of the photoelectric conversion unit 110 and the pixel signal processing unit 120 according to this embodiment. Note that in FIG. 2, the group of driving lines 26 between the vertical scanning circuit 22 and the pixel signal processing unit 120 in FIG.

[0017] The photoelectric conversion unit 110 has an APD 111. The pixel signal processing unit 120 has a quenching element 121, a waveform shaping unit 122, a counter circuit 123, and a selection circuit .

[0018] The APD 111 generates electric charges according to incident light by photoelectric conversion. An electric potential VL is supplied to the anode of the APD 111. A cathode of the APD 111 is connected to a first terminal of the quench element 121 and an input terminal of the waveform shaping unit 122. A electric potential VH higher than the electric potential VL supplied to the anode of the APD 111 is supplied to a second terminal of the quench element 121. As a result, a reverse bias voltage is supplied to the anode and cathode of the APD 111 such that the APD 111 performs an avalanche multiplication operation. When electric charges are generated by incident light in the APD 111 to which the reverse bias voltage is supplied, the electric charges cause avalanche multiplication, generating an avalanche current.

[0019] In addition, the operation modes when a reverse bias voltage is supplied to the APD 111 include a Geiger mode and a linear mode. The Geiger mode is a mode in which the APD 111 operates with a potential difference between the anode and the cathode that is greater than the breakdown voltage. The linear mode is a mode in which the APD 111 operates with a potential difference between the anode and the cathode that is close to or less than the breakdown voltage. The APD 111 may operate in either the linear mode or the Geiger mode.

[0020] An APD operated in Geiger mode is called a SPAD (Single Photon Avalanche Diode). In this case, for example, the potential VL is −30V and the potential VH is 1V.

[0021] The quench element 121 has a function of converting a change in avalanche current occurring in the APD 111 into a voltage signal. The quench element 121 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication. The quench element 121 suppresses the voltage supplied to the APD 111 to suppress avalanche multiplication (quench operation). The quench element 121 may be, for example, a resistor element or a transistor.

[0022] The waveform shaping unit 122 shapes the potential change of the cathode of the APD 111 obtained when a photon is detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 122. Although an example in which one inverter circuit is used as the waveform shaping unit 122 is shown in FIG. 2, the configuration of the waveform shaping unit 122 is not limited to this. For example, the waveform shaping unit 122 may be a circuit in which a plurality of inverter circuits are connected in series, or may be another circuit having a waveform shaping effect.

[0023] The counter circuit 123 counts the pulse signals output from the waveform shaping unit 122 and holds the count value thus obtained. The count value held in the counter circuit 123 is reset in response to a control signal supplied from the vertical scanning circuit 22 shown in FIG. 1 via a driving line 261.

[0024] 1 supplies a control signal to the selection circuit 124 via a drive line 262. In response to this control signal, the selection circuit 124 switches between electrical connection and non-connection between the counter circuit 123 and the pixel output signal line 27. The selection circuit 124 includes, for example, a buffer circuit for outputting a signal held in the counter circuit 123.

[0025] 2, the selection circuit 124 switches between electrical connection and disconnection between the counter circuit 123 and the pixel output signal line 27, but the method of controlling the signal output to the pixel output signal line 27 is not limited to this. For example, a switch such as a transistor may be disposed at a node between the quench element 121 and the APD 111, between the photoelectric conversion unit 110 and the pixel signal processing unit 120, or the like, and the signal output to the pixel output signal line 27 may be controlled by switching between electrical connection and disconnection. Alternatively, the signal output to the pixel output signal line 27 may be controlled by changing the value of the potential VH or potential VL supplied to the photoelectric conversion unit 110 using a switch such as a transistor.

[0026] Figures 3(a), 3(b) and 3(c) are diagrams for explaining the operation of the APD 111 according to this embodiment. Figure 3(a) is a diagram showing the APD 111, the quench element 121 and the waveform shaping unit 122 extracted from Figure 2. As shown in Figure 3(a), a connection node between the input terminals of the APD 111, the quench element 121 and the waveform shaping unit 122 is referred to as nodeA. Also, as shown in Figure 3(a), the output side of the waveform shaping unit 122 is referred to as nodeB.

[0027] FIG. 3(b) is a graph showing the time change of the potential of nodeA in FIG. 3(a). FIG. 3(c) is a graph showing the time change of the potential of nodeB in FIG. 3(a). In the period from time t0 to time t1, a voltage of VH-VL is applied to the APD 111 in FIG. 3(a). When a photon is incident on the APD 111 at time t1, avalanche multiplication occurs in the APD 111. As a result, an avalanche current flows through the quench element 121, and the potential of nodeA drops. After that, the amount of potential drop becomes larger, and the voltage applied to the APD 111 gradually decreases. Then, at time t2, the avalanche multiplication in the APD 111 stops. As a result, the potential level of nodeA does not drop below a certain value. After that, in the period from time t2 to time t3, a current that compensates for the voltage drop flows from the node of potential VH to nodeA, and at time t3, nodeA settles to its original potential.

[0028] In the above process, the potential of node B becomes high during the period when the potential of node A is lower than a certain threshold. In this way, the waveform of the drop in the potential of node A caused by the incidence of a photon is shaped by the waveform shaping unit 122 and output as a pulse to node B.

[0029] In the photoelectric conversion device 1 of this embodiment, the counter unit that counts the pulse signal can operate in two types of modes, a first mode and a second mode. The two types of modes will be described with reference to Fig. 4(a) and Fig. 4(b). Fig. 4(a) is a functional block diagram of the photoelectric conversion device of this embodiment in the first mode. Fig. 4(b) is a functional block diagram of the photoelectric conversion device of this embodiment in the second mode.

[0030] 4(a) and 4(b), the photoelectric conversion device 1 has a photon detection unit 11, a counting unit 12, an output unit 13, an external light information acquisition unit 14, and a count control unit 15. The photon detection unit 11 is a circuit that generates a pulse in response to the incidence of a photon, and corresponds to the APD 111, the quenching element 121, and the waveform shaping unit 122 in FIG. 2. The photon detection unit 11 outputs a pulse generated in response to the incidence of a photon to the counting unit 12.

[0031] The counting unit 12 counts the pulses output from the photon detecting unit 11 and holds the count value obtained by the counting. The counting unit 12 corresponds to the counter circuit 123 in Fig. 2. The counting unit 12 has a memory capacity of multiple bits for holding the count value. The circuit configuration of the counting unit 12 is not particularly limited, but may include, for example, a counter constituted by multiple T flip-flops.

[0032] The output unit 13 corresponds to a circuit subsequent to the counter circuit 123. The output unit 13 reads the count value from the count unit 12, and outputs a frame (a digital signal indicating the result of photon detection in one frame period) based on the count value.

[0033] The count unit 12 can operate in a first mode or a second mode. The operation mode of the count unit 12 is controlled to either the first mode or the second mode by a control signal from the count control unit 15. Fig. 4(a) shows a functional block when the count unit 12 operates in the first mode, and Fig. 4(b) shows a functional block when the count unit 12 operates in the second mode.

[0034] As shown in Fig. 4(a), in the first mode, the counting unit 12 operates as two 2-bit counters including a first 2-bit counter 12a and a second 2-bit counter 12b. A pulse output from the photon detection unit 11 is input to both the first counter 12a and the second counter 12b. The first counter 12a and the second counter 12b can operate in parallel.

[0035] The counting unit 12 assigns the output data of the first counter 12a to the lower two bits and the output data of the second counter 12b to the upper two bits, and outputs the generated output signal P_DATA_O[3:0] to the output unit 13. The [3:0] in "P_DATA_O[3:0]" indicates that this signal is a 4-bit digital signal consisting of the 3rd bit (most significant bit), the 2nd bit, the 1st bit, and the 0th bit (least significant bit). The output unit 13 generates a frame based on the output signal P_DATA_O[3:0] for each frame period and outputs it to the outside. The output data of the two counters can be integrated by assigning the output data of the first counter 12a and the output data of the second counter 12b to different bits of different digital signals.

[0036] As shown in FIG. 4(b), in the second mode, the counting unit 12 operates as a single 4-bit third counter 12c. The counting unit 12 outputs 4-bit output data of the third counter 12c to the output unit 13 as the output signal P_DATA_O[3:0]. The output unit 13 generates a frame based on the output signal P_DATA_O[3:0] for each frame period and outputs it to the outside. In this way, although the number of counters differs between the first mode and the second mode, the total number of bits of the counters is the same. Therefore, the number of bits of the output signal is also the same between the first mode and the second mode.

[0037] The number of bits described above is merely an example and is not limited thereto. The number of bits of each of the first counter 12a and the second counter 12b may be, for example, 1 bit, 4 bits, or 8 bits. The number of bits of the first counter 12a and the second counter 12b may not be the same. For example, the first counter 12a may be a 1-bit counter and the second counter 12b may be a 2-bit counter.

[0038] In the above example, the output signal P_DATA_O[3:0] is generated by assigning the output data of the first counter 12a to the lower two bits and the output data of the second counter 12b to the upper two bits. However, the configuration of the output signal P_DATA_O[3:0] is not limited to this. For example, the output signal P_DATA_O[3:0] may be generated by assigning the output data of the first counter 12a to the upper two bits and the output data of the second counter 12b to the lower two bits.

[0039] In this embodiment, in the first mode, the counting unit 12 operates as two counters, a first counter 12a and a second counter 12b, but the number of counters is not limited to this and may be any number equal to or greater than 2. When the number of counters is three or more, the number of signal output paths from the photon detecting unit 11 to the counting unit 12 is changed to a number corresponding to the number of counters.

[0040] The external light information acquisition unit 14 acquires external light information related to light outside the photoelectric conversion device 1 and supplies the information to the count control unit 15. The external light information is used to determine switching between the first mode and the second mode. That is, the count unit 12 switches between the first mode and the second mode based on the external light information.

[0041] The external light information acquisition unit 14 acquires, as external light information, an amount of environmental light obtained by measuring the environmental light at the place where the photoelectric conversion device 1 is installed using a light sensor or the like. The external light information acquisition unit 14 outputs a determination signal indicating whether the amount of environmental light is equal to or less than a predetermined threshold value or whether the amount of environmental light is greater than a predetermined threshold value to the count control unit 15. When the amount of environmental light is equal to or less than the threshold value, the count control unit 15 outputs a control signal instructing the count unit 12 to operate in the first mode. Also, when the amount of environmental light is greater than the threshold value, the count control unit 15 outputs a control signal instructing the count unit 12 to operate in the second mode. This realizes switching of the operation mode such that the count unit 12 operates in the first mode when the environment in which the photoelectric conversion device 1 is used is a dark place, and the count unit 12 operates in the second mode when the environment in which the photoelectric conversion device 1 is used is a bright place. Note that the function of the above-mentioned optical sensor may be realized by the photon detection unit 11, or it may be a sensor separate from the photon detection unit 11.

[0042] The external light information acquired by the external light information acquisition unit 14 is not limited to the above. For example, in a configuration in which a plurality of photon detection units 11 are arranged as shown in FIG. 1, the number of photon detection units 11 that detect incident light in one frame of an acquired image (light detection number), that is, the number of pixels that detect light in the pixel array 10, may be acquired as the external light information. The external light information acquisition unit 14 outputs a determination signal indicating whether the light detection number is equal to or less than a predetermined threshold value or whether the light detection number is greater than a predetermined threshold value to the count control unit 15. When the light detection number is equal to or less than the threshold value, the count control unit 15 outputs a control signal instructing the count unit 12 to operate in the first mode. Also, when the light detection number is greater than the threshold value, the count control unit 15 outputs a control signal instructing the count unit 12 to operate in the second mode. This realizes switching of the operation mode such that the count unit 12 operates in the first mode when the environment in which the photoelectric conversion device 1 is used is a dark place, and the count unit 12 operates in the second mode when the environment in which the photoelectric conversion device 1 is used is a bright place.

[0043] Fig. 5 is a timing diagram showing the operation of the photoelectric conversion device 1 according to this embodiment in the first and second modes. Fig. 5 shows a count period for outputting a plurality of frames in the first and second modes and the signal output timing of each frame. In Fig. 5, the first and second modes are illustrated vertically for comparison, but in reality, the operation in the first mode and the operation in the second mode are selectively performed.

[0044] In the "First Mode" column of Fig. 5, "Frame (First Counter)", "P_DATA_O[1:0]", "Frame (Second Counter)", "P_DATA_O[3:2]", and "Frame Output" are shown. In the "Second Mode" column of Fig. 5, "Frame (Third Counter)" and "P_DATA_O[3:0]" are shown. "Frame Output" is shown below the "First Mode" and "Second Mode" columns of Fig. 5.

[0045] "Frame (first counter)" indicates a frame period during which the first counter 12a counts pulses and holds and outputs a signal. "Frame (second counter)" indicates a frame period during which the second counter 12b counts pulses and holds and outputs a signal. "Frame (third counter)" indicates a frame period during which the third counter 12c counts pulses and holds and outputs a signal. The left end of the frame indicating the frame indicates the start time of the count period for generating the frame, and the right end of the frame indicates the end time of the count period for generating the frame. In addition, the arrow near the right end of each frame indicates the timing at which the count value of the corresponding frame is read out from the counter.

[0046] The many pulses shown in "P_DATA_O[1:0]", "P_DATA_O[3:2]" and "P_DATA_O[3:0]" show the timing of acquiring a plurality of subframes that are accumulated in generating one frame. "P_DATA_O[1:0]" shows the data of the subframe input to the first counter 12a, and "P_DATA_O[3:2]" shows the data of the subframe input to the second counter 12b. "P_DATA_O[3:0]" shows the data of the subframe input to the third counter 12c. One subframe is one-bit data indicating the presence or absence of an incident photon. The value of this one-bit data is "1" when a photon is incident during the subframe period, and is "0" when no photon is incident. The value of this subframe ("1" or "0") is accumulated over one frame period to generate one frame signal.

[0047] The photon detection unit 11 is configured to output one pulse every time one subframe period elapses if a photon is incident within the subframe period. The first counter 12a, the second counter 12b, and the third counter 12c increase the count value they hold by 1 when this one pulse is input. This allows the first counter 12a, the second counter 12b, and the third counter 12c to accumulate 1-bit data of multiple subframes included in a frame period.

[0048] "Frame output" indicates the timing at which the signal of each frame is output from the output unit 13 in the first mode and the second mode.

[0049] First, the operation of the counting unit 12 in the second mode will be described. In the second mode, the third counter 12c counts the nth frame during the period from time T0 to time T1. Then, at time T1, the third counter 12c outputs the output signal P_DATA_O[3:0] of the nth frame. After that, the third counter 12c counts the (n+1)th frame during the period from time T1 to time T2. Then, at time T2, the third counter 12c outputs the output signal P_DATA_O[3:0] of the (n+1)th frame. The subsequent processes are similar, so the description will be omitted.

[0050] Next, the operation of the counting unit 12 in the first mode will be described. In the first mode, the second counter 12b counts the (n+1)th frame during the period from time T0 to time T2. Then, at time T2 (third time), the second counter 12b outputs the output signal P_DATA_O[3:2] of the (n+1)th frame. After that, the second counter 12b counts the (n+3)th frame during the period from time T2 to time T4 (second period). Then, at time T4, the second counter 12b outputs the output signal P_DATA_O[3:2] of the (n+3)th frame.

[0051] On the other hand, the first counter 12a counts the (n+2)th frame during a period (first period) from time T1 (first time) to time T3 (second time). Then, at time T3, the first counter 12a outputs the output signal P_DATA_O[1:0] of the (n+2)th frame. After that, the first counter 12a starts counting the (n+4)th frame from time T4. Here, time T1 is a time between time T0 and time T2, time T2 is a time between time T1 and time T3, and time T3 is a time between time T2 and time T4.

[0052] As described above, in the first mode, the count periods of the first counter 12a and the second counter 12b overlap in part, and they alternately output signals. That is, the count value is read from the second counter 12b during the count period of the first counter 12a, and the count value is read from the first counter 12a during the count period of the second counter 12b. In this way, by overlapping the count periods of the two counters, the length of the count periods of the first counter 12a and the second counter 12b can be made longer in the first mode than in the second mode. In a configuration in which the two counters, the first counter 12a and the second counter 12b, operate alternately and there is no stop period for the first counter 12a and the second counter 12b, as in this embodiment, the length of the count period in the first mode is twice the length of the count period in the second mode.

[0053] In the second mode, the third counter 12c outputs the signals of the (n-1)th frame, the nth frame, the (n+1)th frame, the (n+2)th frame, and the (n+3)th frame at times T0, T1, T2, T3, and T4, respectively. In contrast, in the first mode, the first counter 12a outputs the signals of the nth frame and the (n+2)th frame at times T1 and T3, respectively. And the second counter 12b outputs the signals of the (n-1)th frame, the (n+1)th frame, and the (n+3)th frame at times T0, T2, and T4, respectively. In this way, the length of the period in which the signals are output is the same in the first mode and the second mode, so that the signal output frequency is the same. In this way, the frame output frequency in the first mode (the number of frame outputs per unit time) is the same as the frame output frequency in the second mode. Furthermore, the sum of the frequency at which the count value is read from the first counter 12a in the first mode and the frequency at which the count value is read from the second counter 12b in the first mode is equal to the frame output frequency in the first mode, which makes it possible to equalize the frame output frequency in the first mode and the second mode regardless of the length of the count period of the first counter 12a and the second counter 12b.

[0054] As described above, in the first mode of this embodiment, the count period can be made longer than in the second mode, and this can improve the accuracy of signal acquisition. Therefore, according to this embodiment, a photoelectric conversion device 1 with improved accuracy is provided.

[0055] Furthermore, since the frame output frequency can be made the same between the first mode and the second mode, the accuracy can be improved without decreasing the frame rate even if the count period is lengthened.

[0056] An application example of the photoelectric conversion device 1 of this embodiment and its effects will be described. The photoelectric conversion device 1 that counts the number of photons using an avalanche photodiode as in this embodiment may be used for applications such as a night vision scope, and is assumed to be used in a dark environment. In imaging in such a dark environment, subframes in which photons are incident and subframes in which photons are not incident occur sporadically, so a noisy image may be acquired. On the other hand, in the method of counting the number of photons using an avalanche photodiode, dark random noise is unlikely to increase even if the number of signal integrations is increased, so that the effect of dark random noise is small even if the count period is extended to increase the probability of photon incidence. Therefore, by applying the driving method of the first mode described above to extend the count period, the accuracy of imaging in a dark environment may be improved.

[0057] As described above, the first mode driving method is effective in improving the accuracy of imaging in a dark environment, but in imaging in a bright environment, the second mode driving method, which has a short counting period and a large number of counter bits, has the advantage of making signal saturation less likely to occur. Therefore, in this embodiment, the external light information acquisition unit 14 acquires external light information indicating the brightness of the usage environment, and switching between the first mode and the second mode is performed based on this information. This realizes appropriate mode switching according to the brightness of the usage environment.

[0058] In this embodiment, an example is shown in which the length of the count period in the first mode is equivalent to two frame periods of the count period in the second mode, but the ratio of the lengths of the count periods is not limited to this. The count period in the first mode may be a period spanning multiple frames in the second mode (i.e., a period longer than one frame period). For example, the length of the count period in the first mode may be a length equivalent to three frame periods of the count period in the second mode, or may be a length equivalent to four frame periods. In that case, the number of counters in the first mode may be changed to a number according to the length of the count period.

[0059] In the present embodiment, the first counter 12a and the second counter 12b operate in the first mode with count periods of the same length, but the lengths of these count periods may be different. For example, the two counters may be a counter that counts in a count period of two frame periods in the second mode, and a counter that counts in a count period of three frame periods in the second mode. Even in such a configuration, the number of counters in the first mode may be appropriately changed according to the length of the count period.

[0060] In this embodiment, the count unit 12 is configured to be inside the pixel 100, but the location where the count unit 12 is disposed and the processing content are not limited to those described in this embodiment. For example, after a signal is read out from the pixel 100, a signal processing circuit disposed inside or outside the photoelectric conversion device 1 may perform a subframe accumulation process.

[0061] In this embodiment, the amount of ambient light and the number of photon detectors 11 that detect incident light (light detection count) are exemplified as examples of the external light information acquired by the external light information acquisition unit 14, but the external light information is not limited thereto. For example, the external light information acquisition unit 14 may determine the presence or absence of a moving object in the installation environment of the photoelectric conversion device 1 based on the imaging result by the photoelectric conversion device 1, and acquire the determination result as the external light information. In this case, the count control unit 15 outputs a control signal to instruct the count unit 12 to operate in the first mode when a moving object is not detected. Also, the count control unit 15 outputs a control signal to instruct the count unit 12 to operate in the second mode when a moving object is detected. This makes it possible to shorten the count period when a moving object is detected, and reduces deterioration in accuracy such as subject blur caused by the movement of the subject. Also, when a moving object is not detected, the possibility of subject blur is low, so that the count period can be lengthened to improve imaging accuracy.

[0062] The length of the count period and the number of bits of the counter are different between the first mode and the second mode. Therefore, the level of the signal obtained may differ between the first mode and the second mode. Therefore, the gain of the output signal may be adjusted in a signal processing circuit arranged inside or outside the photoelectric conversion device 1 so as to correct the level difference according to the operation mode of the counting unit 12.

[0063] [Second embodiment] In the first embodiment, the count period is controlled so that the start time of the count period in the first mode coincides with the timing of the frame output. In other words, when comparing the first mode and the second mode, the start time of the count period in the first mode is controlled so that it corresponds to the start or end timing of the frame period in the second mode. In contrast, in the present embodiment, an example is described in which the count period is controlled so that the start time of the count period in the first mode differs from the timing of the frame output. Descriptions of elements common to the first embodiment may be omitted or simplified as appropriate.

[0064] 6 is a timing diagram showing the operation of the photoelectric conversion device 1 according to this embodiment in the first and second modes. As shown in the "first mode" in FIG. 6, in this embodiment, the start time of the count period of each frame by the first counter 12a and the second counter 12b is set to be later than the output time of the previous frame. This realizes driving in which the count operation is not performed in the period between the frame output and the start of the count period. This operation is realized by controlling the start timing of the count period in the first counter 12a and the second counter 12b on a subframe basis, not on a frame basis as in the first embodiment.

[0065] In this embodiment, the photoelectric conversion device 1 with improved accuracy is provided, similar to the first embodiment. Moreover, in this embodiment, the start timing of the count period is controlled in subframe units, not in frame units, so that the counting operations of the first counter 12a and the second counter 12b can be temporarily stopped and the count period can be shortened. This makes it possible to reduce power consumption in the first mode compared to the configuration of the first embodiment.

[0066] In this embodiment, an example is shown in which the count period in the first mode starts at a timing when the length of the count period in the first mode is greater than one time and less than two times the length of the count period in the second mode. However, the start timing of the count period in the first mode is not limited to this. In the first mode, it is sufficient that the count period starts from a subframe that spans multiple frame periods in the second mode, and for example, the count period may start from a subframe that spans two or more frames. In that case, the number of counters in the first mode may be changed appropriately depending on the length of the count period.

[0067] [Third embodiment] In the first embodiment, counting is performed during the entire counting period in the first mode. In contrast, in the present embodiment, an example will be described in which counting is performed during a portion of the counting period in the first mode and counting is not performed during the other portion.

[0068] 7 is a timing diagram showing the operation of the photoelectric conversion device 1 according to this embodiment in the first and second modes. As shown in "P_DATA_O[1:0]" and "P_DATA_O[3:2]" in FIG. 7, in this embodiment, the first counter 12a and the second counter 12b are set to count intermittently. That is, the first counter 12a and the second counter 12b count in some subframes and do not count in other subframes. This operation is realized by controlling the counting period of the first counter 12a and the second counter 12b on a subframe basis, not on a frame basis as in the first embodiment.

[0069] In this embodiment, the photoelectric conversion device 1 with improved accuracy is provided, similar to the first embodiment. Moreover, in this embodiment, the counting period in the first counter 12a and the second counter 12b is controlled in subframe units, not in frame units, so that the counting period can be reduced. This allows the power consumption in the first mode to be reduced compared to the configuration of the first embodiment.

[0070] In this embodiment, an example is shown in which the length of the period from the start to the end of counting in the first mode is two frames of the counting period in the second mode, but this is not limited to this. The length of the period from the start to the end of counting in the first mode may be a period spanning multiple frames in the second mode. For example, the length of this period may be three frames or four frames of the counting period in the second mode. In this case, the number of counters in the first mode may be appropriately changed according to the length of this period.

[0071] 7 shows an example in which the start timing of the count period is controlled on a frame-by-frame basis, but the start timing of the count period may be controlled on a subframe-by-subframe basis instead of on a frame-by-frame basis. Such a control method may also be an example of an operation in which the first counter 12a and the second counter 12b count in some subframes and do not count in other subframes.

[0072] [Fourth embodiment] A light detection system according to a fourth embodiment of the present invention will be described with reference to Fig. 8. Fig. 8 is a block diagram of the light detection system according to this embodiment. The light detection system according to this embodiment is an imaging system that acquires an image based on incident light.

[0073] The photoelectric conversion device in the above-described embodiment can be applied to various imaging systems. Examples of imaging systems include digital still cameras, digital camcorders, camera heads, copiers, fax machines, mobile phones, vehicle-mounted cameras, observation satellites, and surveillance cameras. Fig. 8 shows a block diagram of a digital still camera as an example of an imaging system.

[0074] The imaging system 7 shown in Fig. 8 includes a barrier 706, a lens 702, an aperture 704, an imaging device 70, a signal processing unit 708, a timing generating unit 720, an overall control and calculation unit 718, a memory unit 710, a recording medium control I / F unit 716, a recording medium 714, and an external I / F unit 712. The barrier 706 protects the lens, and the lens 702 forms an optical image of a subject on the imaging device 70. The aperture 704 varies the amount of light that passes through the lens 702. The imaging device 70 is configured like the photoelectric conversion device of the above-mentioned embodiment, and converts the optical image formed by the lens 702 into image data. The signal processing unit 708 performs various types of correction, data compression, and other processes on the imaging data output from the imaging device 70.

[0075] The timing generating unit 720 outputs various timing signals to the imaging device 70 and the signal processing unit 708. The overall control / calculation unit 718 controls the entire digital still camera, and the memory unit 710 temporarily stores image data. The recording medium control I / F unit 716 is an interface for recording or reading image data to the recording medium 714, and the recording medium 714 is a removable recording medium such as a semiconductor memory for recording or reading image data. The external I / F unit 712 is an interface for communicating with an external computer or the like. Timing signals and the like may be input from outside the imaging system 7, and the imaging system 7 only needs to have at least the imaging device 70 and a signal processing unit 708 for processing image signals output from the imaging device 70.

[0076] In this embodiment, the imaging device 70 and the signal processing unit 708 may be disposed on the same semiconductor substrate, or the imaging device 70 and the signal processing unit 708 may be disposed on separate semiconductor substrates.

[0077] Furthermore, each pixel of the imaging device 70 may include a first photoelectric conversion unit and a second photoelectric conversion unit. The signal processing unit 708 processes a pixel signal based on the charge generated in the first photoelectric conversion unit and a pixel signal based on the charge generated in the second photoelectric conversion unit, and can acquire distance information from the imaging device 70 to the subject.

[0078] [Fifth embodiment] Fig. 9 is a block diagram of a light detection system according to this embodiment, more specifically, a block diagram of a range image sensor using the photoelectric conversion device described in the above embodiment.

[0079] 9, the range image sensor 401 includes an optical system 402, a photoelectric conversion device 403, an image processing circuit 404, a monitor 405, and a memory 406. The range image sensor 401 receives light (modulated light, pulsed light) emitted from a light source device 411 toward a subject and reflected from the surface of the subject. The range image sensor 401 can obtain a range image according to the distance to the subject, based on the time from emission to reception of light.

[0080] The optical system 402 includes one or more lenses, guides image light (incident light) from a subject to the photoelectric conversion device 403 , and forms an image on the light receiving surface (sensor portion) of the photoelectric conversion device 403 .

[0081] The photoelectric conversion device according to each of the above-mentioned embodiments can be applied as the photoelectric conversion device 403. The photoelectric conversion device 403 supplies the image processing circuit 404 with a distance signal indicating a distance determined from a received light signal.

[0082] The image processing circuit 404 performs image processing to construct a distance image based on the distance signal supplied from the photoelectric conversion device 403. The distance image (image data) obtained by the image processing can be displayed on a monitor 405 and stored (recorded) in a memory 406.

[0083] The range image sensor 401 configured in this manner can obtain an accurate range image by applying the above-mentioned photoelectric conversion device.

[0084] [Sixth embodiment] The technology according to the present disclosure may be applied to various products. For example, the technology according to the present disclosure may be applied to an endoscopic surgery system, which is an example of an optical detection system.

[0085] Fig. 10 is a schematic diagram of the endoscopic surgery system in this embodiment. Fig. 10 shows a state in which an operator (doctor) 1131 performs surgery on a patient 1132 on a patient bed 1133 using an endoscopic surgery system 1103. As shown in the figure, the endoscopic surgery system 1103 includes an endoscope 1100, a surgical tool 1110, an arm 1121, and a cart 1134 on which various devices for endoscopic surgery are mounted.

[0086] The endoscope 1100 includes a lens barrel 1101, a region of a predetermined length from the tip of which is inserted into a body cavity of a patient 1132, and a camera head 1102 connected to the base end of the lens barrel 1101. Although Fig. 10 shows the endoscope 1100 configured as a so-called rigid scope having a rigid lens barrel 1101, the endoscope 1100 may also be configured as a so-called flexible scope having a flexible lens barrel.

[0087] An opening into which an objective lens is fitted is provided at the tip of the lens barrel 1101. A light source device 1203 is connected to the endoscope 1100. Light generated by the light source device 1203 is guided to the tip of the lens barrel 1101 by a light guide extending inside the lens barrel 1101, and is irradiated via the objective lens toward an observation target in the body cavity of the patient 1132. The endoscope 1100 may be a direct-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0088] An optical system and a photoelectric conversion device are provided inside the camera head 1102, and reflected light (observation light) from an observation target is focused on the photoelectric conversion device by the optical system. The observation light is photoelectrically converted by the photoelectric conversion device to generate an electrical signal corresponding to the observation light, i.e., an image signal corresponding to an observation image. As the photoelectric conversion device, the photoelectric conversion device described in each of the above-mentioned embodiments may be used. The image signal is transmitted to a camera control unit (CCU) 1135 as RAW data.

[0089] The CCU 1135 is configured with a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and performs overall control of the operations of the endoscope 1100 and the display device 1136. Furthermore, the CCU 1135 receives an image signal from the camera head 1102, and performs various types of image processing on the image signal, such as development processing (demosaic processing), for displaying an image based on the image signal.

[0090] Under the control of the CCU 1135 , the display device 1136 displays an image based on an image signal that has been subjected to image processing by the CCU 1135 .

[0091] The light source device 1203 includes a light source such as an LED (Light Emitting Diode), and supplies the endoscope 1100 with irradiation light when imaging an operation site or the like.

[0092] The input device 1137 is an input interface for the endoscopic surgery system 1103. A user can input various information and instructions to the endoscopic surgery system 1103 via the input device 1137.

[0093] The treatment tool control device 1138 controls the driving of the energy treatment tool 1112 for cauterizing tissue, incising, sealing blood vessels, or the like.

[0094] The light source device 1203 can supply irradiation light to the endoscope 1100 when imaging the surgical site, and can be, for example, a white light source made of an LED, a laser light source, or a combination of these. When a white light source is configured by a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision. Therefore, the light source device 1203 can adjust the white balance of the captured image. In this case, the laser light from each of the RGB laser light sources can be irradiated to the observation target in a time-division manner, and the drive of the image sensor of the camera head 1102 can be controlled in synchronization with the irradiation timing. This makes it possible to capture images corresponding to each of the RGB colors in a time-division manner. According to this method, a color image can be obtained without providing a color filter to the image sensor.

[0095] Furthermore, the driving of the light source device 1203 may be controlled so that the intensity of the light output from the light source device 1203 is changed at predetermined time intervals. By controlling the driving of the image sensor of the camera head 1102 in synchronization with the timing of the change in the light intensity to acquire images in a time-division manner and synthesizing the images, it is possible to generate an image with a high dynamic range that is free of so-called blocked-up shadows and blown-out highlights.

[0096] Furthermore, the light source device 1203 may be configured to be capable of supplying light in a predetermined wavelength band corresponding to the special light observation. In the special light observation, for example, the wavelength dependency of light absorption in body tissue can be utilized. Specifically, a predetermined tissue such as blood vessels on the mucous membrane surface can be photographed with high contrast by irradiating light in a narrower band than the irradiation light (i.e., white light) during normal observation. Alternatively, in the special light observation, a fluorescent observation may be performed in which an image is obtained by fluorescence generated by irradiating excitation light. In the fluorescent observation, it is possible to irradiate excitation light to the body tissue and observe the fluorescence from the body tissue, or to locally inject a reagent such as indocyanine green (ICG) into the body tissue and irradiate the body tissue with excitation light corresponding to the fluorescent wavelength of the reagent to obtain a fluorescent image. The light source device 1203 may be configured to be capable of supplying narrow band light and / or excitation light corresponding to such special light observation.

[0097] [Seventh embodiment] The light detection system and the moving body of this embodiment will be described with reference to Figures 11, 12(a), 12(b), 12(c), and 13. In this embodiment, an example of an in-vehicle camera will be shown as the light detection system.

[0098] FIG. 11 is a schematic diagram of a light detection system according to this embodiment, showing an example of a vehicle system and a light detection system mounted on the vehicle system. The light detection system 1301 includes a photoelectric conversion device 1302, an image preprocessing unit 1315, an integrated circuit 1303, and an optical system 1314. The optical system 1314 forms an optical image of a subject on the photoelectric conversion device 1302. The photoelectric conversion device 1302 converts the optical image of the subject formed by the optical system 1314 into an electrical signal. The photoelectric conversion device 1302 is any of the photoelectric conversion devices according to the above-mentioned embodiments. The image preprocessing unit 1315 performs predetermined signal processing on the signal output from the photoelectric conversion device 1302. The function of the image preprocessing unit 1315 may be incorporated in the photoelectric conversion device 1302. The light detection system 1301 is provided with at least two sets of an optical system 1314 , a photoelectric conversion device 1302 , and an image pre-processing unit 1315 , and the output from each set of the image pre-processing unit 1315 is input to the integrated circuit 1303 .

[0099] The integrated circuit 1303 is an integrated circuit for use in an imaging system, and includes an image processing unit 1304 including a storage medium 1305, an optical distance measuring unit 1306, a parallax calculation unit 1307, an object recognition unit 1308, and an abnormality detection unit 1309. The image processing unit 1304 performs image processing such as development processing and defect correction on the output signal of the image pre-processing unit 1315. The storage medium 1305 performs primary storage of the captured image and stores the defective positions of the captured pixels. The optical distance measuring unit 1306 performs focusing or distance measurement of the subject. The parallax calculation unit 1307 calculates distance measurement information from multiple image data acquired by multiple photoelectric conversion devices 1302. The object recognition unit 1308 recognizes subjects such as cars, roads, signs, and people. When the abnormality detection unit 1309 detects an abnormality in the photoelectric conversion device 1302, it notifies the main control unit 1313 of the abnormality.

[0100] The integrated circuit 1303 may be realized by dedicated hardware, a software module, or a combination of these. It may also be realized by a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like, or a combination of these.

[0101] The main control unit 1313 supervises and controls the operations of the light detection system 1301, the vehicle sensor 1310, the control unit 1320, etc. The main control unit 1313 may not be provided, and the light detection system 1301, the vehicle sensor 1310, and the control unit 1320 may each have a communication interface and transmit and receive control signals via a communication network according to, for example, the CAN standard.

[0102] The integrated circuit 1303 has a function of receiving a control signal from a main control unit 1313 or transmitting a control signal or a set value to the photoelectric conversion device 1302 by its own control unit.

[0103] The optical detection system 1301 is connected to a vehicle sensor 1310, and can detect the vehicle's driving state such as vehicle speed, yaw rate, steering angle, etc., the environment outside the vehicle, and the state of other vehicles and obstacles. The vehicle sensor 1310 is also a distance information acquisition unit that acquires distance information to an object. The optical detection system 1301 is also connected to a driving assistance control unit 1311 (mobile body control unit) that performs various driving assistance functions such as automatic steering, automatic cruising, and collision prevention functions. In particular, regarding the collision determination function, it determines whether or not a collision with another vehicle or obstacle has occurred based on the detection results of the optical detection system 1301 and the vehicle sensor 1310. As a result, avoidance control is performed when a collision is estimated, and a safety device is activated when a collision occurs.

[0104] The light detection system 1301 is also connected to an alarm device 1312 that issues an alarm to the driver based on the result of the determination by the collision determination unit. For example, if the collision determination unit determines that there is a high possibility of a collision, the main control unit 1313 performs vehicle control such as applying the brakes, releasing the accelerator, and suppressing engine output to avoid a collision or reduce damage. The alarm device 1312 issues an alarm to the user using means such as issuing an alarm such as a sound, displaying alarm information on the display screen of a car navigation system or a meter panel, or applying vibration to a seat belt or steering wheel.

[0105] The light detection system 1301 in this embodiment can capture images of the surroundings of the vehicle, for example, the front or rear. Figures 12(a), 12(b), and 12(c) are schematic diagrams of a moving body in this embodiment, showing a configuration in which the light detection system 1301 captures an image of the area in front of the vehicle.

[0106] The two photoelectric conversion devices 1302 are disposed in front of the vehicle 1300. Specifically, it is preferable to regard the center line of the vehicle 1300 relative to the forward / backward direction or the outer shape (e.g., vehicle width) as an axis of symmetry, and to dispose the two photoelectric conversion devices 1302 in line symmetry with respect to the axis of symmetry. This makes it possible to effectively obtain distance information between the vehicle 1300 and an object to be photographed, and to determine the possibility of a collision. It is also preferable that the photoelectric conversion device 1302 is disposed in a position that does not obstruct the driver's field of vision when the driver visually checks the situation outside the vehicle 1300 from the driver's seat. It is preferable that the alarm device 1312 is disposed in a position that is easily within the driver's field of vision.

[0107] Next, a fault detection operation of the photoelectric conversion device 1302 in the photodetection system 1301 will be described with reference to Fig. 13. Fig. 13 is a flowchart showing the operation of the photodetection system in this embodiment. The fault detection operation of the photoelectric conversion device 1302 can be performed in accordance with steps S1410 to S1480 shown in Fig. 13.

[0108] In step S1410, startup settings are made for the photoelectric conversion device 1302. That is, setting information for the operation of the photoelectric conversion device 1302 is transmitted from outside the photodetection system 1301 (e.g., the main control unit 1313) or from inside the photodetection system 1301, and the photoelectric conversion device 1302 starts imaging operation and fault detection operation.

[0109] Next, in step S1420, the photoelectric conversion device 1302 acquires pixel signals from the valid pixels. In addition, in step S1430, the photoelectric conversion device 1302 acquires output values ​​from failure detection pixels provided for failure detection. The failure detection pixels include photoelectric conversion elements, like the valid pixels. A predetermined voltage is written to the photoelectric conversion elements. The failure detection pixels output signals corresponding to the voltage written to the photoelectric conversion elements. Note that steps S1420 and S1430 may be executed in the reverse order.

[0110] Next, in step S1440, the photodetection system 1301 performs a correspondence determination between the output expected value of the fault detection pixel and the actual output value from the fault detection pixel. If the correspondence determination result in step S1440 indicates that the output expected value and the actual output value match, the photodetection system 1301 proceeds to processing in step S1450, determines that the imaging operation is performed normally, and proceeds to processing in step S1460. In step S1460, the photodetection system 1301 transmits the pixel signal of the scanning row to the storage medium 1305 and temporarily stores it. Thereafter, the photodetection system 1301 returns to the processing in step S1420 and continues the fault detection operation. On the other hand, if the correspondence determination result in step S1440 indicates that the output expected value and the actual output value do not match, the photodetection system 1301 proceeds to processing in step S1470. In step S1470, the light detection system 1301 determines that there is an abnormality in the imaging operation, and issues an alarm to the main control unit 1313 or the alarm device 1312. The alarm device 1312 displays on the display unit that an abnormality has been detected. After that, in step S1480, the light detection system 1301 stops the photoelectric conversion device 1302, and ends the operation of the light detection system 1301.

[0111] In this embodiment, the flowchart is looped for each line, but the flowchart may be looped for each set of lines, or the fault detection operation may be performed for each frame. The issuance of the alarm in step S1470 may be notified to the outside of the vehicle via a wireless network.

[0112] In addition, in this embodiment, the control to prevent collision with other vehicles has been described, but the control can also be applied to automatic driving control to follow other vehicles, automatic driving control not to go out of the lane, etc. Furthermore, the light detection system 1301 is not limited to vehicles such as the vehicle itself, but can be applied to moving bodies (moving devices) such as ships, aircraft, and industrial robots. In addition, the light detection system 1301 is not limited to moving bodies, but can be applied to a wide range of devices that use object recognition, such as intelligent transport systems (ITS).

[0113] The photoelectric conversion device of the present invention may further be configured to be capable of acquiring various types of information such as distance information.

[0114] [Eighth embodiment] FIG. 14(a) is a diagram showing a specific example of an electronic device in this embodiment, showing glasses 1600 (smart glasses). The glasses 1600 are provided with a photoelectric conversion device 1602 described in each of the above-mentioned embodiments. That is, the glasses 1600 are an example of a light detection system to which the photoelectric conversion device 1602 described in each of the above-mentioned embodiments can be applied. A display device including a light-emitting device such as an OLED or LED may be provided on the back side of the lens 1601. The photoelectric conversion device 1602 may be one or more. In addition, multiple types of photoelectric conversion devices may be combined. The arrangement position of the photoelectric conversion device 1602 is not limited to that shown in FIG. 14(a).

[0115] The glasses 1600 further include a control device 1603. The control device 1603 functions as a power source that supplies power to the photoelectric conversion device 1602 and the above-mentioned display device. The control device 1603 also controls the operations of the photoelectric conversion device 1602 and the display device. The lens 1601 is provided with an optical system for collecting light on the photoelectric conversion device 1602.

[0116] FIG. 14(b) shows glasses 1610 (smart glasses) according to one application example. The glasses 1610 have a control device 1612, and the control device 1612 is equipped with a photoelectric conversion device corresponding to the photoelectric conversion device 1602 and a display device. The photoelectric conversion device in the control device 1612 and an optical system for projecting light emitted from the display device are arranged in the lens 1611, and an image is projected onto the lens 1611. The control device 1612 functions as a power source that supplies power to the photoelectric conversion device and the display device, and controls the operation of the photoelectric conversion device and the display device. The control device 1612 may have a line of sight detection unit that detects the line of sight of the wearer. Infrared light may be used to detect the line of sight. The infrared light emission unit emits infrared light toward the eyeball of a user gazing at a display image. An image of the eyeball is obtained by detecting the reflected light of the emitted infrared light from the eyeball with an imaging unit having a light receiving element. By providing a reduction unit that reduces the amount of light from the infrared light emitting unit to the display unit in a plan view, degradation of image quality is reduced.

[0117] The control device 1612 detects the user's line of sight with respect to the displayed image from the captured image of the eyeball obtained by capturing infrared light. Any known method can be applied to the line of sight detection using the captured image of the eyeball. As an example, a line of sight detection method based on the Purkinje image formed by the reflection of irradiated light on the cornea can be used.

[0118] More specifically, a gaze detection process based on the pupil-corneal reflex method is performed. Using the pupil-corneal reflex method, a gaze vector that indicates the direction (rotation angle) of the eyeball is calculated based on the pupil image and the Purkinje image included in the captured image of the eyeball, thereby detecting the user's gaze.

[0119] The display device of this embodiment may have a photoelectric conversion device having a light receiving element, and may control the display image of the display device based on information on the line of sight of a user from the photoelectric conversion device.

[0120] Specifically, the display device determines a first field of view area to which the user gazes and a second field of view area other than the first field of view area based on the line-of-sight information. The first field of view area and the second field of view area may be determined by a control device of the display device or an external control device. In the display area of ​​the display device, the display resolution of the first field of view area may be controlled to be higher than the display resolution of the second field of view area. In other words, the resolution of the second field of view area may be lower than that of the first field of view area.

[0121] The display area may include a first display area and a second display area different from the first display area. A high priority area may be determined from the first display area and the second display area based on line-of-sight information. The first field of view area and the second field of view area may be determined by a control device of the display device or an external control device. The resolution of the high priority area may be controlled to be higher than the resolution of areas other than the high priority area. In other words, the resolution of an area with a relatively low priority may be lowered.

[0122] In addition, AI (Artificial Intelligence) may be used in determining the first field of view area and the area with high priority. The AI ​​may be a model configured to estimate the angle of the line of sight and the distance to an object at the end of the line of sight from the image of the eyeball, using an image of the eyeball and the direction in which the eyeball in the image was actually looking as teacher data. The AI ​​program may be provided in either the display device or the photoelectric conversion device, or may be provided in an external device. If the external device has the AI ​​program, it may be transmitted to the display device from a server or the like via communication.

[0123] In the case where display control is performed based on visual recognition detection, the present embodiment can be preferably applied to smart glasses further including a photoelectric conversion device that captures an image of the outside world. The smart glasses can display captured outside information in real time.

[0124] [Modified embodiment] The present invention is not limited to the above-described embodiments and can be modified in various ways. For example, an example in which a part of the configuration of any of the embodiments is added to another embodiment, or an example in which a part of the configuration of any of the embodiments is replaced with a part of the configuration of another embodiment, is also an embodiment of the present invention.

[0125] The disclosure of this specification includes the complement of the concepts described in this specification. In other words, if the specification states, for example, that "A is B" (A=B), the specification is deemed to disclose or suggest that "A is not B" even if the statement that "A is not B" (A≠B) is omitted. This is because when it states that "A is B," it is assumed that the case that "A is not B" is taken into consideration.

[0126] The disclosure of this specification includes the following configurations. (Configuration 1) a photon detector that generates a pulse in response to an incident photon; a counting unit that counts the pulses output from the photon detection unit and holds a count value obtained by the counting; an output unit that reads out the count value from the count unit and outputs a frame based on the count value; having the counting unit operates, in a first mode, as a first counter and a second counter each of which performs the counting and holds the count value; In the first mode, the count value is read from the second counter during a first period in which the first counter performs the counting. A photoelectric conversion device comprising: (Configuration 2) In the first mode, the count value is read from the first counter during a second period in which the second counter performs the counting. 2. The photoelectric conversion device according to configuration 1. (Configuration 3) In the first mode, a start time of the first period and a start time of the second period are different from each other. 3. The photoelectric conversion device according to configuration 2. (Configuration 4) In the first mode, the count value is read from the first counter at a first time and a second time, and the count value is read from the second counter at a third time between the first time and the second time. 4. The photoelectric conversion device according to any one of configurations 1 to 3. (Configuration 5) The first counter starts the counting at the first time to generate the count value that is read out at the second time. 5. The photoelectric conversion device according to configuration 4. (Configuration 6) The first counter starts the counting for generating the count value to be read at the second time at a time later than the first time. 5. The photoelectric conversion device according to configuration 4. (Configuration 7) In the first mode, the first counter counts a part of the pulses output from the photon detection unit during the first period and does not count the other part. 7. The photoelectric conversion device according to any one of configurations 1 to 6. (Configuration 8) The output unit generates and outputs the frame in which the count value read from the first counter and the count value read from the second counter are assigned to different bits in the first mode. 8. The photoelectric conversion device according to any one of configurations 1 to 7. (Configuration 9) The counting unit operates as a single third counter in the second mode. 9. The photoelectric conversion device according to any one of configurations 1 to 8. (Configuration 10) The frequency of outputting the frames from the output unit in the first mode is equal to the frequency of outputting the frames from the output unit in the second mode. 10. The photoelectric conversion device according to configuration 9. (Configuration 11) The frequency at which the frames are output from the output section in the first mode is equal to the sum of the frequency at which the count value is read from the first counter and the frequency at which the count value is read from the second counter. 11. The photoelectric conversion device according to configuration 10. (Configuration 12) The length of the period during which the counting is performed in the first mode is longer than the length of the period during which the counting is performed in the second mode. 12. The photoelectric conversion device according to any one of configurations 9 to 11. (Configuration 13) The counting unit switches between the first mode and the second mode depending on the amount of ambient light. 13. The photoelectric conversion device according to any one of configurations 9 to 12. (Configuration 14) When the amount of ambient light is equal to or less than a threshold value, the counting unit operates in the first mode; When the amount of ambient light is greater than the threshold, the counting unit operates in the second mode. 14. The photoelectric conversion device according to claim 13, (Configuration 15) A photon detector having a plurality of the photon detectors, The counting unit switches between the first mode and the second mode depending on the number of the photon detection units that detect incident light. 13. The photoelectric conversion device according to any one of configurations 9 to 12. (Configuration 16) When the number of the photon detection units that detect the incident light is equal to or smaller than a threshold value, the count unit operates in the first mode; When the number of the photon detection units at which the incident light is detected is greater than the threshold value, the count unit operates in the second mode. 16. The photoelectric conversion device according to configuration 15. (Configuration 17) The counting unit switches between the first mode and the second mode depending on the result of the detection of the moving object. 13. The photoelectric conversion device according to any one of configurations 9 to 12. (Configuration 18) When no moving object is detected, the counting unit operates in the first mode; When a moving object is detected, the counting unit operates in the second mode. 18. The photoelectric conversion device according to claim 17, (Configuration 19) The photoelectric conversion device according to any one of configurations 1 to 18, a signal processing unit that processes a signal output from the photoelectric conversion device; 1. An optical detection system comprising: (Configuration 20) A mobile object, The photoelectric conversion device according to any one of configurations 1 to 18, a moving object control unit that controls the moving object based on distance information generated from a signal output from the photoelectric conversion device; A moving object comprising:

[0127] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0128] It should be noted that the above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these embodiments. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0129] 11 Photon detector 12 Counting section 12a First Counter 12b Second Counter 12c 3rd counter 13 Output section

Claims

1. a photon detector that generates a pulse in response to an incident photon; a counting unit that counts the pulses output from the photon detection unit and holds a count value obtained by the counting; an output unit that reads out the count value from the count unit and outputs a frame based on the count value; having the counting unit operates, in a first mode, as a first counter and a second counter each of which performs the counting and holds the count value; In the first mode, the count value is read from the second counter during a first period in which the first counter performs the counting. A photoelectric conversion device comprising:

2. In the first mode, the count value is read from the first counter during a second period in which the second counter performs the counting.

2. The photoelectric conversion device according to claim 1.

3. In the first mode, a start time of the first period and a start time of the second period are different from each other.

3. The photoelectric conversion device according to claim 2.

4. In the first mode, the count value is read from the first counter at a first time and a second time, and the count value is read from the second counter at a third time between the first time and the second time.

2. The photoelectric conversion device according to claim 1.

5. The first counter starts the counting at the first time to generate the count value that is read out at the second time.

5. The photoelectric conversion device according to claim 4.

6. The first counter starts the counting for generating the count value to be read at the second time at a time later than the first time.

5. The photoelectric conversion device according to claim 4.

7. In the first mode, the first counter counts a part of the pulses output from the photon detection unit during the first period and does not count the other part.

2. The photoelectric conversion device according to claim 1.

8. The output unit generates and outputs the frame in which the count value read from the first counter and the count value read from the second counter are assigned to different bits in the first mode.

2. The photoelectric conversion device according to claim 1.

9. The counting unit operates as a single third counter in the second mode.

2. The photoelectric conversion device according to claim 1.

10. The frequency of outputting the frames from the output unit in the first mode is equal to the frequency of outputting the frames from the output unit in the second mode.

10. The photoelectric conversion device according to claim 9.

11. The frequency at which the frames are output from the output section in the first mode is equal to the sum of the frequency at which the count value is read from the first counter and the frequency at which the count value is read from the second counter. The photoelectric conversion device according to claim 10 .

12. The length of the period during which the counting is performed in the first mode is longer than the length of the period during which the counting is performed in the second mode.

10. The photoelectric conversion device according to claim 9.

13. The counting unit switches between the first mode and the second mode depending on the amount of ambient light.

10. The photoelectric conversion device according to claim 9.

14. When the amount of ambient light is equal to or less than a threshold value, the counting unit operates in the first mode; When the amount of ambient light is greater than the threshold, the counting unit operates in the second mode.

14. The photoelectric conversion device according to claim 13.

15. A photon detector having a plurality of the photon detectors, The counting unit switches between the first mode and the second mode depending on the number of the photon detecting units that detect incident light.

10. The photoelectric conversion device according to claim 9.

16. When the number of the photon detection units that detect the incident light is equal to or smaller than a threshold value, the count unit operates in the first mode; When the number of the photon detection units at which the incident light is detected is greater than the threshold value, the count unit operates in the second mode.

16. The photoelectric conversion device according to claim 15.

17. The counting unit switches between the first mode and the second mode depending on the result of the detection of the moving object.

10. The photoelectric conversion device according to claim 9.

18. When no moving object is detected, the counting unit operates in the first mode; When a moving object is detected, the counting unit operates in the second mode.

18. The photoelectric conversion device according to claim 17.

19. The photoelectric conversion device according to any one of claims 1 to 18, a signal processing unit that processes a signal output from the photoelectric conversion device; 1. An optical detection system comprising:

20. A mobile object, The photoelectric conversion device according to any one of claims 1 to 18, a moving object control unit that controls the moving object based on distance information generated from a signal output from the photoelectric conversion device; A moving object comprising:

Images